Doing something wrong - custom profile terrible results

[eon_designs]

Ok so something is not right here at all. In the past the video quality of this program has been fantastic, but it have just tried to make a custom profile similar on one used in Handbreak and the results are terrible.

Little bit of background. In the past file size has not been a worry, however I am now streaming files over wifi and some of the large files are choking. I downloaded a movie from iTunes to watch, it's quality was absolutely fantastic and the size only 4 gb for a full HD video with 5.1 audio. So I want to try and replicate.

I started with some of the default profiles in VSO converter, but none did what I needed. So I looked back at Handbreak.

Played around with all their settings and got a good result. End test movie came up with these results in MediaInfo

Handbreak version
2.20gb size
1hr 15mins duration
AVC video
AC3 6 Chanel audio
Overall bit rate 4367kbps
Video bit rate 3977kbps

The actual quality is fantastic, no blocky ness or pixelation. Superb. Only downside is that it took 2hrs to encode.

So, went back to VSO to replicate in that. Created a custom profile and used the above file to import settings. Started the encoding and got the results.

VSO version
2.21gb size
1hr 15mins duration
AVC video
AC3 6 Chanel audio
Overall bit rate 4364kbps
Video bit rate 3977kbps

So the end results as recorded by MediaInfo are just about identical to the Handbreak version. Problem is that the VSO version looks terrible. Quality is blocky, terrible pixelation and just really poor quality. The only good point was it only took 14 mins to encode as I was using CUDA.

Question is, why is the VSO version so bad when bit rates are near identical? What am I doing wrong? Even a version encoded through Handbreak using Intel QVS looks better and that took only 11mins to encode!

I am trying a conversion without CUDA to see how that looks, but will take just under 2 hours to encode.

[MyMojo]

Hi eon,

Through my own research while learning to convert/encode videos, I have come across a number of posts on other sites claiming that CUDA can have a negative impact when encoding certain types of codecs. I am interested in your results after deactivating CUDA for that file. I'm guessing the results were better???

Thanks

[eon_designs]

Hello.

Well unfortunately the results were just as bad with CUDA turned off. So really it didn't make any difference on or off. The quality was still not up to what I would have expected.

[MyMojo]

I have an Nvidia card, and will test and see if the same happens with my system when I get home tonight. What type of video file was the original? I see what you were encoding to, but don't see what you were decoding, i.e Blu-ray VC1, DVD Mpeg2, ect...

[eon_designs]

It was the first episode of Boardwalk Empire on Blu Ray. Also forgot to mention I set the custom profile as Main@L4.0 left the size at full 1920 x 1080

[MyMojo]

I don't have that bluray, was it a mpeg2, vc1, mpeg4. Most tv show files I've seen on disc are mpeg2.

[MyMojo]

Also, what version of VC are you using?

[eon_designs]

I'm presuming that it is MPEG 2. I have just ripped the bluray into an ISO file on my desktop. I was using the last but one version. I'm just about to test the latest beta version with the new Nvidia drivers.

[eon_designs]

Okay. Been doing more testing and think I know the problem. I am not comparing apples with apples.

I shall try and explain.

When I was using Handbreak (and dvdfab) I had them using thir constant quality option. In this case a value of 21. This is very different from what VC is doing.

If I then set both Handbreak or Dvdfab to a similar bit rate (4000 ish) then the results are about the same MB size and the quality equally as poor as each other and VC. It's the constant quality option that is the difference.

I also agree that I read that using CUDA can result in reduced quality, but I personally cannot tell the difference. In fact, Handbreak let my use the intelQNC encoding (similar to CUDA) or what ever it is called. I tried an encoding on 1 chapter, it took 13 seconds. Then tried using normal CPU, that took just under 4 minutes for the same chapter. The results were near identical. At least I could not tell the difference visually. The only noticeable difference was that the first file was a fre MB larger.

Anyway. So that is probably why I am getting the poor quality. I think?

[MyMojo]

Hi eon,

Finally got some testing done, after I figured out that beta .12 really doesn't work with my older NVidia card, when at first I thought it did. I had to apologetically rescind my initial 'successful verdict', and now feel like a moron...but I digress....

After running a few short clips, and doing some reading on HandBrake's ConstantQuality feature, I think the answer lies within how each program handles this aspect of video editing.

This taken from HandBrake's site:

Constant Quality: With the average bitrate or target size methods, [meaning other programs and their methods (I think)] you control the size of the output file but give up control over the video's quality.

Constant quality mode does the opposite; you specify a quality level and HandBrake adjusts the bitrate (that is, the size) to meet it.

https://trac.handbrake.fr/wiki/ConstantQuality

If you look at VSO's option for "Target size / quality" setting under "General Settings", you will see what HandBrake is referring to here. VSO is using target size over quality, whereas HandBrake is using quality level over size.

Of course, I could be TOTALLY wrong about all this, but that's what it looks like to me. So if that's right, you'll need to determine which application suits you best for the file you're trying to create.

If anyone reading this knows that I'm wrong, or knows of another way to get VSO to act like HandBrake in this regard, please feel free to way in on this.

Thanks,
Garrett

[cribber]

CUDA doesn't make much difference at all to quality. The fact is that Handbrake can set options for the H264 codec to use Rate Factor, Reference Frames, Variable Bitrate etc. that drastically improves the quality/filesize. VSO Video Converter can't do this and that's why the filesize ends up being absolutely huge compared to Handrake. I asked Cedric a while ago if he could add quality options like this in the h264 codec but I don't know if and when it will happen.

[eon_designs]

I did a couple of screen comparison shots to show the quality differences. All encodings were full 1920 x 1080 and included an AC3 5.1 audio track.

The image on the left is from VSO VC. It is using the standard H264 profile and increased the bitrate to around 6000kbps. The end file is about 3.5GB in size

The image in the middle is from Handbrake. Using the normal profile and setting the bitrate to 6000kbps. The end file is about 3.1GB in size. I got very similar results from DVDFab.

The image on the right is from Handbrake. Using the normal profile but with a Constant Quality value of 21. Looking at the file with MediaInfo the end bitrate is about 4800kbps and the file size only 2.6GB. Again very similar results from DVDFab suing a similar Constant Quality value.

As you can see from the first comparison the results are quite obvious. However, if you then look at the next couple of comparison shots, the Handbrake/DVDFab at a set bitrate and a constant quality value start to even out. The only difference is that the overall file size is smaller. I should point out that I have been getting some playback stutters from files using the constant quality settings. MediaInfo examines these files and cannot read if they are using a constant or variable bitrate where as the files that are generated using a given bitrate are read by MediaInfo as a variable bitrate.

[eon_designs]

Comparison 2

Left VSO 6000kbps
Middle Handbrake/DVDFab 6000kbps
Right Handbrake/DVDFab Constant Quality Value 21

[eon_designs]

Comparison 3

Left VSO 6000kbps
Middle Handbrake/DVDFab 6000kbps
Right Handbrake/DVDFab Constant Quality Value 21

[cribber]

Constant Bitrate of 6000 for BluRay is very low. That's why tools like the Handrake frontend is much better for encoding (at the moment) H264. The RF Quality slider doesn't use a set bitrate and increases it to whats needed for fast/busy scenes. VSO can't do this yet and you end up with a massive filesize of constant bitrate. Only thing is that Handrake doesn't use any GPU acceleration so takes ages. Hopefully Cedric can apply these options to VSO Video Converter with GPU acceleration.

[MyMojo]

Very interesting info. I've played around with HandBrake a little bit, but my workstation doesn't contain any compatible hardware for acceleration using it. Therefore my transcoding experience using HandBrake was like sucking cold honey through a drinking straw....haha!!

Until I upgrade to a Haswell (near future, I hope!), I'll be using VSO. So far, we're very happy with using it, though the files might be bigger, the clarity on our plasma panels look flawless. When I get a bit more serious about streaming over the internet, I'll probably use other means, if I need to. I've seen comparisons using Quick Sync on other sites, and it is downright impressive....to say the least.

[JJ]

ConvertXtoDVD was using CQ very well, so maybe ConvertXtoHD will be able to do the same for HD resolutions.

[MyMojo]

Thanks JJ,

I never looked at ConvertXtoHd, does it support a CQ setting similar to HandBrake?

[JJ]

It is still on Alpha so I will not comment - anything can be added or changed.
You can download it for testing!

[eon_designs]

I know 6000 is a bit low for BluRay but, I was chatting to Cedric today explaining that this is all apples fault really. I downloaded my first HD movie from them and was shocked at how good the quality was and how small the file was. Full HD movie with AC3 5.1 audio. Only 4gb in size and a max bit rate of 4000