The road of x264 setting optimization
--1-pass vs 2-pass--
I recommend 2-pass mode for all of the encodes which you wish to reach a certain filesize. 1-pass mode is strictly for those who are severely short on time or do not expect to attain high-quality output. It is sometimes used to produce streaming content (see the streaming section below) or constant quality content (see the Constant Quality section below).
--Constant Quality--
If you wish your video to maintain a specific, constant quality the whole way through, use only the Constant Quality feature of 1-pass mode.
<> Do not use a quantizer of under 15 unless you are working for archive/reproduction quality.
<> Also, do not use anything over 40: the quality is simply unbearable unless you are encoding from an extremely sharp source of high-contrast edges and plan on streaming it over the internet.
<> A good bet for most people interested in high quality video would be the range of 22-30 (or, more specifically, 24-28). Of course, this varies depending on individual taste and how much free space you have.
<> On animated content with few detailed textures, consider using a higher quantizer.
<> On "real-life" content, especially that with many dark scenes and important subtle textures, consider using a much lower quantizer value.
--STREAMING:--
H.264 can be used for streaming lower-bitrate content over the internet. It can also be used for videoconferencing and is already widely adopted in virtually all new videoconferencing tools. If you wish to produce low-bitrate material and release it to 1000+ individual viewers online, consider streaming H.264 video; it spares users the time otherwise necessary to download the content, and it spares you much bandwidth and webspace.
To stream x264 video, you need to have a player capable of handling streaming; nearly all players I can think of at the moment support this. I suggest using Media Player Classic, but Windows Media Player 9/10 works just as well for many people as long as the ffdshow decoder is installed. Make sure to configure the player to automatically start a stream with the desired file extension (the most commonly made x264 files use the ".avi" extension).
See the Encoding: Streaming Quality section below for the best settings to use if you wish to encode streaming H.264 content.
--Encoding: Streaming Quality--
This particular section of the guide consists of encoding tips for those who wish to stream their non-copyrighted content out on the internet to many viewers without requiring much bandwidth or downloading.
<> Ensure that your source does not have a high framerate. Framerates over 30 practically kill streaming content. The lower the bitrate at which you plan to stream the file, the lower framerate the output must be to maintain good visual quality. Dialup streaming: 12-15fps MAX.
<> Adjust the dimensions of the source video. The lower the dimensions, the better the quality becomes at streaming bitrates. While it is not unheard of to stream at entire DVD dimensions, the maximum recommended streaming video dimensions fall at about 640x480, but 480x360 would actually be a better maximum in most cases of internet streaming. Remember, the decrease in physical frame size should be proportionate to the bitrate you wish to stream at; lower bitrates require more reduction. Dialup streaming: 320x240 MAX.
<> Of course, you can favor one of the above aspects of the source over another. For instance, if you absolutely MUST have perfectly smooth 25fps motion over a low-end DSL connection, compensate by reducing the dimensions to 320x240. Likewise, if you favor a larger viewing area with more visual detail, increase the dimensions to 640x480, but decrease the framerate to 12-15fps.
<> Use 1-3 references in this mode for stream simplicity.
<> Use a maximum of 1 B-frame. It is NOT recommended to use consecutive B-frames on very-low bitrate content. I have proven this with the thread entitled "The Coolness of B-Frames." The WORST place to use a B-frame in H.264 (or many other codecs for that matter) is on low-bitrate, low framerate, streaming, animated content.
<> Reduce the bitrate variability by as much as possible for a stable, "streamable stream." I recommend a 15% maximum variability, and some people actually recommend a maximum under 10%. Of course, this value depends on the situation. If you are streaming for dialup users, decrease this value to 5%: you must maintain as accurate a stream as possible for dialup users to eliminate long "lags/hangs" in decoding by insuring that the dialup user's decoder perform accurate buffering.
<> Follow the rest of either the "High Speed, High Quality" or the "Absolute Max Quality" guides below, whichever is to your liking.
--Encoding: High Speed, High Quality--
Best mix of encoding quality and encoding speed:
These are the recommended settings which produce the maximum quality AVC encodes while maximizing encoding speed.
<> 4-5 references are the typically accepted maximum. However, if you have a little bit of extra time on your hands (or a beefy computer), consider using up to 8 references (which decode just about as fast as 5, according to my Pocket PC AVC benchmarks).
<> I very strongly urge you to use 2-4 consecutive B-frames*, but only with content with a framerate over 23.976fps (I typically use 0-1 on 12-15fps animation or Very-Extremely-Low bitrate streaming content).
<> B-frame reduction (if you choose to use B-frames) should be adjusted according to the desired datarate. For most DVD backups, the default B-frame reduction of 30% seems to be sufficient. However, with certain animated medium bitrate content, I suggest use of up to 50% B-frame reduction.
<> See the "Bitrate Variability" section below for tips on how to set this field.
<> I keep sub-pel refinement on "6 RD-oh!”; the quality increase is very noticeable and well worth the mere seconds lost in the overall encoding job.
<> I check every single one of the checkboxes (for High Profile output) or all but the 8x8 transform (for MP-compliant streams). You may want to consider un-checking chroma estimation for non-animated source, but I find the quality is always helped by it (controversial issue). Of course, you can disable B-frame search if you disabled B-frames, though it makes no difference to the codec.
<> Don't ever use CAVLC... My benchmarks prove that decoding CABLC streams is only slightly faster than CABAC and compression is drastically decreased by its use. In other words, always keep CABAC checked.
<> Personally, I find that the keyframe threshold of 45% and keyframe QP boost: 20 to be the optimal in achieving quality at medium bitrates. However, if you wish to produce Ultra-Low-Bitrate video (DVD backups less than one CD), consider reducing this to 0, for your video's quality (PSNR) per BM will increase.
<> I suggest use of hexagonal motion estimation if encoding speed is desired. It provides sufficient quality for its speed. However, if you have a particularly complex source (or if you really want to scrimp for the best possible quality), try Uneven Multi-Hexagon with a ME range of 16-32, depending on the framerate of the source. Generally lower framerates (12-15 as opposed to 23.976-29.97) need a higher estimation range.
<> Consult the Deblocking Guide near the end of this guide for the best deblocking settings.
I find that this method delivers the best possible quality and filesize while achieving fast encoding speed.
--Encoding: Absolute Max Quality--
These are the recommended settings which produce the maximum quality AVC encodes regardless of encoding speed.
Best Possible Quality (regardless of encoding speed)
<> Use 16 reference frames (the max allowed). The compression quality will be increased slightly at the expense of some encoding time.
<> Use 2-3 max consecutive B-frames* with 30% B-frame size reduction for medium-high bitrates (2-CD backups). Use 1 B-frame with 40% reduction on animated or low bitrate (1-CD backups or less) content. Use 0 B-frames if you are using Very Extremely Low streaming bitrates. Also, use 0 B-frames if you wish to encode archive quality material.
<> Enable All motion searches and CABAC for HP-compliant content (in other words, check all the checkboxes on the lower half of x264's GUI) OR all but the 8x8 DCT for MP-compliant content (although technically, HP content offers significantly better compression quality, it is yet unsupported in many multimedia applications.)
<> Sub-pel refinement should be set to 6 or 7 (CLI), the maximum.
<> In maximum-quality content, I find that the keyframe threshold of 35% and keyframe QP boost of either: 30 or 0 to be the optimal in achieving quality at medium-high bitrates in my "maximum possible quality" mode.
<> Use Exhaustive motion estimation with range of 16 (for pretty darn slow encoding and exemplary quality) or 32 (for painfully slow encoding but highest possible quality). Do not set this value higher than 32 or you will risk loosing quality. Consider 32 to be the peak of the quality mountain, so to speak.
<> See the "Bitrate Variability" section below for tips on how to set this field.
<> Consult the Deblocking Guide near the end of this guide for the best deblocking settings.
<> Take advantage of the newly released custom H.264 quantization matrices. These nifty little numerical spreadsheets control how and where the bitrate goes to each frame. Typically, these matrices are beginning to see some circulation, but they are still being tweaked to provide the best quality possible. The x264 codec currently supports this feature (rev. 365).
*B-frames can be activated in "pyramid" mode, which allows B-frames to serve as references. If you wish to use a lot of references (and thereby increase quality slightly), consider selecting the "Use as references" checkbox. However, this option may cause some crashes with both the encoder and the decoder (if either is old enough). For this reason, I recommend NOT using B-frame references.
Again, these are the best possible settings for optimal performance in the quality-only range.
--Bitrate Variability--
The bitrate variability feature controls how much your datarate can fluxuate at any given time. That is, if you plan on encoding at 500kbps and set this value to 40, the maximum amount of bitrate given to more complex scenes is 700kbps. In a nutshell, the lower you set this value, the better still/non-complex scenes look, but high-motion/complex scenes will look more shabby and garbled. The higher you set this value, the more equal the overall quality will become: still scenes would look worse than with a low value, and high-motion/complex scenes would look a lot better. Overall, it comes down to this: if your source content is an extremely high-paced surge of complex/fast motion (the background moves a lot, there are many scene changes, colors change a lot, high-contrast particles are always flying, etc), make the value higher. Else, leave it be or lower it if your source is...calmer.
---Deblocking Guide---
The deblocker (also known as the "in-loop filter") is an interesting feature of H.264 that was previously unavailable in the older MPEG-4 ASP standard. Such filtering is very useful for eliminating blocks, as its name suggests, but it is also very controversial because it can lead to some pretty bad misuse as well. When enabled (as it always should be, in my opinion, except in lossless, quantizer=0 encoding), it is typically centered at 0, with a threshold of 0.
Think of it as a clothes washing service. The deblocking threshold determines how much of the material needs to be "washed," while the deblocking strength determines how strong of a "washing" effect is needed to eliminate the blocks, or "stains." Naturally, if you don't wash (deblock) enough of the material, there will still be some stains (blocks) left, away from the washed area. Also, if you don't wash (deblock) hard or strong enough, the stains (blocks) simply won't fade/disappear. However, if you wash too much or scrub too hard, you'll ruin the material, because instead of stains, there will be the ugly lack of color where all the texture and detail rubbed out.
Both bars are initially set to 0 for a reason: this is the standard deblocking that will lead to the most accurate mix of deblocking while maintaining detail. If, however, you find the result unsatisfying, look to these tips:
<> If multiple consecutive B-frames are used, you may wish to increase deblocking strength to eliminate between-frame noise. This is especially important if you encode with Recode.
<> If you are encoding an animated source, heavier deblocking is suggested to eliminate all blocks possible. The drawn content is more resistant to smearing due to the high-contrast edges. On the other hand, if you are encoding a "real-life" video, especially one with intricate textures and low/poor lighting, consider decreasing the deblocking to preserve such things without creating a washing effect.
<> To determine what you think is best, try the standard 0/0 settings on a small but indicative sample of the source and experiment with it, keeping in mind the below guidelines.
<> For deblocking strength, try not go out of bounds of the -3 to 3 range. Generally, any more than 3 will turn your result into mush while decreasing PSNR and actually increasing the output file slightly. Any less than -3 may cause the result to look a bit too blocky and any lack of texture will merely become more apparent as all smoothing is taken out.
<> For deblocking threshold, it all depends on the source. Lighting and content type (edge contrast) seem to play the biggest role in determining what needs to be deblocked. Stick to the range of -4 to 4: if the value is set too high, everything is smeared. Setting it too low would mean that hardly anything actually gets deblocked, leaving sharp images with too much mosquito noise).
<> Try to keep a positive correlation between the two settings. That is, if you want heavier deblocking, make sure to increase the threshold so that more gets deblocked, and vise versa. Recall the comparison with the clothes washing: you don't want to heavily wash a small area while the rest remains unwashed; the unwashed areas will stand out more in stark vividness and provide an ugly visual effect.
---Adaptive Quantization---
Present in certain AVC encoders, lumi masking is an encoding technique in which darker areas are awarded less quality than lighter ones, giving more bitrate to the more "visible" parts of the video. This is an example of an encoding tool that can LOWER the PSNR quality measure but actually increase the quality that your eyes see.
<> The main determining factor in whether or not you would like to enable Lumi Masking is the darkness of your source. In films with very dark scenes (dungeons, tunnels, night, shadow), turn this OFF or LOW for good measure. If the source isn't very dark, I'd recommend you try moderate enhancement. Strong lumi masking can lead to problems (the codec might think important semi-dark things are "dark enough" to heavily reduce the quality and create blocking, which may be obvious to the viewer). Check out CiNcH's post below for more info...
Instructions
Install x264-Full package (builds by other ppl are not assured to work with some profiles). The profiles are now included in the package.
Fire up MeGUI-x264 and you will find the profiles in the "Video Profiles" drop down list. Before starting encoding set the bitrate or quantizer (manually or thru the calculator) and adjust the inloop filter accordingly to your needs/taste: keep it 1;1 or higher for anime/toons (max 3;3) and not higher than x264 defaults (0;0) for movies.
Profiles for maximizing Quality and hit a desired bitrate/filesize (2 or more passes, SLOW encoding, High Profile)
HQ-Slow: Pretty slow but not much. Quality is near the best.
HQ-Slower: Slower than HQ-Slow but it tries to squeeze some more compression thru the use of more reference frames.
HQ-Slowest: Same as HQ-Slower with maximized reference frames number and uses --subme=7.
HQ-Insane: Almost identical to HQ-Slowest but 3 passes.
Profiles for Common Encoding (2-passes)
CE-Highprofile: Common settings using high profile options (not compatible with QuickTime)
CE-Mainprofile: Common settings using main profile options (not compatible with QuickTime)
CE-QuickTime: Common settings for QuickTime 7 compatibility.
CE-Baseline: Common settings using baseline profile options (100% compatible with QuickTime 7)
Profiles for maximizing Speed and hit a desired bitrate/filesize (1-pass)
1P-Maxspeed: Everything disabled for max encoding speed.
1P-Intermediate: Intermediate settings for average speed and final quality.
1P-Goodquality: Settings for good quality with 1 pass.
Profiles for constant Quality/Quantizer (1-pass)
CQ-Lossless: Lossless encoding.
CQ-ASP_Q2_equiv: Almost the equivalent of MPEG-4 ASP Q2 encoding.
CQ-ASP_Q2_eq(crf): Same as above but with "constant quality" (constant rate factor).
Profiles for Anime/Toons (2 or more passes)
AE-Standard: 2 passes, Main Profile, standard quality.
AE-Goodquality: Slower than AE-Standard, High Profile, higher quality.
AE-Maxquality: Slowest mode, 3 passes and maxed out settings, best quality.
Profiles for Consoles and Portable Devices
PD-PSP: Profile for Sony PSP playback.
PD-Xbox_Hires: Profile for MS XBOX and higher than 512pixels resolution (latest XBMC is needed).
PD-Xbox_LoRes: Profile for MS XBOX and lower than 512pixels resolutions (latest XBMC is needed).
PD-iPod: Profile for iPod Video based on Arstechnica settings.
Download link: MeGUI-x264 Custom Video Profiles V21.
For questions, comments and requests ask in this thread.
Thanks goes to Sirber for suggestions for Anime profiles.