Background

I recently bought a 4K, 144Hz HDR monitor. The new monitor is quite good, so I wanted to test it out first.

I downloaded a large number of 4K videos, ready to enjoy them. However, I quickly realized that achieving the best possible playback experience is still a deep pit to fall into. When assessing video quality, it's not just about resolution; you also need to consider encoding formats, bitrate, color, HDR, and other factors.

Obviously, these days, the most convenient tool for tinkering with videos is ffmpeg. FFmpeg is a free, open-source software package that can record, convert, and stream audio and video in multiple formats, making it ideal for quickly and batch-processing video modifications.

Before You Begin

Install ffmpeg

The download URL for the latest version of ffmpeg is always: https://www.gyan.dev/ffmpeg/builds/ffmpeg-git-full.7z

Simply download it, extract the archive, place it in a directory of your choice, and add the path containing the exe files to the PATH environment variable.

Understanding Video Encoding Formats

Before getting started, it is necessary to understand the commonly used encoding formats, platform support, and the appropriate encoding to use in different situations.

FLV1

  • Ancient format. Flash Video (abbreviated as FLV) is a web video format used as a streaming format.
  • The typical file extension is .flv.
  • In ffmpeg, this codec format can be specified using flv1.
  • A classic, long-standing encoder. However, it has gradually been phased out over time. Currently, Bilibili's live streaming still utilizes it.
  • Video files tend to be large. However, being a streaming format, the file is resistant to corruption and supports resuming playback after an interruption.
  • Suitable for watching live streams in general scenarios.

H264

  • MPEG-4 Part 10, Advanced Video Coding (abbreviated as MPEG-4 AVC)
  • The common file extension is .mp4, though you may occasionally encounter the less standard .flv extension, which can also use this codec
  • In ffmpeg, you can specify this codec using libx264 (or h264_nvenc for NVIDIA GPUs)
  • A classic, well-established codec with near-universal compatibility
  • Moderate file size
  • Ideal for distributing videos of moderate size in general-purpose scenarios

HEVC

  • Also known as H.265 and MPEG-H Part 2
  • Generally uses the .mp4 extension
  • In ffmpeg, you can specify this encoding format using libx265 (use hevc_nvenc for NVIDIA GPUs)
  • Many players cannot open it by default. HEVC requires licensing fees. Many platforms have not purchased the license. Windows by default and Mozilla have not purchased it, resulting in playback issues.
  • Smaller file size. High encoding and decoding efficiency when using hardware decoding.
  • Suitable for storing and playing high-definition videos and movies that you have shot

VP9

  • An open-source, free encoding format developed by Google
  • The file extension is typically .webm
  • In ffmpeg, you can specify this encoding format using libvpx-vp9
  • Many older players cannot open it by default. For example, the one included with Windows. However, Chrome loves to play this format…
  • The file size is relatively small. However, playback feels very demanding on decoder performance.
  • Ideal for distributing high-quality, small-sized videos efficiently in web browsers

AV1

  • AOMedia Video 1 (abbreviated as AV1) is an open, royalty-free video coding format designed for streaming over the network. Google has invested significant effort in it, and it can be considered the next generation of VP9.
  • The file extension is typically .webm.
  • In ffmpeg, you can specify this codec by using libaom-av1.
  • Many older players cannot open this format by default. For example, the one built into Windows. However, Chrome handles this format very well…
  • The video file size is extremely small. However, playback places a high demand on the decoder's performance.
  • Ideal for efficiently distributing high-definition videos in browsers.

Choose an Encoder That Suits You

Based on the information above, I personally recommend transcoding all self-recorded videos and high-definition movies on disk to HEVC. If you prefer free and open-source software, you might consider VP9.

For large videos intended for distribution on the web, if your target audience primarily uses the Chrome browser, you should prioritize AV1 or VP9. Otherwise, consider H264.

For any video that is not particularly large or high-resolution, you can safely default to using H264.

Choosing the Right Bitrate for You

Bitrate directly impacts clarity. Even with the same resolution and frame rate, different bitrates can significantly affect the level of video compression.

Generally:

  • 1M is very unclear. There are artifacts and noise everywhere. Dynamic details are severely lost. It looks low quality.
  • 2M is barely acceptable. Artifacts are very noticeable, and some dynamic details are lost. It feels like AV (analog video) quality.
  • 4M is watchable. Artifacts and noise are still present, but they are not noticeable unless you look closely. Some details are lost, but it does not significantly affect the overall viewing experience.
  • 8M is quite good. You need to observe carefully to notice some artifacts and noise. In most cases, it will not affect the viewing experience.
  • 16M is very clear. You will not see any signs of compression. However, for high-resolution 4K footage, there is still slight loss.
  • 32M is so clear that most people cannot detect compression artifacts with the naked eye. It is fully comparable to the original source.
  • Going higher generally offers little benefit. It is not recommended to exceed 128M.

Once you have determined the bitrate you want to use, you can specify the video compression bitrate in ffmpeg using the parameter -b:v. For example, I generally use: -b:v 16M to compress my videos.

Usage Overview

ffmpeg has two usage modes: stream copying and stream encoding.

Stream Copying

If stream copying is used, the parameters are simple, efficiency is extremely high, and processor utilization is very low. It can often copy thousands of frames per second, which is hundreds of times faster than real-time video playback. This method is highly recommended.

However, stream copying cannot modify the video's bitrate or resolution, nor can it change the video format.

If live streaming is required, it is recommended to first encode the video into a format suitable for live streaming, and then use the stream copying method to push the stream.

Stream Encoding

If stream encoding is used, the parameters are complex, efficiency is extremely low, and processor utilization is very high. It generally achieves speeds comparable to or significantly slower than real-time video playback. This method is only suitable for non-live streaming scenarios.

However, stream encoding allows you to modify the stream's bitrate and resolution, as well as change the stream's format.

View Video Information

ffmpeg can directly display information such as video resolution, bitrate, frame rate, HDR, and encoder.

Use the command:

ffmpeg -i ./filename.mp4

For example, I recently tested my monitor's performance and obtained the following files:

file

We loaded them one by one using ffmpeg and observed the output.

Among them, Design[4K,SDR,H264,120fps].mp4 is a 4K, SDR, H.264-encoded video at 120fps. Its output shows:

Stream 0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 3840x2160 [SAR 1:1 DAR 16:9], 160934 kb/s, 120 fps, 120 tbr, 15360 tbn (default)

Here, 3840x2160 represents 4K. The absence of terms like bt2020nc/bt2020/smpte2084 indicates that it is not an HDR video. Seeing Video: h264 indicates H264 encoding. Seeing 120 fps indicates a frame rate of 120 fps.

For example, Japan[8K,SDR,AV1,60fps].webm is 8K, SDR, AV1 encoded, at 60fps. Its output can be seen as:

Stream 0:0: Video: av1 (Main), yuv420p(tv, bt709), 7680x4320, SAR 1:1 DAR 16:9, 59.94 fps, 59.94 tbr, 1k tbn (default)

For example, Cyberpunk[8K,HDR,HEVC,60fps].mp4 is an 8K, HDR video with HEVC encoding at 60fps. Its output would look like this:

Stream 0:0[0x1](und): Video: hevc (Main 10) (hev1 / 0x31766568), yuv420p10le(pc, bt2020nc/bt2020/smpte2084, progressive), 7680x4320 [SAR 1:1 DAR 16:9], 95559 kb/s, 60 fps, 60 tbr, 15360 tbn (default)

Note: HEVC is the H265 encoding. Additionally, seeing bt2020nc/bt2020/smpte2084 indicates that this is HDR (High Dynamic Range) video.

Video Transcoding

Video transcoding is the most common use case.

If you add the parameter:

-c copy

Or -c:v copy for video only, and -c:a copy for audio only. This means no transcoding is performed; the streams are copied directly, resulting in higher speed and lower CPU usage, but the encoding cannot be changed.

You can independently control the target encoding for the video and audio streams. Use the following parameters:

-c:v libx265

This allows converting the original video (regardless of its encoding) to HEVC encoding.

Example:

For instance, if I need to convert a video's encoding to HEVC with an 8M bitrate while keeping the audio encoding unchanged:

ffmpeg -i input.mp4 -codec:a copy -codec:v libx265 -b:v 8M output.mp4

Hardware Acceleration

FFmpeg can use NVIDIA graphics cards for hardware acceleration.

The following items need to be configured:

  • A supported graphics card
  • Graphics card drivers
  • You can enable GPU acceleration by simply replacing libx264 with h264_nvenc and libx265 with hevc_nvenc

Note that using NVIDIA's encoder by default may result in a relatively low bitrate. You can specify a higher bitrate. Consider adding the parameter: -b:v 10M

Live Streaming

Stream Copying for Live Streaming

> ffmpeg \
-re \
-i ./small.mp4 \
-c copy \
-f flv\
"rtmp://aaaaa"

Here, ffmpeg refers to the ffmpeg program itself.
The -re parameter simulates a real-time input source. By default, ffmpeg reads the source as fast as possible; this parameter limits the reading speed to one times real-time.
-i specifies the input video. In this example, the video is assumed to be in .mp4 format.
-f forces the output format. RTMP live streaming requires output in flv format.
-c copy indicates stream copying.
Finally, provide the RTMP address. Note that if using a Media Service address, you must append a /random_string to the end of the obtained push stream address; otherwise, an IO error will be returned.

If the server rejects the stream, it is likely because the format was not converted to flv. Videos encoded in H.264 can generally be streamed directly. If the video is not encoded in H.264, try removing the -c copy parameter and adding the -c:v h264 parameter.

If the server rejects the stream, it is likely because the audio format was not converted to aac. Audio encoded in aac can generally be streamed directly, but mp3 format is not supported. If the audio is not encoded in aac, try removing the -c copy parameter and adding the -c:a aac parameter.

In general,

RTMP live streaming servers typically accept video formats with h.264 encoding, usually in mp4 or flv formats.
RTMP live streaming servers typically accept audio formats with aac encoding, usually in aac or mp4 formats.

Warning: mp3 format audio encoded by the Lavc57.89 encoder cannot be streamed.

Real-Time Encoding During Streaming

> ffmpeg \
-re \
-i ./small.mp4 \
-f flv \
-s 1280x720 \
-c:v libx264 \
-c:a aac \
-b:v 2000k \
-b:a 128k \
"rtmp://channel42/fff"

Description

Here, ffmpeg refers to the ffmpeg program itself.
The -re parameter simulates a real-time input source. By default, ffmpeg reads the source as fast as possible; this parameter limits the reading speed to no more than one times the real-time rate.
-i specifies the input video. In this case, it is assumed that the video format is .mp4.
-f forces a specific output format. RTMP live streaming requires the output format to be flv.
-s sets the target resolution. Do not exceed 1280x720, otherwise the streaming bandwidth may not meet the required standards.
-c:v specifies the video encoder. If the input video is in mp4 format, using the h264 encoder is sufficient.
-c:a specifies the audio encoder. If the input video is in mp4 format, its audio is typically encoded with aac, so using the aac encoder is sufficient.
-b:v sets the video bitrate. It is recommended not to exceed 2000k.
-b:a sets the audio bitrate. It is recommended not to exceed 200k.
Finally, provide the RTMP address. Note that if you are using a Media Service address, you must append a /random_string to the copied streaming address; otherwise, an IO error will be returned.

Error Handling

0x20200194 Empty stream
0x2020019c Stream broken

Integrating Video and Audio

ffmpeg \
-i ./voice.mp3 \
-i ./s.mp4 \
-map 0:a \
-map 1:v \
-c:v h264 \
-c:a aac \
-shortest \
-c copy \
test.mp4

Description

ffmpeg refers to the ffmpeg program itself
-i specifies two inputs
-map maps streams
0:a extracts and maps audio from the first input; the first input is the audio
1:v extracts and maps video from the second input; the second input is the video
-c copy indicates no real-time encoding, only stream copying. If both the audio and video formats remain unchanged, you can include this parameter. Otherwise, you should manually specify the audio encoder, video encoder, and bitrate, as shown in the previous command.
-shortest sets the total length of the mapped output to the duration of the shortest stream; the tail of the longer stream will be discarded. This option is optional.

Combined with the streaming command above, you can implement real-time streaming of the integrated media.

Command example:

ffmpeg \
-re -i ./m.mp3 \
-re -i ./v.mp4 \
-map 0:a \
-map 1:v \
-c:v copy \
-c:a aac \
-f flv \
rtmp://autobatcd1265fd66/fff

Integrating Videos and Subtitles

ffmpeg -i subtitle.vtt subtitle.ass

Then burn the subtitles to the video:

ffmpeg -i video.mp4 -vf ass=subtitle.ass out.mp4

or

fmpeg -i input.mp4 -vf "subtitles=sub.vtt" output.mp4

Burning Audio onto Video

ffmpeg adelay document

ffmpeg amix document

ffmpeg loudnorm document

adaplay=milliseconds

inputs: The number of inputs. If unspecified, it defaults to 2.

ffmpeg.exe -i spring_no_audio.mp4 -i laizhe.mp3 -i jiuwen.mp3 -filter_complex "[1]adelay=1000[a1];[2]adelay=5000[a2];[a1][a2]amix=inputs=2,loudnorm[a]" -map 0:v -map "[a]"  -c:v copy output.mp4

Modify Video Resolution

ffmpeg -i in.mp4 -s 1280x720 out.mp4

It is highly recommended to add the parameter -c:a copy to avoid re-encoding the audio.

Adjusting Video Speed

15x faster

ffmpeg -i in.mp4 -filter:v "setpts=PTS/15" out.mp4

Adjust File Volume

This command also works very well with pure audio files like mp3 files.

ffmpeg -i in.mp4 -af 'volume=0.5' out.mp4

Clip a specific segment of a video

Starting from the 3rd second, capture 8 seconds of content.

ffmpeg -i in.mp4 -ss 00:00:03 -t 00:00:08 -async 1 cut.mp4

Extract a Region from a Video Using ffmpeg

ffmpeg -i in.mp4 -filter:v "crop=out_w:out_h:x:y" part.mp4
  • out_w is the width of the output rectangle
  • out_h is the height of the output rectangle
  • x and y specify the top left corner of the output rectangleplex "[0:v:0][1:a:0][2:v:0][2:a:0]concat=n=2:v=1:a=1[outv][outa]" -map "[outv]" -map "[outa]" output.mp4

Reverse Video (TENET Mode)

Traditional reversed video, but with the audio track preserved:

ffmpeg -i in.mp4  -vf reverse rev.mp4

Reverse the video and reverse the audio simultaneously:

ffmpeg -i in.mp4  -vf reverse -af areverse rev.mp4

However, the command above requires ffmpeg to load the entire video into RAM, which can make the computer very sluggish and unstable. Therefore, this approach should only be used for particularly small videos.

For larger videos, I recommend using the command below. (Based on bash, or on Windows, based on git-bash)

You need to first navigate to the folder containing the video you want to reverse in bash. The video to be reversed must be renamed to in.mp4 beforehand. After reversing, you will get out.mp4.

# in.mp4
# out.mp4

rm ./parts -rvf
rm ./fileList.txt

ffmpeg -i in.mp4 -map 0 -c copy -f segment -segment_time 300 -reset_timestamps 1 video_%03d.mp4
mkdir parts
mv ./video_* ./parts

cd parts
for f in *.mp4; do
    ffmpeg -i $f -vf reverse -af areverse ${f/.mp4/_reversed.mp4}
done

rm fileList.txt
touch fileList.txt

for f in ./*_reversed.mp4;  do 
    echo file \'$f\' > tmp.txt
    cat fileList.txt >> tmp.txt
    rm fileList.txt
    mv tmp.txt fileList.txt
done

cd ..

ffmpeg -f concat -safe 0 -i ./parts/fileList.txt -c copy output.mp4

rm ./parts -rvf
rm ./fileList.txt

Other Project Recommendations

Using ffmpeg alone is already sufficient to perform various operations on common videos, and even to start a live stream. However, we can go one step further: if you want to share your live stream with others, ffmpeg alone may not be enough.

Here, we recommend using the SRS project:

https://github.com/ossrs/srs

This is a highly reliable live streaming platform. It can receive RTMP live streams, transcode them in real time to FLV or HLS, and then push the stream to viewers for real-time playback.

  • FLV live streaming offers low latency and high quality, but it is not natively supported by browsers. You will need a dedicated player, which may result in higher power consumption.
  • HLS live streaming typically uses URLs ending in .m3u8. It has moderate latency and average video quality, but it can be played directly using the native <video> tag in browsers, generally with high efficiency.

If the viewer's browser receives an FLV stream, you can use flv.js to load the FLV live stream within a web page.

https://github.com/bilibili/flv.js

Once these components are set up, you can achieve a playback experience similar to Bilibili live streaming locally. Give it a try!