NotesExperiment
VMAF 95, yet the stars were gone: why I compressed an intro video with H.264 instead of AV1
I compared AV1 and H.264 while shrinking the 36.9MB video on the PromptFilm page. AV1 scored higher on quality, but 284 of 439 faint stars disappeared.
At the top of the PromptFilm page, a 35-second space film plays on its own. It is 1080×1080 at 60fps with no sound, and it pulls back from Earth all the way to the observable universe. That one file was 36.9MB. When I first uploaded it, I encoded it at H.264 CRF 16, a very high setting, to keep as much quality as possible.
Sending 36.9MB every time someone opens the page is a burden for visitors and for hosting bandwidth. So on October 16 I decided to compress the video again. The goal was simple: as small as possible while looking no different from the original.
How I compared
Running every test on the full 35 seconds takes a long time, so I cut the first 10 seconds and used that as the reference. This 10-second source is 10.5MB. I encoded it with two codecs at several settings.
- H.264: the codec that plays on almost every device and browser today. x264 encoder,
preset slow. - AV1: a newer codec known for much smaller files at the same quality. SVT-AV1 encoder,
preset 5.
For both, I varied the CRF (a higher number means more compression) and measured how close each result was to the source with VMAF. VMAF is a quality metric made by Netflix; the closer to 100, the more the result looks like the source. A score of 93–95 is usually considered hard for people to tell apart.
# H.264
ffmpeg -i ref10.mp4 -c:v libx264 -preset slow -crf 22 -pix_fmt yuv420p -movflags +faststart -an h264_crf22.mp4
# AV1
ffmpeg -i ref10.mp4 -c:v libsvtav1 -preset 5 -crf 36 -an av1_crf36.mp4
# VMAF (measured every 2nd frame)
ffmpeg -i av1_crf36.mp4 -i ref10.mp4 -lavfi libvmaf=n_subsample=2 -f null -On scores alone, AV1 won
| Setting | 10 s size | VMAF |
|---|---|---|
| Source (H.264 CRF 16) | 10.5MB | reference |
| H.264 CRF 22 | 3.6MB | 94.6 |
| H.264 CRF 24 | 2.8MB | 93.3 |
| H.264 CRF 27 | 1.9MB | 90.5 |
| AV1 CRF 36 | 2.1MB | 95.3 |
| AV1 CRF 32 | 2.9MB | 96.1 |
| AV1 CRF 28 | 4.1MB | 96.8 |
| AV1 CRF 24 | 5.6MB | 97.2 |
AV1 CRF 36 was 40% smaller than H.264 CRF 22 and still scored higher. Re-encoding the full 35 seconds gave the same picture: AV1 CRF 36 was 7.8MB at VMAF 95.0, and H.264 CRF 22 was 16.1MB at 94.7. Going by the numbers, there was nothing to think about.
Then I looked at the screen
Opening the results one by one, I stopped at about 7 seconds. It is the scene showing the group of galaxies around the Milky Way. In the source, hundreds of tiny dots are scattered across the black screen. In the AV1 version, there were visibly fewer of them.
To avoid relying on a feeling, I counted. In the source frame I found every dot clearly brighter than its surroundings (one- or two-pixel dots at least twice as bright as the area around them): 439 in all. Then I read the brightness at the same positions in each encode and counted how many dropped below half their original brightness.
| Setting | 10 s size | VMAF | Dots dimmed below half (of 439) |
|---|---|---|---|
| H.264 CRF 22 | 3.6MB | 94.6 | 10 |
| H.264 CRF 24 | 2.8MB | 93.3 | 27 |
| H.264 CRF 27 | 1.9MB | 90.5 | 156 |
| AV1 CRF 36 | 2.1MB | 95.3 | 284 |
| AV1 CRF 32 | 2.9MB | 96.1 | 255 |
| AV1 CRF 28 | 4.1MB | 96.8 | 194 |
| AV1 CRF 24 | 5.6MB | 97.2 | 126 |
Why the score missed it
VMAF looks at the whole frame and estimates the quality a person would perceive. Hundreds of one- or two-pixel dots cover a tiny share of a 1080×1080 frame. As long as the bright galaxy cores, the text and the large objects stay intact, the score stays high.
To an encoder, dots like these are hard to tell apart from noise. Small dots spread evenly over a dark background cost little in the numbers when they are removed, so the encoder seems to have chosen higher compression. But in this film those dots are the very thing being shown. If the galaxies vanish from the "neighboring galaxies" scene, the scene loses its point.
There was a way to save AV1
SVT-AV1 has an option called variance boost. It gives more bits to flat areas with little change in brightness, so fine detail on a dark background is erased less. I re-encoded at CRF 30 with this option at strength 3.
| Setting | 10 s size | VMAF | Dimmed dots |
|---|---|---|---|
| H.264 CRF 22 | 3.6MB | 94.6 | 10 |
| AV1 CRF 30 + variance boost | 6.2MB | 97.2 | 11 |
The stars came back. But the file grew to 6.2MB, 1.7 times the size of H.264 CRF 22. For this film, the moment the stars were protected, AV1's size advantage disappeared. On top of that, using AV1 means keeping a second H.264 file for browsers and devices that cannot play AV1. That is just more files to manage.
Decision: H.264 CRF 22
In the end I chose H.264 CRF 22, preset slow, with faststart.
- The full 35 seconds went from 36.9MB to 16.1MB, 56% smaller. VMAF over the full length is 94.7.
- Only 10 of the 439 faint dots were dimmed. Without zooming in, it is hard to tell from the source.
faststartputs the file information at the front, so playback starts before the whole file arrives.
CRF 24 was also a candidate. The full 35 seconds came to 12.2MB, and only 27 dots were dimmed, which is not bad. But VMAF dropped to 93.0 and the dot loss nearly tripled, so for the first video people see on the page I kept some margin and went with CRF 22.
What I want to remember
This experiment taught me one thing: a quality score is an average. An average does not know which part of the video actually matters. For a space film where the stars are the point, measure the stars. For a screen recording where the text is the point, measure the text.
The method did not need to be elaborate. Finding what had to be preserved in the source, then reading the same positions in the compressed result, took a script of a few dozen lines. If the score looks good but something feels off, turning that feeling into a number takes about 30 minutes.
The video in this experiment is Cosmic Size Ladder, made with PromptFilm. What the film shows is explained on the films page.