Our open-source SBOM and vulnerability scanning tools Syft and Grype, recently turned four years old. So I did what any nerd would do: render an animated visualization of the development using the now-venerable Gource. Initially, I wanted to render these videos at 120Hz framerate, but that didn't go well. Read on to find out how that panned out.
My employer (perhaps foolishly) gave me the keys to our Anchore YouTube and Anchore Vimeo accounts. You can find the video I rendered on YouTube or embedded below.
For those unaware, Gource is a popular open-source project by Andrew Caudwell. Its purpose is to visualize development with pretty OpenGL-rendered videos. You may have seen these animated glowing renders before, as Gource has been around for a while now.
Syft is Anchore's command-line tool and library for generating a software bill of materials (SBOM) from container images and filesystems. Grype is our vulnerability scanner for container images and filesystems. They're both fundamental components of our Anchore Enterprise platform but are also independently famous.
Generating the video
Plenty of guides online cover how to build Gource visualizations, which are pretty straightforward. Gource analyses the git log of changes in a repository to generate frames of animation which can be viewed or saved to a video. There are settings to control various aspects of the animation, which are well documented in the Gource Wiki.
By default, while Gource is running, a window displaying the animation will appear on your screen. So, if you want to see what the render will look like, most of the defaults are fine when running Gource directly.
Tweak the defaults
I wanted to limit the video duration, and render at a higher resolution than my laptop panel supports. I also wanted the window to be hidden while the process runs.
tl;dr Here's the full command line I used to generate and encode the 4K video in the background.
$ /usr/bin/xvfb-run --server-num=99 -e /dev/stdout \
-s '-screen 0 4096x2160x24 ' /usr/bin/gource \
--max-files 0 --font-scale 4 --output-framerate 60 \
-4096x2160 --auto-skip-seconds 0.1 --seconds-per-day 0.16 \
--bloom-multiplier 0.9 --fullscreen --highlight-users \
--multi-sampling --stop-at-end --high-dpi \
--user-image-dir ../faces/ --start-date 2020-05-07 \
--title 'Syft Development https://github.com/anchore/syft' \
-o - \
ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i - \
-vcodec libx264 -preset veryfast -pix_fmt yuv420p \
-crf 1 -threads 0 -bf 0 ../syft-4096x2160-60.mkvLet's take a step back and examine the preparatory steps and some interesting points to note.
Preparation
The first thing to do is to get Gource and ffmpeg. I'm using Ubuntu 24.04 on my ThinkPad Z13, so a simple sudo apt install gource ffmpeg works.
Grab the Syft and/or Grype source code.
$ mkdir -p ~/Videos/gource/
$ cd ~/Videos/gource
$ git clone https://github.com/anchore/syft
$ git clone https://github.com/anchore/grypeGource can use avatar images in the videos which represent the project contributors. I used gitfaces for this. Gitfaces is available from PyPI, so can be installed with pip install -U gitfaces or similar. Once installed, generate the avatars from within the project folder.
$ cd ~/Videos/gource/syft
$ mkdir ../faces
$ gitfaces . ../facesDo this for each project you wish to render out. I used a central ../faces folder as there would be some duplication between the projects I'm rendering. However, not everyone has an avatar, so they'll show up as an anonymous "head and shoulders" in the animation.
Test render
Perform a quick test to ensure Gource is installed correctly and the avatars are working.
$ cd ~/Videos/gource/syft
$ /usr/bin/gource --user-image-dir ../faces/ A default-sized window of 1052x834 should appear with nicely rendered blobs and lines. If you watch it for any appreciable length, you'll notice it can be boring in the gaps between commits. Gource has some options to improve this.
The --auto-skip-seconds option defines when Gource will skip to the next entry in the git log while there is no activity. The default is 3 seconds, which can be reduced. With --seconds-per-day we can set the render speed so we don't get a very long video.
I used 0.1 and 0.16, respectively. The result is a shorter, faster, more dynamic video. The Gource Wiki details many other options for Gource.
Up the resolution!
While the default 1052x834 video size is fine for a quick render, I wanted something much bigger. Using the '4 years in 4 minutes at 4K' heading would be fun, so I went for 4096x2160. My laptop doesn't have a 4K display (it's 2880x1800 natively), so I decided to render it in the background, saving it to a video.
To run it in the background, I used xvfb-run from the xvfb package on my Ubuntu system. A quick sudo apt install xvfb installed it. To run Gource inside xvfb we simply prefix the command line like this:
(this is not the full command, just a snippet to show the xvfb syntax)
$ /usr/bin/xvfb-run --server-num=99 -e /dev/stdout \
-s '-screen 0 4096x2160x24 ' /usr/bin/gource -4096x2160Note that the XServer's resolution matches the video's, and we use the fullscreen option in Gource to use the whole virtual display. Here we also specify the color bit-depth of the XServer - in this case 24.
Create the video
Using ffmpeg—the Swiss army knife of video encoding—we can turn Gource's output into a video. I used the x264 codec with some reasonable options. We can run these as two separate commands: one to generate a (huge) series of ppm images and the second to compress that into a reasonable file size.
$ /usr/bin/xvfb-run --server-num=99 -e /dev/stdout \
-s '-screen 0 4096x2160x24 ' /usr/bin/gource \
--max-files 0 --font-scale 4 --output-framerate 60 \
-4096x2160 --auto-skip-seconds 0.1 --seconds-per-day 0.16 \
--bloom-multiplier 0.9 --fullscreen --highlight-users \
--multi-sampling --stop-at-end --high-dpi \
--user-image-dir ../faces/ --start-date 2020-05-07 \
--title 'Syft Development: https://github.com/anchore/syft' \
-o ../syft-4096x2160-60.ppm
$ ffmpeg -y -r 60 -f image2pipe -vcodec ppm \
-i ../syft-4096x2160-60.ppm -vcodec libx264 \
-preset veryfast -pix_fmt yuv420p -crf 1 \
-threads 0 -bf 0 ../syft-4096x2160-60.mkvFour years of commits as uncompressed 4K60 images will fill the disk pretty fast. So it's preferable to chain the two commands together so we save time and don't waste too much disk space.
$ /usr/bin/xvfb-run --server-num=99 -e /dev/stdout \
-s '-screen 0 4096x2160x24 ' /usr/bin/gource \
--max-files 0 --font-scale 4 --output-framerate 60 \
-4096x2160 --auto-skip-seconds 0.1 --seconds-per-day 0.16 \
--bloom-multiplier 0.9 --fullscreen --highlight-users \
--multi-sampling --stop-at-end --high-dpi \
--user-image-dir ../faces/ --start-date 2020-05-07 \
--title 'Syft Development: https://github.com/anchore/syft' \
-o - ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i - \
-vcodec libx264 -preset veryfast -pix_fmt yuv420p \
-crf 1 -threads 0 -bf 0 ../syft-4096x2160-60.mkvOn my ThinkPad Z13 equipped with an AMD Ryzen 7 PRO 6860Z CPU, this takes around 42 minutes and generates a ~10GB mkv video. Here's what the resource utilisation looks like while this is running. Fully maxed out all the CPU cores. Toasty!
Challenges
More frames
Initially, I considered creating a video at 120fps rather than the default 60fps that Gource generates. However, Gource is limited in code to 25, 30, and 60fps. As an academic exercise, I patched Gource (diff below) to generate visualizations at the higher frame rate.
I'm not a C++ developer, nor do I play one on TV! But with a bit of grep and a small amount of trial and error, I modified and rebuilt Gource to add support for 120fps.
diff --git a/src/core b/src/core
--- a/src/core
+++ b/src/core
@@ -1 +1 @@
-Subproject commit f7fa400ec164f6fb36bcca5b85d2d2685cd3c7e8
+Subproject commit f7fa400ec164f6fb36bcca5b85d2d2685cd3c7e8-dirty
diff --git a/src/gource.cpp b/src/gource.cpp
index cf86c4f..755745f 100644
--- a/src/gource.cpp
+++ b/src/gource.cpp
@@ -153,7 +153,7 @@ Gource::Gource(FrameExporter* exporter) {
root = 0;
//min physics rate 60fps (ie maximum allowed delta 1.0/60)
- max_tick_rate = 1.0 / 60.0;
+ max_tick_rate = 1.0 / 120.0;
runtime = 0.0f;
frameskip = 0;
framecount = 0;
@@ -511,7 +511,7 @@ void Gource::setFrameExporter(FrameExporter* exporter, int video_framerate) {
this->frameskip = 0;
//calculate appropriate tick rate for video frame rate
- while(gource_framerate<60) {
+ while(gource_framerate<120) {
gource_framerate += video_framerate;
this->frameskip++;
}
I then re-ran Gource with --output-framerate 120 and ffmpeg with -r 120, which successfully generated the higher frame-rate files.
$ ls -lh
-rw-rw-r-- 1 alan alan 7.3G Jun 15 21:42 syft-2560x1440-60.mkv
-rw-rw-r-- 1 alan alan 8.9G Jun 15 22:14 grype-2560x1440-60.mkv
-rw-rw-r-- 1 alan alan 13G Jun 16 22:56 syft-2560x1440-120.mkv
-rw-rw-r-- 1 alan alan 16G Jun 16 22:33 grype-2560x1440-120.mkvAs you can see and probably expect on some test renders, with these settings, double the frames means double the size. I could have fiddled with ffmpeg to use better-optimized options, or a different codec, but decided against it.
There's an even more significant issue here. There are precious few places to host high-frame-rate videos; few people have the hardware, bandwidth, and motivation to watch them. So, I rolled back to 60fps for subsequent renders.
More pixels
While 4K (4096x2160) is fun and fits the story of "4 years in 4 minutes at 4K", I did consider trying to render out at 8K (7680×4320). After all, I had time on my hands at the weekend and spare CPU cycles, so why not?
Sadly, the hardware x264 encoder in my ThinkPad Z13 has a maximum canvas size of 4096x4096, which is far too small for 8K. I could have encoded using software rather than hardware acceleration, but that would have been ludicrously more time-consuming.
I do have an NVIDIA card but don't believe it's new enough to do 8K either, being a 'lowly' (these days) GTX 2080Ti. My work laptop is an M3 MacBook Pro. I didn't attempt rendering there because I couldn't fathom getting xvfb working to do off-screen rendering in Gource on macOS.
I have another four years to figure this out before my ‘8 years of Syft in 8 minutes at 8K’ video, though!
Minor edits
Once Gource and ffmpeg did their work, I used Kdenlive to add some music and our stock "top and tail" animated logo to the video and then rendered it for upload. The default compression settings in Kdenlive dramatically reduced the file size to something more manageable and uploadable!
Conclusion
Syft and Grype are - in open source terms - relatively young, with a small, dedicated team working on them. As such, the Gourse renders aren't as busy or complex as more well-established projects with bigger teams.
We certainly welcome external contributions over on the Syft and Grype repositories. We also have a new Anchore Community Discourse where you can discuss the projects and this article.
If you'd like to see how Syft and Grype are integral to your SBOM generation, vulnerability and policy enforcement tools, contact us and watch the guided tour.
I always find these renders technically neat, beautiful and relaxing to watch. The challenges of rendering them also led me down some interesting technical paths. I'd love to hear feedback and suggestions over on the Anchore Community Discourse
