MariaDB source visualisation with Gource

Gource is a tool which can take a source code tree and create beautiful visualisations out of it. I’ve used it a few times before for various projects. This weekend I spent a little bit of time playing with it and applying it to MairaDB Server to see what it would produce.

The end result can be seen on YouTube.

In this visualisation you can see every file in the source code as a coloured dot. The dots are clustered in directories which are linked together in the directory tree using the lines. Git users swam around the files and spray code into them. It gives you a real sense of just how much work goes into a project.

To create this I got the MariaDB tree from GitHub and switched to branch 10.3. I observed the first MariaDB 5.1 tag was on the date 26th October 2009 so used that as a starting point. I could have gone right back to the beginning of MySQL here but the video would have been a lot longer!

I used my Lenovo ThinkStation S30 to generate the video, this is a 6-core Ivy-Bridge Xeon with 64GB RAM and a GeForce 1050Ti graphics card. The reason I used this over my more powerful machines is the GeForce can be used both for the OpenGL requirements of Gource and to hardware encode the video using FFmpeg.

Gource and FFmpeg are in most Linux distribution repositories. I was using Fedora for this with the RPM Fusion repositories to give me the proprietary NVidia drivers and CUDA based H.264 encoding support.

The command line used was:

gource -1920x1080 --start-date '2009-10-26' --title "MariaDB" --seconds-per-day 0.1 -a 1 --colour-images --hide 'filenames,usernames' -r 60 -o - | ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i - -i mariadb-white.png -filter_complex "overlay=main_w-overlay_w-5:main_h-overlay_h-5" -codec:a copy -vcodec h264_nvenc -preset llhq -rc:v vbr_minqp -qmin:v 19 -qmax:v 21 -profile:v high -b:v 2500k -maxrate:v 5000k output.mp4

Breaking this down we are telling Gource to generate at 1080p starting at 2009-10-26, every day in git should take 0.1 seconds so that this video doesn’t go on for hours. The ‘-a’ flag auto skips if nothing happens for a whole second. We are hiding filenames and usernames simply because there are so many that they would cover up the visualisation. The final part of the Gource command tells it to stream our a PPM video to the pipe at 60 FPS.

For the FFmpeg command we tell it to receive a 60 FPS PPM video feed, add a PNG overlay of the MariaDB logo to the bottom right and pipe this though the NVidia H.264 encoder to generate our output mp4 file. The rest of the settings are to set the quality quite high so that the image is still relatively crisp at the end (YouTube uploading/re-encoding will have reduced the quality a little).

Whilst encoding you get to see the visualisation on the screen, make sure you don’t move your mouse over it as this will bring up context information which will also appear on the video. With FFmpeg using the GeForce to encode the video this is encoded in real time. Before this I tried piping the data over SSH to my larger 16 core HP Z620 workstation to encode but my 1GBit network was only fast enough to do 18 FPS.

I later added a Creative Commons licensed music track on YouTube just to add a bit of ambience.

You may notice the date jumping backwards a little bit due to things such as branches being merged. I haven’t yet seen a way to flatten this out.

Everything here is Open Source and east to tweak for your own software project. I’d be interested to see what others can do with these tool.