One of the many things I’m working on for wolfSSL right now is a Markdown based documentation build system which has to massage some of the input sources quite a bit. The build dependencies for this are getting increasingly difficult to set up. I therefore decided to make it easier for everyone by using Docker Buildkit.
Docker Buildkit basically gets everything you need to build (OS and dependencies) and caches it for subsequent runs, builds the code and allows you to copy the assets out of the build. The following is the file I created, the file format should be similar to anyone who has used a Dockerfile before:
FROM texlive/texlive:latest AS builder WORKDIR /deps RUN apt-get -y update # Pandoc and mkdocs are the tools we use # Doxygen to convert the Doxygen docs # build-essential and cmake to build the dependencies # libfmt-dev nlohmann-json3-dev libspdlog-dev and libcxxopts-dev are needed by Doxybook2 # fonts-noto is needed by our PDF generator RUN apt-get -y install pandoc mkdocs doxygen git build-essential cmake libfmt-dev nlohmann-json3-dev libspdlog-dev libcxxopts-dev fonts-noto # Inja is a dep for Doxybook2 RUN git clone --depth=1 https://github.com/pantor/inja RUN cd inja && cmake . -DBUILD_TESTING=OFF -DBUILD_BENCHMARK=OFF && make install RUN git clone --depth=1 https://github.com/matusnovak/doxybook2 RUN cd doxybook2 && cmake . && make install WORKDIR /src/wolfssl # Cache getting wolfSSL source for Doxygen docs RUN git clone --depth 1 https://github.com/wolfSSL/wolfssl # Copy the source files into Docker, this and any subsequent steps won't be cached COPY . . # Parallel build PDF FROM builder AS stage1 WORKDIR /src/wolfssl RUN make pdf # Parallel build HTML FROM builder AS stage2 WORKDIR /src/wolfssl RUN make html FROM scratch AS export-stage COPY --from=stage1 /src/wolfssl/wolfssl.pdf . COPY --from=stage2 /src/wolfssl/html ./html
Everything up to the
COPY . . will be cached as the output is expected to be the same every time, this copy step will copy the sources over to the Docker image. It is slow to run the first time as the “texlive” Docker image is over 1GB in size and there are a few hundred MB in “apt” dependencies that need to be installed, so having all this cached is very useful.
I’m not too concerned that the wolfSSL git tree clone may be an old cache because the Makefile that is run in subsequent steps will update it. The key thing that makes Docker Buildkit different from just using
docker build is that the
FROM builder stages are executed in parallel, the progress output is also a lot more friendly:
You can see from the title bar here that this is running from a Makefile. The
make docker step simply looks like this:
.PHONY: docker docker: @DOCKER_BUILDKIT=1 docker build --target=export-stage -t doc_build --output=build .
Breaking this down the
@ just hides from the user the command being run.
DOCKER_BUILDKIT=1 enabled the Docker Buildkit mode. The
export-stage as seen in the Dockerfile is what we want to run, that then pulls in
stage2 which in-turn pulls in
builder. Basically if you read the
FROM steps in the Dockerfile from bottom to top you can see the step dependency tree. The
output parameter is where the file stage will write to in the local filesystem.
The final stage just copies the output files from the Docker build to the host filesystem, so that we actually have the required build output. Once the build is complete it looks like this:
The rest of this documentation build system will likely be opened up soon. We are working on cleaning it up and putting some finishing touches to it. But it uses a Markdown base for the chapters and combines it with Doxygen -> Markdown converted API reference. It then does some post-processing so that the Markdown is massaged correctly for Pandoc and mkdocs, which both use slightly different Markdown syntax.
Whilst this is just building documentation, this could be used to build anything where you want people to be able to compile no matter what OS they choose to develop on. It could also be incorporated as part of a CI/CD system.