Setting Up epub Build Workflow

Introduction

This journal entry was written by me, Brandon Nolet. I’m finally able to automatically build the epub files from my source for the book I’ll be writing. This was fun?

Build Flow

It’s important that you understand what the build flow is before I go into the journaling part of this post. I will be writing my book in org-mode, though the existing files I’ve been testing with are in markdown. From markdown, I use pandoc to convert the markdown files to an epub file.

So the build flow is as follows: clone repo>convert markdown files to epub using pandoc>upload epub files as a 'release'

Apt Docker

This was the first page I checked out and it seemed like everything would have been straightforward, but it was anything but. While the nodejs and golang build flows are super clean and worked out, my work/build flow was not. It’s not exactly a clear-cut thing to be working with pandoc, gitea, drone, and docker.

Most of the documentation for any of the docker images I found for pandoc was basically “run docker with this command and anything after this point is considered flags for pandoc”. I wasn’t entirely sure how to convert these “flags” into “commands” in the .drone.yml file so I just tried using pandoc in the commands section. Surprisingly, this worked out without an issue. I used this Docker container for the pandoc conversion. This was probably the most apt Docker container for this purpose.

Especially since it’s so small…relatively. It’s actually kind of large, but for the testing purposes, it was fine, I guess. I want to find something that doesn’t have all the LaTex tools since I really don’t need them. That should cut the image down to sub-100M sizes, but I haven’t tested that yet. We’ll have to see.

Git Push

I originally wanted to just push to the same repository the epub files after they had been built by pandoc. I thought this was going to be awesome.

I searched, however, for a solution that was specific to drone but I couldn’t really find anything. This is where my ignorance really shows because it wasn’t exactly obvious to me that people don’t really use CI for pushing to a repo, at least not unless it’s to mirror a repo. But there are specialized tools that are more suited for that purpose. Those tools are beyond the scope of this post.

But I went along and just found a simple git docker container based on the alpine distro and foraged through the documentation for how to use it. Again, this was one of those “flags come after the docker run commmand” containers. But I tried what I did with pandoc and just ran git commands straight in the container. That worked too!

It worked so well, and I was happy. I was able to build the book and have it available through the repo! But there was a problem. I had only built one version of the book (I have a light and a dark version of the epub. I might abandon that idea, but that’s irrelevant now). I went to modify the .drone.yml file to also include a pandoc command to build the dark-mode version of the epub, and then push it, but I was told the remote had been modified since.

Which made sense, because the commit that Drone did, meant that the remote repo was one commit ahead of my local version. So I pulled the changes and merged them into my local repo, and tried to push the Drone build config changes again. Same problem. This was weird. Why was the remote still ahead of my local repo?

Turns out that Drone was taking every single commit that it made as a change to the repo (which it was) and initiated another build. And another build. And another build. In the end there were 40 or so builds that took place before I had noticed what was going on and was able to turn off the builds for that repo.

I then modified the build config and removed the offending build steps. I had to re-think my strategy.

Releases

Of course! There’s a way to have “releases” in Gitea. These are basically files that are meant to be downloaded “standalone”. This is usually used for new releases of software/applications. Whenever there’s a new version of an app, you can download the installation/patch files from the Releases section of a repo.

I figured this would be a better way to download the epub rather than either cloning the repo or going through the repo to get to the epub files within some directory and then click “Raw” to get the actual file.

So I looked to see if there was a workflow for uploading to releases using Drone and it turns out there was actually a plugin for this very need! So I followed the documentation, creating an API key in Gitea, putting that as a Secret in Drone, and then put the required changes into the .drone.yml for the repo.

It worked first shot! I was so happy and so surprised throughout all of this that the “random” things that I tried just happened to work.

Last Thing

I don’t want to have the Drone runner build every time I push to the repo, so I’ve set it only build whenever I tag a commit. I could build every time I commit, but that seems like a waste, especially since I’d rather only have the epub containing full chapters rather than just abruptly cutting off wherever.

Moving Forward

Now that I’ve actually got the build flow set up, I’ve got some more ideas.

I’ll be rewriting what I have so far in org-mode because I want to take advantage of org-mode’s macro expansion. However, I want those macros to be in a separate file so that I’m not cluttering up the main source file. It’s a little pedantic, I know, but I think I can get it done.

What I imagine the build flow will be is the following: clone repo> docker container with emacs to convert org-mode file to org-mode file, expanding macros> convert org-mode file to epub using pandoc>upload to releases

It seems quite straightforward, but nothing about this was, as much as I was able to get done.

Conclusion

This was an eye-opening experience into the land of continuous integration. I learned that you can find a docker container for any purpose and also to never have a CI runner commit to the repo you’re building from and not have any checks to prevent build loops.

After getting the new build flow working, I’ll start finding a way to get my blog posts automatically posted when I commit them to the personal-blog repo.

I’ll try to put out a bit of an explainer on drone config files on the LinuxLiaison blog tomorrow, if I get the chance.