appimages with nix

static/src/_posts/2021-04-22-composing-processes-into-a-static-binary-with-nix.md

@@ -3,6 +3,7 @@ title: >-
     Composing Processes Into a Static Binary With Nix
 description: >-
     Goodbye, docker-compose!
+tags: tech
 ---
 
 It's pretty frequent that one wants to use a project that requires multiple

static/src/_posts/2021-09-22-building-appimages-with-nix.md

@@ -0,0 +1,267 @@
+---
+title: >-
+    Building AppImages with Nix
+description: >-
+    With some process trees thrown in there for fun.
+series: nebula
+tags: tech
+---
+
+It's been a bit since I've written an update on the cryptic nebula project,
+almost 5 months (since [this post][lastnix], which wasn't officially part of the
+blog series but whatever). Since then it's switched names to "cryptic-net", and
+that we would likely use [MinIO](https://min.io/) as our network storage
+service, but neither of those is the most interesting update.
+
+The project had been stalled because of a lack of a build system which could
+fulfill the following requirements:
+
+* Network configuration (static IP, VPN certificates) of individual hosts is
+  baked into the binary they run.
+
+* Binaries are completely static; no external dependencies need to exist on the
+  host in order to run them.
+
+* Each binary runs a composition of multiple sub-services, each being a separate
+  sub-process, and all of them having been configured to work together (with
+  some possible glue code on our side) to provide the features we want.
+
+* The builder itself should be deterministic; no matter where it runs it should
+  produce the same binary given the same input parameters.
+
+Lacking such a build system we're not able to distribute the program in a way
+which "just works"; it would require some kind of configuration, or some kind of
+runtime environment to be set up, both of which would be a pain for users. And
+lacking a definite build system makes it difficult to move forward on any other
+aspect of a project, as it's not clear what may need to be redone in the future
+when the build system is decided upon.
+
+## Why not nix-bundle?
+
+My usage of [nix-bundle][nix-bundle] in a [previous post][lastnix] was an
+attempt at fulfilling these requirements. Nix in general does very well in
+fulfilling all but the second requirement, and nix-bundle was supposed to
+fulfill even that by packaging a nix derivation into a static binary.
+
+And all of this it did! Except that the mechanism of nix-bundle is a bit odd.
+The process of a nix-bundle'd binary jails itself within a chroot, which it then
+uses to fake the `/nix/store` path which nix built binaries expect to exist.
+
+This might work in a lot of cases, but it did not work in ours. For one, [nebula
+can't create its network interface when run from inside
+nix-bundle's chroot][nix-bundle-issue]. For another, being run in a chroot means
+there's going to be strange restrictions on what our binary is able to do and
+not.
+
+## AppImage
+
+What we really needed was an [AppImage][appimage]. AppImages are static binaries
+which can bundle complex applications, even those which don't expect to be
+bundled into single binaries. In this way the end result is the same as
+nix-bundle, but the mechanism AppImage uses is different and places far fewer
+restrictions on what we can and can't do with our program.
+
+## Building Sub-Services Statically with Nix
+
+It's probably possible to use nix to generate an AppImage which has the
+`/nix/store` built into it, similar to what nix-bundle does, and therefore not
+worry about whether the binaries it's bundling are static or not. But if your
+services are written in sane languages it's not that difficult to build them
+statically and dodge the issue.
+
+For example, here is how you build a go binary statically:
+
+```
+{
+    buildGoModule,
+    fetchFromGitHub,
+}:
+    buildGoModule rec {
+        pname = "nebula";
+        version = "1.4.0";
+
+        src = fetchFromGitHub {
+            owner = "slackhq";
+            repo = pname;
+            rev = "v${version}";
+            sha256 = "lu2/rSB9cFD7VUiK+niuqCX9CI2x+k4Pi+U5yksETSU=";
+        };
+
+        vendorSha256 = "p1inJ9+NAb2d81cn+y+ofhxFz9ObUiLgj+9cACa6Jqg=";
+
+        doCheck = false;
+
+        subPackages = [ "cmd/nebula" "cmd/nebula-cert" ];
+
+        CGO_ENABLED=0;
+        tags = [ "netgo" ];
+        ldflags = [
+            "-X main.Build=${version}"
+            "-w"
+            "-extldflags=-static"
+        ];
+    };
+```
+
+And here's how to statically build a C binary:
+
+```
+{
+    stdenv,
+    glibcStatic, # e.g. pkgs.glibc.static
+}:
+    stdenv.mkDerivation rec {
+        pname = "dnsmasq";
+        version = "2.85";
+
+        src = builtins.fetchurl {
+          url = "https://www.thekelleys.org.uk/dnsmasq/${pname}-${version}.tar.xz";
+          sha256 = "sha256-rZjTgD32h+W5OAgPPSXGKP5ByHh1LQP7xhmXh/7jEvo=";
+        };
+
+        nativeBuildInputs = [ glibcStatic ];
+
+        makeFlags = [
+            "LDFLAGS=-static"
+            "DESTDIR="
+            "BINDIR=$(out)/bin"
+            "MANDIR=$(out)/man"
+            "LOCALEDIR=$(out)/share/locale"
+        ];
+    };
+```
+
+The derivations created by either of these expressions can be plugged right into
+the `pkgs.buildEnv` used to create the AppDir (see AppDir section below).
+
+## Process Manager
+
+An important piece of the puzzle for getting our nebula project into an AppImage
+was a process manager. We need something which can run multiple service
+processes simultaneously, restart processes which exit unexpectedly, gracefully
+handle shutting down all those processes, and coalesce the logs of all processes
+into a single stream.
+
+There are quite a few process managers out there which could fit the bill, but
+finding any which could be statically compiled ended up not being an easy task.
+In the end I decided to see how long it would take me to implement such a
+program in go, and hope it would be less time than it would take to get
+`circus`, a python program, bundled into the AppImage.
+
+2 hours later, [pmux][pmux] was born! Check it out. It's a go program so
+building it looks pretty similar to the nebula builder above, so I won't repeat
+it. However I will show the configuration we're using for it within the
+AppImage, to show how it ties all the processes together:
+
+```yaml
+processes:
+    - name: nebula
+      cmd: bin/nebula
+      args:
+        - "-config"
+        - etc/nebula/nebula.yml
+
+    - name: dnsmasq
+      cmd: bin/dnsmasq
+      args:
+        - "-d"
+        - "-C"
+        - ${dnsmasq}/etc/dnsmasq/dnsmasq.conf
+```
+
+## AppDir -> AppImage
+
+Generating an AppImage requires an AppDir. An AppDir is a directory which
+contains all files required by a program, rooted to the AppDir. For example, if
+the program expects a file to be at `/etc/some/conf`, then that file should be
+places in the AppDir at `<AppDir-path>/etc/some/conf`.
+
+[These docs](https://docs.appimage.org/packaging-guide/manual.html#ref-manual)
+were very helpful for me in figuring out how to construct the AppDir. I then
+used the `pkgs.buildEnv` utility to create an AppDir derivation containing
+everything the nebula project needs to run:
+
+```
+    appDir = pkgs.buildEnv {
+        name = "my-AppDir";
+        paths = [
+
+            # real directory containing non-built files, e.g. the pmux config
+            ./AppDir
+
+            # static binary derivations shown previously
+            nebula
+            dnsmasq
+            pmux
+        ];
+    };
+```
+
+Once the AppDir is built one needs to use `appimagetool` to turn it into an
+AppImage. There is an `appimagetool` build in the standard nixpkgs, but
+unfortunately it doesn't seem to actually work...
+
+Luckily nix-bundle is working on AppImage support, and includes a custom build
+of `appimagetool` which does work!
+
+```
+{
+    fetchFromGitHub,
+    callPackage,
+}: let
+    src = fetchFromGitHub {
+        owner = "matthewbauer";
+        repo = "nix-bundle";
+        rev = "223f4ffc4179aa318c34dc873a08cb00090db829";
+        sha256 = "0pqpx9vnjk9h24h9qlv4la76lh5ykljch6g487b26r1r2s9zg7kh";
+    };
+in
+    callPackage "${src}/appimagetool.nix" {}
+```
+
+Using `callPackage` on this expression will give you a functional `appimagetool`
+derivation. From there's it's a simple matter of writing a derivation which
+generates the AppImage from a created AppDir:
+
+```
+{
+    appDir,
+    appimagetool,
+}:
+    pkgs.stdenv.mkDerivation {
+        name = "program-name-AppImage";
+
+        src = appDir;
+        buildInputs = [ appimagetool ];
+        ARCH = "x86_64"; # required by appimagetool
+
+        builder = builtins.toFile "build.sh" ''
+            source $stdenv/setup
+            cp -rL "$src" buildAppDir
+            chmod +w buildAppDir -R
+            mkdir $out
+
+            # program-name needs to match the desktop file in the AppDir
+            appimagetool program-name "$out/program-name-bin"
+        '';
+    }
+```
+
+Running that derivation deterministically spits out a binary at
+`result/program-name-bin` which can be executed and run immediately, on any
+system using the same CPU architecture.
+
+## Fin
+
+I'm extremely hyped to now have the ability to generate binaries for our nebula
+project that people can _just run_, without them worrying about which
+sub-services that binary is running under-the-hood. From a usability perspective
+it's way nicer than having to tell people to "install docker" or "install nix",
+and from a dev perspective we have a really solid foundation on which to build a
+quite complex application.
+
+[lastnix]: {% post_url 2021-04-22-composing-processes-into-a-static-binary-with-nix %}
+[nix-bundle]: https://github.com/matthewbauer/nix-bundle
+[nix-bundle-issue]: https://github.com/matthewbauer/nix-bundle/issues/78
+[appimage]: https://appimage.org/
+[pmux]: https://github.com/cryptic-io/pmux