parent
ee66563717
commit
9655dc6677
@ -0,0 +1,267 @@ |
|||||||
|
--- |
||||||
|
title: >- |
||||||
|
Building AppImages with Nix |
||||||
|
description: >- |
||||||
|
With some process trees thrown in there for fun. |
||||||
|
series: nebula |
||||||
|
tags: tech |
||||||
|
--- |
||||||
|
|
||||||
|
It's been a bit since I've written an update on the cryptic nebula project, |
||||||
|
almost 5 months (since [this post][lastnix], which wasn't officially part of the |
||||||
|
blog series but whatever). Since then it's switched names to "cryptic-net", and |
||||||
|
that we would likely use [MinIO](https://min.io/) as our network storage |
||||||
|
service, but neither of those is the most interesting update. |
||||||
|
|
||||||
|
The project had been stalled because of a lack of a build system which could |
||||||
|
fulfill the following requirements: |
||||||
|
|
||||||
|
* Network configuration (static IP, VPN certificates) of individual hosts is |
||||||
|
baked into the binary they run. |
||||||
|
|
||||||
|
* Binaries are completely static; no external dependencies need to exist on the |
||||||
|
host in order to run them. |
||||||
|
|
||||||
|
* Each binary runs a composition of multiple sub-services, each being a separate |
||||||
|
sub-process, and all of them having been configured to work together (with |
||||||
|
some possible glue code on our side) to provide the features we want. |
||||||
|
|
||||||
|
* The builder itself should be deterministic; no matter where it runs it should |
||||||
|
produce the same binary given the same input parameters. |
||||||
|
|
||||||
|
Lacking such a build system we're not able to distribute the program in a way |
||||||
|
which "just works"; it would require some kind of configuration, or some kind of |
||||||
|
runtime environment to be set up, both of which would be a pain for users. And |
||||||
|
lacking a definite build system makes it difficult to move forward on any other |
||||||
|
aspect of a project, as it's not clear what may need to be redone in the future |
||||||
|
when the build system is decided upon. |
||||||
|
|
||||||
|
## Why not nix-bundle? |
||||||
|
|
||||||
|
My usage of [nix-bundle][nix-bundle] in a [previous post][lastnix] was an |
||||||
|
attempt at fulfilling these requirements. Nix in general does very well in |
||||||
|
fulfilling all but the second requirement, and nix-bundle was supposed to |
||||||
|
fulfill even that by packaging a nix derivation into a static binary. |
||||||
|
|
||||||
|
And all of this it did! Except that the mechanism of nix-bundle is a bit odd. |
||||||
|
The process of a nix-bundle'd binary jails itself within a chroot, which it then |
||||||
|
uses to fake the `/nix/store` path which nix built binaries expect to exist. |
||||||
|
|
||||||
|
This might work in a lot of cases, but it did not work in ours. For one, [nebula |
||||||
|
can't create its network interface when run from inside |
||||||
|
nix-bundle's chroot][nix-bundle-issue]. For another, being run in a chroot means |
||||||
|
there's going to be strange restrictions on what our binary is able to do and |
||||||
|
not. |
||||||
|
|
||||||
|
## AppImage |
||||||
|
|
||||||
|
What we really needed was an [AppImage][appimage]. AppImages are static binaries |
||||||
|
which can bundle complex applications, even those which don't expect to be |
||||||
|
bundled into single binaries. In this way the end result is the same as |
||||||
|
nix-bundle, but the mechanism AppImage uses is different and places far fewer |
||||||
|
restrictions on what we can and can't do with our program. |
||||||
|
|
||||||
|
## Building Sub-Services Statically with Nix |
||||||
|
|
||||||
|
It's probably possible to use nix to generate an AppImage which has the |
||||||
|
`/nix/store` built into it, similar to what nix-bundle does, and therefore not |
||||||
|
worry about whether the binaries it's bundling are static or not. But if your |
||||||
|
services are written in sane languages it's not that difficult to build them |
||||||
|
statically and dodge the issue. |
||||||
|
|
||||||
|
For example, here is how you build a go binary statically: |
||||||
|
|
||||||
|
``` |
||||||
|
{ |
||||||
|
buildGoModule, |
||||||
|
fetchFromGitHub, |
||||||
|
}: |
||||||
|
buildGoModule rec { |
||||||
|
pname = "nebula"; |
||||||
|
version = "1.4.0"; |
||||||
|
|
||||||
|
src = fetchFromGitHub { |
||||||
|
owner = "slackhq"; |
||||||
|
repo = pname; |
||||||
|
rev = "v${version}"; |
||||||
|
sha256 = "lu2/rSB9cFD7VUiK+niuqCX9CI2x+k4Pi+U5yksETSU="; |
||||||
|
}; |
||||||
|
|
||||||
|
vendorSha256 = "p1inJ9+NAb2d81cn+y+ofhxFz9ObUiLgj+9cACa6Jqg="; |
||||||
|
|
||||||
|
doCheck = false; |
||||||
|
|
||||||
|
subPackages = [ "cmd/nebula" "cmd/nebula-cert" ]; |
||||||
|
|
||||||
|
CGO_ENABLED=0; |
||||||
|
tags = [ "netgo" ]; |
||||||
|
ldflags = [ |
||||||
|
"-X main.Build=${version}" |
||||||
|
"-w" |
||||||
|
"-extldflags=-static" |
||||||
|
]; |
||||||
|
}; |
||||||
|
``` |
||||||
|
|
||||||
|
And here's how to statically build a C binary: |
||||||
|
|
||||||
|
``` |
||||||
|
{ |
||||||
|
stdenv, |
||||||
|
glibcStatic, # e.g. pkgs.glibc.static |
||||||
|
}: |
||||||
|
stdenv.mkDerivation rec { |
||||||
|
pname = "dnsmasq"; |
||||||
|
version = "2.85"; |
||||||
|
|
||||||
|
src = builtins.fetchurl { |
||||||
|
url = "https://www.thekelleys.org.uk/dnsmasq/${pname}-${version}.tar.xz"; |
||||||
|
sha256 = "sha256-rZjTgD32h+W5OAgPPSXGKP5ByHh1LQP7xhmXh/7jEvo="; |
||||||
|
}; |
||||||
|
|
||||||
|
nativeBuildInputs = [ glibcStatic ]; |
||||||
|
|
||||||
|
makeFlags = [ |
||||||
|
"LDFLAGS=-static" |
||||||
|
"DESTDIR=" |
||||||
|
"BINDIR=$(out)/bin" |
||||||
|
"MANDIR=$(out)/man" |
||||||
|
"LOCALEDIR=$(out)/share/locale" |
||||||
|
]; |
||||||
|
}; |
||||||
|
``` |
||||||
|
|
||||||
|
The derivations created by either of these expressions can be plugged right into |
||||||
|
the `pkgs.buildEnv` used to create the AppDir (see AppDir section below). |
||||||
|
|
||||||
|
## Process Manager |
||||||
|
|
||||||
|
An important piece of the puzzle for getting our nebula project into an AppImage |
||||||
|
was a process manager. We need something which can run multiple service |
||||||
|
processes simultaneously, restart processes which exit unexpectedly, gracefully |
||||||
|
handle shutting down all those processes, and coalesce the logs of all processes |
||||||
|
into a single stream. |
||||||
|
|
||||||
|
There are quite a few process managers out there which could fit the bill, but |
||||||
|
finding any which could be statically compiled ended up not being an easy task. |
||||||
|
In the end I decided to see how long it would take me to implement such a |
||||||
|
program in go, and hope it would be less time than it would take to get |
||||||
|
`circus`, a python program, bundled into the AppImage. |
||||||
|
|
||||||
|
2 hours later, [pmux][pmux] was born! Check it out. It's a go program so |
||||||
|
building it looks pretty similar to the nebula builder above, so I won't repeat |
||||||
|
it. However I will show the configuration we're using for it within the |
||||||
|
AppImage, to show how it ties all the processes together: |
||||||
|
|
||||||
|
```yaml |
||||||
|
processes: |
||||||
|
- name: nebula |
||||||
|
cmd: bin/nebula |
||||||
|
args: |
||||||
|
- "-config" |
||||||
|
- etc/nebula/nebula.yml |
||||||
|
|
||||||
|
- name: dnsmasq |
||||||
|
cmd: bin/dnsmasq |
||||||
|
args: |
||||||
|
- "-d" |
||||||
|
- "-C" |
||||||
|
- ${dnsmasq}/etc/dnsmasq/dnsmasq.conf |
||||||
|
``` |
||||||
|
|
||||||
|
## AppDir -> AppImage |
||||||
|
|
||||||
|
Generating an AppImage requires an AppDir. An AppDir is a directory which |
||||||
|
contains all files required by a program, rooted to the AppDir. For example, if |
||||||
|
the program expects a file to be at `/etc/some/conf`, then that file should be |
||||||
|
places in the AppDir at `<AppDir-path>/etc/some/conf`. |
||||||
|
|
||||||
|
[These docs](https://docs.appimage.org/packaging-guide/manual.html#ref-manual) |
||||||
|
were very helpful for me in figuring out how to construct the AppDir. I then |
||||||
|
used the `pkgs.buildEnv` utility to create an AppDir derivation containing |
||||||
|
everything the nebula project needs to run: |
||||||
|
|
||||||
|
``` |
||||||
|
appDir = pkgs.buildEnv { |
||||||
|
name = "my-AppDir"; |
||||||
|
paths = [ |
||||||
|
|
||||||
|
# real directory containing non-built files, e.g. the pmux config |
||||||
|
./AppDir |
||||||
|
|
||||||
|
# static binary derivations shown previously |
||||||
|
nebula |
||||||
|
dnsmasq |
||||||
|
pmux |
||||||
|
]; |
||||||
|
}; |
||||||
|
``` |
||||||
|
|
||||||
|
Once the AppDir is built one needs to use `appimagetool` to turn it into an |
||||||
|
AppImage. There is an `appimagetool` build in the standard nixpkgs, but |
||||||
|
unfortunately it doesn't seem to actually work... |
||||||
|
|
||||||
|
Luckily nix-bundle is working on AppImage support, and includes a custom build |
||||||
|
of `appimagetool` which does work! |
||||||
|
|
||||||
|
``` |
||||||
|
{ |
||||||
|
fetchFromGitHub, |
||||||
|
callPackage, |
||||||
|
}: let |
||||||
|
src = fetchFromGitHub { |
||||||
|
owner = "matthewbauer"; |
||||||
|
repo = "nix-bundle"; |
||||||
|
rev = "223f4ffc4179aa318c34dc873a08cb00090db829"; |
||||||
|
sha256 = "0pqpx9vnjk9h24h9qlv4la76lh5ykljch6g487b26r1r2s9zg7kh"; |
||||||
|
}; |
||||||
|
in |
||||||
|
callPackage "${src}/appimagetool.nix" {} |
||||||
|
``` |
||||||
|
|
||||||
|
Using `callPackage` on this expression will give you a functional `appimagetool` |
||||||
|
derivation. From there's it's a simple matter of writing a derivation which |
||||||
|
generates the AppImage from a created AppDir: |
||||||
|
|
||||||
|
``` |
||||||
|
{ |
||||||
|
appDir, |
||||||
|
appimagetool, |
||||||
|
}: |
||||||
|
pkgs.stdenv.mkDerivation { |
||||||
|
name = "program-name-AppImage"; |
||||||
|
|
||||||
|
src = appDir; |
||||||
|
buildInputs = [ appimagetool ]; |
||||||
|
ARCH = "x86_64"; # required by appimagetool |
||||||
|
|
||||||
|
builder = builtins.toFile "build.sh" '' |
||||||
|
source $stdenv/setup |
||||||
|
cp -rL "$src" buildAppDir |
||||||
|
chmod +w buildAppDir -R |
||||||
|
mkdir $out |
||||||
|
|
||||||
|
# program-name needs to match the desktop file in the AppDir |
||||||
|
appimagetool program-name "$out/program-name-bin" |
||||||
|
''; |
||||||
|
} |
||||||
|
``` |
||||||
|
|
||||||
|
Running that derivation deterministically spits out a binary at |
||||||
|
`result/program-name-bin` which can be executed and run immediately, on any |
||||||
|
system using the same CPU architecture. |
||||||
|
|
||||||
|
## Fin |
||||||
|
|
||||||
|
I'm extremely hyped to now have the ability to generate binaries for our nebula |
||||||
|
project that people can _just run_, without them worrying about which |
||||||
|
sub-services that binary is running under-the-hood. From a usability perspective |
||||||
|
it's way nicer than having to tell people to "install docker" or "install nix", |
||||||
|
and from a dev perspective we have a really solid foundation on which to build a |
||||||
|
quite complex application. |
||||||
|
|
||||||
|
[lastnix]: {% post_url 2021-04-22-composing-processes-into-a-static-binary-with-nix %} |
||||||
|
[nix-bundle]: https://github.com/matthewbauer/nix-bundle |
||||||
|
[nix-bundle-issue]: https://github.com/matthewbauer/nix-bundle/issues/78 |
||||||
|
[appimage]: https://appimage.org/ |
||||||
|
[pmux]: https://github.com/cryptic-io/pmux |
Loading…
Reference in new issue