garage/doc/book/src/development/miscellaneous_notes.md

Miscellaneous Notes

Quirks about cargo2nix/rust in Nix

If you use submodules in your crate (like crdt and replication in garage_table), you must list them in default.nix

The Windows target does not work. it might be solvable through overrides. Indeed, we pass x86_64-pc-windows-gnu but mingw need x86_64-w64-mingw32

We have a simple PR on cargo2nix that fixes critical bugs but the project does not seem very active currently. We must use my patched version of cargo2nix to enable i686 and armv6l compilation. We might need to contribute to cargo2nix in the future.

Nix

Nix has no armv7 + musl toolchains but armv7l is backward compatible with armv6l.

Signing keys are generated with:

nix-store --generate-binary-cache-key nix.web.deuxfleurs.fr cache-priv-key.pem cache-pub-key.pem

We copy the secret key in our nix folder:

cp cache-priv-key.pem /etc/nix/signing-key.sec

Manually sign

We can sign the whole store with:

nix sign-paths --all -k /etc/nix/signing-key.sec

Or simply the current package and its dependencies with:

nix sign-paths --recursive -k /etc/nix/signing-key.sec

Setting a key in nix.conf will do the signature at build time automatically without additional commands, edit the nix.conf of your builder:

secret-key-files = /etc/nix/signing-key.sec
max-jobs = auto
cores = 8

Now you are ready to build your packages:

cat > $HOME/.awsrc <<EOF
export AWS_ACCESS_KEY_ID="xxx"
export AWS_SECRET_ACCESS_KEY="xxx"
EOF

# source each time you want to send on the cache
source ~/.awsrc

# copy garage build dependencies (and not only the output)
nix-build
nix-store -qR --include-outputs $(nix-instantiate default.nix) 
  | xargs nix copy --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage'
  
# copy shell dependencies
nix-build shell.nix -A inputDerivation
nix copy $(nix-store -qR result/) --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage' 

More example of nix-copy

# nix-build produces a result/ symlink
nix copy result/ --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage'

# alternative ways to use nix copy
nix copy nixpkgs.garage --to ...
nix copy /nix/store/3rbb9qsc2w6xl5xccz5ncfhy33nzv3dp-crate-garage-0.3.0 --to ...

Clear the cache:

mc rm --recursive --force garage/nix/

---

A desirable nix.conf for a consumer:

substituters = https://cache.nixos.org https://nix.web.deuxfleurs.fr
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= nix.web.deuxfleurs.fr:eTGL6kvaQn6cDR/F9lDYUIP9nCVR/kkshYfLDJf1yKs=

And now, whenever you run a command like:

nix-shell
nix-build

Our cache will be checked.

Some references about Nix

• https://doc.rust-lang.org/nightly/rustc/platform-support.html
• https://nix.dev/tutorials/cross-compilation
• https://nixos.org/manual/nix/unstable/package-management/s3-substituter.html
• https://fzakaria.com/2020/09/28/nix-copy-closure-your-nix-shell.html
• http://www.lpenz.org/articles/nixchannel/index.html

Drone

Do not try to set a build as trusted from the interface or the CLI tool, your request would be ignored. Instead, directly edit the database (table repos, column repo_trusted).

Drone can do parallelism both at the step and the pipeline level. At the step level, parallelism is restricted to the same runner.

Building Docker containers

We were:

• Unable to use the official Docker plugin because
  • it requires to mount docker socket in the container but it is not recommended
  • you cant set the platform when building
• Unable to use buildah because it needs CLONE_USERNS capability
• Unable to use the kaniko plugin for Drone as we can't set the target platform
• Unable to use the kaniko container provided by Google as we can't run arbitrary logic: we need to put our secret in .docker/config.json.

Finally we chose to build kaniko through nix and use it in a nix-shell.