Update documentation

README.md

@@ -45,13 +45,12 @@ decide which documents they need to care about.
 ### User Docs
 
 Users are participants who use network resources, but do not provide any network
-or storage resources themselves. Users may be accessing the network from a
+resources themselves. Users may be accessing the network from a mobile device,
-laptop, and so are not expected to be online at any particular moment.
+and so are not expected to be online at any particular moment.
 
 Documentation for users:
 
 * [Getting Started](docs/user/getting-started.md)
-* [Creating a daemon.yml File](docs/user/creating-a-daemonyml-file.md)
 * [Using DNS](docs/user/using-dns.md) (advanced)
 * Restic example (TODO)
 
@@ -63,7 +62,7 @@ Operator hosts will need at least one of the following to be useful:
 
 * A static public IP, or a dynamic public IP with [dDNS][ddns] set up.
 
-* At least 100GB of unused storage which can be reserved for the network.
+* At least 100GB of unused storage which can be reserved for the network. (TODO review storage requirements)
 
 Operators are expected to be familiar with server administration, and to not be
 afraid of a terminal.

docs/admin/adding-a-host-to-the-network.md

@@ -4,20 +4,6 @@ This document guides an admin through adding a single host to the network. Keep
 in mind that the steps described here must be done for _each_ host the user
 wishes to add.
 
-There are two ways for a user to add a host to the isle network.
-
-- If the user is savy enough to obtain their own `isle` binary, they can
-  do so. The admin can then generate a `bootstrap.json` file for their host,
-  give that to the user, and the user can run `isle daemon` using that
-  bootstrap file.
-
-- If the user is not so savy, the admin can generate a custom `isle`
-  binary with the `bootstrap.json` embedded into it. The user can be given this
-  binary and run `isle daemon` without any configuration on their end.
-
-From the admin's perspective the only difference between these cases is one
-extra step.
-
 ## Step 1: Choose Hostname
 
 The user will need to provide you with a name for their host. The name should
@@ -75,17 +61,3 @@ gpg -d <path to admin.json.gpg> | isle admin create-bootstrap \
 
 Note that the value of `--admin-path` is `-`, indicating that `admin.json`
 should be read from stdin.
-
-## Step 4: Optionally, Build Binary
-
-If you wish to embed the `bootstrap.json` into a custom binary for the user (to
-make installation _extremely_ easy for them) then you can run the following:
-
-```
-nix-build --arg bootstrap <path to bootstrap.json> -A appImage
-```
-
-The resulting binary can be found in the `result` directory which is created.
-
-This binary should be treated like a `bootstrap.json` in terms of its uniqueness
-and sensitivity.

docs/admin/creating-a-new-network.md

@@ -27,7 +27,7 @@ The requirements for this host are:
     behind a NAT, and/or allowing traffic on that UDP port in your hosts
     firewall.
 
-* At least 300 GB of disk storage space.
+* At least 300 GB of disk storage space. (TODO double check the storage space requirements)
 
   * At least 3 directories should be chosen, each of which will be committing at
     least 100GB. Ideally these directories should be on different physical
@@ -36,16 +36,9 @@ The requirements for this host are:
 * None of the resources being used for this network (the UDP port or storage
   locations) should be being used by other networks.
 
-## Step 1: Create a `daemon.yml` File
+## Step 1: Edit the `daemon.yml` File
 
-A `daemon.yml` will need to be created for use during network creation. You can
+Open `/etc/isle/daemon.yml` in a text editor and perform the following changes:
-create a new `daemon.yml` with default values filled in by doing:
-
-```
-isle admin create-network --dump-config > /path/to/daemon.yml
-```
-
-Open this file in a text editor and perform the following changes:
 
 * Set the `vpn.public_addr` field to the `host:port` your host is accessible on,
   where `host` is the static public IP/DNS name of your host, and `port` is the
@@ -104,7 +97,7 @@ you can run:
 
 ```
 sudo isle admin create-network \
-    --config-path /path/to/daemon.yml \
+    --config-path /etc/isle/daemon.yml \
     --name <name> \
     --ip-net <ip/subnet-prefix> \
     --domain <domain> \
@@ -117,7 +110,8 @@ A couple of notes here:
 
 * The `--ip-net` parameter is formed from both the subnet and the IP you chose
   within it. So if your subnet is `10.10.0.0/16`, and your chosen IP in that
-  subnet is `10.10.4.20`, then your `--ip-net` parameter will be `10.10.4.20/16`.
+  subnet is `10.10.4.20`, then your `--ip-net` parameter will be
+  `10.10.4.20/16`. (TODO expand a bit on what IP is being chosen).
 
 * Only one gpg recipient is specified. If you intend on including other users as
   network administrators you can add them to the recipients list at this step,
@@ -143,6 +137,8 @@ network for the daemon itself.
 
 At this point your host, and your network, are ready to go! You can reference
 the [Getting Started](../user/getting-started.md) document to set up your
-host's daemon process in a more permanent way.
+host's daemon process in a more permanent way. (TODO once creating a network is
+done via RPC then this will be out-of-date. Better to direct them to the
+operator docs, or maybe adding a new host).
 
 [ddns]: https://www.cloudflare.com/learning/dns/glossary/dynamic-dns/

docs/glossary.md

@@ -15,6 +15,10 @@ documentation and source code.
 - "isle network", "network" - A collection of hosts which communicate and share
   resources with each other via the Isle project.
 
+  - "garage cluster" - Garage is one of the sub-processes which isle is able to
+    run. These garage process connect together to form a cluster. We use the
+    term "cluster" in the context of garage to stay consistent with garage's
+    documentation and command-line.
+
 - "user" - A person who takes part in the usage, operation, or administration of
   an isle network.
-

docs/operator/contributing-a-lighthouse.md

@@ -26,18 +26,11 @@ traffic on that port to your host.
 
 Configure your host's firewall to allow all UDP traffic on that port.
 
-## Create daemon.yml
-
-First, if you haven't already, [create a `daemon.yml`
-file](../user/creating-a-daemonyml-file.md). This will be used to
-configure your `isle daemon` process with the public address that other
-hosts can find your daemon on.
-
 ## Edit daemon.yml
 
-Open your `daemon.yml` file  in a text editor, and find the `vpn.public_addr`
+Open your `/etc/isle/daemon.yml` file in a text editor, and find the
-field. Update that field to reflect your host's IP/DNS name and your chosen UDP
+`vpn.public_addr` field. Update that field to reflect your host's IP/DNS name
-port.
+and your chosen UDP port.
 
 ## Restart the Daemon
 

docs/operator/contributing-storage.md

@@ -4,16 +4,9 @@ If your host machine can be reasonably sure of being online most, if not all, of
 the time, and has 100GB or more of unused drive space you'd like to contribute
 to the network, then this document is for you.
 
-## Create `daemon.yml`
-
-First, if you haven't already, [create a `daemon.yml`
-file](../user/creating-a-daemonyml-file.md). This will be used to
-configure your `isle daemon` process with the storage locations and
-capacities you want to contribute.
-
 ## Edit `daemon.yml`
 
-Open your `daemon.yml` file  in a text editor, and find the
+Open your `/etc/isle/daemon.yml` file in a text editor, and find the
 `storage.allocations` section.
 
 Each allocation in the allocations list describes the space being contributed

docs/operator/firewalls.md

@@ -14,8 +14,8 @@ Isle uses the [nebula](https://github.com/slackhq/nebula) project to
 provide its VPN layer. Nebula ships with its own [builtin
 firewall](https://nebula.defined.net/docs/config/firewall), which only applies
 to connections coming in over the virtual network interface which it creates.
-This firewall can be manually configured as part of isle's
+This firewall can be manually configured as part of the `/etc/isle/daemon.yml`
-[`daemon.yml`](../user/creating-a-daemonyml-file.md) file.
+file.
 
 Any storage instances which are defined as part of the `daemon.yml` file will
 have their network ports automatically added to the VPN firewall by isle.

docs/roadmap.md

@@ -8,14 +8,6 @@ order they will be implemented.
 These items are listed more or less in the order they need to be completed, as
 they generally depend on the items previous to them.
 
-### Window Support + GUI
-
-Support for Windows is a must. This requirement also includes a simple GUI,
-which would essentially act as a thin layer on top of `daemon.yml` to start
-with.
-
-Depending on difficulty level, OSX support might be added at this stage as well.
-
 ### NATS
 
 Garage is currently used to handle eventually-consistent persistent storage, but
@@ -23,16 +15,15 @@ there is no mechanism for inter-host realtime communication as of yet. NATS
 would be a good candidate for this, as it uses a gossip protocol which does not
 require a central coordinator (I don't think), and is well supported.
 
-### Integration of [domani](https://code.betamike.com/micropelago/domani)
+### Integration of [Caddy](https://caddyserver.com/docs/)
 
-Integration of domani will require some changes on domani's end. We want domani
+Integration of Caddy's will require some plugins to be developed. We want Caddy
 to be able to store cert information in S3 (garage), so that all isle lighthouse
 nodes can potentially become gateways as well. Once done, it would be possible
 for lighthouses to forward public traffic to inner nodes.
 
-It should also be possible for users within the network to take advantage of
+It should also be possible for users within the network to take use lighthouse
-domani's hosting ability even without an always-on host of their own, without
+Caddy's to host their websites (and eventually gemini capsules) for them.
-requiring a passphrase.
 
 Most likely this integration will require NATS as well, to coordinate cache
 invalidation and cert refreshing.
@@ -45,6 +36,14 @@ files. The bootstrap file would be stored, encrypted, in garage, with the invite
 code being able to both identify and decrypt it. To instantiate a host, the user
 only needs to input the network domain name and the invite code.
 
+### Windows Support + GUI
+
+Support for Windows is a must. This requirement also includes a simple GUI,
+which would essentially act as a thin layer on top of `daemon.yml` to start
+with.
+
+Depending on difficulty level, OSX support might be added at this stage as well.
+
 ### FUSE Mount
 
 KBFS style. Every user should be able to mount virtual directories to their host
@@ -96,11 +95,16 @@ it works.
 ### Proper Linux Packages
 
 Rather than distributing raw binaries for Linux we should instead be
-distributing actual packages, e.g. deb files for debian/ubuntu, PKGBUILD for
+distributing actual packages.
-arch, rpm for fedora (if we care), etc... This will allow for properly setting
+
-capabilities for the binary at install time, so that it can be run as non-root,
+* deb files for debian/ubuntu
-and installing any necessary `.desktop` files so that it can be run as a GUI
+* PKGBUILD for arch (done)
-application.
+* rpm for fedora?
+* flatpak?
+
+This will allow for properly setting capabilities for the binary at install
+time, so that it can be run as non-root, and installing any necessary `.desktop`
+files so that it can be run as a GUI application.
 
 ### Mobile app
 
@@ -109,20 +113,6 @@ would be great. We are not able to use the existing nebula mobile app because it
 is not actually open-source, but we can at least use it as a reference to see
 how this can be accomplished.
 
-### Don't run as root
-
-It's currently a pretty hard requirement for `isle daemon` to run as
-root. This is due to:
-
-- nebula's network interface root to be started.
-
-- dnsmasq listening on port 53, generally a protected port.
-
-On linux it should be fairly straightforward to grant the entrypoint the
-necessary ambient capabilities up-front, and then drop down to a specified user.
-This is how the tests work. Doing this with other OS's will depend on how they
-work.
-
 ### DNS/Firewall Configuration
 
 Ideally Isle could detect the DNS/firewall subsystems being used on a per-OS

docs/user/creating-a-daemonyml-file.md

@@ -1,32 +0,0 @@
-# Creating a daemon.yml File
-
-The `isle daemon` process has generally sane defaults and does not need
-to be configured for most users. This document describes how to use the
-`daemon.yml` file to handle those cases where configuration is necessary.
-
-## Create daemon.yml
-
-First, create a `daemon.yml` file. You can create a new `daemon.yml` with
-default values filled in by doing:
-
-```
-isle daemon --dump-config > /path/to/daemon.yml
-```
-
-If you open that file in a text editor you can view all default values that
-`isle daemon` ships with, as well as documentation for all configurable
-parameters. Feel free to edit this file as needed.
-
-## Using daemon.yml
-
-With the `daemon.yml` created and configured, you can configure your daemon
-process to use it by passing it as the `--config-path` argument:
-
-```
-sudo isle daemon --config-path /path/to/daemon.yml
-```
-
-If you are an operator then your host should be running its `isle daemon`
-process in systemd (see [Getting Started](getting-started.md) if
-not), and you will need to modify the service file accordingly.
-

docs/user/getting-started.md

@@ -6,118 +6,126 @@ binary and joining a network.
 NOTE currently only linux machines with the following architectures are
 supported:
 
-- `x86_64` / `amd64`
+- `x86_64` (aka `amd64`)
-- `aarch64` / `arm64`
+- `aarch64` (aka `arm64`)
 - `i686`
 
-(Only `x86_64` has been tested.)
+(`i686` has not been tested.)
 
 More OSs and architectures coming soon!
 
-## Obtaining an isle Binary
+## Install isle
 
-### The Easy Way
+How isle gets installed depends on which Linux distribution you are using.
 
-Download the latest binary for your platform from
+### Archlinux (also Manjaro)
-[this link](https://code.betamike.com/micropelago/isle/releases/latest).
 
-### The Hard Way
+Download the latest `.pkg.tar.zst` package file for your platform from
+[this link][latest].
 
-Alternatively, you can build your own binary by running the following from the
+Install the package using pacman:
-project's root:
 
 ```
-nix-build -A appImage
+sudo pacman -U /path/to/isle-*.pkg.tar.zst
 ```
 
-(*NOTE* Dependencies of `isle` seemingly compile all of musl and rust
+### Other Distributions
-from scratch (it's not clear why, blame garage!). If you have not otherwise
-configured it, nix might be using a tmpfs as its build directory, and the
-capacity of this tmpfs will probably be exceeded by this build. You can change
-your build directory to somewhere on-disk by setting the TMPDIR environment
-variable for `nix-daemon` (see [this github issue][tmpdir-gh].))
 
-The resulting binary can be found in the `result` directory which is created.
+If a package file is not available for your distribution you can still install
+an AppImage directly. It is assumed that all commands below are run as root.
 
+Download the latest `.AppImage` binary for your platform from
+[this link][latest], and place it in your `/usr/bin` directory.
+
+Create a `daemon.yml` file using default values by doing:
+```
+mkdir -p /etc/isle/
+isle daemon --dump-config > /etc/isle/daemon.yml
+```
+
+Create a system user for the isle daemon to run as:
+```
+useradd -r -s /bin/false -C "isle Daemon" isle
+```
+
+If your distro uses systemd, download [the latest systemd service
+file][serviceFile] and place it in `/etc/systemd/system`. Run `systemctl
+daemon-reload` to ensure systemd has seen the new service file.
+
+If your distro uses an init system other than systemd then you will need to
+configure that yourself. You can use the systemd service file linked above as a
+reference.
+
+[serviceFile]: https://code.betamike.com/micropelago/isle/src/branch/main/dist/linux/isle.service
+
+### From Source
+
+(TODO probably move these instructions into the Dev docs section).
+
+Building from source requires [nix][nix].
+
+You can build your own AppImage by running the following from the project's
+root:
+
+```
+nix-build -A appImageBin
+```
+
+(*NOTE* The first time you run this a lot of things will be built from scratch.
+If you have not otherwise configured it, nix might be using a tmpfs as its build
+directory, and the capacity of this tmpfs will probably be exceeded by this
+build. You can change your build directory to somewhere on-disk by setting the
+TMPDIR environment variable for `nix-daemon` (see
+[this github issue][tmpdir-gh].))
+
+The resulting binary can be found under `result/bin`. From here you can continue
+with the instructions under the "AppImage" section above.
+
+[nix]: https://nixos.wiki/wiki/Nix_package_manager
 [tmpdir-gh]: https://github.com/NixOS/nix/issues/2098#issuecomment-383243838
 
-## Obtaining Your Bootstrap File
+## Add Users to `isle` Group (Optional)
 
-The `bootstrap.json` file contains all information required for your particular
+If you wish to run isle commands as a user other than root, you can add that
-host to join the network, and must be generated and provided to you by an admin
+user to the `isle` group:
-for the network.
-
-## Running the Daemon
-
-Once you have a binary and bootstrap file, you will need to run the `daemon`
-sub-command as the root user. This can most easily be done using the `sudo`
-command, in a terminal:
 
 ```
-sudo /path/to/isle daemon --bootstrap-path /path/to/bootstrap.json
+sudo usermod -aG isle username
 ```
 
-This will start the daemon process, which will keep running until you kill it
+## Start the isle Service
-with `ctrl-c`. The `--bootstrap-path /path/to/bootstrap.json` argument is only
-required the first time the daemon is run, it will be ignored on subsequent
-runs.
 
-You can double check that the daemon is running properly by pinging a private IP
+Once installed and bootstrapped you can enable and start the isle service by
-from the network in a separate terminal:
+doing:
 
 ```
-ping 10.10.0.1
-```
-
-If the pings are successful then your daemon is working!
-
-## Installing the Daemon as a Systemd Service
-
-NOTE in the future we will introduce an `install` sub-command which will
-automate most of this section.
-
-Rather than running the daemon manually, you can install it as a systemd
-service. This way your daemon will automatically start in the background on
-startup, and will be restarted if it has any issues.
-
-To do so, create a file at `/etc/systemd/system/isle.service` with the
-following contents:
-
-```
-[Unit]
-Description=isle
-Requires=network.target
-After=network.target
-
-[Service]
-Restart=always
-RestartSec=1s
-User=root
-ExecStart=/path/to/isle daemon
-
-[Install]
-WantedBy=multi-user.target
-```
-
-Remember to change the `/path/to/isle` part to the actual absolute path
-to your binary!
-
-Once created, perform the following commands in a terminal to enable the
-service:
-
-```
-sudo systemctl daemon-reload
 sudo systemctl enable --now isle
 ```
 
-You can check the service's status by doing:
+(NOTE If your distro uses an init system other than systemd then you will need
+to instead start isle according to that system's requirements.)
+
+## Join a Network
+
+This section will guide you through the process of joining an existing network
+of isle hosts. If instead you wish to create a new network for others to join
+then see the [Creating a New Network][creating-a-new-network] page.
+
+To join an existing network you will need to first obtain a `bootstrap.json`
+file. The `bootstrap.json` file contains all information required for your
+particular host to join the network, and must be generated and provided to you
+by an admin for the network.
+
+Once obtained, you can join the network by doing:
 
 ```
-sudo systemctl status isle
+isle network join --bootstrap-path /path/to/bootstrap.json
 ```
 
-and you can view its full logs by doing:
+After a few moments you will have successfully joined the network!
 
-```
+TODO block the `network join` call until joining has succeeded, or display a failure reason.
-sudo journalctl -lu isle
+
-```
+[creating-a-new-network]: ../admin/creating-a-new-network.md
+
+[latest]: https://code.betamike.com/micropelago/isle/releases/latest

docs/user/using-dns.md

@@ -10,8 +10,7 @@ network's domain name.
 
 If a request for a hostname not within the network's domain is received then the
 server will forward the request to a pre-configured public resolver. The set of
-public resolvers used can be configured using the
+public resolvers used can be configured in the `/etc/isle/daemon.yml` file.
-[daemon.yml](creating-a-daemonyml-file.md) file.
 
 This DNS server is an optional feature of Isle, and not required in general for
 making use of the network.

go/bootstrap/bootstrap.go

@@ -103,16 +103,29 @@ func FromFile(path string) (Bootstrap, error) {
 	defer f.Close()
 
 	var b Bootstrap
-
 	if err := json.NewDecoder(f).Decode(&b); err != nil {
 		return Bootstrap{}, fmt.Errorf("decoding json: %w", err)
 	}
 
-	if b.HostAssigned, err = b.SignedHostAssigned.UnwrapUnsafe(); err != nil {
+	return b, nil
-		return Bootstrap{}, fmt.Errorf("unwrapping host assigned: %w", err)
+}
+
+func (b *Bootstrap) UnmarshalJSON(data []byte) error {
+	type inner Bootstrap
+
+	err := json.Unmarshal(data, (*inner)(b))
+	if err != nil {
+		return err
 	}
 
-	return b, nil
+	b.HostAssigned, err = b.SignedHostAssigned.Unwrap(
+		b.CAPublicCredentials.SigningKey,
+	)
+	if err != nil {
+		return fmt.Errorf("unwrapping HostAssigned: %w", err)
+	}
+
+	return nil
 }
 
 // WriteTo writes the Bootstrap as a new bootstrap to the given io.Writer.
@@ -123,7 +136,6 @@ func (b Bootstrap) WriteTo(into io.Writer) error {
 // ThisHost is a shortcut for b.Hosts[b.HostName], but will panic if the
 // HostName isn't found in the Hosts map.
 func (b Bootstrap) ThisHost() Host {
-
 	host, ok := b.Hosts[b.Name]
 	if !ok {
 		panic(fmt.Sprintf("hostname %q not defined in bootstrap's hosts", b.Name))

go/bootstrap/hosts.go

@@ -78,5 +78,9 @@ type Host struct {
 // This assumes that the Host and its data has already been verified against the
 // CA signing key.
 func (h Host) IP() net.IP {
-	return h.PublicCredentials.Cert.Unwrap().Details.Ips[0].IP
+	cert := h.PublicCredentials.Cert.Unwrap()
+	if len(cert.Details.Ips) == 0 {
+		panic(fmt.Sprintf("host %q not configured with any ips: %+v", h.Name, h))
+	}
+	return cert.Details.Ips[0].IP
 }

go/cmd/entrypoint/daemon.go

@@ -2,15 +2,11 @@ package main
 
 import (
 	"context"
-	"errors"
 	"fmt"
-	"io/fs"
 	"os"
 
-	"isle/bootstrap"
 	"isle/daemon"
 
-	"dev.mediocregopher.com/mediocre-go-lib.git/mctx"
 	"dev.mediocregopher.com/mediocre-go-lib.git/mlog"
 )
 
@@ -31,11 +27,6 @@ var subCmdDaemon = subCmd{
 			"Write the default configuration file to stdout and exit.",
 		)
 
-		bootstrapPath := flags.StringP(
-			"bootstrap-path", "b", "",
-			`Path to a bootstrap.json file. This only needs to be provided the first time the daemon is started, after that it is ignored. If the isle binary has a bootstrap built into it then this argument is always optional.`,
-		)
-
 		logLevelStr := flags.StringP(
 			"log-level", "l", "info",
 			`Maximum log level which should be output. Values can be "debug", "info", "warn", "error", "fatal". Does not apply to sub-processes`,
@@ -64,72 +55,17 @@ var subCmdDaemon = subCmd{
 		}
 		defer runtimeDirCleanup()
 
-		var (
-			bootstrapStateDirPath = bootstrap.StateDirPath(daemonEnvVars.StateDirPath)
-			bootstrapAppDirPath   = bootstrap.AppDirPath(envAppDirPath)
-
-			hostBootstrapPath string
-			hostBootstrap     bootstrap.Bootstrap
-		)
-
-		tryLoadBootstrap := func(path string) bool {
-			ctx := mctx.Annotate(ctx, "bootstrapFilePath", path)
-
-			if err != nil {
-				return false
-
-			} else if hostBootstrap, err = bootstrap.FromFile(path); errors.Is(err, fs.ErrNotExist) {
-				logger.WarnString(ctx, "bootstrap file not found")
-				err = nil
-				return false
-
-			} else if err != nil {
-				err = fmt.Errorf("parsing bootstrap.json at %q: %w", path, err)
-				return false
-			}
-
-			logger.Info(ctx, "bootstrap file found")
-
-			hostBootstrapPath = path
-			return true
-		}
-
-		switch {
-		case tryLoadBootstrap(bootstrapStateDirPath):
-		case *bootstrapPath != "" && tryLoadBootstrap(*bootstrapPath):
-		case tryLoadBootstrap(bootstrapAppDirPath):
-		case err != nil:
-			return fmt.Errorf("attempting to load bootstrap.json file: %w", err)
-		default:
-			return errors.New("No bootstrap.json file could be found, and one is not provided with --bootstrap-path")
-		}
-
-		if hostBootstrapPath != bootstrapStateDirPath {
-
-			// If the bootstrap file is not being stored in the data dir, copy
-			// it there, so it can be loaded from there next time.
-			if err := writeBootstrapToStateDir(hostBootstrap); err != nil {
-				return fmt.Errorf("writing bootstrap.json to data dir: %w", err)
-			}
-		}
-
 		daemonConfig, err := daemon.LoadConfig(envAppDirPath, *daemonConfigPath)
 		if err != nil {
 			return fmt.Errorf("loading daemon config: %w", err)
 		}
 
-		// we update this Host's data using whatever configuration has been
+		daemonInst, err := daemon.NewDaemon(
-		// provided by the daemon config. This way the daemon has the most
+			logger, daemonConfig, envBinDirPath, nil,
-		// up-to-date possible bootstrap. This updated bootstrap will later get
-		// updated in garage as a background daemon task, so other hosts will
-		// see it as well.
-		if hostBootstrap, err = coalesceDaemonConfigAndBootstrap(hostBootstrap, daemonConfig); err != nil {
-			return fmt.Errorf("merging daemon config into bootstrap data: %w", err)
-		}
-
-		daemonInst := daemon.NewDaemon(
-			logger, daemonConfig, envBinDirPath, hostBootstrap, nil,
 		)
+		if err != nil {
+			return fmt.Errorf("starting daemon: %w", err)
+		}
 		defer func() {
 			logger.Info(ctx, "Stopping child processes")
 			if err := daemonInst.Shutdown(); err != nil {

go/cmd/entrypoint/hosts.go

@@ -1,7 +1,6 @@
 package main
 
 import (
-	"encoding/json"
 	"errors"
 	"fmt"
 	"isle/bootstrap"
@@ -30,17 +29,12 @@ var subCmdHostsList = subCmd{
 
 		ctx := subCmdCtx.ctx
 
-		var resRaw json.RawMessage
+		var res daemon.GetHostsResult
-		err := subCmdCtx.daemonRCPClient.Call(ctx, &resRaw, "GetHosts", nil)
+		err := subCmdCtx.daemonRCPClient.Call(ctx, &res, "GetHosts", nil)
 		if err != nil {
 			return fmt.Errorf("calling GetHosts: %w", err)
 		}
 
-		var res daemon.GetHostsResult
-		if err := json.Unmarshal(resRaw, &res); err != nil {
-			return fmt.Errorf("unmarshaling %s into %T: %w", string(resRaw), res, err)
-		}
-
 		type host struct {
 			Name string
 			VPN  struct {

go/cmd/entrypoint/main.go

@@ -66,6 +66,7 @@ func main() {
 		subCmdGarage,
 		subCmdHosts,
 		subCmdNebula,
+		subCmdNetwork,
 		subCmdVersion,
 	)
 

go/cmd/entrypoint/network.go

@@ -0,0 +1,51 @@
+package main
+
+import (
+	"errors"
+	"fmt"
+	"isle/bootstrap"
+)
+
+var subCmdNetworkJoin = subCmd{
+	name:  "join",
+	descr: "Joins this host to an existing network",
+	do: func(subCmdCtx subCmdCtx) error {
+		var (
+			ctx   = subCmdCtx.ctx
+			flags = subCmdCtx.flagSet(false)
+		)
+
+		bootstrapPath := flags.StringP(
+			"bootstrap-path", "b", "", "Path to a bootstrap.json file.",
+		)
+
+		if err := flags.Parse(subCmdCtx.args); err != nil {
+			return fmt.Errorf("parsing flags: %w", err)
+		}
+
+		if *bootstrapPath == "" {
+			return errors.New("--bootstrap-path is required")
+		}
+
+		newBootstrap, err := bootstrap.FromFile(*bootstrapPath)
+		if err != nil {
+			return fmt.Errorf(
+				"loading bootstrap from %q: %w", *bootstrapPath, err,
+			)
+		}
+
+		return subCmdCtx.daemonRCPClient.Call(
+			ctx, nil, "JoinNetwork", newBootstrap,
+		)
+	},
+}
+
+var subCmdNetwork = subCmd{
+	name:  "network",
+	descr: "Sub-commands related to network membership",
+	do: func(subCmdCtx subCmdCtx) error {
+		return subCmdCtx.doSubCmd(
+			subCmdNetworkJoin,
+		)
+	},
+}

go/daemon/bootstrap.go

@@ -1,32 +1,14 @@
 package daemon
 
 import (
-	"errors"
 	"fmt"
-	"io/fs"
 	"os"
 	"path/filepath"
 
 	"isle/bootstrap"
+	"isle/garage/garagesrv"
 )
 
-func loadHostBootstrap(stateDirPath string) (bootstrap.Bootstrap, error) {
-	path := bootstrap.StateDirPath(stateDirPath)
-
-	hostBootstrap, err := bootstrap.FromFile(path)
-	if errors.Is(err, fs.ErrNotExist) {
-		return bootstrap.Bootstrap{}, fmt.Errorf(
-			"%q not found, has the daemon ever been run?",
-			stateDirPath,
-		)
-
-	} else if err != nil {
-		return bootstrap.Bootstrap{}, fmt.Errorf("loading %q: %w", stateDirPath, err)
-	}
-
-	return hostBootstrap, nil
-}
-
 func writeBootstrapToStateDir(
 	stateDirPath string, hostBootstrap bootstrap.Bootstrap,
 ) error {
@@ -48,3 +30,43 @@ func writeBootstrapToStateDir(
 
 	return hostBootstrap.WriteTo(f)
 }
+
+func coalesceDaemonConfigAndBootstrap(
+	daemonConfig Config, hostBootstrap bootstrap.Bootstrap,
+) (
+	bootstrap.Bootstrap, error,
+) {
+	host := bootstrap.Host{
+		HostAssigned: hostBootstrap.HostAssigned,
+		HostConfigured: bootstrap.HostConfigured{
+			Nebula: bootstrap.NebulaHost{
+				PublicAddr: daemonConfig.VPN.PublicAddr,
+			},
+		},
+	}
+
+	if allocs := daemonConfig.Storage.Allocations; len(allocs) > 0 {
+
+		for i, alloc := range allocs {
+
+			id, rpcPort, err := garagesrv.InitAlloc(alloc.MetaPath, alloc.RPCPort)
+			if err != nil {
+				return bootstrap.Bootstrap{}, fmt.Errorf(
+					"initializing alloc at %q: %w", alloc.MetaPath, err,
+				)
+			}
+
+			host.Garage.Instances = append(host.Garage.Instances, bootstrap.GarageHostInstance{
+				ID:        id,
+				RPCPort:   rpcPort,
+				S3APIPort: alloc.S3APIPort,
+			})
+
+			allocs[i].RPCPort = rpcPort
+		}
+	}
+
+	hostBootstrap.Hosts[host.Name] = host
+
+	return hostBootstrap, nil
+}

go/daemon/daemon.go

@@ -8,6 +8,7 @@ import (
 	"errors"
 	"fmt"
 	"io"
+	"io/fs"
 	"isle/bootstrap"
 	"os"
 	"sync"
@@ -20,6 +21,13 @@ import (
 // with isle, typically via the unix socket.
 type Daemon interface {
 
+	// JoinNetwork joins the Daemon to an existing network using the given
+	// Bootstrap.
+	//
+	// Errors:
+	// - ErrAlreadyJoined
+	JoinNetwork(context.Context, bootstrap.Bootstrap) error
+
 	// GetGarageBootstrapHosts loads (and verifies) the <hostname>.json.signed
 	// file for all hosts stored in garage.
 	GetGarageBootstrapHosts(
@@ -67,7 +75,8 @@ func (o *Opts) withDefaults() *Opts {
 }
 
 const (
-	daemonStateInitializing = iota
+	daemonStateNoNetwork = iota
+	daemonStateInitializing
 	daemonStateOk
 	daemonStateRestarting
 	daemonStateShutdown
@@ -79,13 +88,13 @@ type daemon struct {
 	envBinDirPath string
 	opts          *Opts
 
-	l             sync.Mutex
+	l             sync.RWMutex
 	state         int
 	children      *Children
 	currBootstrap bootstrap.Bootstrap
 
-	cancelFn  context.CancelFunc
+	shutdownCh chan struct{}
-	stoppedCh chan struct{}
+	wg         sync.WaitGroup
 }
 
 // NewDaemon initializes and returns a Daemon instance which will manage all
@@ -110,43 +119,89 @@ func NewDaemon(
 	logger *mlog.Logger,
 	daemonConfig Config,
 	envBinDirPath string,
-	currBootstrap bootstrap.Bootstrap,
 	opts *Opts,
-) Daemon {
+) (
-	ctx, cancelFn := context.WithCancel(context.Background())
+	Daemon, error,
+) {
+	var (
+		d = &daemon{
+			logger:        logger,
+			daemonConfig:  daemonConfig,
+			envBinDirPath: envBinDirPath,
+			opts:          opts.withDefaults(),
+			shutdownCh:    make(chan struct{}),
+		}
+		bootstrapFilePath = bootstrap.StateDirPath(d.opts.EnvVars.StateDirPath)
+	)
 
-	d := &daemon{
+	currBootstrap, err := bootstrap.FromFile(bootstrapFilePath)
-		logger:        logger,
+	if errors.Is(err, fs.ErrNotExist) {
-		daemonConfig:  daemonConfig,
+		// daemon has never had a network created or joined
-		envBinDirPath: envBinDirPath,
+	} else if err != nil {
-		opts:          opts.withDefaults(),
+		return nil, fmt.Errorf(
-		currBootstrap: currBootstrap,
+			"loading bootstrap from %q: %w", bootstrapFilePath, err,
-		cancelFn:      cancelFn,
+		)
-		stoppedCh:     make(chan struct{}),
+	} else if err := d.initialize(currBootstrap); err != nil {
+		return nil, fmt.Errorf("initializing with bootstrap: %w", err)
 	}
 
-	go func() {
+	return d, nil
-		d.restartLoop(ctx)
-		d.logger.Debug(ctx, "DaemonRestarter stopped")
-		close(d.stoppedCh)
-	}()
-
-	return d
 }
 
-func withInnerChildren[Res any](
+func (d *daemon) initialize(currBootstrap bootstrap.Bootstrap) error {
-	d *daemon, fn func(*Children) (Res, error),
+	// we update this Host's data using whatever configuration has been provided
+	// by the daemon config. This way the daemon has the most up-to-date
+	// possible bootstrap. This updated bootstrap will later get updated in
+	// garage as a background daemon task, so other hosts will see it as well.
+	currBootstrap, err := coalesceDaemonConfigAndBootstrap(
+		d.daemonConfig, currBootstrap,
+	)
+	if err != nil {
+		return fmt.Errorf("combining daemon configuration into bootstrap: %w", err)
+	}
+
+	err = writeBootstrapToStateDir(d.opts.EnvVars.StateDirPath, currBootstrap)
+	if err != nil {
+		return fmt.Errorf("writing bootstrap to state dir: %w", err)
+	}
+
+	d.currBootstrap = currBootstrap
+	d.state = daemonStateInitializing
+
+	ctx, cancel := context.WithCancel(context.Background())
+	d.wg.Add(1)
+	go func() {
+		defer d.wg.Done()
+		<-d.shutdownCh
+		cancel()
+	}()
+
+	d.wg.Add(1)
+	go func() {
+		defer d.wg.Done()
+		d.restartLoop(ctx)
+		d.logger.Debug(ctx, "Daemon restart loop stopped")
+	}()
+
+	return nil
+}
+
+func withCurrBootstrap[Res any](
+	d *daemon, fn func(bootstrap.Bootstrap) (Res, error),
 ) (Res, error) {
 	var zero Res
-	d.l.Lock()
+	d.l.RLock()
-	children, state := d.children, d.state
+	defer d.l.RUnlock()
-	d.l.Unlock()
+
+	currBootstrap, state := d.currBootstrap, d.state
 
 	switch state {
+	case daemonStateNoNetwork:
+		return zero, ErrNoNetwork
 	case daemonStateInitializing:
 		return zero, ErrInitializing
 	case daemonStateOk:
-		return fn(children)
+		return fn(currBootstrap)
 	case daemonStateRestarting:
 		return zero, ErrRestarting
 	case daemonStateShutdown:
@@ -167,7 +222,7 @@ func (d *daemon) checkBootstrap(
 
 	thisHost := hostBootstrap.ThisHost()
 
-	newHosts, err := d.getGarageBootstrapHosts(ctx)
+	newHosts, err := getGarageBootstrapHosts(ctx, d.logger, hostBootstrap)
 	if err != nil {
 		return bootstrap.Bootstrap{}, false, fmt.Errorf("getting hosts from garage: %w", err)
 	}
@@ -233,19 +288,6 @@ func (d *daemon) watchForChanges(ctx context.Context) bootstrap.Bootstrap {
 }
 
 func (d *daemon) restartLoop(ctx context.Context) {
-	defer func() {
-		d.l.Lock()
-		d.state = daemonStateShutdown
-		children := d.children
-		d.l.Unlock()
-
-		if children != nil {
-			if err := children.Shutdown(); err != nil {
-				d.logger.Fatal(ctx, "Failed to cleanly shutdown daemon children, there may be orphaned child processes", err)
-			}
-		}
-	}()
-
 	wait := func(d time.Duration) bool {
 		select {
 		case <-ctx.Done():
@@ -328,18 +370,46 @@ func (d *daemon) restartLoop(ctx context.Context) {
 	}
 }
 
+func (d *daemon) JoinNetwork(
+	ctx context.Context, newBootstrap bootstrap.Bootstrap,
+) error {
+	d.l.Lock()
+	defer d.l.Unlock()
+
+	if d.state != daemonStateNoNetwork {
+		return ErrAlreadyJoined
+	}
+
+	return d.initialize(newBootstrap)
+}
+
 func (d *daemon) GetGarageBootstrapHosts(
 	ctx context.Context,
 ) (
 	map[string]bootstrap.Host, error,
 ) {
-	return withInnerChildren(d, func(*Children) (map[string]bootstrap.Host, error) {
+	return withCurrBootstrap(d, func(
-		return d.getGarageBootstrapHosts(ctx)
+		currBootstrap bootstrap.Bootstrap,
+	) (
+		map[string]bootstrap.Host, error,
+	) {
+		return getGarageBootstrapHosts(ctx, d.logger, currBootstrap)
 	})
 }
 
 func (d *daemon) Shutdown() error {
-	d.cancelFn()
+	d.l.Lock()
-	<-d.stoppedCh
+	defer d.l.Unlock()
+
+	close(d.shutdownCh)
+	d.wg.Wait()
+	d.state = daemonStateShutdown
+
+	if d.children != nil {
+		if err := d.children.Shutdown(); err != nil {
+			return fmt.Errorf("shutting down children: %w", err)
+		}
+	}
+
 	return nil
 }

go/daemon/errors.go

@@ -3,11 +3,19 @@ package daemon
 import "isle/daemon/jsonrpc2"
 
 var (
+	// ErrNoNetwork is returned when the daemon has never been configured with a
+	// network.
+	ErrNoNetwork = jsonrpc2.NewError(1, "No network configured")
+
 	// ErrInitializing is returned when a network is unavailable due to still
 	// being initialized.
-	ErrInitializing = jsonrpc2.NewError(1, "Network is being initialized")
+	ErrInitializing = jsonrpc2.NewError(2, "Network is being initialized")
 
 	// ErrRestarting is returned when a network is unavailable due to being
 	// restarted.
-	ErrRestarting = jsonrpc2.NewError(2, "Network is being restarted")
+	ErrRestarting = jsonrpc2.NewError(3, "Network is being restarted")
+
+	// ErrAlreadyJoined is returned when the daemon is instructed to create or
+	// join a new network, but it is already joined to a network.
+	ErrAlreadyJoined = jsonrpc2.NewError(4, "Already joined to a network")
 )

go/daemon/global_bucket.go

@@ -11,6 +11,7 @@ import (
 	"path/filepath"
 
 	"dev.mediocregopher.com/mediocre-go-lib.git/mctx"
+	"dev.mediocregopher.com/mediocre-go-lib.git/mlog"
 	"github.com/minio/minio-go/v7"
 )
 
@@ -65,13 +66,13 @@ func (d *daemon) putGarageBoostrapHost(ctx context.Context) error {
 	return nil
 }
 
-func (d *daemon) getGarageBootstrapHosts(
+func getGarageBootstrapHosts(
-	ctx context.Context,
+	ctx context.Context, logger *mlog.Logger, currBootstrap bootstrap.Bootstrap,
 ) (
 	map[string]bootstrap.Host, error,
 ) {
 	var (
-		b      = d.currBootstrap
+		b      = currBootstrap
 		client = b.GlobalBucketS3APIClient()
 		hosts  = map[string]bootstrap.Host{}
 
@@ -106,13 +107,13 @@ func (d *daemon) getGarageBootstrapHosts(
 		obj.Close()
 
 		if err != nil {
-			d.logger.Warn(ctx, "Object contains invalid json", err)
+			logger.Warn(ctx, "Object contains invalid json", err)
 			continue
 		}
 
 		host, err := authedHost.Unwrap(b.CAPublicCredentials)
 		if err != nil {
-			d.logger.Warn(ctx, "Host could not be authenticated", err)
+			logger.Warn(ctx, "Host could not be authenticated", err)
 		}
 
 		hosts[host.Name] = host

go/daemon/rpc.go

@@ -26,6 +26,15 @@ func NewRPC(daemon Daemon) *RPC {
 	return &RPC{daemon}
 }
 
+// JoinNetwork passes through to the Daemon method of the same name.
+func (r *RPC) JoinNetwork(
+	ctx context.Context, req bootstrap.Bootstrap,
+) (
+	struct{}, error,
+) {
+	return struct{}{}, r.daemon.JoinNetwork(ctx, req)
+}
+
 // GetHosts returns all hosts known to the network, sorted by their name.
 func (r *RPC) GetHosts(
 	ctx context.Context, req struct{},

tests/utils/with-1-data-1-empty-node-network.sh

@@ -60,10 +60,9 @@ EOF
 
     isle daemon -l debug --config-path daemon.yml >daemon.log 2>&1 &
     pid="$!"
-    echo "Waiting for primus daemon (process $pid) to initialize"
-
     $SHELL "$UTILS/register-cleanup.sh" "$pid" "1-data-1-empty-node-network/primus"
 
+    echo "Waiting for primus daemon (process $pid) to initialize"
     while ! isle hosts list >/dev/null; do sleep 1; done
 
     echo "Creating secondus bootstrap"
@@ -82,11 +81,17 @@ EOF
             device: isle-secondus
 EOF
 
-        isle daemon -l debug -c daemon.yml -b "$secondus_bootstrap" >daemon.log 2>&1 &
+        isle daemon -l debug -c daemon.yml >daemon.log 2>&1 &
         pid="$!"
-        echo "Waiting for secondus daemon (process $!) to initialize"
-
         $SHELL "$UTILS/register-cleanup.sh" "$pid" "1-data-1-empty-node-network/secondus"
+
+        echo "Waiting for secondus daemon (process $!) to start"
+        while ! [ -e "$ISLE_DAEMON_HTTP_SOCKET_PATH" ]; do sleep 1; done
+
+        echo "Joining secondus to the network"
+        isle network join -b "$secondus_bootstrap"
+
+        echo "Waiting for secondus daemon to join"
         while ! isle hosts list >/dev/null; do sleep 1; done
     )
 fi