Refactor how nebula certs are signed and propagated

I had previously made the mistake of thinking that the Curve25519 key
which is generated for each host to use in nebula communication could
also be used for signing. This is not the case, Ed25519 is used for
signing and is different thant Curve25519.

Rather than figuring out how to convert the Curve25519 key into an
Ed25519 key, which there is no apparent support for in the standard
library, I opted to instead ship a separate key just for signing with
each host. Doing this required a bit of refactoring in order to keep all
the different keys straight and ensure all data which needs a signature
still has it.

AppDir/etc/daemon.yml

@@ -73,7 +73,7 @@ storage:
   #
   # Once assigned (either implicitly or explicitly) the rpc_port of an
   # allocation should not be changed.
-  allocations:
+  #allocations:
 
     #- data_path: /foo/bar/data
     #  meta_path: /foo/bar/meta

README.md

@@ -20,9 +20,9 @@ The core components of cryptic-net, currently, are:
   the network are on a private LAN (e.g. their home WiFi network) or have a
   dynamic IP, they can still communicate directly with each other.
 
-* An S3-compatible network filesystem. Each participant can provide as much
+* An S3-compatible network filesystem. Each users can provide as much storage as
-  storage as they care to, if any. Stored data is sharded and replicated across
+  they care to, if any. Stored data is sharded and replicated across all hosts
-  all hosts that choose to provide storage.
+  that choose to provide storage.
 
 * A DNS server which provides automatic host and service (coming soon) discovery
   within the network.
@@ -44,9 +44,9 @@ decide which documents they need to care about.
 
 ### User Docs
 
-Users are participants who use cryptic-net resources, but do not provide any
+Users are participants who use network resources, but do not provide any network
-network or storage resources themselves. Users may be accessing the network from
+or storage resources themselves. Users may be accessing the network from a
-a laptop, and so are not expected to be online at any particular moment.
+laptop, and so are not expected to be online at any particular moment.
 
 Documentation for users:
 
@@ -57,7 +57,7 @@ Documentation for users:
 
 ### Operator Docs
 
-Operators are participants who own a dedicated host which they can expect to be
+Operators are users who own a dedicated host which they can expect to be
 always-online (to the extent that's possible in a residential environment).
 Operator hosts will need at least one of the following to be useful:
 
@@ -78,8 +78,8 @@ Documentation for operators:
 
 ### Admin Docs
 
-Admins are participants who control membership within the network. They are
+Admins are users who control membership within the network. They are likely
-likely operators as well.
+operators as well.
 
 Documentation for admins:
 
@@ -89,15 +89,15 @@ Documentation for admins:
 
 ### Dev Docs
 
-Devs may or may not be participants in any particular cryptic-net. They instead
+Devs may or may not be users in any particular cryptic network. They instead are
-are those who work on the actual code for cryptic-net.
+those who work on the actual code for cryptic-net.
 
 Documentation for devs:
 
 * [Design Principles](docs/dev/design-principles.md)
 * [`cryptic-net daemon` process tree](docs/dev/daemon-process-tree.svg): Diagram
-  describing the [pmux](https://code.betamike.com/cryptic-io/pmux) process tree created
+  describing the [pmux](https://code.betamike.com/cryptic-io/pmux) process tree
-  by `cryptic-net daemon` at runtime.
+  created by `cryptic-net daemon` at runtime.
 * [Rebuilding Documentation](docs/dev/rebuilding-documentation.md)
 
 ## Misc
@@ -105,5 +105,6 @@ Documentation for devs:
 Besides documentation, there are a few other pages which might be useful:
 
 * [Roadmap][roadmap]
+* [Glossary](docs/glossary.md)
 
 [roadmap]: docs/roadmap.md

docs/admin/adding-a-host-to-the-network.md

@@ -4,7 +4,7 @@ This document guides an admin through adding a single host to the network. Keep
 in mind that the steps described here must be done for _each_ host the user
 wishes to add.
 
-There are two ways for a user to add a host to the cryptic-net network.
+There are two ways for a user to add a host to the cryptic network.
 
 - If the user is savy enough to obtain their own `cryptic-net` binary, they can
   do so. The admin can then generate a `bootstrap.yml` file for their host,
@@ -44,7 +44,7 @@ following command from their own host:
 
 ```
 cryptic-net hosts make-bootstrap \
-    --name <name> \
+    --hostname <name> \
     --ip <ip> \
     --admin-path <path to admin.yml> \
     > bootstrap.yml
@@ -67,7 +67,7 @@ generate a `bootstrap.yml`:
 
 ```
 gpg -d <path to admin.yml.gpg> | cryptic-net hosts make-boostrap \
-    --name <name> \
+    --hostname <name> \
     --ip <ip> \
     --admin-path - \
     > bootstrap.yml

docs/admin/creating-a-new-network.md

@@ -1,9 +1,9 @@
 # Creating a New Network
 
-This guide is for those who wish to start a new cryptic-net network of their
+This guide is for those who wish to start a new cryptic network of their
 own.
 
-By starting a new cryptic-net network, you are becoming the administrator of a
+By starting a new cryptic network, you are becoming the administrator of a
 network. Be aware that being a network administrator is not necessarily easy,
 and the users of your network will frequently need your help in order to have a
 good experience. It can be helpful to have others with which you are
@@ -61,6 +61,9 @@ There are some key parameters which must be chosen when creating a new network.
 These will remain constant throughout the lifetime of the network, and so should
 be chosen with care.
 
+* Name: A human-readable name for the network. This will only be used for
+  display purposes.
+
 * Subnet: The IP subnet (or CIDR) will look something like `10.10.0.0/16`, where
   the `/16` indicates that all IPs from `10.10.0.0` to `10.10.255.255` are
   included. It's recommended to choose from the [ranges reserved for private
@@ -83,8 +86,8 @@ be chosen with care.
 ## Step 3: Prepare to Encrypt `admin.yml`
 
 The `admin.yml` file (which will be created in the next step) is the most
-sensitive part of a cryptic-net network. If it falls into the wrong hands it can
+sensitive part of a cryptic network. If it falls into the wrong hands it can be
-be used to completely compromise your network, impersonate hosts on the network,
+used to completely compromise your network, impersonate hosts on the network,
 and will likely lead to someone stealing or deleting all of your data.
 
 Therefore it is important that the file remains encrypted when it is not being
@@ -101,19 +104,20 @@ you can run:
 
 ```
 sudo cryptic-net admin create-network \
-    --config /path/to/daemon.yml \
+    --config-path /path/to/daemon.yml \
+    --name <name> \
+    --ip-net <ip/subnet-prefix> \
     --domain <domain> \
-    --ip     <ip/subnet-prefix> \
+    --hostname <hostname> \
-    --name   <hostname> \
     | gpg -e -r <my gpg email> \
     > admin.yml.gpg
 ```
 
 A couple of notes here:
 
-* The `--ip` parameter is formed from both the subnet and the IP you chose
+* The `--ip-net` parameter is formed from both the subnet and the IP you chose
   within it. So if your subnet is `10.10.0.0/16`, and your chosen IP in that
-  subnet is `10.10.4.20`, then your `--ip` parameter will be `10.10.4.20/16`.
+  subnet is `10.10.4.20`, then your `--ip-net` parameter will be `10.10.4.20/16`.
 
 * Only one gpg recipient is specified. If you intend on including other users as
   network administrators you can add them to the recipients list at this step,

docs/dev/design-principles.md

@@ -16,6 +16,6 @@ cryptic-net project.
   dispersed.
 
 * It is expected that a single host might be a part of multiple, independent
-  cryptic-net networks. These should not conflict with each other, nor share
+  cryptic networks. These should not conflict with each other, nor share
   resources.
 

docs/glossary.md

@@ -0,0 +1,17 @@
+# Glossary
+
+The purpose of this document is define the specific terms which should be used
+for various concepts, with the goal of establishing consistency throughout
+documentation and source code.
+
+- "user" - a person who takes part in the usage, operation, or administration of
+  a cryptic network.
+
+- "host" - A computer or device used by a user to connect to a cryptic network.
+
+- "cryptic network", "network" - A collection of hosts which communicate and
+  share resources with each other via the mechanisms provided by the cryptic-net
+  project.
+
+- "cryptic-net" - The name of the binary or program which is used to interact
+  with a cryptic network.

docs/user/creating-a-daemonyml-file.md

@@ -20,10 +20,10 @@ parameters. Feel free to edit this file as needed.
 ## Using daemon.yml
 
 With the `daemon.yml` created and configured, you can configure your daemon
-process to use it by passing it as the `-c` argument:
+process to use it by passing it as the `--config-path` argument:
 
 ```
-sudo cryptic-net daemon -c /path/to/daemon.yml
+sudo cryptic-net daemon --config-path /path/to/daemon.yml
 ```
 
 If you are an operator then your host should be running its `cryptic-net daemon`

entrypoint/src/admin/admin.go

@@ -13,6 +13,7 @@ import (
 // are available to all hosts within the network via their bootstrap files.
 type CreationParams struct {
 	ID     string `yaml:"id"`
+	Name   string `yaml:"name"`
 	Domain string `yaml:"domain"`
 }
 

entrypoint/src/bootstrap/bootstrap.go

@@ -37,7 +37,9 @@ type Bootstrap struct {
 	HostName string          `yaml:"hostname"`
 
 	Nebula struct {
+		CAPublicCredentials     nebula.CAPublicCredentials `yaml:"ca_public_credentials"`
 		HostCredentials         nebula.HostCredentials     `yaml:"host_credentials"`
+		SignedPublicCredentials string                     `yaml:"signed_public_credentials"`
 	} `yaml:"nebula"`
 
 	Garage struct {

entrypoint/src/bootstrap/garage_global_bucket.go

@@ -8,6 +8,7 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"strings"
 
 	"github.com/minio/minio-go/v7"
 	"gopkg.in/yaml.v3"
@@ -26,6 +27,12 @@ func (b Bootstrap) PutGarageBoostrapHost(ctx context.Context) error {
 	host := b.ThisHost()
 	client := b.GlobalBucketS3APIClient()
 
+	// the base Bootstrap has the public credentials signed by the CA, but we
+	// need this to be presented in the data stored into garage, so other hosts
+	// can verify that the stored host object is signed by the host public key,
+	// and that the host public key is signed by the CA.
+	host.Nebula.SignedPublicCredentials = b.Nebula.SignedPublicCredentials
+
 	hostB, err := yaml.Marshal(host)
 	if err != nil {
 		return fmt.Errorf("yaml encoding host data: %w", err)
@@ -33,7 +40,7 @@ func (b Bootstrap) PutGarageBoostrapHost(ctx context.Context) error {
 
 	buf := new(bytes.Buffer)
 
-	err = nebula.SignAndWrap(buf, b.Nebula.HostCredentials.HostKeyPEM, hostB)
+	err = nebula.SignAndWrap(buf, b.Nebula.HostCredentials.SigningPrivateKeyPEM, hostB)
 	if err != nil {
 		return fmt.Errorf("signing encoded host data: %w", err)
 	}
@@ -82,7 +89,6 @@ func (b Bootstrap) GetGarageBootstrapHosts(
 	map[string]Host, error,
 ) {
 
-	caCertPEM := b.Nebula.HostCredentials.CACertPEM
 	client := b.GlobalBucketS3APIClient()
 
 	hosts := map[string]Host{}
@@ -109,7 +115,7 @@ func (b Bootstrap) GetGarageBootstrapHosts(
 			return nil, fmt.Errorf("retrieving object %q: %w", objInfo.Key, err)
 		}
 
-		hostB, sig, err := nebula.Unwrap(obj)
+		hostB, hostSig, err := nebula.Unwrap(obj)
 		obj.Close()
 
 		if err != nil {
@@ -121,15 +127,36 @@ func (b Bootstrap) GetGarageBootstrapHosts(
 			return nil, fmt.Errorf("yaml decoding object %q: %w", objInfo.Key, err)
 		}
 
-		hostCertPEM := host.Nebula.CertPEM
+		hostPublicCredsB, hostPublicCredsSig, err := nebula.Unwrap(
+			strings.NewReader(host.Nebula.SignedPublicCredentials),
+		)
 
-		if err := nebula.ValidateSignature(hostCertPEM, hostB, sig); err != nil {
+		if err != nil {
-			fmt.Fprintf(os.Stderr, "invalid host data for %q: %v\n", objInfo.Key, err)
+			fmt.Fprintf(os.Stderr, "unwrapping signed public creds for %q: %v\n", objInfo.Key, err)
 			continue
 		}
 
-		if err := nebula.ValidateHostCertPEM(caCertPEM, hostCertPEM); err != nil {
+		err = nebula.ValidateSignature(
-			fmt.Fprintf(os.Stderr, "invalid nebula cert for %q: %v\n", objInfo.Key, err)
+			b.Nebula.CAPublicCredentials.SigningKeyPEM,
+			hostPublicCredsB,
+			hostPublicCredsSig,
+		)
+
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "invalid signed public creds for %q: %v\n", objInfo.Key, err)
+			continue
+		}
+
+		var hostPublicCreds nebula.HostPublicCredentials
+		if err := yaml.Unmarshal(hostPublicCredsB, &hostPublicCreds); err != nil {
+			fmt.Fprintf(os.Stderr, "yaml unmarshaling signed public creds for %q: %v\n", objInfo.Key, err)
+			continue
+		}
+
+		err = nebula.ValidateSignature(hostPublicCreds.SigningKeyPEM, hostB, hostSig)
+
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "invalid host data for %q: %v\n", objInfo.Key, err)
 			continue
 		}
 

entrypoint/src/bootstrap/hosts.go

@@ -1,18 +1,47 @@
 package bootstrap
 
 import (
+	"bytes"
 	"cryptic-net/nebula"
 	"fmt"
 	"net"
+	"strings"
+
+	"gopkg.in/yaml.v3"
 )
 
 // NebulaHost describes the nebula configuration of a Host which is relevant for
 // other hosts to know.
 type NebulaHost struct {
-	CertPEM    string `yaml:"crt"`
+	SignedPublicCredentials string `yaml:"signed_public_credentials"`
 	PublicAddr              string `yaml:"public_addr,omitempty"`
 }
 
+// NewNebulaHostSignedPublicCredentials constructs the SignedPublicCredentials
+// field of the NebulaHost struct, using the CACredentials to sign the
+// HostPublicCredentials.
+func NewNebulaHostSignedPublicCredentials(
+	caCreds nebula.CACredentials,
+	hostPublicCreds nebula.HostPublicCredentials,
+) (
+	string, error,
+) {
+
+	hostPublicCredsB, err := yaml.Marshal(hostPublicCreds)
+	if err != nil {
+		return "", fmt.Errorf("yaml marshaling host's public credentials: %w", err)
+	}
+
+	buf := new(bytes.Buffer)
+
+	err = nebula.SignAndWrap(buf, caCreds.SigningPrivateKeyPEM, hostPublicCredsB)
+	if err != nil {
+		return "", fmt.Errorf("signing host's public credentials: %w", err)
+	}
+
+	return buf.String(), nil
+}
+
 // GarageHost describes a single garage instance in the GarageHost.
 type GarageHostInstance struct {
 	ID        string `yaml:"id"`
@@ -36,9 +65,25 @@ type Host struct {
 
 // IP returns the IP address encoded in the Host's nebula certificate, or panics
 // if there is an error.
+//
+// This assumes that the Host and its data has already been verified against the
+// CA signing key.
 func (h Host) IP() net.IP {
 
-	ip, err := nebula.IPFromHostCertPEM(h.Nebula.CertPEM)
+	hostPublicCredsB, _, err := nebula.Unwrap(
+		strings.NewReader(h.Nebula.SignedPublicCredentials),
+	)
+
+	if err != nil {
+		panic(fmt.Errorf("unwrapping host's signed public credentials: %w", err))
+	}
+
+	var hostPublicCreds nebula.HostPublicCredentials
+	if err := yaml.Unmarshal(hostPublicCredsB, &hostPublicCreds); err != nil {
+		panic(fmt.Errorf("yaml unmarshaling host's public credentials: %w", err))
+	}
+
+	ip, err := nebula.IPFromHostCertPEM(hostPublicCreds.CertPEM)
 	if err != nil {
 		panic(fmt.Errorf("could not parse IP out of cert for host %q: %w", h.Name, err))
 	}

entrypoint/src/cmd/entrypoint/admin.go

@@ -49,7 +49,7 @@ func readAdmin(path string) (admin.Admin, error) {
 
 var subCmdAdminCreateNetwork = subCmd{
 	name:  "create-network",
-	descr: "Creates a new cryptic-net network, outputting the resulting admin.yml to stdout",
+	descr: "Creates a new cryptic network, outputting the resulting admin.yml to stdout",
 	do: func(subCmdCtx subCmdCtx) error {
 
 		flags := subCmdCtx.flagSet(false)
@@ -64,6 +64,11 @@ var subCmdAdminCreateNetwork = subCmd{
 			"Write the default configuration file to stdout and exit.",
 		)
 
+		name := flags.StringP(
+			"name", "n", "",
+			"Human-readable name to identify the network as.",
+		)
+
 		domain := flags.StringP(
 			"domain", "d", "",
 			"Domain name that should be used as the root domain in the network.",
@@ -75,7 +80,7 @@ var subCmdAdminCreateNetwork = subCmd{
 		)
 
 		hostName := flags.StringP(
-			"name", "n", "",
+			"hostname", "h", "",
 			"Name of this host, which will be the first host in the network",
 		)
 
@@ -87,8 +92,8 @@ var subCmdAdminCreateNetwork = subCmd{
 			return daemon.CopyDefaultConfig(os.Stdout, envAppDirPath)
 		}
 
-		if *domain == "" || *ipNetStr == "" || *hostName == "" {
+		if *name == "" || *domain == "" || *ipNetStr == "" || *hostName == "" {
-			return errors.New("--domain, --ip-net, and --name are required")
+			return errors.New("--name, --domain, --ip-net, and --hostname are required")
 		}
 
 		*domain = strings.TrimRight(strings.TrimLeft(*domain, "."), ".")
@@ -127,8 +132,18 @@ var subCmdAdminCreateNetwork = subCmd{
 			return fmt.Errorf("creating nebula cert for host: %w", err)
 		}
 
+		nebulaHostSignedPublicCreds, err := bootstrap.NewNebulaHostSignedPublicCredentials(
+			nebulaCACreds,
+			nebulaHostCreds.Public,
+		)
+
+		if err != nil {
+			return fmt.Errorf("creating signed public credentials for host: %w", err)
+		}
+
 		adminCreationParams := admin.CreationParams{
 			ID:     randStr(32),
+			Name:   *name,
 			Domain: *domain,
 		}
 
@@ -138,14 +153,17 @@ var subCmdAdminCreateNetwork = subCmd{
 				*hostName: bootstrap.Host{
 					Name: *hostName,
 					Nebula: bootstrap.NebulaHost{
-						CertPEM: nebulaHostCreds.HostCertPEM,
+						SignedPublicCredentials: nebulaHostSignedPublicCreds,
 					},
 				},
 			},
 			HostName: *hostName,
 		}
 
+		hostBootstrap.Nebula.CAPublicCredentials = nebulaCACreds.Public
 		hostBootstrap.Nebula.HostCredentials = nebulaHostCreds
+		hostBootstrap.Nebula.SignedPublicCredentials = nebulaHostSignedPublicCreds
+
 		hostBootstrap.Garage.RPCSecret = randStr(32)
 		hostBootstrap.Garage.AdminToken = randStr(32)
 		hostBootstrap.Garage.GlobalBucketS3APICredentials = garage.NewS3APICredentials()
@@ -235,8 +253,8 @@ var subCmdAdminMakeBootstrap = subCmd{
 
 		flags := subCmdCtx.flagSet(false)
 
-		name := flags.StringP(
+		hostName := flags.StringP(
-			"name", "n", "",
+			"hostname", "h", "",
 			"Name of the host to generate bootstrap.yml for",
 		)
 
@@ -254,12 +272,12 @@ var subCmdAdminMakeBootstrap = subCmd{
 			return fmt.Errorf("parsing flags: %w", err)
 		}
 
-		if *name == "" || *ipStr == "" || *adminPath == "" {
+		if *hostName == "" || *ipStr == "" || *adminPath == "" {
-			return errors.New("--name, --ip, and --admin-path are required")
+			return errors.New("--hostname, --ip, and --admin-path are required")
 		}
 
-		if err := validateHostName(*name); err != nil {
+		if err := validateHostName(*hostName); err != nil {
-			return fmt.Errorf("invalid hostname %q: %w", *name, err)
+			return fmt.Errorf("invalid hostname %q: %w", *hostName, err)
 		}
 
 		ip := net.ParseIP(*ipStr)
@@ -278,19 +296,31 @@ var subCmdAdminMakeBootstrap = subCmd{
 			return fmt.Errorf("loading host bootstrap: %w", err)
 		}
 
-		nebulaHostCreds, err := nebula.NewHostCredentials(adm.Nebula.CACredentials, *name, ip)
+		nebulaHostCreds, err := nebula.NewHostCredentials(adm.Nebula.CACredentials, *hostName, ip)
 		if err != nil {
 			return fmt.Errorf("creating new nebula host key/cert: %w", err)
 		}
 
+		nebulaHostSignedPublicCreds, err := bootstrap.NewNebulaHostSignedPublicCredentials(
+			adm.Nebula.CACredentials,
+			nebulaHostCreds.Public,
+		)
+
+		if err != nil {
+			return fmt.Errorf("creating signed public credentials for host: %w", err)
+		}
+
 		newHostBootstrap := bootstrap.Bootstrap{
 			AdminCreationParams: adm.CreationParams,
 
 			Hosts:    hostBootstrap.Hosts,
-			HostName: *name,
+			HostName: *hostName,
 		}
 
+		newHostBootstrap.Nebula.CAPublicCredentials = adm.Nebula.CACredentials.Public
 		newHostBootstrap.Nebula.HostCredentials = nebulaHostCreds
+		newHostBootstrap.Nebula.SignedPublicCredentials = nebulaHostSignedPublicCreds
+
 		newHostBootstrap.Garage.RPCSecret = adm.Garage.RPCSecret
 		newHostBootstrap.Garage.AdminToken = randStr(32)
 		newHostBootstrap.Garage.GlobalBucketS3APICredentials = adm.Garage.GlobalBucketS3APICredentials

entrypoint/src/cmd/entrypoint/daemon.go

@@ -135,8 +135,12 @@ func runDaemonPmuxOnce(
 		}
 
 		err := doOnce(ctx, func(ctx context.Context) error {
-			fmt.Fprintln(os.Stderr, "updating host info in garage")
+			if err := hostBootstrap.PutGarageBoostrapHost(ctx); err != nil {
-			return hostBootstrap.PutGarageBoostrapHost(ctx)
+				fmt.Fprintf(os.Stderr, "updating host info in garage: %v\n", err)
+				return err
+			}
+
+			return nil
 		})
 
 		if err != nil {
@@ -155,8 +159,12 @@ func runDaemonPmuxOnce(
 			}
 
 			err := doOnce(ctx, func(ctx context.Context) error {
-				fmt.Fprintln(os.Stderr, "applying garage layout")
+				if err := garageApplyLayout(ctx, hostBootstrap, daemonConfig); err != nil {
-				return garageApplyLayout(ctx, hostBootstrap, daemonConfig)
+					fmt.Fprintf(os.Stderr, "applying garage layout: %v\n", err)
+					return err
+				}
+
+				return nil
 			})
 
 			if err != nil {
@@ -236,7 +244,6 @@ var subCmdDaemon = subCmd{
 
 			hostBootstrapPath string
 			hostBootstrap     bootstrap.Bootstrap
-			foundHostBootstrap bool
 		)
 
 		tryLoadBootstrap := func(path string) bool {
@@ -245,6 +252,7 @@ var subCmdDaemon = subCmd{
 				return false
 
 			} else if hostBootstrap, err = bootstrap.FromFile(path); errors.Is(err, fs.ErrNotExist) {
+				fmt.Fprintf(os.Stderr, "bootstrap file not found at %q\n", path)
 				err = nil
 				return false
 
@@ -253,21 +261,22 @@ var subCmdDaemon = subCmd{
 				return false
 			}
 
+			fmt.Fprintf(os.Stderr, "bootstrap file found at %q\n", path)
 			hostBootstrapPath = path
 			return true
 		}
 
-		foundHostBootstrap = tryLoadBootstrap(bootstrapDataDirPath)
+		switch {
-		foundHostBootstrap = !foundHostBootstrap && *bootstrapPath != "" && tryLoadBootstrap(*bootstrapPath)
+		case tryLoadBootstrap(bootstrapDataDirPath):
-		foundHostBootstrap = !foundHostBootstrap && tryLoadBootstrap(bootstrapAppDirPath)
+		case *bootstrapPath != "" && tryLoadBootstrap(*bootstrapPath):
-
+		case tryLoadBootstrap(bootstrapAppDirPath):
-		if err != nil {
+		case err != nil:
 			return fmt.Errorf("attempting to load bootstrap.yml file: %w", err)
-
+		default:
-		} else if !foundHostBootstrap {
 			return errors.New("No bootstrap.yml file could be found, and one is not provided with --bootstrap-path")
+		}
 
-		} else if hostBootstrapPath != bootstrapDataDirPath {
+		if hostBootstrapPath != bootstrapDataDirPath {
 
 			// If the bootstrap file is not being stored in the data dir, copy
 			// it there, so it can be loaded from there next time.

entrypoint/src/cmd/entrypoint/hosts.go

@@ -57,8 +57,8 @@ var subCmdHostsDelete = subCmd{
 
 		flags := subCmdCtx.flagSet(false)
 
-		name := flags.StringP(
+		hostName := flags.StringP(
-			"name", "n", "",
+			"hostname", "h", "",
 			"Name of the host to delete",
 		)
 
@@ -66,8 +66,8 @@ var subCmdHostsDelete = subCmd{
 			return fmt.Errorf("parsing flags: %w", err)
 		}
 
-		if *name == "" {
+		if *hostName == "" {
-			return errors.New("--name is required")
+			return errors.New("--hostname is required")
 		}
 
 		hostBootstrap, err := loadHostBootstrap()
@@ -77,7 +77,7 @@ var subCmdHostsDelete = subCmd{
 
 		client := hostBootstrap.GlobalBucketS3APIClient()
 
-		return bootstrap.RemoveGarageBootstrapHost(subCmdCtx.ctx, client, *name)
+		return bootstrap.RemoveGarageBootstrapHost(subCmdCtx.ctx, client, *hostName)
 	},
 }
 

entrypoint/src/cmd/entrypoint/nebula_util.go

@@ -58,9 +58,9 @@ func nebulaPmuxProcConfig(
 
 	config := map[string]interface{}{
 		"pki": map[string]string{
-			"ca":   hostBootstrap.Nebula.HostCredentials.CACertPEM,
+			"ca":   hostBootstrap.Nebula.CAPublicCredentials.CertPEM,
-			"cert": hostBootstrap.Nebula.HostCredentials.HostCertPEM,
+			"cert": hostBootstrap.Nebula.HostCredentials.Public.CertPEM,
-			"key":  hostBootstrap.Nebula.HostCredentials.HostKeyPEM,
+			"key":  hostBootstrap.Nebula.HostCredentials.KeyPEM,
 		},
 		"static_host_map": staticHostMap,
 		"punchy": map[string]bool{

entrypoint/src/dnsmasq/tpl.go

@@ -32,8 +32,10 @@ no-hosts
 user=
 group=
 
-{{- range $host := .Hosts }}
+{{- $domain := . -}}
-address=/{{ $host.Name }}.hosts.{{ .Domain }}/{{ $host.Nebula.IP }}
+
+{{- range .Hosts }}
+address=/{{ .Name }}.hosts.{{ $domain }}/{{ .IP }}
 {{ end -}}
 
 {{- range .Resolvers }}

entrypoint/src/garage/admin_client.go

@@ -7,6 +7,7 @@ import (
 	"fmt"
 	"io"
 	"net/http"
+	"time"
 )
 
 // AdminClientError gets returned from AdminClient's Do method for non-200
@@ -130,5 +131,7 @@ func (c *AdminClient) Wait(ctx context.Context) error {
 		if numUp >= ReplicationFactor-1 {
 			return nil
 		}
+
+		time.Sleep(250 * time.Millisecond)
 	}
 }

entrypoint/src/nebula/nebula.go

@@ -21,19 +21,34 @@ import (
 // fails.
 var ErrInvalidSignature = errors.New("invalid signature")
 
+// HostPublicCredentials contains certificate and signing public keys which are
+// able to be broadcast publicly.
+type HostPublicCredentials struct {
+	CertPEM       string `yaml:"cert_pem"`
+	SigningKeyPEM string `yaml:"signing_key_pem"`
+}
+
 // HostCredentials contains the certificate and private key files which will
 // need to be present on a particular host. Each file is PEM encoded.
 type HostCredentials struct {
-	CACertPEM   string `yaml:"ca_cert_pem"`
+	Public               HostPublicCredentials `yaml:"public"`
-	HostKeyPEM  string `yaml:"host_key_pem"`
+	KeyPEM               string                `yaml:"key_pem"`
-	HostCertPEM string `yaml:"host_cert_pem"`
+	SigningPrivateKeyPEM string                `yaml:"signing_private_key_pem"`
+}
+
+// CAPublicCredentials contains certificate and signing public keys which are
+// able to be broadcast publicly. The signing public key is the same one which
+// is embedded into the certificate.
+type CAPublicCredentials struct {
+	CertPEM       string `yaml:"cert_pem"`
+	SigningKeyPEM string `yaml:"signing_key_pem"`
 }
 
 // CACredentials contains the certificate and private files which can be used to
 // create and validate HostCredentials. Each file is PEM encoded.
 type CACredentials struct {
-	CACertPEM string `yaml:"ca_cert_pem"`
+	Public               CAPublicCredentials `yaml:"public"`
-	CAKeyPEM  string `yaml:"ca_key_pem"`
+	SigningPrivateKeyPEM string              `yaml:"signing_private_key_pem"`
 }
 
 // NewHostCredentials generates a new key/cert for a nebula host using the CA
@@ -47,12 +62,12 @@ func NewHostCredentials(
 	// The logic here is largely based on
 	// https://github.com/slackhq/nebula/blob/v1.4.0/cmd/nebula-cert/sign.go
 
-	caKey, _, err := cert.UnmarshalEd25519PrivateKey([]byte(caCreds.CAKeyPEM))
+	caSigningKey, _, err := cert.UnmarshalEd25519PrivateKey([]byte(caCreds.SigningPrivateKeyPEM))
 	if err != nil {
 		return HostCredentials{}, fmt.Errorf("unmarshaling ca.key: %w", err)
 	}
 
-	caCert, _, err := cert.UnmarshalNebulaCertificateFromPEM([]byte(caCreds.CACertPEM))
+	caCert, _, err := cert.UnmarshalNebulaCertificateFromPEM([]byte(caCreds.Public.CertPEM))
 	if err != nil {
 		return HostCredentials{}, fmt.Errorf("unmarshaling ca.crt: %w", err)
 	}
@@ -69,6 +84,14 @@ func NewHostCredentials(
 		return HostCredentials{}, fmt.Errorf("invalid ip %q, not contained by network subnet %q", ip, subnet)
 	}
 
+	signingPubKey, signingPrivKey, err := ed25519.GenerateKey(rand.Reader)
+	if err != nil {
+		panic(fmt.Errorf("generating ed25519 key: %w", err))
+	}
+
+	signingPrivKeyPEM := cert.MarshalEd25519PrivateKey(signingPrivKey)
+	signingPubKeyPEM := cert.MarshalEd25519PublicKey(signingPubKey)
+
 	var hostPub, hostKey []byte
 	{
 		var pubkey, privkey [32]byte
@@ -98,7 +121,7 @@ func NewHostCredentials(
 		return HostCredentials{}, fmt.Errorf("validating certificate constraints: %w", err)
 	}
 
-	if err := hostCert.Sign(caKey); err != nil {
+	if err := hostCert.Sign(caSigningKey); err != nil {
 		return HostCredentials{}, fmt.Errorf("signing host cert with ca.key: %w", err)
 	}
 
@@ -110,9 +133,12 @@ func NewHostCredentials(
 	}
 
 	return HostCredentials{
-		CACertPEM:   caCreds.CACertPEM,
+		Public: HostPublicCredentials{
-		HostKeyPEM:  string(hostKeyPEM),
+			CertPEM:       string(hostCertPEM),
-		HostCertPEM: string(hostCertPEM),
+			SigningKeyPEM: string(signingPubKeyPEM),
+		},
+		KeyPEM:               string(hostKeyPEM),
+		SigningPrivateKeyPEM: string(signingPrivKeyPEM),
 	}, nil
 }
 
@@ -120,7 +146,10 @@ func NewHostCredentials(
 // and is included in the signing certificate's Name field.
 func NewCACredentials(domain string, subnet *net.IPNet) (CACredentials, error) {
 
-	pubKey, privKey, err := ed25519.GenerateKey(rand.Reader)
+	// The logic here is largely based on
+	// https://github.com/slackhq/nebula/blob/v1.4.0/cmd/nebula-cert/ca.go
+
+	signingPubKey, signingPrivKey, err := ed25519.GenerateKey(rand.Reader)
 	if err != nil {
 		panic(fmt.Errorf("generating ed25519 key: %w", err))
 	}
@@ -134,51 +163,32 @@ func NewCACredentials(domain string, subnet *net.IPNet) (CACredentials, error) {
 			Subnets:   []*net.IPNet{subnet},
 			NotBefore: now,
 			NotAfter:  expireAt,
-			PublicKey: pubKey,
+			PublicKey: signingPubKey,
 			IsCA:      true,
 		},
 	}
 
-	if err := caCert.Sign(privKey); err != nil {
+	if err := caCert.Sign(signingPrivKey); err != nil {
 		return CACredentials{}, fmt.Errorf("signing caCert: %w", err)
 	}
 
-	caKeyPEM := cert.MarshalEd25519PrivateKey(privKey)
+	signingPrivKeyPEM := cert.MarshalEd25519PrivateKey(signingPrivKey)
+	signingPubKeyPEM := cert.MarshalEd25519PublicKey(signingPubKey)
 
-	caCertPEM, err := caCert.MarshalToPEM()
+	certPEM, err := caCert.MarshalToPEM()
 	if err != nil {
 		return CACredentials{}, fmt.Errorf("marshaling caCert: %w", err)
 	}
 
 	return CACredentials{
-		CACertPEM: string(caCertPEM),
+		Public: CAPublicCredentials{
-		CAKeyPEM:  string(caKeyPEM),
+			CertPEM:       string(certPEM),
+			SigningKeyPEM: string(signingPubKeyPEM),
+		},
+		SigningPrivateKeyPEM: string(signingPrivKeyPEM),
 	}, nil
 }
 
-// ValidateHostCertPEM checks if the given host certificate was signed by the
-// given CA certificate, and returns ErrInvalidSignature if validation fails.
-func ValidateHostCertPEM(caCertPEM, hostCertPEM string) error {
-
-	caCert, _, err := cert.UnmarshalNebulaCertificateFromPEM([]byte(caCertPEM))
-	if err != nil {
-		return fmt.Errorf("unmarshaling CA certificate as PEM: %w", err)
-	}
-
-	hostCert, _, err := cert.UnmarshalNebulaCertificateFromPEM([]byte(hostCertPEM))
-	if err != nil {
-		return fmt.Errorf("unmarshaling host certificate as PEM: %w", err)
-	}
-
-	caPubKey := ed25519.PublicKey(caCert.Details.PublicKey)
-
-	if !hostCert.CheckSignature(caPubKey) {
-		return ErrInvalidSignature
-	}
-
-	return nil
-}
-
 // IPFromHostCertPEM is a convenience function for parsing the IP of a host out
 // of its nebula cert.
 func IPFromHostCertPEM(hostCertPEM string) (net.IP, error) {
@@ -196,11 +206,11 @@ func IPFromHostCertPEM(hostCertPEM string) (net.IP, error) {
 	return ips[0].IP, nil
 }
 
-// SignAndWrap signs the given bytes using the keyPEM, and writes an
+// SignAndWrap signs the given bytes using the host key, and writes an
 // encoded, versioned structure containing the signature and the given bytes.
-func SignAndWrap(into io.Writer, keyPEM string, b []byte) error {
+func SignAndWrap(into io.Writer, signingKeyPEM string, b []byte) error {
 
-	key, _, err := cert.UnmarshalEd25519PrivateKey([]byte(keyPEM))
+	key, _, err := cert.UnmarshalEd25519PrivateKey([]byte(signingKeyPEM))
 	if err != nil {
 		return fmt.Errorf("unmarshaling private key: %w", err)
 	}
@@ -215,7 +225,7 @@ func SignAndWrap(into io.Writer, keyPEM string, b []byte) error {
 	}
 
 	err = pem.Encode(into, &pem.Block{
-		Type:  "SIGNATURE",
+		Type:  "NEBULA ED25519 SIGNATURE",
 		Bytes: sig,
 	})
 
@@ -231,7 +241,7 @@ func SignAndWrap(into io.Writer, keyPEM string, b []byte) error {
 }
 
 // Unwrap reads a stream of bytes which was produced by SignAndWrap, and returns
-// the original inpute to SignAndWrap as well as the signature which was
+// the original input to SignAndWrap as well as the signature which was
 // created. ValidateSignature can be used to validate the signature.
 func Unwrap(from io.Reader) (b, sig []byte, err error) {
 
@@ -255,15 +265,13 @@ func Unwrap(from io.Reader) (b, sig []byte, err error) {
 }
 
 // ValidateSignature can be used to validate a signature produced by Unwrap.
-func ValidateSignature(certPEM string, b, sig []byte) error {
+func ValidateSignature(signingPubKeyPEM string, b, sig []byte) error {
 
-	cert, _, err := cert.UnmarshalNebulaCertificateFromPEM([]byte(certPEM))
+	pubKey, _, err := cert.UnmarshalEd25519PublicKey([]byte(signingPubKeyPEM))
 	if err != nil {
 		return fmt.Errorf("unmarshaling certificate as PEM: %w", err)
 	}
 
-	pubKey := ed25519.PublicKey(cert.Details.PublicKey)
-
 	if !ed25519.Verify(pubKey, b, sig) {
 		return ErrInvalidSignature
 	}

entrypoint/src/nebula/nebula_test.go

@@ -11,6 +11,7 @@ var (
 	ip                     net.IP
 	ipNet                  *net.IPNet
 	caCredsA, caCredsB     CACredentials
+	hostCredsA, hostCredsB HostCredentials
 )
 
 func init() {
@@ -30,24 +31,17 @@ func init() {
 	if err != nil {
 		panic(err)
 	}
-}
 
-func TestValidateHostCredentials(t *testing.T) {
+	hostCredsA, err = NewHostCredentials(caCredsA, "foo", ip)
-
-	hostCreds, err := NewHostCredentials(caCredsA, "foo", ip)
 	if err != nil {
-		t.Fatal(err)
+		panic(err)
 	}
 
-	err = ValidateHostCertPEM(hostCreds.CACertPEM, hostCreds.HostCertPEM)
+	hostCredsB, err = NewHostCredentials(caCredsB, "bar", ip)
 	if err != nil {
-		t.Fatal(err)
+		panic(err)
 	}
 
-	err = ValidateHostCertPEM(caCredsB.CACertPEM, hostCreds.HostCertPEM)
-	if !errors.Is(err, ErrInvalidSignature) {
-		t.Fatalf("expected ErrInvalidSignature, got %v", err)
-	}
 }
 
 func TestSignAndWrap(t *testing.T) {
@@ -55,7 +49,7 @@ func TestSignAndWrap(t *testing.T) {
 	b := []byte("foo bar baz")
 	buf := new(bytes.Buffer)
 
-	if err := SignAndWrap(buf, caCredsA.CAKeyPEM, b); err != nil {
+	if err := SignAndWrap(buf, hostCredsA.SigningPrivateKeyPEM, b); err != nil {
 		t.Fatal(err)
 	}
 
@@ -67,11 +61,11 @@ func TestSignAndWrap(t *testing.T) {
 		t.Fatalf("got %q but expected %q", gotB, b)
 	}
 
-	if err := ValidateSignature(caCredsA.CACertPEM, b, gotSig); err != nil {
+	if err := ValidateSignature(hostCredsA.Public.SigningKeyPEM, b, gotSig); err != nil {
 		t.Fatal(err)
 	}
 
-	if err := ValidateSignature(caCredsB.CACertPEM, b, gotSig); !errors.Is(err, ErrInvalidSignature) {
+	if err := ValidateSignature(hostCredsB.Public.SigningKeyPEM, b, gotSig); !errors.Is(err, ErrInvalidSignature) {
 		t.Fatalf("expected ErrInvalidSignature but got %v", err)
 	}
 }