Compare commits

...

2 Commits

23 changed files with 409 additions and 374 deletions

View File

@ -45,13 +45,12 @@ decide which documents they need to care about.
### User Docs ### User Docs
Users are participants who use network resources, but do not provide any network Users are participants who use network resources, but do not provide any network
or storage resources themselves. Users may be accessing the network from a resources themselves. Users may be accessing the network from a mobile device,
laptop, and so are not expected to be online at any particular moment. and so are not expected to be online at any particular moment.
Documentation for users: Documentation for users:
* [Getting Started](docs/user/getting-started.md) * [Getting Started](docs/user/getting-started.md)
* [Creating a daemon.yml File](docs/user/creating-a-daemonyml-file.md)
* [Using DNS](docs/user/using-dns.md) (advanced) * [Using DNS](docs/user/using-dns.md) (advanced)
* Restic example (TODO) * Restic example (TODO)
@ -63,7 +62,7 @@ Operator hosts will need at least one of the following to be useful:
* A static public IP, or a dynamic public IP with [dDNS][ddns] set up. * A static public IP, or a dynamic public IP with [dDNS][ddns] set up.
* At least 100GB of unused storage which can be reserved for the network. * At least 100GB of unused storage which can be reserved for the network. (TODO review storage requirements)
Operators are expected to be familiar with server administration, and to not be Operators are expected to be familiar with server administration, and to not be
afraid of a terminal. afraid of a terminal.

View File

@ -4,20 +4,6 @@ This document guides an admin through adding a single host to the network. Keep
in mind that the steps described here must be done for _each_ host the user in mind that the steps described here must be done for _each_ host the user
wishes to add. wishes to add.
There are two ways for a user to add a host to the isle network.
- If the user is savy enough to obtain their own `isle` binary, they can
do so. The admin can then generate a `bootstrap.json` file for their host,
give that to the user, and the user can run `isle daemon` using that
bootstrap file.
- If the user is not so savy, the admin can generate a custom `isle`
binary with the `bootstrap.json` embedded into it. The user can be given this
binary and run `isle daemon` without any configuration on their end.
From the admin's perspective the only difference between these cases is one
extra step.
## Step 1: Choose Hostname ## Step 1: Choose Hostname
The user will need to provide you with a name for their host. The name should The user will need to provide you with a name for their host. The name should
@ -75,17 +61,3 @@ gpg -d <path to admin.json.gpg> | isle admin create-bootstrap \
Note that the value of `--admin-path` is `-`, indicating that `admin.json` Note that the value of `--admin-path` is `-`, indicating that `admin.json`
should be read from stdin. should be read from stdin.
## Step 4: Optionally, Build Binary
If you wish to embed the `bootstrap.json` into a custom binary for the user (to
make installation _extremely_ easy for them) then you can run the following:
```
nix-build --arg bootstrap <path to bootstrap.json> -A appImage
```
The resulting binary can be found in the `result` directory which is created.
This binary should be treated like a `bootstrap.json` in terms of its uniqueness
and sensitivity.

View File

@ -27,7 +27,7 @@ The requirements for this host are:
behind a NAT, and/or allowing traffic on that UDP port in your hosts behind a NAT, and/or allowing traffic on that UDP port in your hosts
firewall. firewall.
* At least 300 GB of disk storage space. * At least 300 GB of disk storage space. (TODO double check the storage space requirements)
* At least 3 directories should be chosen, each of which will be committing at * At least 3 directories should be chosen, each of which will be committing at
least 100GB. Ideally these directories should be on different physical least 100GB. Ideally these directories should be on different physical
@ -36,16 +36,9 @@ The requirements for this host are:
* None of the resources being used for this network (the UDP port or storage * None of the resources being used for this network (the UDP port or storage
locations) should be being used by other networks. locations) should be being used by other networks.
## Step 1: Create a `daemon.yml` File ## Step 1: Edit the `daemon.yml` File
A `daemon.yml` will need to be created for use during network creation. You can Open `/etc/isle/daemon.yml` in a text editor and perform the following changes:
create a new `daemon.yml` with default values filled in by doing:
```
isle admin create-network --dump-config > /path/to/daemon.yml
```
Open this file in a text editor and perform the following changes:
* Set the `vpn.public_addr` field to the `host:port` your host is accessible on, * Set the `vpn.public_addr` field to the `host:port` your host is accessible on,
where `host` is the static public IP/DNS name of your host, and `port` is the where `host` is the static public IP/DNS name of your host, and `port` is the
@ -104,7 +97,7 @@ you can run:
``` ```
sudo isle admin create-network \ sudo isle admin create-network \
--config-path /path/to/daemon.yml \ --config-path /etc/isle/daemon.yml \
--name <name> \ --name <name> \
--ip-net <ip/subnet-prefix> \ --ip-net <ip/subnet-prefix> \
--domain <domain> \ --domain <domain> \
@ -117,7 +110,8 @@ A couple of notes here:
* The `--ip-net` parameter is formed from both the subnet and the IP you chose * The `--ip-net` parameter is formed from both the subnet and the IP you chose
within it. So if your subnet is `10.10.0.0/16`, and your chosen IP in that within it. So if your subnet is `10.10.0.0/16`, and your chosen IP in that
subnet is `10.10.4.20`, then your `--ip-net` parameter will be `10.10.4.20/16`. subnet is `10.10.4.20`, then your `--ip-net` parameter will be
`10.10.4.20/16`. (TODO expand a bit on what IP is being chosen).
* Only one gpg recipient is specified. If you intend on including other users as * Only one gpg recipient is specified. If you intend on including other users as
network administrators you can add them to the recipients list at this step, network administrators you can add them to the recipients list at this step,
@ -143,6 +137,8 @@ network for the daemon itself.
At this point your host, and your network, are ready to go! You can reference At this point your host, and your network, are ready to go! You can reference
the [Getting Started](../user/getting-started.md) document to set up your the [Getting Started](../user/getting-started.md) document to set up your
host's daemon process in a more permanent way. host's daemon process in a more permanent way. (TODO once creating a network is
done via RPC then this will be out-of-date. Better to direct them to the
operator docs, or maybe adding a new host).
[ddns]: https://www.cloudflare.com/learning/dns/glossary/dynamic-dns/ [ddns]: https://www.cloudflare.com/learning/dns/glossary/dynamic-dns/

View File

@ -15,6 +15,10 @@ documentation and source code.
- "isle network", "network" - A collection of hosts which communicate and share - "isle network", "network" - A collection of hosts which communicate and share
resources with each other via the Isle project. resources with each other via the Isle project.
- "garage cluster" - Garage is one of the sub-processes which isle is able to
run. These garage process connect together to form a cluster. We use the
term "cluster" in the context of garage to stay consistent with garage's
documentation and command-line.
- "user" - A person who takes part in the usage, operation, or administration of - "user" - A person who takes part in the usage, operation, or administration of
an isle network. an isle network.

View File

@ -26,18 +26,11 @@ traffic on that port to your host.
Configure your host's firewall to allow all UDP traffic on that port. Configure your host's firewall to allow all UDP traffic on that port.
## Create daemon.yml
First, if you haven't already, [create a `daemon.yml`
file](../user/creating-a-daemonyml-file.md). This will be used to
configure your `isle daemon` process with the public address that other
hosts can find your daemon on.
## Edit daemon.yml ## Edit daemon.yml
Open your `daemon.yml` file in a text editor, and find the `vpn.public_addr` Open your `/etc/isle/daemon.yml` file in a text editor, and find the
field. Update that field to reflect your host's IP/DNS name and your chosen UDP `vpn.public_addr` field. Update that field to reflect your host's IP/DNS name
port. and your chosen UDP port.
## Restart the Daemon ## Restart the Daemon

View File

@ -4,16 +4,9 @@ If your host machine can be reasonably sure of being online most, if not all, of
the time, and has 100GB or more of unused drive space you'd like to contribute the time, and has 100GB or more of unused drive space you'd like to contribute
to the network, then this document is for you. to the network, then this document is for you.
## Create `daemon.yml`
First, if you haven't already, [create a `daemon.yml`
file](../user/creating-a-daemonyml-file.md). This will be used to
configure your `isle daemon` process with the storage locations and
capacities you want to contribute.
## Edit `daemon.yml` ## Edit `daemon.yml`
Open your `daemon.yml` file in a text editor, and find the Open your `/etc/isle/daemon.yml` file in a text editor, and find the
`storage.allocations` section. `storage.allocations` section.
Each allocation in the allocations list describes the space being contributed Each allocation in the allocations list describes the space being contributed

View File

@ -14,8 +14,8 @@ Isle uses the [nebula](https://github.com/slackhq/nebula) project to
provide its VPN layer. Nebula ships with its own [builtin provide its VPN layer. Nebula ships with its own [builtin
firewall](https://nebula.defined.net/docs/config/firewall), which only applies firewall](https://nebula.defined.net/docs/config/firewall), which only applies
to connections coming in over the virtual network interface which it creates. to connections coming in over the virtual network interface which it creates.
This firewall can be manually configured as part of isle's This firewall can be manually configured as part of the `/etc/isle/daemon.yml`
[`daemon.yml`](../user/creating-a-daemonyml-file.md) file. file.
Any storage instances which are defined as part of the `daemon.yml` file will Any storage instances which are defined as part of the `daemon.yml` file will
have their network ports automatically added to the VPN firewall by isle. have their network ports automatically added to the VPN firewall by isle.

View File

@ -8,14 +8,6 @@ order they will be implemented.
These items are listed more or less in the order they need to be completed, as These items are listed more or less in the order they need to be completed, as
they generally depend on the items previous to them. they generally depend on the items previous to them.
### Window Support + GUI
Support for Windows is a must. This requirement also includes a simple GUI,
which would essentially act as a thin layer on top of `daemon.yml` to start
with.
Depending on difficulty level, OSX support might be added at this stage as well.
### NATS ### NATS
Garage is currently used to handle eventually-consistent persistent storage, but Garage is currently used to handle eventually-consistent persistent storage, but
@ -23,16 +15,15 @@ there is no mechanism for inter-host realtime communication as of yet. NATS
would be a good candidate for this, as it uses a gossip protocol which does not would be a good candidate for this, as it uses a gossip protocol which does not
require a central coordinator (I don't think), and is well supported. require a central coordinator (I don't think), and is well supported.
### Integration of [domani](https://code.betamike.com/micropelago/domani) ### Integration of [Caddy](https://caddyserver.com/docs/)
Integration of domani will require some changes on domani's end. We want domani Integration of Caddy's will require some plugins to be developed. We want Caddy
to be able to store cert information in S3 (garage), so that all isle lighthouse to be able to store cert information in S3 (garage), so that all isle lighthouse
nodes can potentially become gateways as well. Once done, it would be possible nodes can potentially become gateways as well. Once done, it would be possible
for lighthouses to forward public traffic to inner nodes. for lighthouses to forward public traffic to inner nodes.
It should also be possible for users within the network to take advantage of It should also be possible for users within the network to take use lighthouse
domani's hosting ability even without an always-on host of their own, without Caddy's to host their websites (and eventually gemini capsules) for them.
requiring a passphrase.
Most likely this integration will require NATS as well, to coordinate cache Most likely this integration will require NATS as well, to coordinate cache
invalidation and cert refreshing. invalidation and cert refreshing.
@ -45,6 +36,14 @@ files. The bootstrap file would be stored, encrypted, in garage, with the invite
code being able to both identify and decrypt it. To instantiate a host, the user code being able to both identify and decrypt it. To instantiate a host, the user
only needs to input the network domain name and the invite code. only needs to input the network domain name and the invite code.
### Windows Support + GUI
Support for Windows is a must. This requirement also includes a simple GUI,
which would essentially act as a thin layer on top of `daemon.yml` to start
with.
Depending on difficulty level, OSX support might be added at this stage as well.
### FUSE Mount ### FUSE Mount
KBFS style. Every user should be able to mount virtual directories to their host KBFS style. Every user should be able to mount virtual directories to their host
@ -96,11 +95,16 @@ it works.
### Proper Linux Packages ### Proper Linux Packages
Rather than distributing raw binaries for Linux we should instead be Rather than distributing raw binaries for Linux we should instead be
distributing actual packages, e.g. deb files for debian/ubuntu, PKGBUILD for distributing actual packages.
arch, rpm for fedora (if we care), etc... This will allow for properly setting
capabilities for the binary at install time, so that it can be run as non-root, * deb files for debian/ubuntu
and installing any necessary `.desktop` files so that it can be run as a GUI * PKGBUILD for arch (done)
application. * rpm for fedora?
* flatpak?
This will allow for properly setting capabilities for the binary at install
time, so that it can be run as non-root, and installing any necessary `.desktop`
files so that it can be run as a GUI application.
### Mobile app ### Mobile app
@ -109,20 +113,6 @@ would be great. We are not able to use the existing nebula mobile app because it
is not actually open-source, but we can at least use it as a reference to see is not actually open-source, but we can at least use it as a reference to see
how this can be accomplished. how this can be accomplished.
### Don't run as root
It's currently a pretty hard requirement for `isle daemon` to run as
root. This is due to:
- nebula's network interface root to be started.
- dnsmasq listening on port 53, generally a protected port.
On linux it should be fairly straightforward to grant the entrypoint the
necessary ambient capabilities up-front, and then drop down to a specified user.
This is how the tests work. Doing this with other OS's will depend on how they
work.
### DNS/Firewall Configuration ### DNS/Firewall Configuration
Ideally Isle could detect the DNS/firewall subsystems being used on a per-OS Ideally Isle could detect the DNS/firewall subsystems being used on a per-OS

View File

@ -1,32 +0,0 @@
# Creating a daemon.yml File
The `isle daemon` process has generally sane defaults and does not need
to be configured for most users. This document describes how to use the
`daemon.yml` file to handle those cases where configuration is necessary.
## Create daemon.yml
First, create a `daemon.yml` file. You can create a new `daemon.yml` with
default values filled in by doing:
```
isle daemon --dump-config > /path/to/daemon.yml
```
If you open that file in a text editor you can view all default values that
`isle daemon` ships with, as well as documentation for all configurable
parameters. Feel free to edit this file as needed.
## Using daemon.yml
With the `daemon.yml` created and configured, you can configure your daemon
process to use it by passing it as the `--config-path` argument:
```
sudo isle daemon --config-path /path/to/daemon.yml
```
If you are an operator then your host should be running its `isle daemon`
process in systemd (see [Getting Started](getting-started.md) if
not), and you will need to modify the service file accordingly.

View File

@ -6,118 +6,126 @@ binary and joining a network.
NOTE currently only linux machines with the following architectures are NOTE currently only linux machines with the following architectures are
supported: supported:
- `x86_64` / `amd64` - `x86_64` (aka `amd64`)
- `aarch64` / `arm64` - `aarch64` (aka `arm64`)
- `i686` - `i686`
(Only `x86_64` has been tested.) (`i686` has not been tested.)
More OSs and architectures coming soon! More OSs and architectures coming soon!
## Obtaining an isle Binary ## Install isle
### The Easy Way How isle gets installed depends on which Linux distribution you are using.
Download the latest binary for your platform from ### Archlinux (also Manjaro)
[this link](https://code.betamike.com/micropelago/isle/releases/latest).
### The Hard Way Download the latest `.pkg.tar.zst` package file for your platform from
[this link][latest].
Alternatively, you can build your own binary by running the following from the Install the package using pacman:
project's root:
``` ```
nix-build -A appImage sudo pacman -U /path/to/isle-*.pkg.tar.zst
``` ```
(*NOTE* Dependencies of `isle` seemingly compile all of musl and rust ### Other Distributions
from scratch (it's not clear why, blame garage!). If you have not otherwise
configured it, nix might be using a tmpfs as its build directory, and the
capacity of this tmpfs will probably be exceeded by this build. You can change
your build directory to somewhere on-disk by setting the TMPDIR environment
variable for `nix-daemon` (see [this github issue][tmpdir-gh].))
The resulting binary can be found in the `result` directory which is created. If a package file is not available for your distribution you can still install
an AppImage directly. It is assumed that all commands below are run as root.
Download the latest `.AppImage` binary for your platform from
[this link][latest], and place it in your `/usr/bin` directory.
Create a `daemon.yml` file using default values by doing:
```
mkdir -p /etc/isle/
isle daemon --dump-config > /etc/isle/daemon.yml
```
Create a system user for the isle daemon to run as:
```
useradd -r -s /bin/false -C "isle Daemon" isle
```
If your distro uses systemd, download [the latest systemd service
file][serviceFile] and place it in `/etc/systemd/system`. Run `systemctl
daemon-reload` to ensure systemd has seen the new service file.
If your distro uses an init system other than systemd then you will need to
configure that yourself. You can use the systemd service file linked above as a
reference.
[serviceFile]: https://code.betamike.com/micropelago/isle/src/branch/main/dist/linux/isle.service
### From Source
(TODO probably move these instructions into the Dev docs section).
Building from source requires [nix][nix].
You can build your own AppImage by running the following from the project's
root:
```
nix-build -A appImageBin
```
(*NOTE* The first time you run this a lot of things will be built from scratch.
If you have not otherwise configured it, nix might be using a tmpfs as its build
directory, and the capacity of this tmpfs will probably be exceeded by this
build. You can change your build directory to somewhere on-disk by setting the
TMPDIR environment variable for `nix-daemon` (see
[this github issue][tmpdir-gh].))
The resulting binary can be found under `result/bin`. From here you can continue
with the instructions under the "AppImage" section above.
[nix]: https://nixos.wiki/wiki/Nix_package_manager
[tmpdir-gh]: https://github.com/NixOS/nix/issues/2098#issuecomment-383243838 [tmpdir-gh]: https://github.com/NixOS/nix/issues/2098#issuecomment-383243838
## Obtaining Your Bootstrap File ## Add Users to `isle` Group (Optional)
The `bootstrap.json` file contains all information required for your particular If you wish to run isle commands as a user other than root, you can add that
host to join the network, and must be generated and provided to you by an admin user to the `isle` group:
for the network.
## Running the Daemon
Once you have a binary and bootstrap file, you will need to run the `daemon`
sub-command as the root user. This can most easily be done using the `sudo`
command, in a terminal:
``` ```
sudo /path/to/isle daemon --bootstrap-path /path/to/bootstrap.json sudo usermod -aG isle username
``` ```
This will start the daemon process, which will keep running until you kill it ## Start the isle Service
with `ctrl-c`. The `--bootstrap-path /path/to/bootstrap.json` argument is only
required the first time the daemon is run, it will be ignored on subsequent
runs.
You can double check that the daemon is running properly by pinging a private IP Once installed and bootstrapped you can enable and start the isle service by
from the network in a separate terminal: doing:
``` ```
ping 10.10.0.1
```
If the pings are successful then your daemon is working!
## Installing the Daemon as a Systemd Service
NOTE in the future we will introduce an `install` sub-command which will
automate most of this section.
Rather than running the daemon manually, you can install it as a systemd
service. This way your daemon will automatically start in the background on
startup, and will be restarted if it has any issues.
To do so, create a file at `/etc/systemd/system/isle.service` with the
following contents:
```
[Unit]
Description=isle
Requires=network.target
After=network.target
[Service]
Restart=always
RestartSec=1s
User=root
ExecStart=/path/to/isle daemon
[Install]
WantedBy=multi-user.target
```
Remember to change the `/path/to/isle` part to the actual absolute path
to your binary!
Once created, perform the following commands in a terminal to enable the
service:
```
sudo systemctl daemon-reload
sudo systemctl enable --now isle sudo systemctl enable --now isle
``` ```
You can check the service's status by doing: (NOTE If your distro uses an init system other than systemd then you will need
to instead start isle according to that system's requirements.)
## Join a Network
This section will guide you through the process of joining an existing network
of isle hosts. If instead you wish to create a new network for others to join
then see the [Creating a New Network][creating-a-new-network] page.
To join an existing network you will need to first obtain a `bootstrap.json`
file. The `bootstrap.json` file contains all information required for your
particular host to join the network, and must be generated and provided to you
by an admin for the network.
Once obtained, you can join the network by doing:
``` ```
sudo systemctl status isle isle network join --bootstrap-path /path/to/bootstrap.json
``` ```
and you can view its full logs by doing: After a few moments you will have successfully joined the network!
``` TODO block the `network join` call until joining has succeeded, or display a failure reason.
sudo journalctl -lu isle
``` [creating-a-new-network]: ../admin/creating-a-new-network.md
[latest]: https://code.betamike.com/micropelago/isle/releases/latest

View File

@ -10,8 +10,7 @@ network's domain name.
If a request for a hostname not within the network's domain is received then the If a request for a hostname not within the network's domain is received then the
server will forward the request to a pre-configured public resolver. The set of server will forward the request to a pre-configured public resolver. The set of
public resolvers used can be configured using the public resolvers used can be configured in the `/etc/isle/daemon.yml` file.
[daemon.yml](creating-a-daemonyml-file.md) file.
This DNS server is an optional feature of Isle, and not required in general for This DNS server is an optional feature of Isle, and not required in general for
making use of the network. making use of the network.

View File

@ -103,16 +103,29 @@ func FromFile(path string) (Bootstrap, error) {
defer f.Close() defer f.Close()
var b Bootstrap var b Bootstrap
if err := json.NewDecoder(f).Decode(&b); err != nil { if err := json.NewDecoder(f).Decode(&b); err != nil {
return Bootstrap{}, fmt.Errorf("decoding json: %w", err) return Bootstrap{}, fmt.Errorf("decoding json: %w", err)
} }
if b.HostAssigned, err = b.SignedHostAssigned.UnwrapUnsafe(); err != nil { return b, nil
return Bootstrap{}, fmt.Errorf("unwrapping host assigned: %w", err) }
func (b *Bootstrap) UnmarshalJSON(data []byte) error {
type inner Bootstrap
err := json.Unmarshal(data, (*inner)(b))
if err != nil {
return err
} }
return b, nil b.HostAssigned, err = b.SignedHostAssigned.Unwrap(
b.CAPublicCredentials.SigningKey,
)
if err != nil {
return fmt.Errorf("unwrapping HostAssigned: %w", err)
}
return nil
} }
// WriteTo writes the Bootstrap as a new bootstrap to the given io.Writer. // WriteTo writes the Bootstrap as a new bootstrap to the given io.Writer.
@ -123,7 +136,6 @@ func (b Bootstrap) WriteTo(into io.Writer) error {
// ThisHost is a shortcut for b.Hosts[b.HostName], but will panic if the // ThisHost is a shortcut for b.Hosts[b.HostName], but will panic if the
// HostName isn't found in the Hosts map. // HostName isn't found in the Hosts map.
func (b Bootstrap) ThisHost() Host { func (b Bootstrap) ThisHost() Host {
host, ok := b.Hosts[b.Name] host, ok := b.Hosts[b.Name]
if !ok { if !ok {
panic(fmt.Sprintf("hostname %q not defined in bootstrap's hosts", b.Name)) panic(fmt.Sprintf("hostname %q not defined in bootstrap's hosts", b.Name))

View File

@ -78,5 +78,9 @@ type Host struct {
// This assumes that the Host and its data has already been verified against the // This assumes that the Host and its data has already been verified against the
// CA signing key. // CA signing key.
func (h Host) IP() net.IP { func (h Host) IP() net.IP {
return h.PublicCredentials.Cert.Unwrap().Details.Ips[0].IP cert := h.PublicCredentials.Cert.Unwrap()
if len(cert.Details.Ips) == 0 {
panic(fmt.Sprintf("host %q not configured with any ips: %+v", h.Name, h))
}
return cert.Details.Ips[0].IP
} }

View File

@ -2,15 +2,11 @@ package main
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"io/fs"
"os" "os"
"isle/bootstrap"
"isle/daemon" "isle/daemon"
"dev.mediocregopher.com/mediocre-go-lib.git/mctx"
"dev.mediocregopher.com/mediocre-go-lib.git/mlog" "dev.mediocregopher.com/mediocre-go-lib.git/mlog"
) )
@ -31,11 +27,6 @@ var subCmdDaemon = subCmd{
"Write the default configuration file to stdout and exit.", "Write the default configuration file to stdout and exit.",
) )
bootstrapPath := flags.StringP(
"bootstrap-path", "b", "",
`Path to a bootstrap.json file. This only needs to be provided the first time the daemon is started, after that it is ignored. If the isle binary has a bootstrap built into it then this argument is always optional.`,
)
logLevelStr := flags.StringP( logLevelStr := flags.StringP(
"log-level", "l", "info", "log-level", "l", "info",
`Maximum log level which should be output. Values can be "debug", "info", "warn", "error", "fatal". Does not apply to sub-processes`, `Maximum log level which should be output. Values can be "debug", "info", "warn", "error", "fatal". Does not apply to sub-processes`,
@ -64,72 +55,17 @@ var subCmdDaemon = subCmd{
} }
defer runtimeDirCleanup() defer runtimeDirCleanup()
var (
bootstrapStateDirPath = bootstrap.StateDirPath(daemonEnvVars.StateDirPath)
bootstrapAppDirPath = bootstrap.AppDirPath(envAppDirPath)
hostBootstrapPath string
hostBootstrap bootstrap.Bootstrap
)
tryLoadBootstrap := func(path string) bool {
ctx := mctx.Annotate(ctx, "bootstrapFilePath", path)
if err != nil {
return false
} else if hostBootstrap, err = bootstrap.FromFile(path); errors.Is(err, fs.ErrNotExist) {
logger.WarnString(ctx, "bootstrap file not found")
err = nil
return false
} else if err != nil {
err = fmt.Errorf("parsing bootstrap.json at %q: %w", path, err)
return false
}
logger.Info(ctx, "bootstrap file found")
hostBootstrapPath = path
return true
}
switch {
case tryLoadBootstrap(bootstrapStateDirPath):
case *bootstrapPath != "" && tryLoadBootstrap(*bootstrapPath):
case tryLoadBootstrap(bootstrapAppDirPath):
case err != nil:
return fmt.Errorf("attempting to load bootstrap.json file: %w", err)
default:
return errors.New("No bootstrap.json file could be found, and one is not provided with --bootstrap-path")
}
if hostBootstrapPath != bootstrapStateDirPath {
// If the bootstrap file is not being stored in the data dir, copy
// it there, so it can be loaded from there next time.
if err := writeBootstrapToStateDir(hostBootstrap); err != nil {
return fmt.Errorf("writing bootstrap.json to data dir: %w", err)
}
}
daemonConfig, err := daemon.LoadConfig(envAppDirPath, *daemonConfigPath) daemonConfig, err := daemon.LoadConfig(envAppDirPath, *daemonConfigPath)
if err != nil { if err != nil {
return fmt.Errorf("loading daemon config: %w", err) return fmt.Errorf("loading daemon config: %w", err)
} }
// we update this Host's data using whatever configuration has been daemonInst, err := daemon.NewDaemon(
// provided by the daemon config. This way the daemon has the most logger, daemonConfig, envBinDirPath, nil,
// up-to-date possible bootstrap. This updated bootstrap will later get
// updated in garage as a background daemon task, so other hosts will
// see it as well.
if hostBootstrap, err = coalesceDaemonConfigAndBootstrap(hostBootstrap, daemonConfig); err != nil {
return fmt.Errorf("merging daemon config into bootstrap data: %w", err)
}
daemonInst := daemon.NewDaemon(
logger, daemonConfig, envBinDirPath, hostBootstrap, nil,
) )
if err != nil {
return fmt.Errorf("starting daemon: %w", err)
}
defer func() { defer func() {
logger.Info(ctx, "Stopping child processes") logger.Info(ctx, "Stopping child processes")
if err := daemonInst.Shutdown(); err != nil { if err := daemonInst.Shutdown(); err != nil {

View File

@ -1,7 +1,6 @@
package main package main
import ( import (
"encoding/json"
"errors" "errors"
"fmt" "fmt"
"isle/bootstrap" "isle/bootstrap"
@ -30,17 +29,12 @@ var subCmdHostsList = subCmd{
ctx := subCmdCtx.ctx ctx := subCmdCtx.ctx
var resRaw json.RawMessage var res daemon.GetHostsResult
err := subCmdCtx.daemonRCPClient.Call(ctx, &resRaw, "GetHosts", nil) err := subCmdCtx.daemonRCPClient.Call(ctx, &res, "GetHosts", nil)
if err != nil { if err != nil {
return fmt.Errorf("calling GetHosts: %w", err) return fmt.Errorf("calling GetHosts: %w", err)
} }
var res daemon.GetHostsResult
if err := json.Unmarshal(resRaw, &res); err != nil {
return fmt.Errorf("unmarshaling %s into %T: %w", string(resRaw), res, err)
}
type host struct { type host struct {
Name string Name string
VPN struct { VPN struct {

View File

@ -66,6 +66,7 @@ func main() {
subCmdGarage, subCmdGarage,
subCmdHosts, subCmdHosts,
subCmdNebula, subCmdNebula,
subCmdNetwork,
subCmdVersion, subCmdVersion,
) )

View File

@ -0,0 +1,51 @@
package main
import (
"errors"
"fmt"
"isle/bootstrap"
)
var subCmdNetworkJoin = subCmd{
name: "join",
descr: "Joins this host to an existing network",
do: func(subCmdCtx subCmdCtx) error {
var (
ctx = subCmdCtx.ctx
flags = subCmdCtx.flagSet(false)
)
bootstrapPath := flags.StringP(
"bootstrap-path", "b", "", "Path to a bootstrap.json file.",
)
if err := flags.Parse(subCmdCtx.args); err != nil {
return fmt.Errorf("parsing flags: %w", err)
}
if *bootstrapPath == "" {
return errors.New("--bootstrap-path is required")
}
newBootstrap, err := bootstrap.FromFile(*bootstrapPath)
if err != nil {
return fmt.Errorf(
"loading bootstrap from %q: %w", *bootstrapPath, err,
)
}
return subCmdCtx.daemonRCPClient.Call(
ctx, nil, "JoinNetwork", newBootstrap,
)
},
}
var subCmdNetwork = subCmd{
name: "network",
descr: "Sub-commands related to network membership",
do: func(subCmdCtx subCmdCtx) error {
return subCmdCtx.doSubCmd(
subCmdNetworkJoin,
)
},
}

View File

@ -1,32 +1,14 @@
package daemon package daemon
import ( import (
"errors"
"fmt" "fmt"
"io/fs"
"os" "os"
"path/filepath" "path/filepath"
"isle/bootstrap" "isle/bootstrap"
"isle/garage/garagesrv"
) )
func loadHostBootstrap(stateDirPath string) (bootstrap.Bootstrap, error) {
path := bootstrap.StateDirPath(stateDirPath)
hostBootstrap, err := bootstrap.FromFile(path)
if errors.Is(err, fs.ErrNotExist) {
return bootstrap.Bootstrap{}, fmt.Errorf(
"%q not found, has the daemon ever been run?",
stateDirPath,
)
} else if err != nil {
return bootstrap.Bootstrap{}, fmt.Errorf("loading %q: %w", stateDirPath, err)
}
return hostBootstrap, nil
}
func writeBootstrapToStateDir( func writeBootstrapToStateDir(
stateDirPath string, hostBootstrap bootstrap.Bootstrap, stateDirPath string, hostBootstrap bootstrap.Bootstrap,
) error { ) error {
@ -48,3 +30,43 @@ func writeBootstrapToStateDir(
return hostBootstrap.WriteTo(f) return hostBootstrap.WriteTo(f)
} }
func coalesceDaemonConfigAndBootstrap(
daemonConfig Config, hostBootstrap bootstrap.Bootstrap,
) (
bootstrap.Bootstrap, error,
) {
host := bootstrap.Host{
HostAssigned: hostBootstrap.HostAssigned,
HostConfigured: bootstrap.HostConfigured{
Nebula: bootstrap.NebulaHost{
PublicAddr: daemonConfig.VPN.PublicAddr,
},
},
}
if allocs := daemonConfig.Storage.Allocations; len(allocs) > 0 {
for i, alloc := range allocs {
id, rpcPort, err := garagesrv.InitAlloc(alloc.MetaPath, alloc.RPCPort)
if err != nil {
return bootstrap.Bootstrap{}, fmt.Errorf(
"initializing alloc at %q: %w", alloc.MetaPath, err,
)
}
host.Garage.Instances = append(host.Garage.Instances, bootstrap.GarageHostInstance{
ID: id,
RPCPort: rpcPort,
S3APIPort: alloc.S3APIPort,
})
allocs[i].RPCPort = rpcPort
}
}
hostBootstrap.Hosts[host.Name] = host
return hostBootstrap, nil
}

View File

@ -8,6 +8,7 @@ import (
"errors" "errors"
"fmt" "fmt"
"io" "io"
"io/fs"
"isle/bootstrap" "isle/bootstrap"
"os" "os"
"sync" "sync"
@ -20,6 +21,13 @@ import (
// with isle, typically via the unix socket. // with isle, typically via the unix socket.
type Daemon interface { type Daemon interface {
// JoinNetwork joins the Daemon to an existing network using the given
// Bootstrap.
//
// Errors:
// - ErrAlreadyJoined
JoinNetwork(context.Context, bootstrap.Bootstrap) error
// GetGarageBootstrapHosts loads (and verifies) the <hostname>.json.signed // GetGarageBootstrapHosts loads (and verifies) the <hostname>.json.signed
// file for all hosts stored in garage. // file for all hosts stored in garage.
GetGarageBootstrapHosts( GetGarageBootstrapHosts(
@ -67,7 +75,8 @@ func (o *Opts) withDefaults() *Opts {
} }
const ( const (
daemonStateInitializing = iota daemonStateNoNetwork = iota
daemonStateInitializing
daemonStateOk daemonStateOk
daemonStateRestarting daemonStateRestarting
daemonStateShutdown daemonStateShutdown
@ -79,13 +88,13 @@ type daemon struct {
envBinDirPath string envBinDirPath string
opts *Opts opts *Opts
l sync.Mutex l sync.RWMutex
state int state int
children *Children children *Children
currBootstrap bootstrap.Bootstrap currBootstrap bootstrap.Bootstrap
cancelFn context.CancelFunc shutdownCh chan struct{}
stoppedCh chan struct{} wg sync.WaitGroup
} }
// NewDaemon initializes and returns a Daemon instance which will manage all // NewDaemon initializes and returns a Daemon instance which will manage all
@ -110,43 +119,89 @@ func NewDaemon(
logger *mlog.Logger, logger *mlog.Logger,
daemonConfig Config, daemonConfig Config,
envBinDirPath string, envBinDirPath string,
currBootstrap bootstrap.Bootstrap,
opts *Opts, opts *Opts,
) Daemon { ) (
ctx, cancelFn := context.WithCancel(context.Background()) Daemon, error,
) {
var (
d = &daemon{
logger: logger,
daemonConfig: daemonConfig,
envBinDirPath: envBinDirPath,
opts: opts.withDefaults(),
shutdownCh: make(chan struct{}),
}
bootstrapFilePath = bootstrap.StateDirPath(d.opts.EnvVars.StateDirPath)
)
d := &daemon{ currBootstrap, err := bootstrap.FromFile(bootstrapFilePath)
logger: logger, if errors.Is(err, fs.ErrNotExist) {
daemonConfig: daemonConfig, // daemon has never had a network created or joined
envBinDirPath: envBinDirPath, } else if err != nil {
opts: opts.withDefaults(), return nil, fmt.Errorf(
currBootstrap: currBootstrap, "loading bootstrap from %q: %w", bootstrapFilePath, err,
cancelFn: cancelFn, )
stoppedCh: make(chan struct{}), } else if err := d.initialize(currBootstrap); err != nil {
return nil, fmt.Errorf("initializing with bootstrap: %w", err)
} }
go func() { return d, nil
d.restartLoop(ctx)
d.logger.Debug(ctx, "DaemonRestarter stopped")
close(d.stoppedCh)
}()
return d
} }
func withInnerChildren[Res any]( func (d *daemon) initialize(currBootstrap bootstrap.Bootstrap) error {
d *daemon, fn func(*Children) (Res, error), // we update this Host's data using whatever configuration has been provided
// by the daemon config. This way the daemon has the most up-to-date
// possible bootstrap. This updated bootstrap will later get updated in
// garage as a background daemon task, so other hosts will see it as well.
currBootstrap, err := coalesceDaemonConfigAndBootstrap(
d.daemonConfig, currBootstrap,
)
if err != nil {
return fmt.Errorf("combining daemon configuration into bootstrap: %w", err)
}
err = writeBootstrapToStateDir(d.opts.EnvVars.StateDirPath, currBootstrap)
if err != nil {
return fmt.Errorf("writing bootstrap to state dir: %w", err)
}
d.currBootstrap = currBootstrap
d.state = daemonStateInitializing
ctx, cancel := context.WithCancel(context.Background())
d.wg.Add(1)
go func() {
defer d.wg.Done()
<-d.shutdownCh
cancel()
}()
d.wg.Add(1)
go func() {
defer d.wg.Done()
d.restartLoop(ctx)
d.logger.Debug(ctx, "Daemon restart loop stopped")
}()
return nil
}
func withCurrBootstrap[Res any](
d *daemon, fn func(bootstrap.Bootstrap) (Res, error),
) (Res, error) { ) (Res, error) {
var zero Res var zero Res
d.l.Lock() d.l.RLock()
children, state := d.children, d.state defer d.l.RUnlock()
d.l.Unlock()
currBootstrap, state := d.currBootstrap, d.state
switch state { switch state {
case daemonStateNoNetwork:
return zero, ErrNoNetwork
case daemonStateInitializing: case daemonStateInitializing:
return zero, ErrInitializing return zero, ErrInitializing
case daemonStateOk: case daemonStateOk:
return fn(children) return fn(currBootstrap)
case daemonStateRestarting: case daemonStateRestarting:
return zero, ErrRestarting return zero, ErrRestarting
case daemonStateShutdown: case daemonStateShutdown:
@ -167,7 +222,7 @@ func (d *daemon) checkBootstrap(
thisHost := hostBootstrap.ThisHost() thisHost := hostBootstrap.ThisHost()
newHosts, err := d.getGarageBootstrapHosts(ctx) newHosts, err := getGarageBootstrapHosts(ctx, d.logger, hostBootstrap)
if err != nil { if err != nil {
return bootstrap.Bootstrap{}, false, fmt.Errorf("getting hosts from garage: %w", err) return bootstrap.Bootstrap{}, false, fmt.Errorf("getting hosts from garage: %w", err)
} }
@ -233,19 +288,6 @@ func (d *daemon) watchForChanges(ctx context.Context) bootstrap.Bootstrap {
} }
func (d *daemon) restartLoop(ctx context.Context) { func (d *daemon) restartLoop(ctx context.Context) {
defer func() {
d.l.Lock()
d.state = daemonStateShutdown
children := d.children
d.l.Unlock()
if children != nil {
if err := children.Shutdown(); err != nil {
d.logger.Fatal(ctx, "Failed to cleanly shutdown daemon children, there may be orphaned child processes", err)
}
}
}()
wait := func(d time.Duration) bool { wait := func(d time.Duration) bool {
select { select {
case <-ctx.Done(): case <-ctx.Done():
@ -328,18 +370,46 @@ func (d *daemon) restartLoop(ctx context.Context) {
} }
} }
func (d *daemon) JoinNetwork(
ctx context.Context, newBootstrap bootstrap.Bootstrap,
) error {
d.l.Lock()
defer d.l.Unlock()
if d.state != daemonStateNoNetwork {
return ErrAlreadyJoined
}
return d.initialize(newBootstrap)
}
func (d *daemon) GetGarageBootstrapHosts( func (d *daemon) GetGarageBootstrapHosts(
ctx context.Context, ctx context.Context,
) ( ) (
map[string]bootstrap.Host, error, map[string]bootstrap.Host, error,
) { ) {
return withInnerChildren(d, func(*Children) (map[string]bootstrap.Host, error) { return withCurrBootstrap(d, func(
return d.getGarageBootstrapHosts(ctx) currBootstrap bootstrap.Bootstrap,
) (
map[string]bootstrap.Host, error,
) {
return getGarageBootstrapHosts(ctx, d.logger, currBootstrap)
}) })
} }
func (d *daemon) Shutdown() error { func (d *daemon) Shutdown() error {
d.cancelFn() d.l.Lock()
<-d.stoppedCh defer d.l.Unlock()
close(d.shutdownCh)
d.wg.Wait()
d.state = daemonStateShutdown
if d.children != nil {
if err := d.children.Shutdown(); err != nil {
return fmt.Errorf("shutting down children: %w", err)
}
}
return nil return nil
} }

View File

@ -3,11 +3,19 @@ package daemon
import "isle/daemon/jsonrpc2" import "isle/daemon/jsonrpc2"
var ( var (
// ErrNoNetwork is returned when the daemon has never been configured with a
// network.
ErrNoNetwork = jsonrpc2.NewError(1, "No network configured")
// ErrInitializing is returned when a network is unavailable due to still // ErrInitializing is returned when a network is unavailable due to still
// being initialized. // being initialized.
ErrInitializing = jsonrpc2.NewError(1, "Network is being initialized") ErrInitializing = jsonrpc2.NewError(2, "Network is being initialized")
// ErrRestarting is returned when a network is unavailable due to being // ErrRestarting is returned when a network is unavailable due to being
// restarted. // restarted.
ErrRestarting = jsonrpc2.NewError(2, "Network is being restarted") ErrRestarting = jsonrpc2.NewError(3, "Network is being restarted")
// ErrAlreadyJoined is returned when the daemon is instructed to create or
// join a new network, but it is already joined to a network.
ErrAlreadyJoined = jsonrpc2.NewError(4, "Already joined to a network")
) )

View File

@ -11,6 +11,7 @@ import (
"path/filepath" "path/filepath"
"dev.mediocregopher.com/mediocre-go-lib.git/mctx" "dev.mediocregopher.com/mediocre-go-lib.git/mctx"
"dev.mediocregopher.com/mediocre-go-lib.git/mlog"
"github.com/minio/minio-go/v7" "github.com/minio/minio-go/v7"
) )
@ -65,13 +66,13 @@ func (d *daemon) putGarageBoostrapHost(ctx context.Context) error {
return nil return nil
} }
func (d *daemon) getGarageBootstrapHosts( func getGarageBootstrapHosts(
ctx context.Context, ctx context.Context, logger *mlog.Logger, currBootstrap bootstrap.Bootstrap,
) ( ) (
map[string]bootstrap.Host, error, map[string]bootstrap.Host, error,
) { ) {
var ( var (
b = d.currBootstrap b = currBootstrap
client = b.GlobalBucketS3APIClient() client = b.GlobalBucketS3APIClient()
hosts = map[string]bootstrap.Host{} hosts = map[string]bootstrap.Host{}
@ -106,13 +107,13 @@ func (d *daemon) getGarageBootstrapHosts(
obj.Close() obj.Close()
if err != nil { if err != nil {
d.logger.Warn(ctx, "Object contains invalid json", err) logger.Warn(ctx, "Object contains invalid json", err)
continue continue
} }
host, err := authedHost.Unwrap(b.CAPublicCredentials) host, err := authedHost.Unwrap(b.CAPublicCredentials)
if err != nil { if err != nil {
d.logger.Warn(ctx, "Host could not be authenticated", err) logger.Warn(ctx, "Host could not be authenticated", err)
} }
hosts[host.Name] = host hosts[host.Name] = host

View File

@ -26,6 +26,15 @@ func NewRPC(daemon Daemon) *RPC {
return &RPC{daemon} return &RPC{daemon}
} }
// JoinNetwork passes through to the Daemon method of the same name.
func (r *RPC) JoinNetwork(
ctx context.Context, req bootstrap.Bootstrap,
) (
struct{}, error,
) {
return struct{}{}, r.daemon.JoinNetwork(ctx, req)
}
// GetHosts returns all hosts known to the network, sorted by their name. // GetHosts returns all hosts known to the network, sorted by their name.
func (r *RPC) GetHosts( func (r *RPC) GetHosts(
ctx context.Context, req struct{}, ctx context.Context, req struct{},

View File

@ -60,10 +60,9 @@ EOF
isle daemon -l debug --config-path daemon.yml >daemon.log 2>&1 & isle daemon -l debug --config-path daemon.yml >daemon.log 2>&1 &
pid="$!" pid="$!"
echo "Waiting for primus daemon (process $pid) to initialize"
$SHELL "$UTILS/register-cleanup.sh" "$pid" "1-data-1-empty-node-network/primus" $SHELL "$UTILS/register-cleanup.sh" "$pid" "1-data-1-empty-node-network/primus"
echo "Waiting for primus daemon (process $pid) to initialize"
while ! isle hosts list >/dev/null; do sleep 1; done while ! isle hosts list >/dev/null; do sleep 1; done
echo "Creating secondus bootstrap" echo "Creating secondus bootstrap"
@ -82,11 +81,17 @@ EOF
device: isle-secondus device: isle-secondus
EOF EOF
isle daemon -l debug -c daemon.yml -b "$secondus_bootstrap" >daemon.log 2>&1 & isle daemon -l debug -c daemon.yml >daemon.log 2>&1 &
pid="$!" pid="$!"
echo "Waiting for secondus daemon (process $!) to initialize"
$SHELL "$UTILS/register-cleanup.sh" "$pid" "1-data-1-empty-node-network/secondus" $SHELL "$UTILS/register-cleanup.sh" "$pid" "1-data-1-empty-node-network/secondus"
echo "Waiting for secondus daemon (process $!) to start"
while ! [ -e "$ISLE_DAEMON_HTTP_SOCKET_PATH" ]; do sleep 1; done
echo "Joining secondus to the network"
isle network join -b "$secondus_bootstrap"
echo "Waiting for secondus daemon to join"
while ! isle hosts list >/dev/null; do sleep 1; done while ! isle hosts list >/dev/null; do sleep 1; done
) )
fi fi