micropelago/isle
3
0
Fork 0
Code Issues 1 Pull Requests Projects Releases 5 Activity

Replace admin create-network with network create over RPC

Browse Source
This commit is contained in:
Brian Picciano 2024-07-09 11:43:17 +02:00
parent f9d033b89f
commit 279c79a9f1
11 changed files with 145 additions and 361 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@ -62,7 +62,7 @@ Operator hosts will need at least one of the following to be useful:
* A static public IP, or a dynamic public IP with [dDNS][ddns] set up.
* At least 100GB of unused storage which can be reserved for the network. (TODO review storage requirements)
* At least 1GB of unused storage which can be reserved for the network.
Operators are expected to be familiar with server administration, and to not be
afraid of a terminal.

49
docs/admin/creating-a-new-network.md
View File

@ -27,16 +27,16 @@ The requirements for this host are:
behind a NAT, and/or allowing traffic on that UDP port in your hosts
firewall.
* At least 300 GB of disk storage space. (TODO double check the storage space requirements)
* At least 3 GB of disk storage space.
* At least 3 directories should be chosen, each of which will be committing at
least 100GB. Ideally these directories should be on different physical
disks, but if that's not possible it's ok. See the Next Steps section.
least 1GB. Ideally these directories should be on different physical disks,
but if that's not possible it's ok. See the Next Steps section.
* None of the resources being used for this network (the UDP port or storage
locations) should be being used by other networks.
## Step 1: Edit the `daemon.yml` File
## Step 1: Configure the isle Daemon
Open `/etc/isle/daemon.yml` in a text editor and perform the following changes:
@ -48,6 +48,12 @@ Open `/etc/isle/daemon.yml` in a text editor and perform the following changes:
Save and close the file.
Run the following to restart the daemon with the new configuration:
```
sudo systemctl restart isle
```
## Step 2: Choose Parameters
There are some key parameters which must be chosen when creating a new network.
@ -92,12 +98,10 @@ if you care to use a different method.
## Step 4: Create the `admin.json` File
To create the `admin.json` file, which effectively creates the network itself,
you can run:
To create the network, and the `admin.json` file in the process, run:
```
sudo isle admin create-network \
--config-path /etc/isle/daemon.yml \
sudo isle network create \
--name <name> \
--ip-net <ip/subnet-prefix> \
--domain <domain> \
@ -111,34 +115,19 @@ A couple of notes here:
* The `--ip-net` parameter is formed from both the subnet and the IP you chose
within it. So if your subnet is `10.10.0.0/16`, and your chosen IP in that
subnet is `10.10.4.20`, then your `--ip-net` parameter will be
`10.10.4.20/16`. (TODO expand a bit on what IP is being chosen).
`10.10.4.20/16`.
* Only one gpg recipient is specified. If you intend on including other users as
network administrators you can add them to the recipients list at this step,
so they will be able to use the `admin.json` file as well. You can also
manually add them as recipients later.
You will see a lot of output, as `create-network` starts up many child processes
in order to set the network up. It should exit successfully on its own after a
few seconds.
The `isle network create` command may take up to a minute to complete. Once
completed you should have an `admin.json.gpg` file in your current directory.
At this point you should have an `admin.json.gpg` file in your current directory.
## Step 5: Run the Daemon
The isle daemon can be run now, using the following command:
```
sudo isle daemon -c /path/to/daemon.yml
```
**NOTE** that you _must_ use the same `daemon.yml` file used when creating the
network for the daemon itself.
At this point your host, and your network, are ready to go! You can reference
the [Getting Started](../user/getting-started.md) document to set up your
host's daemon process in a more permanent way. (TODO once creating a network is
done via RPC then this will be out-of-date. Better to direct them to the
operator docs, or maybe adding a new host).
At this point your host, and your network, are ready to go! To add other hosts
to the network you can reference the [Adding a Host to the Network][add-host]
document.
[add-host]: ./adding-a-host-to-the-network.md
[ddns]: https://www.cloudflare.com/learning/dns/glossary/dynamic-dns/

4
docs/operator/contributing-storage.md
View File

@ -1,8 +1,8 @@
# Contributing Storage
If your host machine can be reasonably sure of being online most, if not all, of
the time, and has 100GB or more of unused drive space you'd like to contribute
to the network, then this document is for you.
the time, and has 1GB or more of unused drive space you'd like to contribute to
the network, then this document is for you.
## Edit `daemon.yml`

192
go/cmd/entrypoint/admin.go
View File

@ -7,14 +7,9 @@ import (
"fmt"
"isle/admin"
"isle/bootstrap"
"isle/daemon"
"isle/garage"
"isle/nebula"
"net"
"os"
"strings"
"dev.mediocregopher.com/mediocre-go-lib.git/mlog"
)
func randStr(l int) string {
@ -46,192 +41,6 @@ func readAdmin(path string) (admin.Admin, error) {
return admin.FromReader(f)
}
var subCmdAdminCreateNetwork = subCmd{
name: "create-network",
descr: "Creates a new isle network, outputting the resulting admin.json to stdout",
do: func(subCmdCtx subCmdCtx) error {
flags := subCmdCtx.flagSet(false)
daemonConfigPath := flags.StringP(
"config-path", "c", "",
"Optional path to a daemon.yml file to load configuration from.",
)
dumpConfig := flags.Bool(
"dump-config", false,
"Write the default configuration file to stdout and exit.",
)
name := flags.StringP(
"name", "n", "",
"Human-readable name to identify the network as.",
)
domain := flags.StringP(
"domain", "d", "",
"Domain name that should be used as the root domain in the network.",
)
ipNetStr := flags.StringP(
"ip-net", "i", "",
`IP+prefix (e.g. "10.10.0.1/16") which denotes the IP of this host, which will be the first host in the network, and the range of IPs which other hosts in the network can be assigned`,
)
hostName := flags.StringP(
"hostname", "h", "",
"Name of this host, which will be the first host in the network",
)
logLevelStr := flags.StringP(
"log-level", "l", "info",
`Maximum log level which should be output. Values can be "debug", "info", "warn", "error", "fatal". Does not apply to sub-processes`,
)
if err := flags.Parse(subCmdCtx.args); err != nil {
return fmt.Errorf("parsing flags: %w", err)
}
ctx := subCmdCtx.ctx
if *dumpConfig {
return daemon.CopyDefaultConfig(os.Stdout, envAppDirPath)
}
if *name == "" || *domain == "" || *ipNetStr == "" || *hostName == "" {
return errors.New("--name, --domain, --ip-net, and --hostname are required")
}
logLevel := mlog.LevelFromString(*logLevelStr)
if logLevel == nil {
return fmt.Errorf("couldn't parse log level %q", *logLevelStr)
}
logger := subCmdCtx.logger.WithMaxLevel(logLevel.Int())
*domain = strings.TrimRight(strings.TrimLeft(*domain, "."), ".")
ip, subnet, err := net.ParseCIDR(*ipNetStr)
if err != nil {
return fmt.Errorf("parsing %q as a CIDR: %w", *ipNetStr, err)
}
if err := validateHostName(*hostName); err != nil {
return fmt.Errorf("invalid hostname %q: %w", *hostName, err)
}
runtimeDirCleanup, err := setupAndLockRuntimeDir(ctx, logger)
if err != nil {
return fmt.Errorf("setting up runtime directory: %w", err)
}
defer runtimeDirCleanup()
daemonConfig, err := daemon.LoadConfig(envAppDirPath, *daemonConfigPath)
if err != nil {
return fmt.Errorf("loading daemon config: %w", err)
}
if len(daemonConfig.Storage.Allocations) < 3 {
return fmt.Errorf("daemon config with at least 3 allocations was not provided")
}
nebulaCACreds, err := nebula.NewCACredentials(*domain, subnet)
if err != nil {
return fmt.Errorf("creating nebula CA cert: %w", err)
}
adminCreationParams := admin.CreationParams{
ID: randStr(32),
Name: *name,
Domain: *domain,
}
garageBootstrap := bootstrap.Garage{
RPCSecret: randStr(32),
AdminToken: randStr(32),
}
hostBootstrap, err := bootstrap.New(
nebulaCACreds,
adminCreationParams,
garageBootstrap,
*hostName,
ip,
)
if err != nil {
return fmt.Errorf("initializing bootstrap data: %w", err)
}
if hostBootstrap, err = coalesceDaemonConfigAndBootstrap(hostBootstrap, daemonConfig); err != nil {
return fmt.Errorf("merging daemon config into bootstrap data: %w", err)
}
children, err := daemon.NewChildren(
ctx,
logger.WithNamespace("daemon"),
daemonConfig,
hostBootstrap,
envBinDirPath,
&daemon.Opts{
// NOTE both stdout and stderr are sent to stderr, so that the
// user can pipe the resulting admin.json to stdout.
Stdout: os.Stderr,
},
)
if err != nil {
return fmt.Errorf("initializing children: %w", err)
}
defer func() {
logger.Info(ctx, "Shutting down child processes")
if err := children.Shutdown(); err != nil {
logger.Error(ctx, "Failed to shut down children cleanly, there may be zombie children leftover", err)
}
}()
logger.Info(ctx, "Applying garage layout")
if err := daemon.GarageApplyLayout(
ctx, logger, daemonConfig, hostBootstrap,
); err != nil {
return fmt.Errorf("applying garage layout: %w", err)
}
logger.Info(ctx, "initializing garage shared global bucket")
garageGlobalBucketCreds, err := garageInitializeGlobalBucket(
ctx, logger, hostBootstrap, daemonConfig,
)
if cErr := (garage.AdminClientError{}); errors.As(err, &cErr) && cErr.StatusCode == 409 {
return fmt.Errorf("shared global bucket has already been created, are the storage allocations from a previously initialized isle being used?")
} else if err != nil {
return fmt.Errorf("initializing garage shared global bucket: %w", err)
}
hostBootstrap.Garage.GlobalBucketS3APICredentials = garageGlobalBucketCreds
// rewrite the bootstrap now that the global bucket creds have been
// added to it.
if err := writeBootstrapToStateDir(hostBootstrap); err != nil {
return fmt.Errorf("writing bootstrap file: %w", err)
}
logger.Info(ctx, "Network initialized successfully, writing admin.json to stdout")
adm := admin.Admin{
CreationParams: adminCreationParams,
}
adm.Nebula.CACredentials = nebulaCACreds
adm.Garage.RPCSecret = hostBootstrap.Garage.RPCSecret
adm.Garage.GlobalBucketS3APICredentials = hostBootstrap.Garage.GlobalBucketS3APICredentials
if err := adm.WriteTo(os.Stdout); err != nil {
return fmt.Errorf("writing admin.json to stdout")
}
return nil
},
}
var subCmdAdminCreateBootstrap = subCmd{
name: "create-bootstrap",
descr: "Creates a new bootstrap.json file for a particular host and writes it to stdout",
@ -392,7 +201,6 @@ var subCmdAdmin = subCmd{
descr: "Sub-commands which only admins can run",
do: func(subCmdCtx subCmdCtx) error {
return subCmdCtx.doSubCmd(
subCmdAdminCreateNetwork,
subCmdAdminCreateBootstrap,
subCmdAdminCreateNebulaCert,
)

21
go/cmd/entrypoint/bootstrap_util.go
View File

@ -5,8 +5,6 @@ import (
"fmt"
"io/fs"
"isle/bootstrap"
"os"
"path/filepath"
)
func loadHostBootstrap() (bootstrap.Bootstrap, error) {
@ -26,22 +24,3 @@ func loadHostBootstrap() (bootstrap.Bootstrap, error) {
return hostBootstrap, nil
}
func writeBootstrapToStateDir(hostBootstrap bootstrap.Bootstrap) error {
path := bootstrap.StateDirPath(daemonEnvVars.StateDirPath)
dirPath := filepath.Dir(path)
if err := os.MkdirAll(dirPath, 0700); err != nil {
return fmt.Errorf("creating directory %q: %w", dirPath, err)
}
f, err := os.Create(path)
if err != nil {
return fmt.Errorf("creating file %q: %w", path, err)
}
defer f.Close()
return hostBootstrap.WriteTo(f)
}

46
go/cmd/entrypoint/daemon_util.go
View File

@ -4,10 +4,8 @@ import (
"context"
"errors"
"fmt"
"isle/bootstrap"
"isle/daemon"
"isle/daemon/jsonrpc2"
"isle/garage/garagesrv"
"net"
"net/http"
@ -15,50 +13,6 @@ import (
"dev.mediocregopher.com/mediocre-go-lib.git/mlog"
)
func coalesceDaemonConfigAndBootstrap(
hostBootstrap bootstrap.Bootstrap, daemonConfig daemon.Config,
) (
bootstrap.Bootstrap, error,
) {
host := bootstrap.Host{
HostAssigned: hostBootstrap.HostAssigned,
HostConfigured: bootstrap.HostConfigured{
Nebula: bootstrap.NebulaHost{
PublicAddr: daemonConfig.VPN.PublicAddr,
},
},
}
if allocs := daemonConfig.Storage.Allocations; len(allocs) > 0 {
for i, alloc := range allocs {
id, rpcPort, err := garagesrv.InitAlloc(alloc.MetaPath, alloc.RPCPort)
if err != nil {
return bootstrap.Bootstrap{}, fmt.Errorf(
"initializing alloc at %q: %w", alloc.MetaPath, err,
)
}
host.Garage.Instances = append(host.Garage.Instances, bootstrap.GarageHostInstance{
ID: id,
RPCPort: rpcPort,
S3APIPort: alloc.S3APIPort,
})
allocs[i].RPCPort = rpcPort
}
}
hostBootstrap.Hosts[host.Name] = host
if err := writeBootstrapToStateDir(hostBootstrap); err != nil {
return bootstrap.Bootstrap{}, fmt.Errorf("writing bootstrap file: %w", err)
}
return hostBootstrap, nil
}
const daemonHTTPRPCPath = "/rpc/v0.json"
func newHTTPServer(

50
go/cmd/entrypoint/garage_util.go
View File

@ -1,50 +0,0 @@
package main
import (
"context"
"fmt"
"isle/bootstrap"
"isle/daemon"
"isle/garage"
"dev.mediocregopher.com/mediocre-go-lib.git/mlog"
)
func garageInitializeGlobalBucket(
ctx context.Context,
logger *mlog.Logger,
hostBootstrap bootstrap.Bootstrap,
daemonConfig daemon.Config,
) (
garage.S3APICredentials, error,
) {
adminClient := daemon.NewGarageAdminClient(
logger, daemonConfig, hostBootstrap,
)
creds, err := adminClient.CreateS3APICredentials(
ctx, garage.GlobalBucketS3APICredentialsName,
)
if err != nil {
return creds, fmt.Errorf("creating global bucket credentials: %w", err)
}
bucketID, err := adminClient.CreateBucket(ctx, garage.GlobalBucket)
if err != nil {
return creds, fmt.Errorf("creating global bucket: %w", err)
}
if err := adminClient.GrantBucketPermissions(
ctx,
bucketID,
creds.ID,
garage.BucketPermissionRead,
garage.BucketPermissionWrite,
); err != nil {
return creds, fmt.Errorf(
"granting permissions to shared global bucket key: %w", err,
)
}
return creds, nil
}

62
go/cmd/entrypoint/network.go
View File

@ -3,9 +3,70 @@ package main
import (
"errors"
"fmt"
"isle/admin"
"isle/bootstrap"
"isle/daemon"
"os"
)
var subCmdNetworkCreate = subCmd{
name: "create",
descr: "Create's a new network, with this host being the first host in that network. The resulting admin.json is output to stdout.",
do: func(subCmdCtx subCmdCtx) error {
var (
ctx = subCmdCtx.ctx
flags = subCmdCtx.flagSet(false)
req daemon.CreateNetworkRequest
)
flags.StringVarP(
&req.Name, "name", "n", "",
"Human-readable name to identify the network as.",
)
flags.StringVarP(
&req.Domain, "domain", "d", "",
"Domain name that should be used as the root domain in the network.",
)
flags.StringVarP(
&req.IPNet, "ip-net", "i", "",
`IP+prefix (e.g. "10.10.0.1/16") which denotes the IP of this`+
` host, which will be the first host in the network, and the`+
` range of IPs which other hosts in the network can be`+
` assigned`,
)
flags.StringVarP(
&req.HostName, "hostname", "h", "",
"Name of this host, which will be the first host in the network",
)
if err := flags.Parse(subCmdCtx.args); err != nil {
return fmt.Errorf("parsing flags: %w", err)
}
if req.Name == "" ||
req.Domain == "" ||
req.IPNet == "" ||
req.HostName == "" {
return errors.New("--name, --domain, --ip-net, and --hostname are required")
}
var adm admin.Admin
err := subCmdCtx.daemonRCPClient.Call(ctx, &adm, "CreateNetwork", req)
if err != nil {
return fmt.Errorf("creating network: %w", err)
}
if err := adm.WriteTo(os.Stdout); err != nil {
return fmt.Errorf("writing admin.json to stdout")
}
return nil
},
}
var subCmdNetworkJoin = subCmd{
name: "join",
descr: "Joins this host to an existing network",
@ -45,6 +106,7 @@ var subCmdNetwork = subCmd{
descr: "Sub-commands related to network membership",
do: func(subCmdCtx subCmdCtx) error {
return subCmdCtx.doSubCmd(
subCmdNetworkCreate,
subCmdNetworkJoin,
)
},

23
go/daemon/daemon.go
View File

@ -349,24 +349,37 @@ func (d *daemon) postInit(ctx context.Context) bool {
// TODO this is pretty hacky, but there doesn't seem to be a better way to
// manage it at the moment.
if d.currBootstrap.Garage.GlobalBucketS3APICredentials == (garage.S3APICredentials{}) {
var garageGlobalBucketCreds garage.S3APICredentials
currBootstrap := d.currBootstrap
if !until(
ctx,
d.logger,
"Initializing garage shared global bucket",
func(ctx context.Context) error {
var err error
garageGlobalBucketCreds, err = garageInitializeGlobalBucket(
garageGlobalBucketCreds, err := garageInitializeGlobalBucket(
ctx, d.logger, d.daemonConfig, d.currBootstrap,
)
return err
if err != nil {
return fmt.Errorf("initializing global bucket: %w", err)
}
currBootstrap.Garage.GlobalBucketS3APICredentials = garageGlobalBucketCreds
d.logger.Info(ctx, "Writing bootstrap to state directory")
err = writeBootstrapToStateDir(
d.opts.EnvVars.StateDirPath, currBootstrap,
)
if err != nil {
return fmt.Errorf("writing bootstrap to state dir: %w", err)
}
return nil
},
) {
return false
}
d.l.Lock()
d.currBootstrap.Garage.GlobalBucketS3APICredentials = garageGlobalBucketCreds
d.currBootstrap = currBootstrap
d.l.Unlock()
}

40
go/daemon/rpc.go
View File

@ -4,17 +4,13 @@ import (
"cmp"
"context"
"fmt"
"isle/admin"
"isle/bootstrap"
"slices"
"golang.org/x/exp/maps"
)
// GetHostsResult wraps the results from the GetHosts RPC method.
type GetHostsResult struct {
Hosts []bootstrap.Host
}
// RPC exposes all RPC methods which are available to be called over the RPC
// interface.
type RPC struct {
@ -26,6 +22,35 @@ func NewRPC(daemon Daemon) *RPC {
return &RPC{daemon}
}
// CreateNetworkRequest contains the arguments to the CreateNetwork RPC method.
//
// All fields are required.
type CreateNetworkRequest struct {
// Human-readable name of the network.
Name string
// Primary domain name that network services are served under.
Domain string
// An IP + subnet mask which represents both the IP of this first host in
// the network, as well as the overall range of possible IPs in the network.
IPNet string
// The name of this first host in the network.
HostName string
}
// CreateNetwork passes through to the Daemon method of the same name.
func (r *RPC) CreateNetwork(
ctx context.Context, req CreateNetworkRequest,
) (
admin.Admin, error,
) {
return r.daemon.CreateNetwork(
ctx, req.Name, req.Domain, req.IPNet, req.HostName,
)
}
// JoinNetwork passes through to the Daemon method of the same name.
func (r *RPC) JoinNetwork(
ctx context.Context, req bootstrap.Bootstrap,
@ -35,6 +60,11 @@ func (r *RPC) JoinNetwork(
return struct{}{}, r.daemon.JoinNetwork(ctx, req)
}
// GetHostsResult wraps the results from the GetHosts RPC method.
type GetHostsResult struct {
Hosts []bootstrap.Host
}
// GetHosts returns all hosts known to the network, sorted by their name.
func (r *RPC) GetHosts(
ctx context.Context, req struct{},

17
tests/utils/with-1-data-1-empty-node-network.sh
View File

@ -49,22 +49,21 @@ if [ ! -d "$XDG_RUNTIME_DIR/isle" ]; then
capacity: 1
EOF
isle daemon -l debug --config-path daemon.yml >daemon.log 2>&1 &
pid="$!"
$SHELL "$UTILS/register-cleanup.sh" "$pid" "1-data-1-empty-node-network/primus"
echo "Waiting for primus daemon (process $pid) to start"
while ! [ -e "$ISLE_DAEMON_HTTP_SOCKET_PATH" ]; do sleep 1; done
echo "Creating 1-data-1-empty network"
isle admin create-network \
--config-path daemon.yml \
isle network create \
--domain shared.test \
--hostname primus \
--ip-net "$current_ip/24" \
--name "testing" \
> admin.json
isle daemon -l debug --config-path daemon.yml >daemon.log 2>&1 &
pid="$!"
$SHELL "$UTILS/register-cleanup.sh" "$pid" "1-data-1-empty-node-network/primus"
echo "Waiting for primus daemon (process $pid) to initialize"
while ! isle hosts list >/dev/null; do sleep 1; done
echo "Creating secondus bootstrap"
isle admin create-bootstrap \
--admin-path admin.json \