Update documentation

This commit is contained in:
Brian Picciano 2024-10-03 18:59:48 +02:00
parent 61c5338b84
commit 48611df2cb
16 changed files with 240 additions and 185 deletions

116
README.md
View File

@ -6,103 +6,41 @@ rely on it for anything._**
# Isle # Isle
The Isle project provides the foundation for an **autonomous community Welcome to Isle's technical documentation. You can find a less technical
cloud infrastructure**. entrypoint to Isle on [the Micropelago website][isle].
This project targets communities of individuals, where certain members of the Isle runs on a host as a server daemon, and connects to other isle instances to
community would like to host services and applications from servers running in form a peer-to-peer network. Isle networks are completely self-hosted; no
their homes or offices. These servers can range from simple Raspberry Pis to third-parties are required for a network to function.
full-sized home PCs.
The core components of Isle, currently, are: Members of a network are able to build upon the capabilities provided by Isle to
host services for themselves and others. These capabilities include:
* A VPN which enables direct peer-to-peer communication. Even if most hosts in * A VPN which enables direct peer-to-peer communication between network members.
the network are on a private LAN (e.g. their home WiFi network) or have a Even if most hosts in the network are on a private LAN (e.g. their home WiFi
dynamic IP, they can still communicate directly with each other. network) or have a dynamic IP, they can still communicate directly with each
other.
* An S3-compatible network filesystem. Each users can provide as much storage as * An S3-compatible network filesystem. Each member can provide as much storage
they care to, if any. Stored data is sharded and replicated across all hosts as they care to, if any. Stored data is sharded and replicated across all
that choose to provide storage. hosts that choose to provide storage.
* A DNS server which provides automatic host and service (coming soon) discovery * A DNS server which provides automatic host discovery within the network.
within the network.
These components are wrapped into a single binary, with all setup being Every isle daemon is able to create or join multiple independent networks. In
automated. Isle takes "just works" very seriously. this case the networks remain siloed from each other, such that members of one
network are unable to access resources or communicate with members of the other.
Participants are able to build upon these foundations to host services for [isle]: http://portal.mozz.us/gemini/micropelago.net/isle/
themselves and others. They can be assured that their communications are private
and their storage is reliable, all with zero administrative overhead and zero
third parties involved.
## Documentation ## Getting Started
Isle users fall into different roles, depending on their level of The following pages will guide you through setup of Isle, joining an existing
involvement and expertise within their particular network. The documentation network, and all other functionality available via the command-line.
is broken down by these categories, so that the reader can easily
decide which documents they need to care about.
### User Docs * [Installation](./docs/install.md)
* [Command-line Usage](./docs/command-line.md)
* [Join a Network](./docs/user/join-a-network.md)
Users are participants who use network resources, but do not provide any network Those who want to dive in and contribute to the Isle codebase should check out
resources themselves. Users may be accessing the network from a mobile device, the [Developer Documentation](./docs/dev/index.md).
and so are not expected to be online at any particular moment.
Documentation for users:
* [Getting Started](docs/user/getting-started.md)
* [Using DNS](docs/user/using-dns.md) (advanced)
* Restic example (TODO)
### Operator Docs
Operators are users who own a dedicated host which they can expect to be
always-online (to the extent that's possible in a residential environment).
Operator hosts will need at least one of the following to be useful:
* A static public IP, or a dynamic public IP with [dDNS][ddns] set up.
* At least 1GB of unused storage which can be reserved for the network.
Operators are expected to be familiar with server administration, and to not be
afraid of a terminal.
Documentation for operators:
* [Contributing Storage](docs/operator/contributing-storage.md)
* [Contributing a Lighthouse](docs/operator/contributing-a-lighthouse.md)
* [Managing garage](docs/operator/managing-garage.md)
* [Firewalls](doc/operator/firewall.md)
[ddns]: https://www.cloudflare.com/learning/dns/glossary/dynamic-dns/
### Admin Docs
Admins are users who control membership within the network. They are likely
operators as well.
Documentation for admins:
* [Creating a New Network](docs/admin/creating-a-new-network.md)
* [Adding a Host to the Network](docs/admin/adding-a-host-to-the-network.md)
* Removing a Host From the Network (TODO)
### Dev Docs
Devs may or may not be users in any particular isle network. They instead are
those who work on the actual code for Isle.
Documentation for devs:
* [Design Principles](docs/dev/design-principles.md)
* [`isle daemon` process tree](docs/dev/daemon-process-tree.svg): Diagram
describing the [pmux](https://code.betamike.com/micropelago/pmux) process tree
created by `isle daemon` at runtime.
* [Rebuilding Documentation](docs/dev/rebuilding-documentation.md)
* [Releases](docs/dev/releases.md)
## Misc
Besides documentation, there are a few other pages which might be useful:
* [Glossary](docs/glossary.md)

View File

@ -7,7 +7,7 @@
revision ? "dev", revision ? "dev",
releaseName ? "dev", releaseName ? "dev",
bootstrap ? null, bootstrap ? null, # TODO remove this
}: let }: let

View File

@ -1,7 +1,6 @@
# Creating a New Network # Creating a New Network
This guide is for those who wish to start a new isle network of their This guide is for those who wish to start a new isle network of their own.
own.
By starting a new isle network, you are becoming the administrator of a By starting a new isle network, you are becoming the administrator of a
network. Be aware that being a network administrator is not necessarily easy, network. Be aware that being a network administrator is not necessarily easy,

54
docs/command-line.md Normal file
View File

@ -0,0 +1,54 @@
# Command-line Usage
This documents provides examples of how to accomplish various tasks from Isle's
command-line interface.
Isle network members fall into different roles, depending on their level of
involvement and expertise within their particular network. The documentation is
broken down by these categories, so that the reader can easily decide which
documents they need to care about.
### User Docs
Users are members who use network resources, but do not provide any network
resources themselves. Users may be accessing the network from a mobile device,
and so are not expected to be online at any particular moment.
Documentation for users:
* [Joining a Network](./user/join-a-network.md)
* [Using DNS](./user/using-dns.md) (advanced)
* Restic example (TODO)
### Operator Docs
Operators are members who own a dedicated host which they can expect to be
always-online (to the extent that's possible in a residential environment).
Operator hosts will need at least one of the following to be useful:
* A static public IP, or a dynamic public IP with [dDNS][ddns] set up.
* At least 1GB of unused storage which can be reserved for the network.
Operators are expected to be familiar with server administration, and to not be
afraid of a terminal.
Documentation for operators:
* [Contributing Storage](./operator/contributing-storage.md)
* [Contributing a Lighthouse](./operator/contributing-a-lighthouse.md)
* [Managing garage](./operator/managing-garage.md)
* [Firewalls](./operator/firewall.md)
[ddns]: https://www.cloudflare.com/learning/dns/glossary/dynamic-dns/
### Admin Docs
Admins are members who control membership within the network. They are likely
operators as well.
Documentation for admins:
* [Creating a New Network](./admin/creating-a-new-network.md)
* [Adding a Host to the Network](./admin/adding-a-host-to-the-network.md)
* Removing a Host From the Network (TODO)

25
docs/dev/architecture.md Normal file
View File

@ -0,0 +1,25 @@
# Architecture
The isle daemon is the central point around which all operations occur. Users of
the isle command-line tool (or, soon, a GUI) communicate with the daemon over a
local RPC socket to issue commands, for example to tell it to join a new network
or to retrieve the names of hosts in an already-joined network.
For every network which is joined, the isle daemon will create and manage
sub-processes to provide certain parts of its functionality. These include:
* A [nebula][nebula] instance to provide VPN functionality.
* A [dnsmasq][dnsmasq] instance to act as DNS server.
* Zero or more [garage][garage] instances, depending on how storage is
configured on the host, to provide the S3 storage layer.
The isle daemon considers the configuration and management of these
sub-processes to be an implementation detail. In other words, its RPC interface,
and therefore all user interfaces, do not expose members to the fact that these
sub-processes exist, or even that these projects are being used under the hood.
![Architectural diagram](./architecture.svg)
[nebula]: https://github.com/slackhq/nebula
[garage]: https://garagehq.deuxfleurs.fr/
[dnsmasq]: https://dnsmasq.org/doc.html

View File

@ -0,0 +1,49 @@
@startuml
skinparam componentStyle rectangle
[isle command line] as isleCommand
() "daemon.RPC" as isleDaemonRPC
[daemon.Client] as daemonClient
frame "isle daemon process" {
portin "RPC Socket" as rpcSocket
[RPC Server] as rpcServer
() "daemon.RPC" as daemonRPC
[daemon.Daemon] as daemon
[network.Network (A)] as networkA
[network.Network (B)] as networkB
rpcServer --> rpcSocket : handle
rpcServer ..> daemonRPC : dispatch
daemon - daemonRPC
daemon --> networkA
daemon --> networkB
}
isleCommand ..> isleDaemonRPC : issue commands
daemonClient - isleDaemonRPC
daemonClient --> rpcSocket
package "network A child processes" {
[nebula] as networkANebula
[garage (alloc 1)] as networkAGarage1
[garage (alloc 2)] as networkAGarage2
[dnsmasq] as networkADNSMasq
}
networkA --> networkANebula
networkA --> networkAGarage1
networkA --> networkAGarage2
networkA --> networkADNSMasq
package "network B child processes" {
[nebula] as networkBNebula
[dnsmasq] as networkBDNSMasq
}
networkB --> networkBNebula
networkB --> networkBDNSMasq
@enduml

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 13 KiB

34
docs/dev/building.md Normal file
View File

@ -0,0 +1,34 @@
# Building Isle
Building from source requires [nix][nix].
(*NOTE* The first time you run some of these builds a lot of things will be
built from scratch. If you have not otherwise configured it, nix might be using
a tmpfs as its build directory, and the capacity of this tmpfs will probably be
exceeded by this build. You can change your build directory to somewhere on-disk
by setting the TMPDIR environment variable for `nix-daemon` (see [this github
issue][tmpdir-gh].))
[nix]: https://nixos.wiki/wiki/Nix_package_manager
[tmpdir-gh]: https://github.com/NixOS/nix/issues/2098#issuecomment-383243838
## Current System
You can build an AppImage for your current system by running the following from
the project's root:
```
nix-build -A appImageBin
```
The resulting binary can be found under `result/bin`.
## Cross-Compile
An AppImage can be cross-compiled by passing the `hostSystem` argument:
```
nix-build --argstr hostSystem "x86_64-linux" -A appImageBin
```
Supported system strings are enumerated in `nix/pkgs.nix`.

View File

@ -1,37 +0,0 @@
@startuml
hide empty description
state "./isle daemon -c ./daemon.yml" as init
state AppDir {
note "All relative paths are relative to the root of the AppDir" as N1
state "./bin/entrypoint daemon -c ./daemon.yml" as entrypoint {
entrypoint : * Create runtime dir
entrypoint : * Lock runtime dir
entrypoint : * Merge given and default daemon.yml files
entrypoint : * Copy bootstrap.json into state directory, if it's not there
entrypoint : * Merge daemon.yml config into bootstrap.json
entrypoint : * Create $RUNTIME_DIRECTORY/dnsmasq.conf
entrypoint : * Create $RUNTIME_DIRECTORY/nebula.yml
entrypoint : * Create $RUNTIME_DIRECTORY/garage-N.toml\n (one per storage allocation)
entrypoint : * Spawn child processes
entrypoint : * Wait for nebula & garage to initialize
entrypoint : * Updates garage cluster layout
entrypoint : * Stores host info in global bucket, based on latest bootstrap.json
}
init --> entrypoint : exec
state "./bin/dnsmasq -d -C $RUNTIME_DIRECTORY/dnsmasq.conf" as dnsmasq
entrypoint --> dnsmasq : child
state "./bin/nebula -config $RUNTIME_DIRECTORY/nebula.yml" as nebula
entrypoint --> nebula : child
state "./bin/garage -c $RUNTIME_DIRECTORY/garage-N.toml server" as garage
entrypoint --> garage : child (one per storage allocation)
}
@enduml

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 8.9 KiB

View File

@ -3,7 +3,7 @@
The following points form the basis for all design decisions made within the The following points form the basis for all design decisions made within the
Isle project. Isle project.
* The UX is aggressively optimized to eliminate manual intervention by users. * The UX is aggressively optimized to eliminate manual intervention by members.
All other concerns are secondary. The concept of "UX" extends beyond GUI All other concerns are secondary. The concept of "UX" extends beyond GUI
interfaces, and encompasses all interactions of any sort with an isle interfaces, and encompasses all interactions of any sort with an isle
process. process.

View File

@ -8,7 +8,8 @@ documentation and source code.
be capitalized. be capitalized.
- "isle" - The name of the binary or program produced by the Isle project. isle - "isle" - The name of the binary or program produced by the Isle project. isle
is the name of the file itself, and so is always lower-case. is the name of the file itself, as well as an instance of the process, and so
is always lower-case.
- "host" - A computer or device on which isle is run. - "host" - A computer or device on which isle is run.
@ -20,5 +21,5 @@ documentation and source code.
term "cluster" in the context of garage to stay consistent with garage's term "cluster" in the context of garage to stay consistent with garage's
documentation and command-line. documentation and command-line.
- "user" - A person who takes part in the usage, operation, or administration of - "member" - A person who takes part in the usage, operation, or administration
an isle network. of an isle network.

15
docs/dev/index.md Normal file
View File

@ -0,0 +1,15 @@
# Developer Documentation
This section of the documentation is targeted at those who are contributing to
the Isle codebase. It is recommended to start with the following pages, in order
to better understand how to navigate and work on the codebase.
* [Glossary](./glossary.md)
* [Design Principles](./design-principles.md)
* [Architecture](./architecture.md)
These pages can be helpful in specific situations.
* [Building Isle](./building.md)
* [Rebuilding Documentation](./rebuilding-documentation.md)
* [Releases](./releases.md)

View File

@ -2,8 +2,8 @@
A release consists of: A release consists of:
- A full set of binaries for all supported platforms, compiled from the same - A full set of `isle` binaries for all supported platforms, compiled from the
source. same source.
- A text file containing hashes of each binary. - A text file containing hashes of each binary.
- A file containing a signature of the hash file, created by whoever is building - A file containing a signature of the hash file, created by whoever is building
the release. the release.

View File

@ -1,7 +1,7 @@
# Getting Started # Installation
This document will guide you through the process of obtaining an isle This document will guide you through the process of obtaining and installing
binary and joining a network. Isle on your machine.
NOTE currently only linux machines with the following architectures are NOTE currently only linux machines with the following architectures are
supported: supported:
@ -56,33 +56,13 @@ If your distro uses an init system other than systemd then you will need to
configure that yourself. You can use the systemd service file linked above as a configure that yourself. You can use the systemd service file linked above as a
reference. reference.
[latest]: https://code.betamike.com/micropelago/isle/releases/latest
[serviceFile]: https://code.betamike.com/micropelago/isle/src/branch/main/dist/linux/isle.service [serviceFile]: https://code.betamike.com/micropelago/isle/src/branch/main/dist/linux/isle.service
### From Source ### From Source
(TODO probably move these instructions into the Dev docs section). If you'd like to build your own `isle` binary from scratch, see the [Building
Isle](./dev/building.md) document.
Building from source requires [nix][nix].
You can build your own AppImage by running the following from the project's
root:
```
nix-build -A appImageBin
```
(*NOTE* The first time you run this a lot of things will be built from scratch.
If you have not otherwise configured it, nix might be using a tmpfs as its build
directory, and the capacity of this tmpfs will probably be exceeded by this
build. You can change your build directory to somewhere on-disk by setting the
TMPDIR environment variable for `nix-daemon` (see
[this github issue][tmpdir-gh].))
The resulting binary can be found under `result/bin`. From here you can continue
with the instructions under the "AppImage" section above.
[nix]: https://nixos.wiki/wiki/Nix_package_manager
[tmpdir-gh]: https://github.com/NixOS/nix/issues/2098#issuecomment-383243838
## Add Users to the `isle` Group (Optional) ## Add Users to the `isle` Group (Optional)
@ -104,26 +84,3 @@ sudo systemctl enable --now isle
(NOTE If your distro uses an init system other than systemd then you will need (NOTE If your distro uses an init system other than systemd then you will need
to instead start isle according to that system's requirements.) to instead start isle according to that system's requirements.)
## Join a Network
This section will guide you through the process of joining an existing network
of isle hosts. If instead you wish to create a new network for others to join
then see the [Creating a New Network][creating-a-new-network] page.
To join an existing network you will need to first obtain a `bootstrap.json`
file. The `bootstrap.json` file contains all information required for your
particular host to join the network, and must be generated and provided to you
by an admin for the network.
Once obtained, you can join the network by doing:
```
isle network join --bootstrap-path /path/to/bootstrap.json
```
After a few moments you will have successfully joined the network!
[creating-a-new-network]: ../admin/creating-a-new-network.md
[latest]: https://code.betamike.com/micropelago/isle/releases/latest

View File

@ -0,0 +1,20 @@
# Join a Network
This document guide you through the process of joining an existing network
of isle hosts. If instead you wish to create a new network for others to join
then see the [Creating a New Network][creating-a-new-network] page.
To join an existing network you will need to first obtain a `bootstrap.json`
file. The `bootstrap.json` file contains all information required for your
particular host to join the network, and must be generated and provided to you
by an admin for the network.
Once obtained, you can join the network by doing:
```
isle network join --bootstrap-path /path/to/bootstrap.json
```
After a few moments you will have successfully joined the network!
[creating-a-new-network]: ../admin/creating-a-new-network.md