isle/docs/dev/gateway.md

201 lines
8.2 KiB
Markdown
Raw Normal View History

2023-04-24 19:31:59 +00:00
# Gateway
This document acts as the spec for a general purpose gateway system built into
the isle daemon.
2023-04-24 19:31:59 +00:00
The high level goal of the gateway is to remove significant barriers to entry
for individuals hosting data and services at their own domain names:
* Maintaining a public IP or dDNS
* Keeping a server always online
If a network operator has configured their host as a lighthouse for their
network they will have already solved these problems. By providing gateway
functionality they can share that work with their community.
The gateway functions by serving requests for a user's DNS domain on their
behalf. Requests are served by publicly facing hosts of an isle network using
2023-04-24 19:31:59 +00:00
a backend configured by the user.
## Terminology
* "gateway process" refers to the the system process running on a single host
which serves requests for an isle network's gateway.
2023-04-24 19:31:59 +00:00
* "gateway" refers to the collection of all gateway processes which are
configured and running for an isle network. All gateway processes exhibit
2023-04-24 19:31:59 +00:00
the same behavior, and serve the same domain names, and so are conceived as
being a single unit.
* "gateway DNS name" refers to the single DNS domain or subdomain which contains
a A/AAAA record for each of the gateway processes of an isle network.
2023-04-24 19:31:59 +00:00
## Gateway DNS Setup
In order to support the gateway feature, the operators of a network will need to
agree on a single public DNS name (which could be any arbitrary domain or
sub-domain) which will be used by the gateway. The operators will need to agree
on who is able to edit this name, taking into consideration "bus factor"
concerns.
For an operator to enable their host's gateway process they need to:
* Enable the option within their `daemon.yml`, and include the appropriate
public IP or DNS name for their host.
* Make ports 80/443 publicly available on their host.
* Have that same public IP/DNS name added as an A/AAAA record on the gateway DNS
name.
## Backends
Within the context of the gateway, a backend is any method used by the gateway
to serve requests at a domain. Backends could include:
- Public git repositories (like how github pages works)
- S3 buckets on the private garage cluster
- IPFS files or IPNS names
- Services which are available privately over an isle network
2023-04-24 19:31:59 +00:00
A backend is uniquely described and identified by an object containing a `type`
field, and then further fields which depend on the type. The object for a git
repository backend would look like:
```json
{
"type":"git",
"origin":"URL",
"branch":"main"
}
```
All backends essentially act as a static filesystem, from which requests are
served by the gateway. For example, given a domain `example.com` which has been
configured with an isle network gateway to serve a git repo, if
2023-04-24 19:31:59 +00:00
`example.com/foo/bar` is requested then the file at `/foo/bar` within the
current tip of the git repo is served.
## User Experience
It should be noted up-front that a gateway user should not need to have any
hosts registered on an isle network, or have any other material connection to
2023-04-24 19:31:59 +00:00
the network, in order to use the gateway.
### Setup a Domain
The user navigates to a webpage hosted at the gateway DNS name, and selects the
option to "setup domain". This brings them to a new webpage.
This page presents the user with a dropdown input containing options for each
supported backend type. When the user selects a backend type, the page updates
to present further fields depending on the backend type. The presented fields
are all those required to construct a backend object of the selected type.
Once the user has input all necessary information, they click a button to
continue to the next step.
At the next step they are prompted for a secret passphrase that is only known by
the network operators. The user must get this passphrase from an operator, if
they haven't already. Once successfully input, they continue to the next step.
At the next step they are presented with a unique, opaque string, along with
instructions. The instructions prompt the user to:
* Add a CNAME record to their DNS domain, pointing to the gateway DNS name.
* Add a TXT record to their DNS domain containing `isle_gateway=<opaque
2023-04-24 19:31:59 +00:00
string>`.
The page provides the user with an input box to input their domain name into,
and a "Check" button to click once they have completed these instructions. If
they have not successfully completed the instructions then clicking "Check" will
display an appropriate error to them. If they have successfully completed the
instructions then clicking "Check" will result in a "Success" screen.
At this point navigating to their domain should result in their chosen backend
being served.
### Checking a Domain
The user navigates to a webpage hosted at the gateway DNS name, and selects the
option to "check existing domain". This brings them to a new webpage.
This page presents the user with a text box to input a domain name into, and a
"Check" button. When the button is clicked and a domain without a backend
already set up on it is given then an error is displayed. When the button is
clicked and a domain with a backend set up is given then the information for
that backend is displayed in a read-only mode.
### Updating a Domain
The user navigates to a webpage hosted at the gateway DNS name, and selects the
option to "update existing domain". This brings them to a new webpage.
This webpage works almost the same as the "Checking a Domain" page. The
difference comes at the final step, when the "Check" button is clicked and a
domain with a backend set up is given. At this point the existing backend
information for that domain is displayed in a _read-write_ mode, i.e. in a form
similar to that used in the "Setup a Domain" step for the same backend. From
here the user can edit any information they want, and the flow continues exactly
as if they were in the "Setup a Domain" flow.
## Implementation Notes
The following are general notes and ideas about how this can be implemented.
They are not meant to be set in stone.
* A backend object can be deterministically hashed to form a unique ID for that
backend. The "opaque string" which is added to a user's domain's TXT record is
this ID.
* The gateway processes use a shared garage S3 bucket for persistence of
domain/backend mappings.
* Each process periodically re-reads all mapping data from S3 and uses that as
its currently active set of mappings. A pubsub system like NATS can be added
later to ensure that re-syncing happens at the same time.
* Each file in the bucket corresponds to a domain name which has been
successfully setup. The value contains the backend which is set up for the
domain.
* When the "Check" button is clicked during the "Setup/Update a Domain" flows,
the gateway process which handles the call does the following:
* Checks that a TXT record with the expected value exists on the domain being
added.
* Checks that a CNAME record with the expected value exists on the domain
being added.
* Performs ACME challenge for SSL cert. It will do this via an
[HTTP-01][http01] challenge. The challenge token will be put in the shared
garage S3 bucket, so that all gateway processes can potentially serve it
when requested.
* Depending on the backend, it may be required to sync some data into the
shared garage S3 bucket. For example, if the git backend is being used, then
the tip of the repo+branch will be synced into the S3 bucket.
* Adds a file to the shared garage S3 bucket corresponding to the new
domain/backend mapping.
* Each gateway process is responsible for keeping its own SSL certificates up to
date via periodic ACME challenges.
* Backend data in the shared garage S3 bucket will need to be periodically kept
up to date. A mechanism needs to be devised to do this reasonably reliably,
but without _all_ gateway processes constantly doing it. A communication layer
like NATS might come in handy here.
[http01]: https://letsencrypt.org/docs/challenge-types/#http-01-challenge
## Future Work
It should be possible for someone accessing a private domain on the isle
2023-04-24 19:31:59 +00:00
network to access the gateway web panel and use it _without_ requiring a secret
passphrase. This way community members who have already been vetted (by being
added to the VPN) can have even less friction.