isle/docs/dev/gateway.md

# Gateway

This document acts as the spec for a general purpose gateway system built into
the isle daemon.

The high level goal of the gateway is to remove significant barriers to entry
for individuals hosting data and services at their own domain names:

* Maintaining a public IP or dDNS

* Keeping a server always online

If a network operator has configured their host as a lighthouse for their
network they will have already solved these problems. By providing gateway
functionality they can share that work with their community.

The gateway functions by serving requests for a user's DNS domain on their
behalf. Requests are served by publicly facing hosts of an isle network using
a backend configured by the user.

## Terminology

* "gateway process" refers to the the system process running on a single host
  which serves requests for an isle network's gateway.

* "gateway" refers to the collection of all gateway processes which are
  configured and running for an isle network. All gateway processes exhibit
  the same behavior, and serve the same domain names, and so are conceived as
  being a single unit.

* "gateway DNS name" refers to the single DNS domain or subdomain which contains
  a A/AAAA record for each of the gateway processes of an isle network.

## Gateway DNS Setup

In order to support the gateway feature, the operators of a network will need to
agree on a single public DNS name (which could be any arbitrary domain or
sub-domain) which will be used by the gateway. The operators will need to agree
on who is able to edit this name, taking into consideration "bus factor"
concerns.

For an operator to enable their host's gateway process they need to:

* Enable the option within their `daemon.yml`, and include the appropriate
  public IP or DNS name for their host.

* Make ports 80/443 publicly available on their host.

* Have that same public IP/DNS name added as an A/AAAA record on the gateway DNS
  name.

## Backends

Within the context of the gateway, a backend is any method used by the gateway
to serve requests at a domain. Backends could include:

- Public git repositories (like how github pages works)
- S3 buckets on the private garage cluster
- IPFS files or IPNS names
- Services which are available privately over an isle network

A backend is uniquely described and identified by an object containing a `type`
field, and then further fields which depend on the type. The object for a git
repository backend would look like:

```json
{
  "type":"git",
  "origin":"URL",
  "branch":"main"
}
```

All backends essentially act as a static filesystem, from which requests are
served by the gateway. For example, given a domain `example.com` which has been
configured with an isle network gateway to serve a git repo, if
`example.com/foo/bar` is requested then the file at `/foo/bar` within the
current tip of the git repo is served.

## User Experience

It should be noted up-front that a gateway user should not need to have any
hosts registered on an isle network, or have any other material connection to
the network, in order to use the gateway.

### Setup a Domain

The user navigates to a webpage hosted at the gateway DNS name, and selects the
option to "setup domain". This brings them to a new webpage.

This page presents the user with a dropdown input containing options for each
supported backend type. When the user selects a backend type, the page updates
to present further fields depending on the backend type. The presented fields
are all those required to construct a backend object of the selected type.

Once the user has input all necessary information, they click a button to
continue to the next step.

At the next step they are prompted for a secret passphrase that is only known by
the network operators. The user must get this passphrase from an operator, if
they haven't already. Once successfully input, they continue to the next step.

At the next step they are presented with a unique, opaque string, along with
instructions. The instructions prompt the user to:

* Add a CNAME record to their DNS domain, pointing to the gateway DNS name.

* Add a TXT record to their DNS domain containing `isle_gateway=<opaque
  string>`.

The page provides the user with an input box to input their domain name into,
and a "Check" button to click once they have completed these instructions. If
they have not successfully completed the instructions then clicking "Check" will
display an appropriate error to them. If they have successfully completed the
instructions then clicking "Check" will result in a "Success" screen.

At this point navigating to their domain should result in their chosen backend
being served.

### Checking a Domain

The user navigates to a webpage hosted at the gateway DNS name, and selects the
option to "check existing domain". This brings them to a new webpage.

This page presents the user with a text box to input a domain name into, and a
"Check" button. When the button is clicked and a domain without a backend
already set up on it is given then an error is displayed. When the button is
clicked and a domain with a backend set up is given then the information for
that backend is displayed in a read-only mode.

### Updating a Domain

The user navigates to a webpage hosted at the gateway DNS name, and selects the
option to "update existing domain". This brings them to a new webpage.

This webpage works almost the same as the "Checking a Domain" page. The
difference comes at the final step, when the "Check" button is clicked and a
domain with a backend set up is given. At this point the existing backend
information for that domain is displayed in a _read-write_ mode, i.e. in a form
similar to that used in the "Setup a Domain" step for the same backend. From
here the user can edit any information they want, and the flow continues exactly
as if they were in the "Setup a Domain" flow.

## Implementation Notes

The following are general notes and ideas about how this can be implemented.
They are not meant to be set in stone.

* A backend object can be deterministically hashed to form a unique ID for that
  backend. The "opaque string" which is added to a user's domain's TXT record is
  this ID.

* The gateway processes use a shared garage S3 bucket for persistence of
  domain/backend mappings.

  * Each process periodically re-reads all mapping data from S3 and uses that as
    its currently active set of mappings. A pubsub system like NATS can be added
    later to ensure that re-syncing happens at the same time.

  * Each file in the bucket corresponds to a domain name which has been
    successfully setup. The value contains the backend which is set up for the
    domain.

* When the "Check" button is clicked during the "Setup/Update a Domain" flows,
  the gateway process which handles the call does the following:

  * Checks that a TXT record with the expected value exists on the domain being
    added.

  * Checks that a CNAME record with the expected value exists on the domain
    being added.

  * Performs ACME challenge for SSL cert. It will do this via an
    [HTTP-01][http01] challenge. The challenge token will be put in the shared
    garage S3 bucket, so that all gateway processes can potentially serve it
    when requested.

  * Depending on the backend, it may be required to sync some data into the
    shared garage S3 bucket. For example, if the git backend is being used, then
    the tip of the repo+branch will be synced into the S3 bucket.

  * Adds a file to the shared garage S3 bucket corresponding to the new
    domain/backend mapping.

* Each gateway process is responsible for keeping its own SSL certificates up to
  date via periodic ACME challenges.

* Backend data in the shared garage S3 bucket will need to be periodically kept
  up to date. A mechanism needs to be devised to do this reasonably reliably,
  but without _all_ gateway processes constantly doing it. A communication layer
  like NATS might come in handy here.

[http01]: https://letsencrypt.org/docs/challenge-types/#http-01-challenge

## Future Work

It should be possible for someone accessing a private domain on the isle
network to access the gateway web panel and use it _without_ requiring a secret
passphrase. This way community members who have already been vetted (by being
added to the VPN) can have even less friction.