Gateway

This document acts as the spec for a general purpose gateway system built into the isle daemon.

The high level goal of the gateway is to remove significant barriers to entry for individuals hosting data and services at their own domain names:

Maintaining a public IP or dDNS
Keeping a server always online

If a network operator has configured their host as a lighthouse for their network they will have already solved these problems. By providing gateway functionality they can share that work with their community.

The gateway functions by serving requests for a user's DNS domain on their behalf. Requests are served by publicly facing hosts of an isle network using a backend configured by the user.

Terminology

"gateway process" refers to the the system process running on a single host which serves requests for an isle network's gateway.
"gateway" refers to the collection of all gateway processes which are configured and running for an isle network. All gateway processes exhibit the same behavior, and serve the same domain names, and so are conceived as being a single unit.
"gateway DNS name" refers to the single DNS domain or subdomain which contains a A/AAAA record for each of the gateway processes of an isle network.

Gateway DNS Setup

In order to support the gateway feature, the operators of a network will need to agree on a single public DNS name (which could be any arbitrary domain or sub-domain) which will be used by the gateway. The operators will need to agree on who is able to edit this name, taking into consideration "bus factor" concerns.

For an operator to enable their host's gateway process they need to:

Enable the option within their daemon.yml, and include the appropriate public IP or DNS name for their host.
Make ports 80/443 publicly available on their host.
Have that same public IP/DNS name added as an A/AAAA record on the gateway DNS name.

Backends

Within the context of the gateway, a backend is any method used by the gateway to serve requests at a domain. Backends could include:

Public git repositories (like how github pages works)
S3 buckets on the private garage cluster
IPFS files or IPNS names
Services which are available privately over an isle network

A backend is uniquely described and identified by an object containing a type field, and then further fields which depend on the type. The object for a git repository backend would look like:

{
  "type":"git",
  "origin":"URL",
  "branch":"main"
}

All backends essentially act as a static filesystem, from which requests are served by the gateway. For example, given a domain example.com which has been configured with an isle network gateway to serve a git repo, if example.com/foo/bar is requested then the file at /foo/bar within the current tip of the git repo is served.

User Experience

It should be noted up-front that a gateway user should not need to have any hosts registered on an isle network, or have any other material connection to the network, in order to use the gateway.

Setup a Domain

The user navigates to a webpage hosted at the gateway DNS name, and selects the option to "setup domain". This brings them to a new webpage.

This page presents the user with a dropdown input containing options for each supported backend type. When the user selects a backend type, the page updates to present further fields depending on the backend type. The presented fields are all those required to construct a backend object of the selected type.

Once the user has input all necessary information, they click a button to continue to the next step.

At the next step they are prompted for a secret passphrase that is only known by the network operators. The user must get this passphrase from an operator, if they haven't already. Once successfully input, they continue to the next step.

At the next step they are presented with a unique, opaque string, along with instructions. The instructions prompt the user to:

Add a CNAME record to their DNS domain, pointing to the gateway DNS name.
Add a TXT record to their DNS domain containing isle_gateway=<opaque string>.

The page provides the user with an input box to input their domain name into, and a "Check" button to click once they have completed these instructions. If they have not successfully completed the instructions then clicking "Check" will display an appropriate error to them. If they have successfully completed the instructions then clicking "Check" will result in a "Success" screen.

At this point navigating to their domain should result in their chosen backend being served.

Checking a Domain

The user navigates to a webpage hosted at the gateway DNS name, and selects the option to "check existing domain". This brings them to a new webpage.

This page presents the user with a text box to input a domain name into, and a "Check" button. When the button is clicked and a domain without a backend already set up on it is given then an error is displayed. When the button is clicked and a domain with a backend set up is given then the information for that backend is displayed in a read-only mode.

Updating a Domain

The user navigates to a webpage hosted at the gateway DNS name, and selects the option to "update existing domain". This brings them to a new webpage.

This webpage works almost the same as the "Checking a Domain" page. The difference comes at the final step, when the "Check" button is clicked and a domain with a backend set up is given. At this point the existing backend information for that domain is displayed in a read-write mode, i.e. in a form similar to that used in the "Setup a Domain" step for the same backend. From here the user can edit any information they want, and the flow continues exactly as if they were in the "Setup a Domain" flow.

Implementation Notes

The following are general notes and ideas about how this can be implemented. They are not meant to be set in stone.

A backend object can be deterministically hashed to form a unique ID for that backend. The "opaque string" which is added to a user's domain's TXT record is this ID.
The gateway processes use a shared garage S3 bucket for persistence of domain/backend mappings.
- Each process periodically re-reads all mapping data from S3 and uses that as its currently active set of mappings. A pubsub system like NATS can be added later to ensure that re-syncing happens at the same time.
- Each file in the bucket corresponds to a domain name which has been successfully setup. The value contains the backend which is set up for the domain.
When the "Check" button is clicked during the "Setup/Update a Domain" flows, the gateway process which handles the call does the following:
- Checks that a TXT record with the expected value exists on the domain being added.
- Checks that a CNAME record with the expected value exists on the domain being added.
- Performs ACME challenge for SSL cert. It will do this via an HTTP-01 challenge. The challenge token will be put in the shared garage S3 bucket, so that all gateway processes can potentially serve it when requested.
- Depending on the backend, it may be required to sync some data into the shared garage S3 bucket. For example, if the git backend is being used, then the tip of the repo+branch will be synced into the S3 bucket.
- Adds a file to the shared garage S3 bucket corresponding to the new domain/backend mapping.
Each gateway process is responsible for keeping its own SSL certificates up to date via periodic ACME challenges.
Backend data in the shared garage S3 bucket will need to be periodically kept up to date. A mechanism needs to be devised to do this reasonably reliably, but without all gateway processes constantly doing it. A communication layer like NATS might come in handy here.

Future Work

It should be possible for someone accessing a private domain on the isle network to access the gateway web panel and use it without requiring a secret passphrase. This way community members who have already been vetted (by being added to the VPN) can have even less friction.

8.2 KiB Raw Blame History