diff --git a/README.md b/README.md index 75e1ae8..182f3a6 100644 --- a/README.md +++ b/README.md @@ -101,6 +101,8 @@ Documentation for devs: created by `cryptic-net daemon` at runtime. * [Rebuilding Documentation](docs/dev/rebuilding-documentation.md) * [Releases](docs/dev/releases.md) +* [Gateway](docs/dev/gateway.md): Tentative spec for gateway functionality in + cryptic-net ## Misc diff --git a/docs/dev/gateway.md b/docs/dev/gateway.md new file mode 100644 index 0000000..b0665c5 --- /dev/null +++ b/docs/dev/gateway.md @@ -0,0 +1,200 @@ +# Gateway + +This document acts as the spec for a general purpose gateway system built into +the cryptic-net daemon. + +The high level goal of the gateway is to remove significant barriers to entry +for individuals hosting data and services at their own domain names: + +* Maintaining a public IP or dDNS + +* Keeping a server always online + +If a network operator has configured their host as a lighthouse for their +network they will have already solved these problems. By providing gateway +functionality they can share that work with their community. + +The gateway functions by serving requests for a user's DNS domain on their +behalf. Requests are served by publicly facing hosts of a cryptic network using +a backend configured by the user. + +## Terminology + +* "gateway process" refers to the the system process running on a single host + which serves requests for a cryptic network's gateway. + +* "gateway" refers to the collection of all gateway processes which are + configured and running for a cryptic network. All gateway processes exhibit + the same behavior, and serve the same domain names, and so are conceived as + being a single unit. + +* "gateway DNS name" refers to the single DNS domain or subdomain which contains + a A/AAAA record for each of the gateway processes of a cryptic network. + +## Gateway DNS Setup + +In order to support the gateway feature, the operators of a network will need to +agree on a single public DNS name (which could be any arbitrary domain or +sub-domain) which will be used by the gateway. The operators will need to agree +on who is able to edit this name, taking into consideration "bus factor" +concerns. + +For an operator to enable their host's gateway process they need to: + +* Enable the option within their `daemon.yml`, and include the appropriate + public IP or DNS name for their host. + +* Make ports 80/443 publicly available on their host. + +* Have that same public IP/DNS name added as an A/AAAA record on the gateway DNS + name. + +## Backends + +Within the context of the gateway, a backend is any method used by the gateway +to serve requests at a domain. Backends could include: + +- Public git repositories (like how github pages works) +- S3 buckets on the private garage cluster +- IPFS files or IPNS names +- Services which are available privately over cryptic network + +A backend is uniquely described and identified by an object containing a `type` +field, and then further fields which depend on the type. The object for a git +repository backend would look like: + +```json +{ + "type":"git", + "origin":"URL", + "branch":"main" +} +``` + +All backends essentially act as a static filesystem, from which requests are +served by the gateway. For example, given a domain `example.com` which has been +configured with a cryptic network gateway to serve a git repo, if +`example.com/foo/bar` is requested then the file at `/foo/bar` within the +current tip of the git repo is served. + +## User Experience + +It should be noted up-front that a gateway user should not need to have any +hosts registered on a cryptic network, or have any other material connection to +the network, in order to use the gateway. + +### Setup a Domain + +The user navigates to a webpage hosted at the gateway DNS name, and selects the +option to "setup domain". This brings them to a new webpage. + +This page presents the user with a dropdown input containing options for each +supported backend type. When the user selects a backend type, the page updates +to present further fields depending on the backend type. The presented fields +are all those required to construct a backend object of the selected type. + +Once the user has input all necessary information, they click a button to +continue to the next step. + +At the next step they are prompted for a secret passphrase that is only known by +the network operators. The user must get this passphrase from an operator, if +they haven't already. Once successfully input, they continue to the next step. + +At the next step they are presented with a unique, opaque string, along with +instructions. The instructions prompt the user to: + +* Add a CNAME record to their DNS domain, pointing to the gateway DNS name. + +* Add a TXT record to their DNS domain containing `cryptic_net_gateway=`. + +The page provides the user with an input box to input their domain name into, +and a "Check" button to click once they have completed these instructions. If +they have not successfully completed the instructions then clicking "Check" will +display an appropriate error to them. If they have successfully completed the +instructions then clicking "Check" will result in a "Success" screen. + +At this point navigating to their domain should result in their chosen backend +being served. + +### Checking a Domain + +The user navigates to a webpage hosted at the gateway DNS name, and selects the +option to "check existing domain". This brings them to a new webpage. + +This page presents the user with a text box to input a domain name into, and a +"Check" button. When the button is clicked and a domain without a backend +already set up on it is given then an error is displayed. When the button is +clicked and a domain with a backend set up is given then the information for +that backend is displayed in a read-only mode. + +### Updating a Domain + +The user navigates to a webpage hosted at the gateway DNS name, and selects the +option to "update existing domain". This brings them to a new webpage. + +This webpage works almost the same as the "Checking a Domain" page. The +difference comes at the final step, when the "Check" button is clicked and a +domain with a backend set up is given. At this point the existing backend +information for that domain is displayed in a _read-write_ mode, i.e. in a form +similar to that used in the "Setup a Domain" step for the same backend. From +here the user can edit any information they want, and the flow continues exactly +as if they were in the "Setup a Domain" flow. + +## Implementation Notes + +The following are general notes and ideas about how this can be implemented. +They are not meant to be set in stone. + +* A backend object can be deterministically hashed to form a unique ID for that + backend. The "opaque string" which is added to a user's domain's TXT record is + this ID. + +* The gateway processes use a shared garage S3 bucket for persistence of + domain/backend mappings. + + * Each process periodically re-reads all mapping data from S3 and uses that as + its currently active set of mappings. A pubsub system like NATS can be added + later to ensure that re-syncing happens at the same time. + + * Each file in the bucket corresponds to a domain name which has been + successfully setup. The value contains the backend which is set up for the + domain. + +* When the "Check" button is clicked during the "Setup/Update a Domain" flows, + the gateway process which handles the call does the following: + + * Checks that a TXT record with the expected value exists on the domain being + added. + + * Checks that a CNAME record with the expected value exists on the domain + being added. + + * Performs ACME challenge for SSL cert. It will do this via an + [HTTP-01][http01] challenge. The challenge token will be put in the shared + garage S3 bucket, so that all gateway processes can potentially serve it + when requested. + + * Depending on the backend, it may be required to sync some data into the + shared garage S3 bucket. For example, if the git backend is being used, then + the tip of the repo+branch will be synced into the S3 bucket. + + * Adds a file to the shared garage S3 bucket corresponding to the new + domain/backend mapping. + +* Each gateway process is responsible for keeping its own SSL certificates up to + date via periodic ACME challenges. + +* Backend data in the shared garage S3 bucket will need to be periodically kept + up to date. A mechanism needs to be devised to do this reasonably reliably, + but without _all_ gateway processes constantly doing it. A communication layer + like NATS might come in handy here. + +[http01]: https://letsencrypt.org/docs/challenge-types/#http-01-challenge + +## Future Work + +It should be possible for someone accessing a private domain on the cryptic +network to access the gateway web panel and use it _without_ requiring a secret +passphrase. This way community members who have already been vetted (by being +added to the VPN) can have even less friction. diff --git a/docs/roadmap.md b/docs/roadmap.md index 8ac05e1..2518f34 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -39,16 +39,7 @@ auto-detect the installed operating system and install the daemon automatically. One idea is to have every `cryptic-net daemon` run a webserver as one of its sub-processes. This server could serve multiple functions: -- Possible public gateway, if the host is configured to be public, into internal - cryptic-net services. This will require some sort of service discovery - mechanism that other hosts in the network can easily hook into via their - `daemon.yml`s. Ideally this mechanism would involve little beyond updating - entries in garage, which the public gateways then pick up on. - - One wrinkle here will be TLS certificates. Ideally internal hosts could host - websites publicly via the gateways on their network, using arbitrary domains - that the users own. The gateways will need some way of obtaining certs for - these domains automatically (or as automatically as possible). +- [Gateway service](./dev/gateway.md). - Local interface for the `cryptic-net daemon` process. For example, status and connectivity information for the local host could be provided via a simple web