Gateway doc

README.md

@@ -101,6 +101,8 @@ Documentation for devs:
   created by `cryptic-net daemon` at runtime.
 * [Rebuilding Documentation](docs/dev/rebuilding-documentation.md)
 * [Releases](docs/dev/releases.md)
+* [Gateway](docs/dev/gateway.md): Tentative spec for gateway functionality in
+  cryptic-net
 
 ## Misc
 

docs/dev/gateway.md

@@ -0,0 +1,200 @@
+# Gateway
+
+This document acts as the spec for a general purpose gateway system built into
+the cryptic-net daemon.
+
+The high level goal of the gateway is to remove significant barriers to entry
+for individuals hosting data and services at their own domain names:
+
+* Maintaining a public IP or dDNS
+
+* Keeping a server always online
+
+If a network operator has configured their host as a lighthouse for their
+network they will have already solved these problems. By providing gateway
+functionality they can share that work with their community.
+
+The gateway functions by serving requests for a user's DNS domain on their
+behalf. Requests are served by publicly facing hosts of a cryptic network using
+a backend configured by the user.
+
+## Terminology
+
+* "gateway process" refers to the the system process running on a single host
+  which serves requests for a cryptic network's gateway.
+
+* "gateway" refers to the collection of all gateway processes which are
+  configured and running for a cryptic network. All gateway processes exhibit
+  the same behavior, and serve the same domain names, and so are conceived as
+  being a single unit.
+
+* "gateway DNS name" refers to the single DNS domain or subdomain which contains
+  a A/AAAA record for each of the gateway processes of a cryptic network.
+
+## Gateway DNS Setup
+
+In order to support the gateway feature, the operators of a network will need to
+agree on a single public DNS name (which could be any arbitrary domain or
+sub-domain) which will be used by the gateway. The operators will need to agree
+on who is able to edit this name, taking into consideration "bus factor"
+concerns.
+
+For an operator to enable their host's gateway process they need to:
+
+* Enable the option within their `daemon.yml`, and include the appropriate
+  public IP or DNS name for their host.
+
+* Make ports 80/443 publicly available on their host.
+
+* Have that same public IP/DNS name added as an A/AAAA record on the gateway DNS
+  name.
+
+## Backends
+
+Within the context of the gateway, a backend is any method used by the gateway
+to serve requests at a domain. Backends could include:
+
+- Public git repositories (like how github pages works)
+- S3 buckets on the private garage cluster
+- IPFS files or IPNS names
+- Services which are available privately over cryptic network
+
+A backend is uniquely described and identified by an object containing a `type`
+field, and then further fields which depend on the type. The object for a git
+repository backend would look like:
+
+```json
+{
+  "type":"git",
+  "origin":"URL",
+  "branch":"main"
+}
+```
+
+All backends essentially act as a static filesystem, from which requests are
+served by the gateway. For example, given a domain `example.com` which has been
+configured with a cryptic network gateway to serve a git repo, if
+`example.com/foo/bar` is requested then the file at `/foo/bar` within the
+current tip of the git repo is served.
+
+## User Experience
+
+It should be noted up-front that a gateway user should not need to have any
+hosts registered on a cryptic network, or have any other material connection to
+the network, in order to use the gateway.
+
+### Setup a Domain
+
+The user navigates to a webpage hosted at the gateway DNS name, and selects the
+option to "setup domain". This brings them to a new webpage.
+
+This page presents the user with a dropdown input containing options for each
+supported backend type. When the user selects a backend type, the page updates
+to present further fields depending on the backend type. The presented fields
+are all those required to construct a backend object of the selected type.
+
+Once the user has input all necessary information, they click a button to
+continue to the next step.
+
+At the next step they are prompted for a secret passphrase that is only known by
+the network operators. The user must get this passphrase from an operator, if
+they haven't already. Once successfully input, they continue to the next step.
+
+At the next step they are presented with a unique, opaque string, along with
+instructions. The instructions prompt the user to:
+
+* Add a CNAME record to their DNS domain, pointing to the gateway DNS name.
+
+* Add a TXT record to their DNS domain containing `cryptic_net_gateway=<opaque
+  string>`.
+
+The page provides the user with an input box to input their domain name into,
+and a "Check" button to click once they have completed these instructions. If
+they have not successfully completed the instructions then clicking "Check" will
+display an appropriate error to them. If they have successfully completed the
+instructions then clicking "Check" will result in a "Success" screen.
+
+At this point navigating to their domain should result in their chosen backend
+being served.
+
+### Checking a Domain
+
+The user navigates to a webpage hosted at the gateway DNS name, and selects the
+option to "check existing domain". This brings them to a new webpage.
+
+This page presents the user with a text box to input a domain name into, and a
+"Check" button. When the button is clicked and a domain without a backend
+already set up on it is given then an error is displayed. When the button is
+clicked and a domain with a backend set up is given then the information for
+that backend is displayed in a read-only mode.
+
+### Updating a Domain
+
+The user navigates to a webpage hosted at the gateway DNS name, and selects the
+option to "update existing domain". This brings them to a new webpage.
+
+This webpage works almost the same as the "Checking a Domain" page. The
+difference comes at the final step, when the "Check" button is clicked and a
+domain with a backend set up is given. At this point the existing backend
+information for that domain is displayed in a _read-write_ mode, i.e. in a form
+similar to that used in the "Setup a Domain" step for the same backend. From
+here the user can edit any information they want, and the flow continues exactly
+as if they were in the "Setup a Domain" flow.
+
+## Implementation Notes
+
+The following are general notes and ideas about how this can be implemented.
+They are not meant to be set in stone.
+
+* A backend object can be deterministically hashed to form a unique ID for that
+  backend. The "opaque string" which is added to a user's domain's TXT record is
+  this ID.
+
+* The gateway processes use a shared garage S3 bucket for persistence of
+  domain/backend mappings.
+
+  * Each process periodically re-reads all mapping data from S3 and uses that as
+    its currently active set of mappings. A pubsub system like NATS can be added
+    later to ensure that re-syncing happens at the same time.
+
+  * Each file in the bucket corresponds to a domain name which has been
+    successfully setup. The value contains the backend which is set up for the
+    domain.
+
+* When the "Check" button is clicked during the "Setup/Update a Domain" flows,
+  the gateway process which handles the call does the following:
+
+  * Checks that a TXT record with the expected value exists on the domain being
+    added.
+
+  * Checks that a CNAME record with the expected value exists on the domain
+    being added.
+
+  * Performs ACME challenge for SSL cert. It will do this via an
+    [HTTP-01][http01] challenge. The challenge token will be put in the shared
+    garage S3 bucket, so that all gateway processes can potentially serve it
+    when requested.
+
+  * Depending on the backend, it may be required to sync some data into the
+    shared garage S3 bucket. For example, if the git backend is being used, then
+    the tip of the repo+branch will be synced into the S3 bucket.
+
+  * Adds a file to the shared garage S3 bucket corresponding to the new
+    domain/backend mapping.
+
+* Each gateway process is responsible for keeping its own SSL certificates up to
+  date via periodic ACME challenges.
+
+* Backend data in the shared garage S3 bucket will need to be periodically kept
+  up to date. A mechanism needs to be devised to do this reasonably reliably,
+  but without _all_ gateway processes constantly doing it. A communication layer
+  like NATS might come in handy here.
+
+[http01]: https://letsencrypt.org/docs/challenge-types/#http-01-challenge
+
+## Future Work
+
+It should be possible for someone accessing a private domain on the cryptic
+network to access the gateway web panel and use it _without_ requiring a secret
+passphrase. This way community members who have already been vetted (by being
+added to the VPN) can have even less friction.

docs/roadmap.md

@@ -39,16 +39,7 @@ auto-detect the installed operating system and install the daemon automatically.
 One idea is to have every `cryptic-net daemon` run a webserver as one of its
 sub-processes. This server could serve multiple functions:
 
-- Possible public gateway, if the host is configured to be public, into internal
+- [Gateway service](./dev/gateway.md).
-  cryptic-net services. This will require some sort of service discovery
-  mechanism that other hosts in the network can easily hook into via their
-  `daemon.yml`s. Ideally this mechanism would involve little beyond updating
-  entries in garage, which the public gateways then pick up on.
-
-  One wrinkle here will be TLS certificates. Ideally internal hosts could host
-  websites publicly via the gateways on their network, using arbitrary domains
-  that the users own. The gateways will need some way of obtaining certs for
-  these domains automatically (or as automatically as possible).
 
 - Local interface for the `cryptic-net daemon` process. For example, status and
   connectivity information for the local host could be provided via a simple web