diff --git a/INTRODUCTION.md b/INTRODUCTION.md deleted file mode 100644 index f5ee4ba..0000000 --- a/INTRODUCTION.md +++ /dev/null @@ -1,230 +0,0 @@ -# dehub - -**Embed project coordination into your git history.** - -## Gettin Started - -``` -git clone https://dehub.mediocregopher.com/dehub.git -``` - -and check out the project! dehub is still very very alpha, but it will be -"eating its own dogfood" from the start. - -Check out the `cmd/http-server` directory if you'd like to host your own. - -## Motivation - -Any active git project has a set of requirements which are not met by the git -protocol directly, for example: - -* Authenticating committers -* Some kind of ticket system for bugs and proposals -* Change reviews -* Signoff of changes by one or more maintainers -* Release management (git tags are mutable and therefore generally ineffective) - -To solve these requirements developers generally turn to centralized services -like GitHub or Bitbucket, or self-hosted server solutions like Gitlab or gogs. -These platforms become a point of hindrance; their sheer size makes developers -dependent on the platform developers to implement features and fix bugs, as well -as making developers dependent on non-trivial amounts of devops (whether -provided by the service, or self-hosted) in order to function. - -## Enter dehub - -By embedding project meta-information into git messages, as yaml encoded data -structures, dehub is able to incept all the features generally provided by git -platforms into the git history itself, including dehub's own configuration. - -By doing this, the server-side git component can be reduced to a mere -pre-receive hook (if anything at all). This opens the door for much more -lightweight and flexible hosting of git projects, as well as even more radical -solutions; dehub can enable hosting git projects on completely decentralized -platforms like IPFS. - -### Example - -MyProject wants to ensure that at least 2 of the 3 maintainers sign off on a -commit which changes files before the commit can be placed into the `main` -branch (dehub's equivalent of the `master` branch). MyProject's repo would -contain a `.dehub/config.yml` file with the following access controls set: - -``` -# ... -access_controls: - - action: allow - filters: - - type: branch - pattern: main - - - type: commit_type - commit_type: change - - - type: signature - account_ids: - - alice - - bob - - carol - count: 2 - - - action: deny - filters: - - type: branch - branch: main -``` - -A commit in the `main` branch would have a message with the following form: - -``` -This is the first line of the commit message. It remains human readable - ---- -type: change -message: | - This is the first line of the commit message. It remains human readable - - The rest of the message body is a yaml encoded object. The message field of - that object repeats the first line of the commit message, followed by the - rest of the commit message (if there is one). The first line is duplicated - so that commands like `git log` are more usable, while at the same time - allowing the full commit message to be signed off on. - -# A hash of the diff between the previous commit and this one. -change_hash: ABCDEFGHIJKLMNOPQRSTUVWXYZ - -credentials: - - type: pgp_signature - pub_key_id: 01234 - body: SIGNATUREBODY - account: alice - - - type: pgp_signature - pub_key_id: 56789 - body: SIGNATUREBODY - account: carol -``` - -The `credentials` contains signatures of both the commit message and its -changes, allowing it to be added to the `main`. A simple git hook is all that's -needed to verify commits in `main` when they are pushed or pulled. - -## dehub Thread Branches - -The `main` branch is the project's source-of-truth. Other branches, called -threads, are used to coordinate new changes, and then coalesce those changes -into a commit suitable for `main`. - -### Example - -Alice creates and pushes a thread branch on the git repo called `featureBranch`, -and pushes to it a commit with the following commit message: - -``` -This commit adds some really cool features - ---- -type: change -message: This commit adds some really cool features -change_hash: SOMECHANGEHASH -credentials: - - type: pgp_signature - pub_key_id: 01234 - body: SIGNATUREBODY - account: alice - -# Note that this commit does not have enough credentials to be allowed in the -# main branch. -``` - -Bob sees the new thread branch and looks through it. He pushes the following -commit (with no file changes): - -``` -A small comment - ---- -type: comment -message: | - A small comment - - I think you should change the code at file:line to be more like the code at - otherFile:otherLine - -# Comment credentials sign the comment itself, so you can be sure of its -# authenticity. -credentials: - - type: pgp_signature - pub_key_id: 01234 - body: SIGNATUREBODY - account: bob -``` - -Alice sees Bob's comment, and agrees with his suggestion. She pushes a new -commit to the thread, which contains a slight modification of the original -commit message plus the suggested changes: - -``` -This commit adds some really cool features - ---- -type: change -message: | - This commit adds some really cool features - - The pattern used at file:line was suggested by Bob. Thanks Bob! - -change_hash: NEWCHANGEHASH -credentials: - - type: pgp_signature - pub_key_id: 01234 - body: NEWSIGNATUREBODY - account: alice - -# Note that this commit does not have enough credentials to be allowed in the -# main branch. -``` - -Bob, happy with these changes, pushes a commit to the thread which adds his own -signature for the latest commit message and all file changes in the branch: - -``` -bob's signature for this branch's changes - ---- -type: credential -change_hash: NEWCHANGEHASH -credentials: - - type: pgp_signature - pub_key_id: 56789 - body: SIGNATUREBODY - account: bob -``` - -_Finally_ the thread branch is ready to be coalesced, which is a step anyone -can do once all the required credentials are available. - -To coalesce, the following is done: All file changes in the branch are squashed -into a single change commit, using the latest commit message which was pushed by -Alice. Bob's signature is added to the change commit message as a credential. -The commit can then be pushed to `main` (because it now has two credentials) and -`featureBranch` can be deleted. - -## Pre-emptively Answered Questions - -**How can I trust that the git history I've received is legitimate?** - -Each commit in `main` can have its credentials verified locally. Credentials -are currently provided by pgp signatures, so your trust in the git chain can be -as strong as your trust in those signatures. Support for other kinds of -credentials (e.g. keybase signatures) will increase the number of options for -trust the user has. - -**Why `main`?** - -The primary branch in most git projects is called `master`. It makes sense to -use a different one, `main`, for dehub, since the commits on it will be -following a specific protocol which is not compatible with most `master` -branches. By having a different primary branch convention we can prevent undue -conflict, as well as make it easy to tell at a glance what kind of project is -being worked with. diff --git a/README.md b/README.md new file mode 100644 index 0000000..8db4a11 --- /dev/null +++ b/README.md @@ -0,0 +1,97 @@ +# dehub + +dehub aims to provide all the features of a git hosting platform, but without +the hosting part. These features include: + +**User management** - Authentication that commits come from the user they say +they do, and fine-grained control over which users can do what. + +**Pull requests and issues** - Facilitation of discussion via comment commits, +and fine-grained (down to the file level) sign-off requirements. + +**Tags and releases**\* - Mark releases in the repo itself, and provide +immutable and verifiable git tags so there's never any funny business. + +**Plugins**\*: Extend all aspects of dehub functionality via executables managed +in the repo itself (in the same style as git hooks). + +## Key Concepts + +To implement these features, dehub combines two key concepts: + +First, repo configuration is defined in the repo itself. A file called +`.dehub/config.yml` contains all information related to user accounts, their pgp +keys, branch and file level access controls, and more. Every commit must adhere +to the configuration of its parent in order to be considered _verifiable_. The +configuration file is committed to the repo like any other file would be, and so +is even able to define the access controls on itself. + +Second, the commit message of every dehub commit contains YAML encoded metadata, +which allows dehub to extend git and provide multiple commit types, each with +its own capabilities and restrictions. Some example dehub commit types are +`change` commits, `comment` commits, and `credential` commits. + +## Infrastructure (or lack thereof) + +Because a dehub project is entirely housed within a traditional git project, +which is merely a collection of files, any existing git or network filesystem +infrastructure can be used to host any dehub project: + +* The most barebones [git + daemon](https://git-scm.com/book/en/v2/Git-on-the-Server-Git-Daemon) server + (with a simple pre-receive hook set up). + +* A remote SSH endpoint. + +* A mailing list (aka the old school way). + +* Network file syncing utilities such as dropbox, + [syncthing](https://github.com/syncthing/syncthing), or + [NFS](https://en.wikipedia.org/wiki/Network_File_System). + +* Existing git project hosts like GitHub, Bitbucket, or Keybase. + +* Decentralized filesystems such as IPFS\*. + +_\* Planned feature, but not yet implemented._ + +# Getting Started + +The dehub project itself can be found by cloning +`https://dehub.dev/src/dehub.git`. + +Installation of the dehub tool is currently done via the `go get` command: + +``` +go get dehub.dev/src/dehub.git/cmd/dehub +``` + +This will install the binary to your `$GOBIN` path, which you'll want to put in +your `$PATH`. Run `go env` if you're not sure where your `$GOBIN` is. + +Once installed, running `dehub -h` should show you the help output of the +command. You can continue on to the tutorials if you're not sure where to go +from here. + +## Tutorials + +The following tutorials will guide you through the basic usage of dehub. As +dehub is still very much in development a high level of git and PGP profiency is +still required in order to use dehub effectively. + +TODO + +## Documentation + +The [SPEC](/SPEC.html) is the best place to see every possible nitty-gritty +detail of how dehub works. It attempts to be both human-readable and exhaustive +in its coverage. + +## Other links + +[ROADMAP](/ROADMAP.html) documents upcoming features and other work required on +the project. If you're looking to contribute, this is a great place to start. + +[dehub-remote](/cmd/dehub-remote) TODO + +[git-http-server](/cmd/git-http-server) TODO diff --git a/ROADMAP.md b/ROADMAP.md index d801ac3..01b1637 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -4,12 +4,29 @@ This document describes currently planned features and events related to the dehub project. It's intention is to help prioritize work. There are no dates set, only a sequence of milestones and the requirements to hit them. -## Milestone: Be able to add other developers to the project +## Milestone: Publicize project + +Must be able to feel good about showing the project publicly, as well as be able +to accept help from people asking to help. + +* Restrict new branches so that they must be ancestors of main. +* Fast-forward perms on branches (so they can be deleted) +* `init` command. +* Ammending commits. +* Figure out commit range syntax, use that everywhere. +* Support short hash names +* Ability to specify a pgp key manually, even if it's not in the project. +* Ability to require _any_ signature on a commit, even if it's not in the + config. +* Create a branch which is just a public "welcome thread", which can be part of + the tutorials. +* Tutorials +* Update SPEC; it's sloppy, out-of-date, and incomplete. +* Maybe move external host? -* Coalesce command -* Fast-forward perms on branches -* Authorship in commit messages -* README with a "Getting Started" section +## Milestone: IPFS support + +* Big ol' question mark on this one. ## Milestone: Versions @@ -17,50 +34,56 @@ set, only a sequence of milestones and the requirements to hit them. * Add dehub version to the SPEC, make binary aware of it * Figure out a release system? -## Milestone: Enough polish to show off the project +## Milestone: Minimal plugin support + +* SPEC and implement. Things which should be pluggable, initially: + * Conditions + * Signifiers + * Filters + * Commits??? + +## Milestone: Minimal notifications support + +* Some way to store notification settings locally, and run a command which shows + a sequence of events since the last time you ran it. + * The command should keep a history of all of its outputs, and allow the + user to see that history (in case they run the command, then clear the + output by accident). + * The user should be able to specifically get notifications on threads + they're a part of, threads by branch name pattern, files by path pattern, + and keywords in commit messages. + +# Misc Polish + +These tasks aren't necessarily scheduled for any particular milestone, but they +are things that could use doing anyway. * Maybe coalesce the `accessctl`, `fs`, and `sigcred` packages back into the root "dehub" package. -* Polish all error messages +* Polish all error messages. A good error message has the following qualities: + * If wrapping an error which was returned from a sub-call: + * Uses `fmt.Errorf` with the `%w` format directive at the end. + * Phrased as if the sentence starts with the word "while", e.g. "opening + file: %w". + * Only includes information the caller of that function/method couldn't + already know. * Polish commands - - New flag system, some kind of interactivity support (e.g. user doesn't + * New flag system, some kind of interactivity support (e.g. user doesn't specify required argument, give them a prompt on the CLI to input it rather than an error). This is partially done, in that a new flag system has been started. Needs further work. - - Review flags, probably make some of them into positional arguments, - document everything better. - - - Possibly save state locally in order to speed things along, such as - "account id" which probably isn't going to change often for a user. + * Review flags: + * probably make some of them into positional arguments + * add flag shortcuts + * document everything better. -* HTTP docs - - Some kind of addition to the docker server to auto-serve md files. - - Reorganize all the md files under a `docs` directory, except the README. - - Make sure linking between documents works. + * POSIX compatible-ish flags? -* More/better tests? - -## Milestone: IPFS support - -* Big ol' question mark on this one. - -## Milestone: Minimal plugin support - -* SPEC and implement. Things which should be pluggable, initially: - - Conditions - - Signifiers - -## Milestone: Minimal notifications support - -* Some way to store notification settings locally, and run a command which shows - a sequence of events since the last time you ran it. - - The command should keep a history of all of its outputs, and allow the - user to see that history (in case they run the command, then clear the - output by accident). - - The user should be able to specifically get notifications on threads - they're a part of, threads by branch name pattern, files by path - pattern, and keywords in commit messages. + * Possibly save state locally in order to speed things along, such as + "account id" which probably isn't going to change often for a user. +* More/better tests + * Commits need much better test coverage. diff --git a/SPEC.md b/SPEC.md index 09ef7b2..bc1a441 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,9 +1,15 @@ -# .dehub +# SPEC + +This document attempts to describe, at minimum, every aspect of the dehub +protocol which would be required to know in order to create an alternate +implementation of dehub. It is currently sloppily written and incomplete. + +## .dehub The `.dehub` directory contains all meta information related to decentralized repository management and access control. -## config.yml +### config.yml The `.dehub/config.yml` file takes the following structure: @@ -60,7 +66,7 @@ access_controls: - action: deny ``` -# Change Hash +## Change Hash When a change commit (see Commits section) is being signed by a signifier there is an expected data format for the data to be signed. The format is a SHA-256 @@ -86,12 +92,12 @@ hash of the following pieces of data concatenated together: The raw output from the SHA-256 is then prepended with a `0` byte (for forward compatibility). The result is the raw change hash. -# Comment Message Hash +## Comment Message Hash When a comment commit (see Commits section) is being signed by the signifier of the author there is an expected data format for the data to be signed, very similar to how change hashes are signed. The format is a SHA-256 hash of the -following pieces of data communicated together: +following pieces of data concatenated together: * A uvarint indicating the number of bytes in the comment message. * The message. @@ -99,7 +105,7 @@ following pieces of data communicated together: The raw output from the SHA-256 is then prepended with a `0` byte (for forward compatibility). The result is the raw comment hash. -# Credentials +## Credentials All file changes need to have some kind of credential to be accepted into the `main` branch (see Main Branch section). Each credential is encoded as a yaml @@ -108,7 +114,7 @@ object with a `type` field. All credentials contain enough information to correspond them to a specific signifier in the `config.yml`, so as to be able to verify them. -## PGP Signature Credential +### PGP Signature Credential Currently there is only a single credential type, the `pgp_signature`, which signs a raw change hash (which is communicated out-of-band of the object): @@ -120,7 +126,7 @@ pub_key_id: XXX body: "base-64 signature body" ``` -# Commits +## Commits All commit messages in dehub repositories are expected to follow the following template (newlines included, yaml comments start with `#` and are only for @@ -138,7 +144,7 @@ fieldA: valueA fieldB: valueB ``` -## Change Commits +### Change Commits Commits of type `change` correspond to the standard git commit; they encompass a set of file changes as well as a message describing the changes which occurred. @@ -180,7 +186,7 @@ credentials: body: "base-64 signature body" ``` -## Credential Commits +### Credential Commits Commits of type `credential` contain one or more credentials for some hash (presumably a change hash, but in the future there may be other types). The @@ -202,7 +208,7 @@ credentials: body: "base-64 signature body" ``` -## Comment Commits +### Comment Commits Commits of type `comment` contain a message for others to read. The commit message head is not spec'd, but should be a human-readable description of "who @@ -220,7 +226,7 @@ type: comment # forwards compatibility. See the Comment Message Hash section. message_hash: XXX message: > - Heay all, how's it going? + Hey all, how's it going? Just wanted to pop by and say howdy. @@ -233,39 +239,7 @@ credentials: body: "base-64 signature body" ``` -# Branches - -dehub branches correspond 1-to-1 with branches in the underlying git repo. All -commits in a dehub branch should contain an encoded message as specified in the -Commits section of this document, and possibly file changes as appropriate. - -## Main Branch - -The "primary" branch of a dehub repo is the `main` branch. All new commits -being appended to the HEAD of the `main` branch are subject to the following -requirements: - -* Must be `change` commits. - -* Must conform to all requirements defined by the `access_controls` section of - the `config.yml`, as found in the current HEAD. If the commit is the initial - commit of the branch then it instead uses the `config.yml` found in itself. - -* Must be a "fast-forward" commit (this may be amended later, but at present it - simplifies implementation). - - -## Thread Branches - -Branches which are not the `main` branch are referred to as "threads", and have -much less stringent requirements than the `main` branch: - -* They can contain commits of any type, as long as the commits come from those - with an account defined in the `config.yml`. - -* `change` commits are not subject `access_controls` requirements. - -# TODO +## TODO -* access control patterns related to who may push to MR branches, and what types - of commits they can push. +* Access controls +* Update credential commit section