mediocregopher/mediocre-go-lib
1
0
Fork 0
Code Releases Activity

WIP mrpc update proto.md to use jstream

Browse Source
This commit is contained in:
Brian Picciano 2018-04-12 11:35:04 +00:00
parent fc853e739c
commit 5626f9171b
1 changed files with 71 additions and 177 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

248
mrpc/proto.md
View File

@ -1,208 +1,102 @@
# Protocol # Protocol
The mediocre-rpc protocol is designed to operate over nearly any network The mediocre-rpc protocol is an RPC protocol with support for streaming
protocol. It exists entirely in the data layer, and only relies on being carried arbitrary numbers of both request and response objects, byte blobs of unknown
by a protocol which supports a request/response paradaigm, and garauntees length, and managing request/response debug data.
order/reception. This includes HTTP and raw TCP sockets, and likely many others.
RPC calls and responses are JSON encoded objects. The protocol supports single The protocol itself is carried via the jstream protocol, which is specified and
object calls/responses, as well as streaming multiple objects for either. There implemented in this repo.
is also support for streaming raw byte blobs.
## General ## General
A couple common rules which apply across all subsequent documentation for this Common rules and terminology which apply across all subsequent documentation for
spec: this spec:
* An "RPC call", or just "call", is composed of two events: a "request" and a
"response".
* The entity which initiates the call by sending a request is the "caller".
* The entity which serves the call by responding to a request is the "server".
* In all JSON object specs, a field which is not required can be omitted * In all JSON object specs, a field which is not required can be omitted
entirely, and its value is assumed to be the expected type's empty value (e.g. entirely, and its value is assumed to be the expected type's empty value (e.g.
`""` for strings, `0` for numbers, `{}` for objects, `null` for any-JSON `""` for strings, `0` for numbers, `{}` for objects, `null` for any-JSON
types). types).
* For multiple JSON objects appearing back-to-back on the wire there may or may ## Debug
not be white-space separating them.
## Calls Many components of this RPC protocol carry a `debug` field, whose value may be
some arbitrary set of data as desired by the user. The use and purpose of the
`debug` field will be different for everyone, but some example use-cases would
be a unique ID useful for tracing, metadata useful for logging in case of an
error, and request/response timings from both the caller and server sides
(useful for determining RTT).
### Single call When determining if some piece of data should be considered debug data or part
of the request/response proper, imagine that some piece of code was completely
removing the `debug` field in random places at random times. Your application
should run _identically_ in that scenario as in real-life.
A single call looks like the following: In other words: if some field in `debug` effects the behavior of a call directly
then it should not be carried via `debug`. This could mean duplicating data
between `debug` and the request/response proper, e.g. the IP address of the
caller.
{ ## Call request
"method":"methodName (required)",
"args":"anyJSON",
"debug":{ "foo":"anyJSON" }
}
`method` is required and indicates the name of the method being called. Its A call request is defined as being three jstream elements read off the pipe by
value has no restrictions on the protocol level, it's up to the caller and the server. Once all three elements have been read the request is considered to
handler to know ahead of time which methods are available. be completely consumed and the pipe may be used for a new request.
`args` are the arguments to the method, and can be anything. The three elements of the request stream are specified as follows:
`debug` is metadata about the call which can be made accessible to the handler * The first element, the head, is a JSON value with an object containing a
for purposes of tracing, logging, etc... `debug` must be a JSON object. A good `name` field, which identifies the call being made, and optionally a `debug`
rule to know if something should be debug or part of the arguments is that if field.
any `debug` field were to be deleted on any request the response wouldn't
change. If the response were to change then that field should be in `args`.
### Stream call * The second element is the argument to the call. This may be a JSON value, a
byte blob, or an embedded stream containing even more elements, depending on
the call. It's up to the caller and server to coordinate beforehand what to
expect here.
A stream call consists of a leading JSON object which looks like a single call, * The third element, the tail, is a JSON value with an object optionally
a body of zero or more JSON objects containing single elements of a stream, and containing a `debug` field.
a tail which indicates the end of the stream.
A stream method call whose body is all JSON strings might look this (newlines ## Call response
added for clarity, they are optional in the protocol):
{ A call response almost the same as the call request. The only difference is the
"method":"methodName (required)", lack of `name` field in the head, and the addition of the `err` field in the
"args":"anyJSON", tail.
"debug":{ "foo":"anyJSON" },
"streamStart":true,
"streamLen":3
}
{ "el":"anyJSON" } A call response is defined as being three jstream elements read off the pipe by
the caller. Once all three elements have been read the response is considered to
be completely consumed and the pipe may be used for a new request.
{ "elBytesFrame":"RANDFRAME" } The three elements of the response stream are specified as follows:
RANDFRAMEsome very cool arbitrary bytes
which may contain whitespace or anythingRANDFRAME
{ * The first element, the head, is a JSON value with an object containing
"elBytesFrame":"RANDFRAME", optionally containing a `debug` field.
"elBytesLen":10
}
RANDFRAMEsome-bytesRANDFRAME
{ * The second element is the response from the call. This may be a JSON value, a
"args":"anyJSON", byte blob, or an embedded stream containing even more elements, depending on
"debug":{ "foo":"anyJSON" }, the call. It's up to the caller and server to coordinate beforehand what to
"streamEnd":true expect here.
}
The head is the first JSON value in the stream. It looks and operates much like a * The third element, the tail, is a JSON value with an object optionally
single call's head, but with the added `streamStart` field. The `streamLen` containing an `err` field, and optionally containing `debug` field. The value
field is optional, but may be used if the number of elements in the stream is of `err` may be any JSON value which is meaningful to the caller and server.
known beforehand.
Each element in the stream is a JSON object, and can be either a single JSON ## Pipelining
value or a blob of arbitrary bytes.
If the element is a JSON value the JSON object will have an `el` field whose The protocol allows for the server to begin sending back a response, and even to
value is the JSON value. send back a complete response, _as soon as_ it receives the request head. In
effect this means that the server can be sending back response data while the
caller is still sending request data.
If the element is a blob of arbitrary bytes the JSON object will have an Once the server has sent the response tail it can assume the call has completed
`elBytesFrame` field whose value is a set of random alphanumeric characters successfully and ignore all subsequent request data (though it must still fully
which will be used to prefix and suffix the bytes. Following the JSON object may read the three request elements off the pipe in order to use it again).
be some whitespace, and then the arbitrary bytes with the `elBytesFrame` Likewise, once a caller receives the response tail it can cancel whatever it's
prefixed and suffixed immediately around it. The JSON object for the element may doing, finish sending the request argument and tail as soon as possible, and
optionally have the `elBytesLen` field if the length of the blob is known assume the call has been completed.
beforehand. The arbitrary bytes _must_ be prefixed/suffixed by the frame even if
`elBytesLen` is given.
TODO elBytesFrame size recommendation
The tail is the last JSON value in the stream and indicates the stream
has ended. It is required even if `streamLen` was given in the head. The tail
can also have its own `args` and `debug`, independent of the head's but subject
to the same rules.
## Response
### Single response
A single response looks like the following:
{
"res":"anyJSON",
"err":{
"msg":"some presumably helpful string",
"meta":"anyJSON"
},
"debug":{ "foo":"anyJSON" }
}
`res` is the response from the call, and can be anything.
`err` is mutually exclusive with `res` (if one is set the other should be `null`
or unset). The `msg` is be any arbitrary string. `meta` is optional and contains
any extra information which might be actionable by the client receiving the
error.
`debug` is metadata about the response which can be made accessible to the
caller for purposes of tracing, logging, etc... `debug` must be a JSON object. A
good rule to know if something should be debug or part of the results is that
if any `debug` field were to be deleted on any response the client would act in
the same way. If the client's subsequent actions were to change then that field
should be in `res`.
### Stream response
A stream response looks and acts very similar to a stream call, and the
documentation for the stream call can be referenced for details on the
following:
{
"res":"anyJSON",
"debug":{ "foo":"anyJSON" },
"streamStart":true,
"streamLen":3
}
{ "el":"anyJSON" }
{ "elBytesFrame":"RANDFRAME" }
RANDFRAMEsome very cool arbitrary bytes
which may contain whitespace or anythingRANDFRAME
{
"elBytesFrame":"RANDFRAME",
"elBytesLen":10
}
RANDFRAMEsome-bytesRANDFRAME
{
"args":"anyJSON",
"debug":{ "foo":"anyJSON" },
"streamEnd":true
}
The one note specific to stream responses is that a stream response _cannot_
contain an `err` field.
## Network details
In the case of a stream call the handler may send back a response before the
stream has been completely sent. In this case the client should ignore the fact
that the stream wasn't completely sent and return the response as it receives
it.
In the case of a stream call and a stream response both the client and handler
can be sending their respective streams to the other simultaneously. As in the
previous case, if the handler ends the stream with the tail object the caller
should treat the call as successfully completed.
The general rule is: If the client receives either a single response or the tail
of a stream response then the call should be treated as completed.
## TODO
* In the case of pipelining then short-circuiting ought to be implemented for
the case of the stream response having been completed but the stream call
hasn't. In that case the stream call should short-circuit the stream so the
connection can be reused asap
* Figure out the naming
* Len is weird, cause if a stream is short-circuited then the length will have
just been a hint
* Maybe just make bytes a thing which can be bolted onto any of the JSON
objects, either the single, head, elements, or tail.
* Maybe instead of merely defining a single-level stream, do something where all
elements in the stream are the same (either a json value or bytes), and
additionally each can declare that it is the beginning of a stream of further
objects. That gets weird with both `method` and `err`, but it's kinda nice in
that it would potentially simplify the code interface greatly.