mediocregopher/mediocre-go-lib
1
0
Fork 0
Code Releases Activity

WIP mrpc update proto.md to use jstream

Browse Source
This commit is contained in:
Brian Picciano 2018-04-12 11:35:04 +00:00
parent fc853e739c
commit 5626f9171b
1 changed files with 71 additions and 177 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

248
mrpc/proto.md
View File

@ -1,208 +1,102 @@
# Protocol
The mediocre-rpc protocol is designed to operate over nearly any network
protocol. It exists entirely in the data layer, and only relies on being carried
by a protocol which supports a request/response paradaigm, and garauntees
order/reception. This includes HTTP and raw TCP sockets, and likely many others.
The mediocre-rpc protocol is an RPC protocol with support for streaming
arbitrary numbers of both request and response objects, byte blobs of unknown
length, and managing request/response debug data.
RPC calls and responses are JSON encoded objects. The protocol supports single
object calls/responses, as well as streaming multiple objects for either. There
is also support for streaming raw byte blobs.
The protocol itself is carried via the jstream protocol, which is specified and
implemented in this repo.
## General
A couple common rules which apply across all subsequent documentation for this
spec:
Common rules and terminology which apply across all subsequent documentation for
this spec:
* An "RPC call", or just "call", is composed of two events: a "request" and a
"response".
* The entity which initiates the call by sending a request is the "caller".
* The entity which serves the call by responding to a request is the "server".
* In all JSON object specs, a field which is not required can be omitted
entirely, and its value is assumed to be the expected type's empty value (e.g.
`""` for strings, `0` for numbers, `{}` for objects, `null` for any-JSON
types).
* For multiple JSON objects appearing back-to-back on the wire there may or may
not be white-space separating them.
## Debug
## Calls
Many components of this RPC protocol carry a `debug` field, whose value may be
some arbitrary set of data as desired by the user. The use and purpose of the
`debug` field will be different for everyone, but some example use-cases would
be a unique ID useful for tracing, metadata useful for logging in case of an
error, and request/response timings from both the caller and server sides
(useful for determining RTT).
### Single call
When determining if some piece of data should be considered debug data or part
of the request/response proper, imagine that some piece of code was completely
removing the `debug` field in random places at random times. Your application
should run _identically_ in that scenario as in real-life.
A single call looks like the following:
In other words: if some field in `debug` effects the behavior of a call directly
then it should not be carried via `debug`. This could mean duplicating data
between `debug` and the request/response proper, e.g. the IP address of the
caller.
{
"method":"methodName (required)",
"args":"anyJSON",
"debug":{ "foo":"anyJSON" }
}
## Call request
`method` is required and indicates the name of the method being called. Its
value has no restrictions on the protocol level, it's up to the caller and
handler to know ahead of time which methods are available.
A call request is defined as being three jstream elements read off the pipe by
the server. Once all three elements have been read the request is considered to
be completely consumed and the pipe may be used for a new request.
`args` are the arguments to the method, and can be anything.
The three elements of the request stream are specified as follows:
`debug` is metadata about the call which can be made accessible to the handler
for purposes of tracing, logging, etc... `debug` must be a JSON object. A good
rule to know if something should be debug or part of the arguments is that if
any `debug` field were to be deleted on any request the response wouldn't
change. If the response were to change then that field should be in `args`.
* The first element, the head, is a JSON value with an object containing a
`name` field, which identifies the call being made, and optionally a `debug`
field.
### Stream call
* The second element is the argument to the call. This may be a JSON value, a
byte blob, or an embedded stream containing even more elements, depending on
the call. It's up to the caller and server to coordinate beforehand what to
expect here.
A stream call consists of a leading JSON object which looks like a single call,
a body of zero or more JSON objects containing single elements of a stream, and
a tail which indicates the end of the stream.
* The third element, the tail, is a JSON value with an object optionally
containing a `debug` field.
A stream method call whose body is all JSON strings might look this (newlines
added for clarity, they are optional in the protocol):
## Call response
{
"method":"methodName (required)",
"args":"anyJSON",
"debug":{ "foo":"anyJSON" },
"streamStart":true,
"streamLen":3
}
A call response almost the same as the call request. The only difference is the
lack of `name` field in the head, and the addition of the `err` field in the
tail.
{ "el":"anyJSON" }
A call response is defined as being three jstream elements read off the pipe by
the caller. Once all three elements have been read the response is considered to
be completely consumed and the pipe may be used for a new request.
{ "elBytesFrame":"RANDFRAME" }
RANDFRAMEsome very cool arbitrary bytes
which may contain whitespace or anythingRANDFRAME
The three elements of the response stream are specified as follows:
{
"elBytesFrame":"RANDFRAME",
"elBytesLen":10
}
RANDFRAMEsome-bytesRANDFRAME
* The first element, the head, is a JSON value with an object containing
optionally containing a `debug` field.
{
"args":"anyJSON",
"debug":{ "foo":"anyJSON" },
"streamEnd":true
}
* The second element is the response from the call. This may be a JSON value, a
byte blob, or an embedded stream containing even more elements, depending on
the call. It's up to the caller and server to coordinate beforehand what to
expect here.
The head is the first JSON value in the stream. It looks and operates much like a
single call's head, but with the added `streamStart` field. The `streamLen`
field is optional, but may be used if the number of elements in the stream is
known beforehand.
* The third element, the tail, is a JSON value with an object optionally
containing an `err` field, and optionally containing `debug` field. The value
of `err` may be any JSON value which is meaningful to the caller and server.
Each element in the stream is a JSON object, and can be either a single JSON
value or a blob of arbitrary bytes.
## Pipelining
If the element is a JSON value the JSON object will have an `el` field whose
value is the JSON value.
The protocol allows for the server to begin sending back a response, and even to
send back a complete response, _as soon as_ it receives the request head. In
effect this means that the server can be sending back response data while the
caller is still sending request data.
If the element is a blob of arbitrary bytes the JSON object will have an
`elBytesFrame` field whose value is a set of random alphanumeric characters
which will be used to prefix and suffix the bytes. Following the JSON object may
be some whitespace, and then the arbitrary bytes with the `elBytesFrame`
prefixed and suffixed immediately around it. The JSON object for the element may
optionally have the `elBytesLen` field if the length of the blob is known
beforehand. The arbitrary bytes _must_ be prefixed/suffixed by the frame even if
`elBytesLen` is given.
TODO elBytesFrame size recommendation
The tail is the last JSON value in the stream and indicates the stream
has ended. It is required even if `streamLen` was given in the head. The tail
can also have its own `args` and `debug`, independent of the head's but subject
to the same rules.
## Response
### Single response
A single response looks like the following:
{
"res":"anyJSON",
"err":{
"msg":"some presumably helpful string",
"meta":"anyJSON"
},
"debug":{ "foo":"anyJSON" }
}
`res` is the response from the call, and can be anything.
`err` is mutually exclusive with `res` (if one is set the other should be `null`
or unset). The `msg` is be any arbitrary string. `meta` is optional and contains
any extra information which might be actionable by the client receiving the
error.
`debug` is metadata about the response which can be made accessible to the
caller for purposes of tracing, logging, etc... `debug` must be a JSON object. A
good rule to know if something should be debug or part of the results is that
if any `debug` field were to be deleted on any response the client would act in
the same way. If the client's subsequent actions were to change then that field
should be in `res`.
### Stream response
A stream response looks and acts very similar to a stream call, and the
documentation for the stream call can be referenced for details on the
following:
{
"res":"anyJSON",
"debug":{ "foo":"anyJSON" },
"streamStart":true,
"streamLen":3
}
{ "el":"anyJSON" }
{ "elBytesFrame":"RANDFRAME" }
RANDFRAMEsome very cool arbitrary bytes
which may contain whitespace or anythingRANDFRAME
{
"elBytesFrame":"RANDFRAME",
"elBytesLen":10
}
RANDFRAMEsome-bytesRANDFRAME
{
"args":"anyJSON",
"debug":{ "foo":"anyJSON" },
"streamEnd":true
}
The one note specific to stream responses is that a stream response _cannot_
contain an `err` field.
## Network details
In the case of a stream call the handler may send back a response before the
stream has been completely sent. In this case the client should ignore the fact
that the stream wasn't completely sent and return the response as it receives
it.
In the case of a stream call and a stream response both the client and handler
can be sending their respective streams to the other simultaneously. As in the
previous case, if the handler ends the stream with the tail object the caller
should treat the call as successfully completed.
The general rule is: If the client receives either a single response or the tail
of a stream response then the call should be treated as completed.
## TODO
* In the case of pipelining then short-circuiting ought to be implemented for
the case of the stream response having been completed but the stream call
hasn't. In that case the stream call should short-circuit the stream so the
connection can be reused asap
* Figure out the naming
* Len is weird, cause if a stream is short-circuited then the length will have
just been a hint
* Maybe just make bytes a thing which can be bolted onto any of the JSON
objects, either the single, head, elements, or tail.
* Maybe instead of merely defining a single-level stream, do something where all
elements in the stream are the same (either a json value or bytes), and
additionally each can declare that it is the beginning of a stream of further
objects. That gets weird with both `method` and `err`, but it's kinda nice in
that it would potentially simplify the code interface greatly.
Once the server has sent the response tail it can assume the call has completed
successfully and ignore all subsequent request data (though it must still fully
read the three request elements off the pipe in order to use it again).
Likewise, once a caller receives the response tail it can cancel whatever it's
doing, finish sending the request argument and tail as soon as possible, and
assume the call has been completed.