424 lines
21 KiB
Plaintext
424 lines
21 KiB
Plaintext
I need to prioritize the future of this project a bit more. I've been thinking
|
|
I'm going to figure this thing out at this level, but I shouldn't even be
|
|
working here without a higher level view.
|
|
|
|
I can't finish this project without financial help. I don't think I can get a v0
|
|
up without financial help. What this means at minimum, no matter what, I'm going
|
|
to have to:
|
|
|
|
- Develop a full concept of the language that can get it to where I want to go
|
|
- Figure out where I want it to go
|
|
- Write the concept into a manifesto of the language
|
|
- Write the concept into a proposal for course of action to take in developing
|
|
the language further
|
|
|
|
I'm unsure about what this language actually is, or is actually going to look
|
|
like, but I'm sure of those things. So those are the lowest hanging fruit, and I
|
|
should start working on them pronto. It's likely I'll need to experiment with
|
|
some ideas which will require coding, and maybe even some big ideas, but those
|
|
should all be done under the auspices of developing the concepts of the
|
|
language, and not the compiler of the language itself.
|
|
|
|
#########
|
|
|
|
Elemental types:
|
|
|
|
* Tuples
|
|
* Arrays
|
|
* Integers
|
|
|
|
#########
|
|
|
|
Been doing thinking and research on ginger's elemental types and what their
|
|
properties should be. Ran into roadblock where I was asking myself these
|
|
questions:
|
|
|
|
* Can I do this without atoms?
|
|
* What are different ways atoms can be encoded?
|
|
* Can I define language types (elementals) without defining an encoding for
|
|
them?
|
|
|
|
I also came up with two new possible types:
|
|
|
|
* Stream, effectively an interface which produces discreet packets (each has a
|
|
length), where the production of one packet indicates the size of the next one
|
|
at the same time.
|
|
* Tagged, sort of like a stream, effectively a type which says "We don't know
|
|
what this will be at compile-time, but we know it will be prefixed with some
|
|
kind of tag indicating its type and size.
|
|
* Maybe only the size is important
|
|
* Maybe precludes user defined types that aren't composites of the
|
|
elementals? Maybe that's ok?
|
|
|
|
Ran into this:
|
|
https://www.ps.uni-saarland.de/~duchier/python/continuations.htm://www.ps.uni-saarland.de/~duchier/python/continuations.html
|
|
https://en.wikipedia.org/wiki/Continuation#First-class_continuations
|
|
|
|
which is interesting. A lot of my problems now are derived from stack-based
|
|
systems and their need for knowing the size input and output data, continuations
|
|
seem to be an alternative system?
|
|
|
|
I found this:
|
|
|
|
http://lambda-the-ultimate.org/node/4512
|
|
|
|
I don't understand any of it, I should definitely learn feather
|
|
|
|
I should finish reading this:
|
|
http://www.blackhat.com/presentations/bh-usa-07/Ferguson/Whitepaper/bh-usa-07-ferguson-WP.pdf
|
|
|
|
#########
|
|
|
|
Ok, so I'm back at this for the first time in a while, and I've got a good thing
|
|
going. The vm package is working out well, Using tuples and atoms as the basis
|
|
of a language is pretty effective (thanks erlang!). I've got basic variable
|
|
assignment working as well. No functions yet. Here's the things I still need to
|
|
figure out or implement:
|
|
|
|
* lang
|
|
* constant size arrays
|
|
* using them for a "do" macro
|
|
* figure out constant, string, int, etc... look at what erlang's actual
|
|
primitive types are for a hint
|
|
* figure out all needed macros for creating and working with lang types
|
|
* vm
|
|
* figure out the differentiation between compiler macros and runtime calls
|
|
* probably separate the two into two separate call systems
|
|
* the current use of varCtx is still pretty ugly, the do macro might help
|
|
clean it up
|
|
* functions
|
|
* are they a primitive? I guess so....
|
|
* declaration and type
|
|
* variable deconstruction
|
|
* scoping/closures
|
|
* compiler macros, need vm's Run to output a lang.Term
|
|
* need to learn about linking
|
|
* figure out how to include llvm library in compiled binary and make it
|
|
callable. runtime macros will come from this
|
|
* linking in of other ginger code? or how to import in general
|
|
* comiler, a general purpose binary for taking ginger code and turning it
|
|
into machine code using the vm package
|
|
* swappable syntax, including syntax-dependent macros
|
|
* close the loop?
|
|
|
|
############
|
|
|
|
I really want contexts to work. They _feel_ right, as far as abstractions go.
|
|
And they're clean, if I can work out the details.
|
|
|
|
Just had a stupid idea, might as well write it down though.
|
|
|
|
Similar to how the DNA and RNA in our cells work, each Context is created with
|
|
some starting set of data on it. This will be the initial protein block. Based
|
|
on the data there some set of Statements (the RNA) will "latch" on and do
|
|
whatever work they're programmed to do. That work could include making new
|
|
Contexts and "releasing" them into the ether, where they would get latched onto
|
|
(or not).
|
|
|
|
There's so many problems with this idea, it's not even a little viable. But here
|
|
goes:
|
|
|
|
* Order of execution becomes super duper fuzzy. It would be really difficult to
|
|
think about how your program is actually going to work.
|
|
|
|
* Having Statement sets just latch onto Contexts is super janky. They would get
|
|
registered I guess, and it would be pretty straightforward to differentiate
|
|
one Context from another, but what about conflicts? If two Statements want to
|
|
latch onto the same Context then what? If we wanted to keep the metaphor one
|
|
would just get randomly chosen over the other, but obviously that's insane.
|
|
|
|
############
|
|
|
|
I explained some of this to ibrahim already, but I might as well get it all
|
|
down, cause I've expanded on it a bit since.
|
|
|
|
Basically, ops (functions) are fucking everything up. The biggest reason for
|
|
this is that they are really really hard to implement without a type annotation
|
|
system. The previous big braindump is about that, but basically I can't figure
|
|
out a way that feels clean and good enough to be called a "solution" to type
|
|
inference. I really don't want to have to add type annotations just to support
|
|
functions, at least not until I explore all of my options.
|
|
|
|
The only other option I've come up with so far is the context thing. It's nice
|
|
because it covers a lot of ground without adding a lot of complexity. Really the
|
|
biggest problem with it is it doesn't allow for creating new things which look
|
|
like operations. Instead, everything is done with the %do operator, which feels
|
|
janky.
|
|
|
|
One solution I just thought of is to get rid of the %do operator and simply make
|
|
it so that a list of Statements can be used as the operator in another
|
|
Statement. This would _probably_ allow for everything that I want to do. One
|
|
outstanding problem I'm facing is figuring out if all Statements should take a
|
|
Context or not.
|
|
|
|
* If they did it would be a lot more explicit what's going on. There wouldn't be
|
|
an ethereal "this context" that would need to be managed and thought about. It
|
|
would also make things like using a set of Statements as an operator a lot
|
|
more straightforward, since without Contexts in the Statement it'll be weird
|
|
to "do" a set of Statements in another Context.
|
|
|
|
* On the other hand, it's quite a bit more boilerplate. For the most part most
|
|
Statements are going to want to be run in "this" context. Also this wouldn't
|
|
really decrease the number of necessary macros, since one would still be
|
|
needed in order to retrieve the "root" Context.
|
|
|
|
* One option would be for a Statement's Context to be optional. I don't really
|
|
like this option, it makes a very fundamental datatype (a Statement) a bit
|
|
fuzzier.
|
|
|
|
* Another thing to think about is that I might just rethink how %bind works so
|
|
that it doesn't operate on an ethereal "this" Context. %ctxbind is one attempt
|
|
at this, but there's probably other ways.
|
|
|
|
* One issue I just thought of with having a set of Statements be used as an
|
|
operator is that the argument to that Statement becomes.... weird. What even
|
|
is it? Something the set of Statements can access somehow? Then we still need
|
|
something like the %in operator.
|
|
|
|
Let me backtrack a bit. What's the actual problem? The actual thing I'm
|
|
struggling with is allowing for code re-use, specifically pure functions. I
|
|
don't think there's any way anyone could argue that pure functions are not an
|
|
effective building block in all of programming, so I think I can make that my
|
|
statement of faith: pure functions are good and worthwhile, impure functions
|
|
are.... fine.
|
|
|
|
Implementing them, however, is quite difficult. Moreso than I thought it would
|
|
be. The big inhibitor is the method by which I actually pass input data into the
|
|
function's body. From an implementation standpoint it's difficult because I
|
|
*need* to know how many bytes on the stack the arguments take up. From a syntax
|
|
standpoint this is difficult without a type annotation system. And from a
|
|
usability standpoint this is difficult because it's a task the programmer has to
|
|
do which doesn't really have to do with the actual purpose or content of the
|
|
function, it's just a book-keeping exercise.
|
|
|
|
So the stack is what's screwing us over here. It's a nice idea, but ultimately
|
|
makes what we're trying to do difficult. I'm not sure if there's ever going to
|
|
be a method of implementing pure functions that doesn't involve argument/return
|
|
value copying though, and therefore which doesn't involve knowing the byte size
|
|
of your arguments ahead of time.
|
|
|
|
It's probably not worth backtracking this much either. For starters, cpus are
|
|
heavily optimized for stack based operations, and much of the way we currently
|
|
think about programming is also based on the stack. It would take a lot of
|
|
backtracking if we ever moved to something else, if there even is anything else
|
|
worth moving to.
|
|
|
|
If that's the case, how is the stack actually used then?
|
|
|
|
* There's a stack pointer which points at an address on the stack, the stack
|
|
being a contiguous range of memory addresses. The place the stack points to is
|
|
the "top" of the stack, all higher addresses are considered unused (no matter
|
|
what's in them). All the values in the stack are available to the currently
|
|
executing code, it simply needs to know either their absolute address or their
|
|
relative position to the stack pointer.
|
|
|
|
* When a function is "called" the arguments to it are copied onto the top of the
|
|
stack, the stack pointer is increased to reflect the new stack height, and the
|
|
function's body is jumped to. Inside the body the function need only pop
|
|
values off the stack as it expects them, as long as it was called properly it
|
|
doesn't matter how or when the function was called. Once it's done operating
|
|
the function ensures all the input values have been popped off the stack, and
|
|
subsequently pushes the return values onto the stack, and jumps back to the
|
|
caller (the return address was also stored on the stack).
|
|
|
|
That's not quite right, but it's close enough for most cases. The more I'm
|
|
reading about this the more I think it's not going to be worth it to backtrack
|
|
passed the stack. There's a lot of compiler and machine specific crap that gets
|
|
involved at that low of a level, and I don't think it's worth getting into it.
|
|
LLVM did all of that for me, I should learn how to make use of that to make what
|
|
I want happen.
|
|
|
|
But what do I actually want? That's the hard part. I guess I've come full
|
|
circle. I pretty much *need* to use llvm functions. But I can't do it without
|
|
declaring the types ahead of time. Ugghh.
|
|
|
|
################################
|
|
|
|
So here's the current problem:
|
|
|
|
I have the concept of a list of statements representing a code block. It's
|
|
possible/probable that more than this will be needed to represent a code block,
|
|
but we'll see.
|
|
|
|
There's two different ways I think it's logical to use a block:
|
|
|
|
* As a way of running statements within a new context which inherits all of its
|
|
bindings from the parent. This would be used for things like if statements and
|
|
loops, and behaves the way a code block behaves in most other languages.
|
|
|
|
* To define a operator body. An operator's body is effectively the same as the
|
|
first use-case, except that it has input/output as well. An operator can be
|
|
bound to an identifier and used in any statement.
|
|
|
|
So the hard part, really, is that second point. I have the first done already.
|
|
The second one isn't too hard to "fake" using our current context system, but it
|
|
can't be made to be used as an operator in a statement. Here's how to fake it
|
|
though:
|
|
|
|
* Define the list of statements
|
|
* Make a new context
|
|
* Bind the "input" bindings into the new context
|
|
* Run %do with that new context and list of statements
|
|
* Pull the "output" bindings out of that new context
|
|
|
|
And that's it. It's a bit complicated but it ultimately works and effectively
|
|
inlines a function call.
|
|
|
|
It's important that this looks like a normal operator call though, because I
|
|
believe in guy steele. Here's the current problems I'm having:
|
|
|
|
* Defining the input/output values is the big one. In the inline method those
|
|
were defined implicitly based on what the statements actually use, and the
|
|
compiler would fail if any were missing or the wrong type. But here we ideally
|
|
want to define an actual llvm function and not inline everytime. So we need to
|
|
somehow "know" what the input/output is, and their types.
|
|
|
|
* The output value isn't actually *that* difficult. We just look at the
|
|
output type of the last statement in the list and use that.
|
|
|
|
* The input is where it gets tricky. One idea would be to use a statement
|
|
with no input as the first statement in the list, and that would define
|
|
the input type. The way macros work this could potentially "just work",
|
|
but it's tricky.
|
|
|
|
* It would also be kind of difficult to make work with operators that take
|
|
in multiple parameters too. For example, `bind A, 1` would be the normal
|
|
syntax for binding, but if we want to bind an input value it gets weirder.
|
|
|
|
* We could use a "future" kind of syntax, like `bind A, _` or something
|
|
like that, but that would requre a new expression type and also just
|
|
be kind of weird.
|
|
|
|
* We could have a single macro which always returns the input, like
|
|
`%in` or something. So the bind would become `bind A, %in` or
|
|
`bind (A, B), %in` if we ever get destructuring. This isn't a terrible
|
|
solution, though a bit unfortunate in that it could get confusing with
|
|
different operators all using the same input variable effectively. It
|
|
also might be a bit difficult to implement, since it kind of forces us
|
|
to only have a single argument to the LLVM function? Hard to say how
|
|
that would work. Possibly all llvm functions could be made to take in
|
|
a struct, but that would be ghetto af. Not doing a struct would take a
|
|
special interaction though.... It might not be possible to do this
|
|
without a struct =/
|
|
|
|
* Somehow allowing to define the context which gets used on each call to the
|
|
operator, instead of always using a blank one, would be nice.
|
|
|
|
* The big part of this problem is actually the syntax for calling the
|
|
operator. It's pretty easy to have this handled within the operator by the
|
|
%thisctx macro. But we want the operator to be callable by the same syntax
|
|
as all other operator calls, and currently that doesn't have any way of
|
|
passing in a new context.
|
|
|
|
* Additionally, if we're implementing the operator as an LLVM function then
|
|
there's not really any way to pass in that context to it without making
|
|
those variables global or something, which is shitty.
|
|
|
|
* So writing all this out it really feels like I'm dealing with two separate
|
|
types that just happen to look similar:
|
|
|
|
* Block: a list of statements which run with a variable context.
|
|
|
|
* Operator: a list of statements which run with a fixed (empty?) context,
|
|
and have input/output.
|
|
|
|
* There's so very nearly a symmetry there. Things that are inconsistent:
|
|
|
|
* A block doesn't have input/output
|
|
|
|
* It sort of does, in the form of the context it's being run with and
|
|
%ctxget, but not an explicit input/output like the operator has.
|
|
|
|
* If this could be reconciled I think this whole shitshow could be made
|
|
to have some consistency.
|
|
|
|
* Using %in this pretty much "just works". But it's still weird. Really
|
|
we'd want to turn the block into a one-off operator everytime we use
|
|
it. This is possible.
|
|
|
|
* An operator's context must be empty
|
|
|
|
* It doesn't *have* to be, defining the ctx which goes with the operator
|
|
could be part of however an operator is created.
|
|
|
|
* So after all of that, I think operators and blocks are kind of the same.
|
|
|
|
* They both use %in to take in input, and both output using the last statement
|
|
in their list of statements.
|
|
|
|
* They both have a context bound to them, operators are fixed but a block
|
|
changes.
|
|
|
|
* An operator is a block with a bound context.
|
|
|
|
##############@@@@@@@@@#$%^&^%$#@#$%^&*
|
|
|
|
* New problem: type inference. LLVM requires that a function's definition have
|
|
the type specified up-front. This kind of blows. Well actually, it blows a lot
|
|
more than kind of. There's two things that need to be infered from a List of
|
|
Statements then: the input type and the output type. There's two approaches
|
|
I've thought of in the current setup.
|
|
|
|
* There's two approaches to determining the type of an operator: analyze the
|
|
code as ginger expressions, or build the actual llvm structures and
|
|
analyze those.
|
|
|
|
* Looking at the ginger expressions is definitely somewhat fuzzy. We can
|
|
look at all the statements and sub-statements until we find an
|
|
instance of %in, then look at what that's in input into. But if it's
|
|
simply binding into an Identifier then we have to find the identifier.
|
|
If it's destructuring then that gets even *more* complicated.
|
|
|
|
* Destructuring is what really makes this approach difficult.
|
|
Presumably there's going to be a function that takes in an
|
|
Identifier (or %in I guess?) and a set of Statements and returns
|
|
the type for that Identifier. If we find that %in is destructured
|
|
into a tuple then we would run that function for each constituent
|
|
Identifier and put it all together. But then this inference
|
|
function is really coupled to %bind, which kind of blows. Also we
|
|
may one day want to support destructuring into non-tuples as well,
|
|
which would make this even harder.
|
|
|
|
* We could make it the job of the macro definition to know its input
|
|
and output types, as well as the types of any bindings it makes.
|
|
That places some burden on user macros in the future, but then
|
|
maybe it can be inferred for user macros? That's a lot of hope. It
|
|
would also mean the macro would need the full set of statements
|
|
that will ever run in the same Context as it, so it can determine
|
|
the types of any bindings it makes.
|
|
|
|
* The second method is to build the statements into LLVM structures and
|
|
then look at those structures. This has the benefit of being
|
|
non-ambiguous once we actually find the answer. LLVM is super strongly
|
|
typed, and re-iterates the types involved for every operation. So if
|
|
the llvm builder builds it then we need only look for the first usage
|
|
of every argument/return and we'll know the types involved.
|
|
|
|
* This requires us to use structs for tuples, and not actually use
|
|
multiple arguments. Otherwise it won't be possible to know the
|
|
difference between a 3 argument function and a 4 argument one
|
|
which doesn't use its 4th argument (which shouldn't really happen,
|
|
but could).
|
|
|
|
* The main hinderence is that the llvm builder is really not
|
|
designed for this sort of thing. We could conceivably create a
|
|
"dummy" function with bogus types and write the body, analyze the
|
|
body, erase the function, and start over with a non-dummy
|
|
function. But it's the "analyze the body" step that's difficult.
|
|
It's difficult to find the types of things without the llvm.Value
|
|
objects in hand, but since building is set up as a recursive
|
|
process that becomes non-trivial. This really feels like the way
|
|
to go though, I think it's actually doable.
|
|
|
|
* This could be something we tack onto llvmVal, and then make
|
|
Build return extra data about what types the Statements it
|
|
handled input and output.
|
|
|
|
* For other setups that would enable this a bit better, the one that keeps
|
|
coming to mind is a more pipeline style system. Things like %bind would need
|
|
to be refactored from something that takes a Tuple to something that only
|
|
takes an Identifier and returns a macro which will bind to that Identifier.
|
|
This doesn't *really* solve the type problem I guess, since whatever is input
|
|
into the Identifier's bind doesn't necessarily have a type attached to it.
|
|
Sooo yeah nvm.
|