micropelago/garage
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
92 Commits 25 Branches 62 Tags 26 MiB
4ef84a0558
Commit Graph

32 Commits

Author SHA1 Message Date
Alex Auvolat
c9c6b0dbd4 Reorganize code 2020-04-23 17:05:46 +00:00
Alex Auvolat
2fe82be3bc RPC to ourself do not pass through serialization + HTTPS 2020-04-23 14:40:59 +00:00
Alex Auvolat
e8214cb180 Better concurrency: 
Use Notify instead of stupid sleep in background worker
Use Semaphore to limit concurrent requests in rpc_client
Make more background tasks cancellable
 2020-04-22 16:51:52 +00:00
Alex Auvolat
cc4f2f1cfb Pretty logging 2020-04-21 12:54:55 +00:00
Alex Auvolat
ea75564851 More aggressive sync timings & improve other stuff 2020-04-19 17:59:59 +00:00
Alex Auvolat
a6129d8626 Begin implement bucket management & admin commands 2020-04-19 17:15:48 +02:00
Alex Auvolat
7131553c53 Refactor sharding logic; coming next: full replication with epidemic dissemination 2020-04-19 13:22:28 +02:00
Alex Auvolat
4ba54ccfca Reorder imports. 
Trying to separate:
1. Stuff for handling the swarm of nodes and generic table data replication
2. Stuff for the object store core application: metadata tables and block management
3. Stuff for the S3 API
 2020-04-18 19:39:57 +02:00
Alex Auvolat
bd1618e78e Remove proto.rs & move some definitions out of data.rs 2020-04-18 19:30:05 +02:00
Alex Auvolat
f41583e1b7 Massive RPC refactoring 2020-04-18 19:21:34 +02:00
Alex Auvolat
69f1d8fef2 WIP 
TODOs:
- ensure sync goes both way
- finish sending blocks to other nodes when they need them before deleting
 2020-04-17 17:09:57 +02:00
Alex Auvolat
e41ce4d815 Implement getting missing blocks when RC increases 
Issue: RC increases also when the block ref entry is first put by the actual client.
At that point the client is probably already sending us the block content,
so we don't need to do a get...
We should add a delay before the task is added or find something to do.
 2020-04-17 15:40:13 +02:00
Alex Auvolat
e8d750175d Implement ring comparison algorithm 2020-04-16 17:04:28 +02:00
Alex Auvolat
f01c1e71b5 Begin work on sync... 2020-04-16 14:50:49 +02:00
Alex Auvolat
2bea76ce16 Small refactorings 2020-04-12 22:24:53 +02:00
Alex Auvolat
d1e8f78b2c Trying to do TLS 2020-04-12 15:51:19 +02:00
Alex Auvolat
5967c5a5af Refactor a bit 2020-04-12 13:03:55 +02:00
Alex Auvolat
9c931f5eda Keep network status & ring in a tokio::sync::watch 
advantages
- reads don't prevent preparing writes
- can be followed from other parts of the system by cloning the receiver
 2020-04-11 23:53:32 +02:00
Alex Auvolat
5dd59e437d Local refcounting of blocks 2020-04-11 23:00:26 +02:00
Alex Auvolat
dcf58499a4 table::insert_many, version_table::updated 2020-04-11 19:43:29 +02:00
Alex Auvolat
53289b69e5 Background task runner that replaces tokio::spawn 2020-04-11 18:51:11 +02:00
Alex Auvolat
cf8fd948fc Add block ref table 2020-04-10 23:11:52 +02:00
Alex Auvolat
3477864142 Fix the Sync issue. Details: 
So the HTTP client future of Hyper is not Sync, thus the stream
that read blocks wasn't either. However Hyper's default Body type
requires a stream to be Sync for wrap_stream. Solution: reimplement
a custom HTTP body type.
 2020-04-10 22:01:48 +02:00
Alex Auvolat
d66c0d6833 Why is it not Sync?? 2020-04-09 23:45:07 +02:00
Alex Auvolat
1d786c2c66 Something works 2020-04-09 18:43:53 +02:00
Alex Auvolat
101444abb3 Some progress 2020-04-09 17:32:28 +02:00
Alex Auvolat
cc580da0ae Some work 2020-04-08 23:01:49 +02:00
Alex Auvolat
bacc76a057 Some work in actually storing things 2020-04-08 22:00:41 +02:00
Alex Auvolat
d50edcdb4f Basic workflow for client PUT, next is do actual storage 2020-04-08 00:39:07 +02:00
Alex Auvolat
90cdffb425 custom data type for hashes and identifiers 2020-04-07 18:10:20 +02:00
Alex Auvolat
82b7fcd280 Config management & ring 2020-04-07 17:00:48 +02:00
Alex Auvolat
061e676136 Refactor; ability to update network config 2020-04-07 16:26:22 +02:00