micropelago/garage
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,304 Commits 25 Branches 62 Tags 26 MiB
3d477906d4
Commit Graph

193 Commits

Author SHA1 Message Date
Alex Auvolat
1bcd6fabbd
New buckets for 0.6.0: small changes 
- Fix bucket delete

- fix merge of bucket creation date

- Replace deletable with option in aliases
    Rationale: if two aliases point to conflicting bucket, resolving
    by making an arbitrary choice risks making data accessible when it
    shouldn't be. We'd rather resolve to deleting the alias until
    someone puts it back.
 2022-01-04 12:52:47 +01:00
Alex Auvolat
ba7f268b99
Rename and change query filters 2022-01-04 12:52:46 +01:00
Alex Auvolat
e59c23a69d
Refactor logic for setting/unsetting aliases 2022-01-04 12:52:46 +01:00
Alex Auvolat
2140cd7205
Remove website redirects 2022-01-04 12:52:46 +01:00
Alex Auvolat
beeef4758e
Some movement of helper code and refactoring of error handling 2022-01-04 12:52:46 +01:00
Alex Auvolat
d8ab5bdc3e
New buckets for 0.6.0: fix model and migration 2022-01-04 12:47:28 +01:00
Alex Auvolat
b1cfd16913
New buckets for 0.6.0: small fixes, including: 
- ensure bucket names are correct aws s3 names
- when making aliases, ensure timestamps of links in both ways are the
  same
- fix small remarks by trinity
- don't have a separate website_access field
 2022-01-04 12:46:41 +01:00
Alex Auvolat
4d30e62db4
New buckets for 0.6.0: migration code and build files 2022-01-04 12:46:13 +01:00
Alex Auvolat
0bbb6673e7
Model changes 2022-01-04 12:45:52 +01:00
Alex Auvolat
53f71b3a57
Implement bucket alias and bucket unalias 2022-01-04 12:45:51 +01:00
Alex Auvolat
5b1117e582
New model for buckets 2022-01-04 12:45:46 +01:00
Alex Auvolat
8f6026de5e
Make table name a const in trait 2021-12-15 15:39:10 +01:00
trinity-1686a
1eb972b1ac Add compression using zstd (#173) 
fix #27

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/173
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2021-12-15 11:26:43 +01:00
Alex Auvolat
c94406f428
Improve how node roles are assigned in Garage 
- change the terminology: the network configuration becomes the role
  table, the configuration of a nodes becomes a node's role
- the modification of the role table takes place in two steps: first,
  changes are staged in a CRDT data structure. Then, once the user is
  happy with the changes, they can commit them all at once (or revert
  them).
- update documentation
- fix tests
- implement smarter partition assignation algorithm

This patch breaks the format of the network configuration: when
migrating, the cluster will be in a state where no roles are assigned.
All roles must be re-assigned and commited at once. This migration
should not pose an issue.
 2021-11-16 16:05:53 +01:00
Alex Auvolat
ad7ab31411
Implement GC delay for table data 2021-11-08 15:47:47 +01:00
Alex Auvolat
74a7a550eb
Safety: never voluntarily delete block in 10min interval after RC reaches zero 2021-11-08 15:47:47 +01:00
Alex Auvolat
2090a6187f
Add tranquilizer mechanism to improve on token bucket mechanism 2021-11-04 13:26:59 +01:00
Alex Auvolat
69b89fb46d
Fix race in block resync 2021-10-27 12:01:12 +02:00
Alex Auvolat
6b47c294f5
Refactoring on repair commands 2021-10-27 11:14:55 +02:00
Trinity Pointard
28c015d9ff
add cli parameter to verify local bloc integrity 
reuse code for listing local blocks
add disk i/o speed limit on integrity check
 2021-10-27 10:31:03 +02:00
Alex Auvolat
43e13a501d
Use published netapp crate instead of git repo 2021-10-26 10:36:57 +02:00
Alex Auvolat
ada7899b24
Fix clippy lints (fix #121) 2021-10-26 10:20:05 +02:00
Alex Auvolat
df8a4068d9
Refactor block manager code, and hopefully fix deadlock 2021-10-25 14:21:51 +02:00
Alex Auvolat
de4276202a
Improve CLI, adapt tests, update documentation 2021-10-25 14:21:48 +02:00
Alex Auvolat
1b450c4b49
Improvements to CLI and various fixes for netapp version 
Discovery via consul, persist peer list to file
 2021-10-22 16:55:24 +02:00
Alex Auvolat
4067797d01
First port of Garage to Netapp 2021-10-22 15:55:18 +02:00
Alex Auvolat
b9127dd6f8
Prepare for v0.3.0 and add migration path from v0.2.1.x 2021-05-28 15:29:58 +02:00
Alex Auvolat
b490ebc7f6
Many improvements on ring/replication and its configuration: 
- Explicit "replication_mode" configuration parameters that takes
  either "none", "2" or "3" as values, instead of letting user configure
  replication factor themselves. These are presets whose corresponding
  replication/quorum values can be found in replication/mode.rs

- Explicit support for single-node and two-node deployments
  (number of nodes must be at least "replication_mode", with "none"
  we can have only one node)

- Ring is now stored much more compactly with 256*8 + n*32 bytes,
  instead of 256*32 bytes

- Support for gateway-only nodes that do not store data
  (these nodes still need a metadata_directory to store the list
  of bucket and keys since those are stored on all nodes; it also
  technically needs a data_directory to start but it will stay
  empty unless we have bugs)
 2021-05-28 14:07:36 +02:00
Trinity Pointard
e4b9e4e24d
rename types to CamelCase 2021-05-03 22:15:09 +02:00
Trinity Pointard
4a1e079e8f
fix clippy warnings on model 2021-05-03 22:11:41 +02:00
Alex Auvolat
119217f9f6
change a few comments 2021-04-27 16:53:47 +02:00
Trinity Pointard
2812a027ea
change some more comments and revert changes on TableSchema 2021-04-27 16:49:07 +02:00
Trinity Pointard
74373aebcf
make most requested changes 2021-04-27 16:47:08 +02:00
Trinity Pointard
1e3df189d0
document api crate 2021-04-27 16:37:10 +02:00
Trinity Pointard
67585a4ffa
attempt at documenting model crate 2021-04-27 16:37:10 +02:00
Trinity Pointard
b437610812
attempt at documenting table crate 2021-04-27 16:37:10 +02:00
Alex Auvolat
9ced9f78dc
Improve bootstraping: do it regularly; persist peer list 2021-04-27 16:37:08 +02:00
Alex Auvolat
f859d15062 update to v0.2.1 2021-03-19 13:39:18 +01:00
Alex Auvolat
4c26a0b9c1 Update Cargo.toml files with AGPL license info 2021-03-18 21:59:17 +01:00
Alex Auvolat
dead945c8f Prepare for release 0.2 2021-03-18 19:33:15 +01:00
Alex Auvolat
4eb16e8863 Allow to import keys from previous Garage instance 2021-03-18 19:24:59 +01:00
Alex Auvolat
08bcd51956 GC object table in a specific case 2021-03-16 16:51:15 +01:00
Alex Auvolat
f4346cc5f4 Update dependencies 2021-03-16 15:58:40 +01:00
Alex Auvolat
1d9961e411 Simplify replication logic 2021-03-16 11:14:27 +01:00
Alex Auvolat
6a8439fd13 Some improvements in background worker but we terminate late 2021-03-15 23:14:12 +01:00
Alex Auvolat
0cd5b2ae19 WIP migrate to tokio 1 2021-03-15 22:36:41 +01:00
Alex Auvolat
4d4117f2b4 Refactor block resync loop; make workers infaillible 2021-03-15 20:09:44 +01:00
Alex Auvolat
667e4e72a8 Small fixes 2021-03-15 19:51:16 +01:00
Alex Auvolat
642bed601f Make it case-insensitive 2021-03-15 19:16:42 +01:00
Alex Auvolat
5ee1d956b6 Allow manipulation of keys by their shorthand in the CLI 2021-03-15 19:14:26 +01:00
Alex Auvolat
0290afe1f8 Make block rc code more understandable 2021-03-15 18:27:26 +01:00
Alex Auvolat
3bf2df622a Time and metadata improvements 2021-03-15 16:21:41 +01:00
Alex Auvolat
097c339d98 Fix race condition 2021-03-15 15:26:29 +01:00
Alex Auvolat
bdcbdd1cd8 Fix list API bug 2021-03-15 14:46:37 +01:00
Alex Auvolat
831eb35763 cargo fmt 2021-03-12 21:52:19 +01:00
Alex Auvolat
c475471e7a Implement table gc, currently for block_ref and version only 2021-03-12 19:57:37 +01:00
Alex Auvolat
3f7a496355 More security: don't delete stuff too easily 2021-03-11 19:06:27 +01:00
Alex Auvolat
046b649bcc (not well tested) use merkle tree for sync 2021-03-11 18:28:27 +01:00
Alex Auvolat
94f3d28774 WIP big refactoring 2021-03-11 16:54:15 +01:00
Alex Auvolat
8d63738cb0 Checkpoint: add merkle tree in data table 2021-03-11 13:47:21 +01:00
Alex Auvolat
3214dd52dd Very minor changes 2021-03-10 21:50:09 +01:00
Alex Auvolat
af7600f989 Correctly implement CompleteMultipartUpload with etag check of parts 2021-03-10 17:01:05 +01:00
Alex Auvolat
445912dc6a Remove migration paths from 0.1 branch 2021-03-10 16:38:31 +01:00
Alex Auvolat
0fd7df8fa0 Switch to blake2 sum for identifying blocks by their data 2021-03-10 16:33:31 +01:00
Alex Auvolat
f319a7d374 Refactor model stuff, including cleaner CRDTs 2021-03-10 16:21:56 +01:00
Alex Auvolat
3882d5ba36 Remove epidemic propagation for fully replicated stuff: write directly to all nodes 2021-03-05 15:09:18 +01:00
Alex Auvolat
49c25a1509 Simulate stuff moving around 2021-02-25 10:53:33 +01:00
Alex Auvolat
5fe95ebae7 fix tracing 2021-02-24 12:18:01 +01:00
Alex Auvolat
13e2eda0c2 Arrange block manager 2021-02-24 11:58:03 +01:00
Alex Auvolat
20e6e9fa20 Update sled & try to debug deadlock (but its in sled...) 2021-02-23 21:27:28 +01:00
Alex Auvolat
bf25c95fe2 Make updated() be a sync function that doesn't fail 2021-02-23 20:25:15 +01:00
Alex Auvolat
b1b640ae8b rename hash() to sha256sum(), we might want to change it at some places 2021-02-21 15:24:30 +01:00
Alex Auvolat
ceeb0732a2 Use 0.1.0b instead of 0.1.0 (for compatibility with new Error type) 2021-01-15 18:27:58 +01:00
Alex Auvolat
1d1d497e2b Bump everything to 0.1.1 2021-01-15 17:54:48 +01:00
Quentin Dufour
ccda9ab1ca Merge branch 'master' into feature/website 2020-12-17 21:09:50 +01:00
Quentin
3bc4d57a0f First implementation of the CLI 2020-12-15 12:48:24 +01:00
Quentin
d0eb6a457f Migrate RPC to new schema 2020-12-14 21:46:49 +01:00
Quentin
96388acf23 Implement migration 2020-12-12 21:35:29 +01:00
Quentin
e1ce2b228a WIP table migration 2020-12-12 17:00:31 +01:00
Quentin
8722e27600 CRDT doc 2020-11-23 17:49:21 +01:00
Alex Auvolat
68be5072e5 simplify 2020-11-20 23:53:54 +01:00
Alex Auvolat
f8a04852a2 Convert bucket table to better CRDT representation 2020-11-20 23:53:54 +01:00
Alex Auvolat
e02e9e035e Begin improve model to use better CRDTs 2020-11-20 23:53:54 +01:00
Alex Auvolat
2dc9a48022 Fix reconciliation logic 2020-11-20 20:12:32 +01:00
Alex Auvolat
e9fd265ce6 Slight refactoring to make things clearer with DeletedFilter 2020-11-20 20:11:04 +01:00
Alex Auvolat
a50fa70d45 Refactor error management in API part 2020-11-08 15:05:28 +01:00
Alex Auvolat
44dba0e53c cargo fmt 2020-07-08 17:34:37 +02:00
Alex Auvolat
84bbbfaa7b Add multiple headers to object model 2020-07-08 17:33:14 +02:00
Alex Auvolat
a5fa2a136b (WIP) New object table model, TODO: update API calls to use it 2020-07-08 16:46:47 +02:00
Alex Auvolat
86bf4dedac Add support for model migrations 2020-07-08 16:10:53 +02:00
Alex Auvolat
3b0b11085e Add versions to dependencies 2020-07-07 14:18:47 +02:00
Alex Auvolat
cc65cdc0fe Add license, description and repository to .toml files 2020-07-07 14:14:58 +02:00
Alex Auvolat
bec26a1312 Rename garage_core to garage_model 2020-07-07 13:59:22 +02:00