micropelago/garage
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
764 Commits 25 Branches 62 Tags 26 MiB
2cad656a03
Commit Graph

174 Commits

Author SHA1 Message Date
Alex Auvolat
3586c7257c
Clarify error message 2022-01-10 14:10:04 +01:00
Alex Auvolat
7ee11f0eb6
Fix unit tests 2022-01-05 17:34:48 +01:00
Alex Auvolat
168a90dfb5
Fix some error codes 2022-01-05 17:07:36 +01:00
Alex Auvolat
135858d067
Implement DeleteBucket 2022-01-05 16:28:19 +01:00
Alex Auvolat
8395030e48
Implement CreateBucket 2022-01-05 15:56:48 +01:00
Alex Auvolat
677ab60cc1
Small changes in key model and refactoring 2022-01-04 18:59:17 +01:00
Alex Auvolat
df35feba18
New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updates 2022-01-04 12:53:14 +01:00
Alex Auvolat
1bcd6fabbd
New buckets for 0.6.0: small changes 
- Fix bucket delete

- fix merge of bucket creation date

- Replace deletable with option in aliases
    Rationale: if two aliases point to conflicting bucket, resolving
    by making an arbitrary choice risks making data accessible when it
    shouldn't be. We'd rather resolve to deleting the alias until
    someone puts it back.
 2022-01-04 12:52:47 +01:00
Alex Auvolat
2140cd7205
Remove website redirects 2022-01-04 12:52:46 +01:00
Alex Auvolat
beeef4758e
Some movement of helper code and refactoring of error handling 2022-01-04 12:52:46 +01:00
Alex Auvolat
d8ab5bdc3e
New buckets for 0.6.0: fix model and migration 2022-01-04 12:47:28 +01:00
Alex Auvolat
b76d0580a0
Fix forgotten flag 2022-01-04 12:47:28 +01:00
Alex Auvolat
b1cfd16913
New buckets for 0.6.0: small fixes, including: 
- ensure bucket names are correct aws s3 names
- when making aliases, ensure timestamps of links in both ways are the
  same
- fix small remarks by trinity
- don't have a separate website_access field
 2022-01-04 12:46:41 +01:00
Alex Auvolat
0bbb6673e7
Model changes 2022-01-04 12:45:52 +01:00
Alex Auvolat
5b1117e582
New model for buckets 2022-01-04 12:45:46 +01:00
trinity-1686a
ca7b438f3f less strict ListBuckets (#178) 
fix #175

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/178
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2021-12-15 15:05:36 +01:00
trinity-1686a
60d4459926 BucketWebsite (#174) 
fix #77

this does not store anything but a on/off switch for website, and does not implement GetBucketWebsite as it would require storing more. GetBucketWebsite should be pretty easy to implement once data is stored though.

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/174
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2021-12-15 10:41:39 +01:00
trinity-1686a
c4ac8835d3 add proper request router for s3 api (#163) 
fix #161

Current request router was organically grown, and is getting messier and messier with each addition.
This router cover exaustively existing API endpoints (with exceptions listed in [#161(comment)](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/161#issuecomment-1773) either because new and old api endpoint can't feasabily be differentied, or it's more lambda than s3).

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/163
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2021-12-06 15:17:47 +01:00
trinity-1686a
7f26ed55cd Improved handling of HTTP ranges 
- correct HTTP code when range syntax is invalid (fix #140)
- when multiple ranges are given, simply ignore and send whole file

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/157
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2021-11-29 11:52:42 +01:00
Quentin Dufour
8811bb08e6
In ListBuckets, hide entry if no perms 2021-11-22 12:10:28 +01:00
Alex Auvolat
c94406f428
Improve how node roles are assigned in Garage 
- change the terminology: the network configuration becomes the role
  table, the configuration of a nodes becomes a node's role
- the modification of the role table takes place in two steps: first,
  changes are staged in a CRDT data structure. Then, once the user is
  happy with the changes, they can commit them all at once (or revert
  them).
- update documentation
- fix tests
- implement smarter partition assignation algorithm

This patch breaks the format of the network configuration: when
migrating, the cluster will be in a state where no roles are assigned.
All roles must be re-assigned and commited at once. This migration
should not pose an issue.
 2021-11-16 16:05:53 +01:00
Trinity Pointard
53888995bd update doc and comments 2021-11-16 15:41:41 +01:00
Trinity Pointard
396fe4c702 clippy 2021-11-16 15:41:41 +01:00
Trinity Pointard
02158ee666 fix issue where list on vhost-bucket would list bucket instead of bucket content 2021-11-16 15:41:41 +01:00
Trinity Pointard
9c58ec28d3 add support for vhost-style s3 bucket 2021-11-16 15:41:41 +01:00
Alex Auvolat
ada7899b24
Fix clippy lints (fix #121) 2021-10-26 10:20:05 +02:00
Alex Auvolat
de4276202a
Improve CLI, adapt tests, update documentation 2021-10-25 14:21:48 +02:00
Alex Auvolat
1b450c4b49
Improvements to CLI and various fixes for netapp version 
Discovery via consul, persist peer list to file
 2021-10-22 16:55:24 +02:00
Alex Auvolat
4067797d01
First port of Garage to Netapp 2021-10-22 15:55:18 +02:00
Alex Auvolat
f6060b92aa
Fix HTTP return code for DeleteObject (fix #98) 2021-10-11 14:24:49 +02:00
Alex Auvolat
f3a097abdf
WIP: try to fix #93, and improve S3 ListObjects (v1 and v2) API calls 2021-10-11 11:15:47 +02:00
Alex Auvolat
b9127dd6f8
Prepare for v0.3.0 and add migration path from v0.2.1.x 2021-05-28 15:29:58 +02:00
Quentin Dufour
5fdabf3e75
Add basic support for the "Versioning" command 2021-05-14 22:33:26 +02:00
Alex Auvolat
6ccffc3162
Improved XML serialization 
- Use quick_xml and serde for all XML response returned by the S3 API.
- Include tests for all structs used to generate XML
- Remove old manual XML escaping function which was unsafe
 2021-05-06 22:37:15 +02:00
Trinity Pointard
e4b9e4e24d
rename types to CamelCase 2021-05-03 22:15:09 +02:00
Trinity Pointard
84856e84e5
fix clippy warnings on api 2021-05-03 22:11:41 +02:00
Quentin Dufour
631c36b3ff S3 API: support ListBuckets 2021-05-03 21:55:30 +02:00
Alex Auvolat
ee2a3d363b
Remove STREAMING-AWS4-HMAC-SHA256-PAYLOAD (see #64) 2021-05-03 17:30:40 +02:00
Alex Auvolat
dcfc32cf85
Many S3 compatibility improvements: 
- return XML errors
- implement AuthorizationHeaderMalformed error to redirect clients to
  correct location (used by minio client)
- implement GetBucketLocation
- fix DeleteObjects XML parsing and response
 2021-04-28 01:05:40 +02:00
Alex Auvolat
642186c530
Fix #59 (& issue) 2021-04-27 23:10:43 +02:00
Trinity Pointard
74373aebcf
make most requested changes 2021-04-27 16:47:08 +02:00
Trinity Pointard
1e3df189d0
document api crate 2021-04-27 16:37:10 +02:00
Alex Auvolat
f859d15062 update to v0.2.1 2021-03-19 13:39:18 +01:00
Alex Auvolat
4c26a0b9c1 Update Cargo.toml files with AGPL license info 2021-03-18 21:59:17 +01:00
Alex Auvolat
6edbc65847 Add trinity's comment in the code 2021-03-18 19:46:43 +01:00
Alex
bfa0ff8f82 Merge pull request 'add support for caching headers' (#49) from trinity-1686a/garage:cache-headers into master 
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/49
 2021-03-18 19:45:02 +01:00
Alex Auvolat
dead945c8f Prepare for release 0.2 2021-03-18 19:33:15 +01:00
Trinity Pointard
b4c903371c add support for caching headers 2021-03-18 15:46:33 +01:00
Alex Auvolat
f4346cc5f4 Update dependencies 2021-03-16 15:58:40 +01:00
Alex Auvolat
0cd5b2ae19 WIP migrate to tokio 1 2021-03-15 22:36:41 +01:00
Alex Auvolat
667e4e72a8 Small fixes 2021-03-15 19:51:16 +01:00
Alex Auvolat
3bf2df622a Time and metadata improvements 2021-03-15 16:21:41 +01:00
Alex Auvolat
097c339d98 Fix race condition 2021-03-15 15:26:29 +01:00
Alex Auvolat
bdcbdd1cd8 Fix list API bug 2021-03-15 14:46:37 +01:00
Alex Auvolat
831eb35763 cargo fmt 2021-03-12 21:52:19 +01:00
Alex Auvolat
c475471e7a Implement table gc, currently for block_ref and version only 2021-03-12 19:57:37 +01:00
Alex Auvolat
7fdaf7aef0 Fix merkle updater not being notified; improved logging 2021-03-12 14:37:46 +01:00
Alex Auvolat
3214dd52dd Very minor changes 2021-03-10 21:50:09 +01:00
Alex Auvolat
af7600f989 Correctly implement CompleteMultipartUpload with etag check of parts 2021-03-10 17:01:05 +01:00
Alex Auvolat
0fd7df8fa0 Switch to blake2 sum for identifying blocks by their data 2021-03-10 16:33:31 +01:00
Alex Auvolat
f319a7d374 Refactor model stuff, including cleaner CRDTs 2021-03-10 16:21:56 +01:00
Alex Auvolat
40763fd749 Cargo fmt 2021-02-23 18:46:25 +01:00
Alex Auvolat
b1b640ae8b rename hash() to sha256sum(), we might want to change it at some places 2021-02-21 15:24:30 +01:00
Alex Auvolat
3bcbbe1e31 More precise logging (warn only when returning a 500) 2021-02-20 00:30:39 +01:00
Alex Auvolat
10b983b8e7 Add verification of part numbers in CompleteMultipartUpload (WIP #30) 2021-02-20 00:13:07 +01:00
Alex Auvolat
1de96248e0 add application/xml header and missing xml escapes 2021-02-19 23:40:18 +01:00
Alex Auvolat
5d1fa591d9 Add compatibility list 2021-02-19 19:10:23 +01:00
Alex Auvolat
e64ecbdccd S3 compatibility: return 404 instead of 400 on some multipart commands 2021-02-19 18:51:05 +01:00
Alex Auvolat
55a2a636ca Implement ListObjectsV2 2021-02-19 16:44:06 +01:00
Alex Auvolat
02d512f3fd Fix #28, extra headers being ignored (because of profound stupidity) 2021-02-19 12:38:22 +01:00
Alex Auvolat
76390085ef Small improvements in the S3 put workflow 2021-02-19 12:11:02 +01:00
Alex Auvolat
36814be447 Fix S3 ListObjects result and replace println!s by debug!s 2021-01-16 16:05:54 +01:00
Alex Auvolat
e818f51073 Forgot a bump 2021-01-15 18:36:51 +01:00
Alex Auvolat
022b386a50 Improved compatibility on list API call 2020-12-06 15:39:03 +01:00
Alex Auvolat
4a5bbbb810 Propose ETag fix 2020-12-05 19:23:46 +01:00
Alex Auvolat
76b489f3d3 Reformulate patch 2020-12-05 19:20:07 +01:00
Quentin
bd7e3d1bd1 Fix Content-Length 2020-12-05 18:57:22 +01:00
Alex Auvolat
9f46fb699a Content-range fix 2020-12-05 16:37:59 +01:00
Alex Auvolat
d54f15b2c6 Small optimisation 2020-11-29 17:07:14 +01:00
Alex Auvolat
fed97f37e1 ETag patch 2020-11-29 16:38:01 +01:00
Alex Auvolat
601ae25ad2 Small refactorings 2020-11-29 16:21:28 +01:00
Quentin
17dc610f8a Also check hash for < 3KB files 2020-11-22 11:14:46 +01:00
Quentin
435d5f9205 Fix base64/hex checksum comparison 2020-11-22 11:04:33 +01:00
Alex Auvolat
f8a04852a2 Convert bucket table to better CRDT representation 2020-11-20 23:53:54 +01:00
Alex Auvolat
e02e9e035e Begin improve model to use better CRDTs 2020-11-20 23:53:54 +01:00
Alex Auvolat
e9fd265ce6 Slight refactoring to make things clearer with DeletedFilter 2020-11-20 20:11:04 +01:00
Alex Auvolat
7d7b9e95a9 Simplify and_then(Some) as map() and remove move 2020-11-11 16:36:48 +01:00
Quentin
5a5592c176 Replace with option syntaxic sugar 2020-11-11 16:12:42 +01:00
Alex Auvolat
a50fa70d45 Refactor error management in API part 2020-11-08 15:05:28 +01:00
Alex Auvolat
6a8b972f3a Modif parse_bucket_key to fail when bucket is "" 2020-11-08 13:39:44 +01:00
Quentin
e1415f5976 Add a second test 2020-11-07 15:34:53 +01:00
Quentin
9a50ce12a8 Fix formatting 2020-11-07 13:59:30 +01:00
Quentin
8f4ada1965 Add a test for parse_bucket_key 2020-11-07 13:53:32 +01:00
Alex Auvolat
5faf069e33 trace test 2020-07-15 15:41:49 +02:00
Alex Auvolat
27a0d0d859 pretty 2020-07-15 15:37:35 +02:00
Alex Auvolat
1c70552f95 Validate content MD5 and SHA256 sums for PutObject and UploadPart 2020-07-15 15:31:13 +02:00
Alex Auvolat
6c7f9704ea Implement correct ETag for objects created with PutObject 2020-07-13 16:51:30 +02:00
Alex Auvolat
9305e5e87f More headers taken into account 2020-07-09 17:04:43 +02:00
Alex Auvolat
44dba0e53c cargo fmt 2020-07-08 17:34:37 +02:00
Alex Auvolat
64a6eda0d2 Migrate S3 api to use new model 2020-07-08 17:33:24 +02:00