micropelago
/
isle
3
0
Fork 0
Code Issues 1 Pull Requests Projects Releases 4 Activity
Labels Milestones
New Issue

`Forbidden: No such key` error when adding garage for first time #1

Open
opened 1 year ago by mediocregopher · 0 comments
mediocregopher commented 1 year ago
Owner

It seems that when a garage instance initially starts up and join the network it does not immediately replicate bucket api keys locally, nor does it forward S3 API requests to other instances which do have those keys.

This means that upon first starting a cryptic-net host which is adding storage capacity we see errors like this:

Dec 13 15:32:08 boobytrap cryptic-net[1421430]: garage-3900 » 2022-12-13T22:32:08.448372Z  INFO garage_api::generic_server: 10.242.0.3:38408 PUT /cryptic-net-global/bootstrap/hosts/boobytrap.yml.signed
Dec 13 15:32:08 boobytrap cryptic-net[1421430]: garage-3900 » 2022-12-13T22:32:08.448535Z  INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: No such key: cb6aa5fbc992dfa5
Dec 13 15:32:08 boobytrap cryptic-net[1421430]: garage-3900 » 2022-12-13T22:32:08.448545Z  INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: No such key: cb6aa5fbc992dfa5
Dec 13 15:32:08 boobytrap cryptic-net[1421430]: ERROR updating host info in garage "errMsg"="writing to \"bootstrap/hosts/boobytrap.yml.signed\" in global bucket: Forbidden: No such key: cb6aa5fbc992dfa5"

These will continue until the local garage instance has synced the API keys with the other instances.

It seems that when a garage instance initially starts up and join the network it does not immediately replicate bucket api keys locally, nor does it forward S3 API requests to other instances which _do_ have those keys. This means that upon first starting a cryptic-net host which is adding storage capacity we see errors like this: ``` Dec 13 15:32:08 boobytrap cryptic-net[1421430]: garage-3900 » 2022-12-13T22:32:08.448372Z INFO garage_api::generic_server: 10.242.0.3:38408 PUT /cryptic-net-global/bootstrap/hosts/boobytrap.yml.signed Dec 13 15:32:08 boobytrap cryptic-net[1421430]: garage-3900 » 2022-12-13T22:32:08.448535Z INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: No such key: cb6aa5fbc992dfa5 Dec 13 15:32:08 boobytrap cryptic-net[1421430]: garage-3900 » 2022-12-13T22:32:08.448545Z INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: No such key: cb6aa5fbc992dfa5 Dec 13 15:32:08 boobytrap cryptic-net[1421430]: ERROR updating host info in garage "errMsg"="writing to \"bootstrap/hosts/boobytrap.yml.signed\" in global bucket: Forbidden: No such key: cb6aa5fbc992dfa5" ``` These will continue until the local garage instance has synced the API keys with the other instances.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
Labels
Apply labels
Clear labels
No items
No Label
Milestone
Set milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Assign users
Clear assignees
No Assignees
1 Participants
Notifications
Due Date

No due date set.

Dependencies

No dependencies set.

Reference: micropelago/isle#1
Write Preview
Loading…
Cancel
Save
There is no content yet.
Delete Branch '%!s(<nil>)'

Deleting a branch is permanent. It CANNOT be undone. Continue?

No
Yes