micropelago/isle
4
1
Fork 0
Code Issues 1 Pull Requests Projects Releases 6 Activity

Forbidden: No such key error when adding garage for first time #1

New Issue
Open
opened 2022-12-14 19:24:59 +00:00 by mediocregopher · 0 comments
mediocregopher commented 2022-12-14 19:24:59 +00:00
Owner
Copy Link

It seems that when a garage instance initially starts up and join the network it does not immediately replicate bucket api keys locally, nor does it forward S3 API requests to other instances which do have those keys.

This means that upon first starting a cryptic-net host which is adding storage capacity we see errors like this:

Dec 13 15:32:08 boobytrap cryptic-net[1421430]: garage-3900 » 2022-12-13T22:32:08.448372Z  INFO garage_api::generic_server: 10.242.0.3:38408 PUT /cryptic-net-global/bootstrap/hosts/boobytrap.yml.signed
Dec 13 15:32:08 boobytrap cryptic-net[1421430]: garage-3900 » 2022-12-13T22:32:08.448535Z  INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: No such key: cb6aa5fbc992dfa5
Dec 13 15:32:08 boobytrap cryptic-net[1421430]: garage-3900 » 2022-12-13T22:32:08.448545Z  INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: No such key: cb6aa5fbc992dfa5
Dec 13 15:32:08 boobytrap cryptic-net[1421430]: ERROR updating host info in garage "errMsg"="writing to \"bootstrap/hosts/boobytrap.yml.signed\" in global bucket: Forbidden: No such key: cb6aa5fbc992dfa5"

These will continue until the local garage instance has synced the API keys with the other instances.

It seems that when a garage instance initially starts up and join the network it does not immediately replicate bucket api keys locally, nor does it forward S3 API requests to other instances which _do_ have those keys. This means that upon first starting a cryptic-net host which is adding storage capacity we see errors like this: ``` Dec 13 15:32:08 boobytrap cryptic-net[1421430]: garage-3900 » 2022-12-13T22:32:08.448372Z INFO garage_api::generic_server: 10.242.0.3:38408 PUT /cryptic-net-global/bootstrap/hosts/boobytrap.yml.signed Dec 13 15:32:08 boobytrap cryptic-net[1421430]: garage-3900 » 2022-12-13T22:32:08.448535Z INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: No such key: cb6aa5fbc992dfa5 Dec 13 15:32:08 boobytrap cryptic-net[1421430]: garage-3900 » 2022-12-13T22:32:08.448545Z INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: No such key: cb6aa5fbc992dfa5 Dec 13 15:32:08 boobytrap cryptic-net[1421430]: ERROR updating host info in garage "errMsg"="writing to \"bootstrap/hosts/boobytrap.yml.signed\" in global bucket: Forbidden: No such key: cb6aa5fbc992dfa5" ``` These will continue until the local garage instance has synced the API keys with the other instances.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: micropelago/isle#1
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?