storage: add s3 backend support (without GC and dedupe) #216

eusebiu-constantin-petu-dbk · 2021-08-09T16:43:18Z

using: https://github.com/distribution/distribution/tree/main/registry/storage/driver

Supports a lot of storages types.

rchincha · 2021-08-09T16:47:26Z

examples/config-s3.json

+    "version": "0.1.0-dev",
+    "storage": {
+        "rootDirectory": "/zot",
+        "objectStoreParams": {


let's review this part carefully.

Instead of "objectStorageParams", let's just call it "storageDriver"

codecov · 2021-08-27T13:51:39Z

Codecov Report

Merging #216 (5096ba5) into main (8e4d828) will decrease coverage by 1.28%.
The diff coverage is 75.21%.

❗ Current head 5096ba5 differs from pull request most recent head 4ecc97c. Consider uploading reports for the commit 4ecc97c to get more accurate results

@@            Coverage Diff             @@
##             main     #216      +/-   ##
==========================================
- Coverage   79.25%   77.97%   -1.29%     
==========================================
  Files          43       30      -13     
  Lines        5360     5066     -294     
==========================================
- Hits         4248     3950     -298     
+ Misses        818      804      -14     
- Partials      294      312      +18

Impacted Files	Coverage Δ
pkg/api/config.go	`78.57% <ø> (ø)`
pkg/storage/s3/storage.go	`74.68% <74.68%> (ø)`
pkg/api/controller.go	`88.67% <86.95%> (-0.69%)`	⬇️
pkg/storage/storage_fs.go	`76.09% <100.00%> (-0.67%)`	⬇️
pkg/extensions/search/cve/cve.go	`74.64% <0.00%> (-12.20%)`	⬇️
pkg/api/errors.go	`98.14% <0.00%> (-1.86%)`	⬇️
pkg/api/routes.go	`55.67% <0.00%> (-1.36%)`	⬇️
pkg/storage/cache.go	`57.89% <0.00%> (-1.29%)`	⬇️
... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8e4d828...4ecc97c. Read the comment docs.

rchincha · 2021-10-26T15:57:07Z

https://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3iface/

eusebiu-constantin-petu-dbk · 2021-10-26T16:01:46Z

https://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3iface/

But I'm not using the sdk directly, I'm using "github.com/docker/distribution/registry/storage/driver/factory"

Look here: https://github.com/anuvu/zot/pull/216/files#diff-4edf1087b440a5696d89e156f567d352995d832217650afee1c116c54fc032d1R103

rchincha · 2021-10-14T19:51:00Z

examples/config-s3.json

+    "version": "0.1.0-dev",
+    "storage": {
+        "rootDirectory": "/zot",
+        "objectStoreParams": {


Instead of "objectStorageParams", let's just call it "storageDriver"

rchincha · 2021-10-14T19:51:31Z

examples/config-s3.json

+        },
+        "subPaths": {
+            "/devops": {
+                "rootDirectory": "/zot-devops"


Each of these "rootDirectory" could potentially have their own "storageDriver"s

Can we mix filesystem and s3? In the current implementation we can.

rchincha · 2021-10-14T19:53:00Z

examples/config-s3.json

+    "storage": {
+        "rootDirectory": "/zot",
+        "objectStoreParams": {
+            "name": "s3",


Is this the name of the driver or the KeyID for aws storage?

it's both, I will add a document specifying all the parameters

rchincha · 2021-10-26T17:23:59Z

pkg/api/controller.go

+			storeName := fmt.Sprintf("%v", c.Config.Storage.ObjectStoreParams["name"])
+
+			// Init a Storager from connection string.
+			store, err := factory.Create(storeName, c.Config.Storage.ObjectStoreParams)


Can we create a factory.Create("s3mock"...)? so we control its behaviour?
And specify driver=s3mock in config.

I don;t understand the question.

You can do that already if you want with regionendpoint parameter.

"storageDriver": { "name": "s3", "region": "us-east-2", "regionendpoint": "localhost:4566", "bucket": "zot-storage", "secure": false, "skipverify": false }

localhost:4566 is https://github.com/localstack/localstack

pkg/api/controller.go

rchincha · 2021-11-12T17:50:25Z

pkg/api/controller_test.go

+
+		storageDriverParams := map[string]interface{}{
+			"rootDir":        "zot",
+			"name":           "s3",


Make this S3StorageDriverName

and other places too

rchincha · 2021-11-14T00:16:25Z

pkg/api/controller_test.go

+		conf.HTTP.Port = port
+		storageDriverParams := map[string]interface{}{
+			"rootDir": "zot",
+			"name":    "s3",


Change this to the string constant.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>

rchincha

lgtm

rchincha · 2021-11-14T00:16:36Z

pkg/api/controller_test.go

+
+		storageDriverParams := map[string]interface{}{
+			"rootDir":        "zot",
+			"name":           "s3",


eusebiu-constantin-petu-dbk requested review from andaaron, rchincha and shimish2 August 9, 2021 16:44

rchincha reviewed Aug 9, 2021

View reviewed changes

eusebiu-constantin-petu-dbk force-pushed the s3_docker branch 4 times, most recently from 052bd63 to e32460c Compare August 27, 2021 13:39

eusebiu-constantin-petu-dbk changed the title ~~POC S3 backend without GC, Dedupe, Tests~~ POC S3 backend without GC and Dedupe Aug 27, 2021

eusebiu-constantin-petu-dbk force-pushed the s3_docker branch 8 times, most recently from 55a0cb0 to 266c4a3 Compare September 3, 2021 12:56

eusebiu-constantin-petu-dbk force-pushed the s3_docker branch 2 times, most recently from f4c208d to 2378d65 Compare September 10, 2021 16:28

eusebiu-constantin-petu-dbk force-pushed the s3_docker branch 4 times, most recently from 7f10c1e to 3c643e0 Compare September 15, 2021 15:58

eusebiu-constantin-petu-dbk force-pushed the s3_docker branch 5 times, most recently from fecccff to cbe0f4d Compare September 22, 2021 13:22

rchincha changed the title ~~POC S3 backend without GC and Dedupe~~ storage: add s3 backend support (without GC and dedupe) Oct 26, 2021

eusebiu-constantin-petu-dbk force-pushed the s3_docker branch from 181122b to e650fd7 Compare November 3, 2021 16:20

rchincha reviewed Nov 5, 2021

View reviewed changes

eusebiu-constantin-petu-dbk force-pushed the s3_docker branch 4 times, most recently from 8d1d632 to 140c309 Compare November 9, 2021 12:56

rchincha reviewed Nov 10, 2021

View reviewed changes

pkg/api/controller.go Show resolved Hide resolved

pkg/api/controller.go Show resolved Hide resolved

eusebiu-constantin-petu-dbk force-pushed the s3_docker branch 3 times, most recently from f563ac1 to 6053051 Compare November 10, 2021 14:13

rchincha reviewed Nov 10, 2021

View reviewed changes

pkg/api/controller.go Show resolved Hide resolved

pkg/api/controller.go Outdated Show resolved Hide resolved

eusebiu-constantin-petu-dbk force-pushed the s3_docker branch 2 times, most recently from e5c9af7 to 54536c2 Compare November 12, 2021 15:51

rchincha reviewed Nov 12, 2021

View reviewed changes

rchincha reviewed Nov 14, 2021

View reviewed changes

eusebiu-constantin-petu-dbk force-pushed the s3_docker branch from 54536c2 to 6cb11ee Compare November 15, 2021 11:29

storage: add s3 backend support (without GC and dedupe)

4ecc97c

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>

eusebiu-constantin-petu-dbk force-pushed the s3_docker branch from 6cb11ee to 4ecc97c Compare November 15, 2021 14:10

rchincha modified the milestones: v1.3.2, v1.3.3 Nov 15, 2021

rchincha approved these changes Nov 15, 2021

View reviewed changes

pkg/api/controller_test.go Outdated

storageDriverParams := map[string]interface{}{

"rootDir": "zot",

"name": "s3",

Copy link

Contributor

rchincha Nov 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

rchincha merged commit 9c568c0 into project-zot:main Nov 15, 2021

This was referenced Nov 16, 2021

proposal: Integrate with beyondstorage/go-storage to add multiple storage backend supports #206

Closed

Other storage backends? #179

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

storage: add s3 backend support (without GC and dedupe) #216

storage: add s3 backend support (without GC and dedupe) #216

eusebiu-constantin-petu-dbk commented Aug 9, 2021

rchincha Aug 9, 2021

rchincha Oct 14, 2021

eusebiu-constantin-petu-dbk Nov 9, 2021

codecov bot commented Aug 27, 2021 •

edited

Loading

rchincha commented Oct 26, 2021

eusebiu-constantin-petu-dbk commented Oct 26, 2021 •

edited

Loading

rchincha Oct 14, 2021

rchincha Oct 14, 2021

eusebiu-constantin-petu-dbk Nov 9, 2021

rchincha Oct 14, 2021

eusebiu-constantin-petu-dbk Nov 9, 2021

rchincha Oct 26, 2021

eusebiu-constantin-petu-dbk Nov 9, 2021

rchincha Nov 12, 2021

rchincha Nov 12, 2021

rchincha Nov 14, 2021

rchincha left a comment

rchincha Nov 14, 2021

storage: add s3 backend support (without GC and dedupe) #216

storage: add s3 backend support (without GC and dedupe) #216

Conversation

eusebiu-constantin-petu-dbk commented Aug 9, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Aug 27, 2021 • edited Loading

Codecov Report

rchincha commented Oct 26, 2021

eusebiu-constantin-petu-dbk commented Oct 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rchincha left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Aug 27, 2021 •

edited

Loading

eusebiu-constantin-petu-dbk commented Oct 26, 2021 •

edited

Loading