✨[patch] Implementation of agent-sidecar storage backup logic #409

rinx · 2020-05-27T10:50:29Z

Description:

In this PR, a blob storage backup logic of agent-sidecar is implemented.
Currently, it only supports S3.

Bugfix:

internal/core/ngt/ngt.go ... remove useless C.frees

Related Issue:

Nothing.

How Has This Been Tested?:

Nothing.

Environment:

Golang Version: 1.14.3
Docker Version: 19.03.8
Kubernetes Version: 1.18.2
NGT Version: 1.11.5

Types of changes:

Bug fix [type/bug]
New feature [type/feature]
Add tests [type/test]
Security related changes [type/security]
Add documents [type/documentation]
Refactoring [type/refactoring]
Update dependencies [type/dependency]
Update benchmarks and performances [type/bench]
Update CI [type/ci]

Changes to Core Features:

Have you added an explanation of what your changes do and why you'd like us to include them?
Have you written new tests for your core changes, as applicable?
Have you successfully ran tests with your changes locally?

Checklist:

I have read the CONTRIBUTING document.
I have checked open Pull Requests for the similar feature or fixes?
I have added tests and benchmarks to cover my changes.
I have ensured all new and existing tests passed.
I have commented my code, particularly in hard-to-understand areas
I have updated the documentation accordingly.

pull-assistant · 2020-05-27T10:50:48Z

Best reviewed: commit by commit

Optimal code review plan (10 warnings)

✨ Implement S3 multipart upload

...l/db/storage/blob/s3/option.go 55% changes removed in 🚚 rename s3cli...

.../db/storage/blob/s3/s3_test.go 43% changes removed in ✅ f...

✨ Added single upload mode for s3 writer

...orage/blob/s3/writer/option.go 57% changes removed in ✨ Add max_p...

...orage/blob/s3/writer/writer.go 51% changes removed in ✨ Add max_p...

🔥 Remove C.frees in NGT.Close func

✨ Add Reader Writer funcs to compress structs

✨ Add sidecar backup implementations

...ernal/db/storage/blob/s3/s3.go 67% changes removed in ✨ Add max_p...

...l/db/storage/blob/s3/option.go 56% changes removed in 🚚 rename s3cli...

...ar/service/storage_observer.go 43% changes removed in 🐛 backup trigger...

🐳 Fix Dockerfile for agent-sidecar

✅ Add tests

♻️ use single dir for watch target

...ar/service/storage_observer.go 70% changes removed in ♻️ revise obs...

⚡ Using io.Pipe

...ar/service/storage_observer.go 47% changes removed in ♻️ revise obs...

✅ gen tests

...ce/blob_storage_option_test.go 61% changes removed in ✨ Add initc...

✨ Add initcontainer mode for agent-sidecar

✅ gen tests

.../db/storage/blob/s3/s3_test.go 50% changes removed in ✅ f...

internal/config/blob_test.go 50% changes removed in ✅ f...

☸️ Add sidecar to chart

🔥 Remove C.frees

✅ Fix tests

.../db/storage/blob/s3/s3_test.go 55% changes removed in ✅ f...

☸️ remove agent.sidecar.volumeMounts

♻️ fix behavior of agent-sidecar initContainer mode

🐛 backup triggered from a single goroutine

🔥 Remove auto_backup_duration_limit

☸️ Add agent-sidecar service

☸️ Fix templates for agent initContainers

♻️ Fix order of defer

✨ Add wg

♻️ Add another wg

✅ Fix tests

♻️ Fix order of closing io.Writer, io.Reader

✏️ Fix typo

🐛 Fix

...r/service/observer/observer.go 86% changes removed in ♻️ revise obs...

✨ Add poststop

🔧 revise max-part-size

🔧 Fix maxPartSize

✨ Add max_part_size_mb parameter

...ernal/db/storage/blob/s3/s3.go 50% changes removed in 🚚 rename s3cli...

✅ fix broken tests

🚨 fix warnings of deepsource go

☸️ update terminationGracePeriodSeconds of agent

♻️ revise observer backup function

✅ fix tests and fix permissions

🚚 rename s3client to client / add validation for s3 partsize

✅ fix tests

Powered by Pull Assistant. Last update cdd9817 ... 5a23ba3. Read the comment docs.

vdaas-ci · 2020-05-27T10:50:48Z

[WARNING] Changes in interal/config may require you to change Helm charts. Please check.

internal/core/ngt/ngt.go

codecov · 2020-05-27T10:57:50Z

Codecov Report

Merging #409 into master will decrease coverage by 0.36%.
The diff coverage is 0.91%.

@@            Coverage Diff            @@
##           master    #409      +/-   ##
=========================================
- Coverage    7.71%   7.35%   -0.37%     
=========================================
  Files         372     383      +11     
  Lines       18845   19664     +819     
=========================================
- Hits         1454    1446       -8     
- Misses      17178   17999     +821     
- Partials      213     219       +6

Impacted Files	Coverage Δ
internal/compress/gob.go	`27.50% <0.00%> (-33.62%)`	⬇️
internal/compress/gzip.go	`34.00% <0.00%> (-17.52%)`	⬇️
internal/compress/lz4.go	`41.66% <0.00%> (-17.16%)`	⬇️
internal/compress/zstd.go	`35.29% <0.00%> (-17.65%)`	⬇️
internal/config/blob.go	`0.00% <0.00%> (ø)`
internal/config/compress.go	`0.00% <0.00%> (ø)`
internal/config/sidecar.go	`0.00% <0.00%> (ø)`
internal/db/storage/blob/s3/reader/option.go	`0.00% <0.00%> (ø)`
internal/db/storage/blob/s3/reader/reader.go	`0.00% <0.00%> (ø)`
internal/db/storage/blob/s3/s3.go	`0.00% <0.00%> (ø)`
... and 33 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7bf7dbf...5a23ba3. Read the comment docs.

vdaas-ci · 2020-05-27T11:50:13Z

[WARNING] Changes in interal/config may require you to change Helm charts. Please check.

vdaas-ci · 2020-05-27T11:59:12Z

[WARNING] Changes in interal/config may require you to change Helm charts. Please check.

pkg/agent/sidecar/service/storage_observer.go

internal/config/sidecar.go

vdaas-ci · 2020-05-28T03:41:13Z

[WARNING] Changes in interal/config may require you to change Helm charts. Please check.

vdaas-ci · 2020-05-28T03:44:47Z

[WARNING] Changes in interal/config may require you to change Helm charts. Please check.

vdaas-ci · 2020-05-28T07:13:43Z

[WARNING] Changes in interal/config may require you to change Helm charts. Please check.

vdaas-ci · 2020-05-28T07:22:03Z

[WARNING] Changes in interal/config may require you to change Helm charts. Please check.

vdaas-ci · 2020-05-28T07:40:26Z

[WARNING] Changes in interal/config may require you to change Helm charts. Please check.

vdaas-ci · 2020-05-28T09:12:45Z

[WARNING] Changes in interal/config may require you to change Helm charts. Please check.

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

vdaas-ci · 2020-06-03T09:01:43Z

[WARNING] Changes in interal/config may require you to change Helm charts. Please check.

pkg/agent/sidecar/service/storage/option_test.go

pkg/agent/sidecar/service/observer/observer.go

github-actions bot added area/internal area/makefile team/set SET team team/sre SRE team type/ci size/XXL type/feature New feature priority/medium labels May 27, 2020

rinx commented May 27, 2020

View reviewed changes

internal/core/ngt/ngt.go Outdated Show resolved Hide resolved

rinx force-pushed the feature/agent-sidecar/implement-storage-backup-logic branch from 32e50d0 to a548f15 Compare May 27, 2020 11:49

github-actions bot added size/XXXL and removed size/XXL labels May 27, 2020

rinx commented May 27, 2020

View reviewed changes

pkg/agent/sidecar/service/storage_observer.go Outdated Show resolved Hide resolved

pkg/agent/sidecar/service/storage_observer.go Outdated Show resolved Hide resolved

internal/config/sidecar.go Outdated Show resolved Hide resolved

rinx force-pushed the feature/agent-sidecar/implement-storage-backup-logic branch from ffdef45 to 27c3bad Compare May 28, 2020 07:21

rinx force-pushed the feature/agent-sidecar/implement-storage-backup-logic branch from 27c3bad to db07fab Compare May 28, 2020 07:40

rinx force-pushed the feature/agent-sidecar/implement-storage-backup-logic branch from db07fab to c18cf63 Compare May 28, 2020 09:12

github-actions bot added the area/helm label May 28, 2020

rinx marked this pull request as ready for review May 28, 2020 09:13

rinx force-pushed the feature/agent-sidecar/implement-storage-backup-logic branch from c18cf63 to 6f61a72 Compare May 28, 2020 09:37

rinx added 25 commits June 3, 2020 09:01

✅ Fix tests

a88bd33

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

☸️ remove agent.sidecar.volumeMounts

e176877

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

♻️ fix behavior of agent-sidecar initContainer mode

e6defb5

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

🐛 backup triggered from a single goroutine

1900c6c

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

🔥 Remove auto_backup_duration_limit

f87c8a8

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

☸️ Add agent-sidecar service

caa8872

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

☸️ Fix templates for agent initContainers

4bd7a9b

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

♻️ Fix order of defer

27cbbd9

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

✨ Add wg

b1450e1

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

♻️ Add another wg

aa72a2f

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

✅ Fix tests

edc8047

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

♻️ Fix order of closing io.Writer, io.Reader

ba9370b

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

✏️ Fix typo

c633a48

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

🐛 Fix

2b34cf9

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

✨ Add poststop

3e887f2

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

🔧 revise max-part-size

d71f17f

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

🔧 Fix maxPartSize

8da1122

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

✨ Add max_part_size_mb parameter

9d7f26f

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

✅ fix broken tests

b5966b0

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

🚨 fix warnings of deepsource go

147703b

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

☸️ update terminationGracePeriodSeconds of agent

c5bab10

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

♻️ revise observer backup function

5b4aa6e

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

✅ fix tests and fix permissions

5a833d5

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

🚚 rename s3client to client / add validation for s3 partsize

e8fc964

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

✅ fix tests

5a23ba3

Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>

vdaas-ci force-pushed the feature/agent-sidecar/implement-storage-backup-logic branch from d36d7ee to 5a23ba3 Compare June 3, 2020 09:01

github-actions bot reviewed Jun 3, 2020

View reviewed changes

rinx merged commit 58ed429 into master Jun 3, 2020

rinx deleted the feature/agent-sidecar/implement-storage-backup-logic branch June 3, 2020 09:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨[patch] Implementation of agent-sidecar storage backup logic #409

✨[patch] Implementation of agent-sidecar storage backup logic #409

rinx commented May 27, 2020 •

edited

Loading

pull-assistant bot commented May 27, 2020 •

edited

Loading

vdaas-ci commented May 27, 2020

codecov bot commented May 27, 2020 •

edited

Loading

vdaas-ci commented May 27, 2020

vdaas-ci commented May 27, 2020

vdaas-ci commented May 28, 2020

vdaas-ci commented May 28, 2020

vdaas-ci commented May 28, 2020

vdaas-ci commented May 28, 2020

vdaas-ci commented May 28, 2020

vdaas-ci commented May 28, 2020

vdaas-ci commented Jun 3, 2020

✨[patch] Implementation of agent-sidecar storage backup logic #409

✨[patch] Implementation of agent-sidecar storage backup logic #409

Conversation

rinx commented May 27, 2020 • edited Loading

Description:

Related Issue:

How Has This Been Tested?:

Environment:

Types of changes:

Changes to Core Features:

Checklist:

pull-assistant bot commented May 27, 2020 • edited Loading

vdaas-ci commented May 27, 2020

codecov bot commented May 27, 2020 • edited Loading

Codecov Report

vdaas-ci commented May 27, 2020

vdaas-ci commented May 27, 2020

vdaas-ci commented May 28, 2020

vdaas-ci commented May 28, 2020

vdaas-ci commented May 28, 2020

vdaas-ci commented May 28, 2020

vdaas-ci commented May 28, 2020

vdaas-ci commented May 28, 2020

vdaas-ci commented Jun 3, 2020

rinx commented May 27, 2020 •

edited

Loading

pull-assistant bot commented May 27, 2020 •

edited

Loading

codecov bot commented May 27, 2020 •

edited

Loading