Support of snapshot copy to StorPool primary storage in different zones #9478

slavkap · 2024-08-01T13:47:53Z

Description

This PR supports copying a snapshot to StorPool primary storage in different zones.

Added additional API param storageids in CloudStack API calls:

createSnapshot
copySnapshot
createSnapshotPolicy

The option snapshot.backup.to.secondary does not apply to the copy functionality. The snapshots will be copied only on the required primary storage in a different zone.
The user can create volumes/templates from the copied snapshots. The option to copy the snapshots to both primary and secondary storage is available by providing in the create/copy snapshot the destination zone IDs and storage IDs

For other storage plugins that want to adopt this functionality:

The Primary storage driver should have the capability CAN_COPY_SNAPSHOT_BETWEEN_ZONES
the respective plugin needs to implement the copySnapshot method in their SnapshotStrategy and that the driver can handle the COPY operation

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale

Major
Minor

Screenshots (if appropriate):

How Has This Been Tested?

Manual and smoke tests with StorPool primary storage on multiple zones

codecov · 2024-08-01T13:56:23Z

Codecov Report

Attention: Patch coverage is 4.60358% with 746 lines in your changes missing coverage. Please review.

Project coverage is 15.79%. Comparing base (1d4700a) to head (457b33f).

Files with missing lines	Patch %	Lines
...om/cloud/storage/snapshot/SnapshotManagerImpl.java	8.88%	155 Missing and 9 partials ⚠️
...ack/storage/snapshot/StorPoolSnapshotStrategy.java	0.00%	140 Missing ⚠️
...tastore/driver/StorPoolPrimaryDataStoreDriver.java	0.00%	73 Missing ⚠️
...age/collector/StorPoolAbandonObjectsCollector.java	0.00%	63 Missing ⚠️
...loudstack/storage/datastore/util/StorPoolUtil.java	0.00%	36 Missing ⚠️
...n/java/com/cloud/storage/VolumeApiServiceImpl.java	6.25%	29 Missing and 1 partial ⚠️
...org/apache/cloudstack/snapshot/SnapshotHelper.java	21.62%	24 Missing and 5 partials ⚠️
...oudstack/storage/snapshot/SnapshotServiceImpl.java	0.00%	27 Missing ⚠️
...udstack/storage/datastore/util/StorPoolHelper.java	0.00%	22 Missing ⚠️
...e/wrapper/StorPoolModifyStorageCommandWrapper.java	0.00%	21 Missing ⚠️
... and 24 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9478      +/-   ##
============================================
- Coverage     15.81%   15.79%   -0.02%     
- Complexity    12554    12555       +1     
============================================
  Files          5629     5629              
  Lines        492029   492642     +613     
  Branches      62293    60282    -2011     
============================================
+ Hits          77811    77816       +5     
- Misses       405894   406493     +599     
- Partials       8324     8333       +9

Flag	Coverage Δ
uitests	`4.47% <ø> (-0.01%)`	⬇️
unittests	`16.58% <4.60%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-08-21T08:21:53Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

DaanHoogland · 2024-08-22T13:48:15Z

@blueorangutan package

blueorangutan · 2024-08-22T13:50:03Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-08-22T15:24:36Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10756

DaanHoogland · 2024-08-23T07:43:27Z

@blueorangutan test

blueorangutan · 2024-08-23T07:44:03Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-08-23T22:40:23Z

[SF] Trillian test result (tid-11155)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 51322 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9478-t11155-kvm-ol8.zip
Smoke tests completed. 137 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_snapshot_usage	`Failure`	38.32	test_usage.py
test_01_vpc_site2site_vpn	`Failure`	308.08	test_vpc_vpn.py

github-actions · 2024-08-29T08:41:48Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

api/src/main/java/org/apache/cloudstack/api/command/user/snapshot/CopySnapshotCmd.java

.../src/main/java/org/apache/cloudstack/engine/subsystem/api/storage/DataStoreCapabilities.java

api/src/main/java/org/apache/cloudstack/api/command/user/snapshot/CopySnapshotCmd.java

github-actions · 2024-09-04T08:40:33Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

.../src/main/java/org/apache/cloudstack/engine/subsystem/api/storage/DataStoreCapabilities.java

sureshanaparti · 2024-09-04T12:06:59Z

...l/src/main/java/org/apache/cloudstack/storage/collector/StorPoolAbandonObjectsCollector.java

+                        continue;
+                    }
+                    for (SnapshotDetailsVO snapshot : snapshotDetails) {
+                        String name = snapshot.getValue().split(";")[0];


make sure the snapshot detail after .split() has length >= 2 & then get name and location from it (chance of Index Out of Bound error).

sureshanaparti · 2024-09-04T12:07:57Z

...l/src/main/java/org/apache/cloudstack/storage/collector/StorPoolAbandonObjectsCollector.java

+        }
+    }
+
+    private static List<String> snapshotsForRcovery(JsonArray arr) {


Suggested change

private static List<String> snapshotsForRcovery(JsonArray arr) {

private static List<String> snapshotsForRecovery(JsonArray arr) {

thanks, it's fixed in the latest commits

sureshanaparti · 2024-09-04T12:14:19Z

server/src/main/java/com/cloud/storage/snapshot/SnapshotManagerImpl.java

+//            logger.info(String.format("Cannot copy snapshot to another storage in different zone. It's not in the right state %s", snapshotOnStore.getStatus()));
+//            return false;
+//        }
+


remove these comments if not needed

thanks, it's removed in the latest commits

server/src/main/java/com/cloud/storage/snapshot/SnapshotManagerImpl.java

ui/src/views/storage/FormSchedule.vue

sureshanaparti · 2024-09-04T12:17:20Z

ui/src/views/storage/SnapshotZones.vue

+        const listStoragePools = json.liststoragepoolsresponse.storagepool
+        if (listStoragePools) {
+          this.storagePools = listStoragePools
+          this.storagePools = this.storagePools.filter(pool => pool.storagecapabilities.CAN_COPY_SNAPSHOT_BETWEEN_ZONES && pool.zoneid !== this.resource.zoneid)


Suggested change

this.storagePools = this.storagePools.filter(pool => pool.storagecapabilities.CAN_COPY_SNAPSHOT_BETWEEN_ZONES && pool.zoneid !== this.resource.zoneid)

this.storagePools = this.storagePools.filter(pool => pool.storagecapabilities.CAN_COPY_SNAPSHOT_BETWEEN_ZONES_AND_SAME_POOL_TYPE && pool.zoneid !== this.resource.zoneid)

ui/src/views/storage/TakeSnapshot.vue

…n zones Added support to copy a snapshot to another StorPool primary storage in different zones.

Added drop down to choose the primary storage pools to copy a snapshot Small fixes

boring-cyborg bot added component:api component:ceph component:orchestration component:primary-storage component:storage labels Aug 1, 2024

slavkap force-pushed the support-snapshot-copy-on-primary branch from 8cc7b49 to 0df6764 Compare August 14, 2024 09:14

slavkap marked this pull request as ready for review August 14, 2024 09:51

slavkap changed the title ~~[WIP] Support of snapshot copy to StorPool primary storage in different zones~~ Support of snapshot copy to StorPool primary storage in different zones Aug 14, 2024

github-actions bot added the status:has-conflicts label Aug 21, 2024

slavkap force-pushed the support-snapshot-copy-on-primary branch from 0df6764 to e0f4283 Compare August 22, 2024 12:49

github-actions bot removed the status:has-conflicts label Aug 22, 2024

github-actions bot added the status:has-conflicts label Aug 29, 2024

slavkap force-pushed the support-snapshot-copy-on-primary branch from e0f4283 to 6f209d8 Compare August 30, 2024 09:08

github-actions bot removed the status:has-conflicts label Aug 30, 2024

sureshanaparti reviewed Aug 30, 2024

View reviewed changes

api/src/main/java/org/apache/cloudstack/api/command/user/snapshot/CopySnapshotCmd.java Show resolved Hide resolved

sureshanaparti reviewed Aug 30, 2024

View reviewed changes

.../src/main/java/org/apache/cloudstack/engine/subsystem/api/storage/DataStoreCapabilities.java Outdated Show resolved Hide resolved

sureshanaparti reviewed Aug 30, 2024

View reviewed changes

api/src/main/java/org/apache/cloudstack/api/command/user/snapshot/CopySnapshotCmd.java Outdated Show resolved Hide resolved

rohityadavcloud added this to the 4.20.0.0 milestone Sep 4, 2024

github-actions bot added the status:has-conflicts label Sep 4, 2024

sureshanaparti reviewed Sep 4, 2024

View reviewed changes

.../src/main/java/org/apache/cloudstack/engine/subsystem/api/storage/DataStoreCapabilities.java Outdated Show resolved Hide resolved

sureshanaparti reviewed Sep 4, 2024

View reviewed changes

server/src/main/java/com/cloud/storage/snapshot/SnapshotManagerImpl.java Outdated Show resolved Hide resolved

sureshanaparti reviewed Sep 4, 2024

View reviewed changes

ui/src/views/storage/FormSchedule.vue Outdated Show resolved Hide resolved

sureshanaparti reviewed Sep 4, 2024

View reviewed changes

ui/src/views/storage/TakeSnapshot.vue Outdated Show resolved Hide resolved

JoaoJandre modified the milestones: 4.20.0.0, 4.21.0.0 Sep 10, 2024

slavkap added 5 commits September 16, 2024 17:24

Support of snapshot copy to different StorPool primary storage betwee…

a0ab1fb

…n zones Added support to copy a snapshot to another StorPool primary storage in different zones.

added option in the UI for the storage pools

8d1dc43

Added drop down to choose the primary storage pools to copy a snapshot Small fixes

Addressed comments

f4492c5

Address reviews

7034941

fix snapshot deletion

457b33f

slavkap force-pushed the support-snapshot-copy-on-primary branch from 524f382 to 457b33f Compare September 16, 2024 14:24

github-actions bot removed the status:has-conflicts label Sep 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support of snapshot copy to StorPool primary storage in different zones #9478

Support of snapshot copy to StorPool primary storage in different zones #9478

slavkap commented Aug 1, 2024 •

edited

Loading

codecov bot commented Aug 1, 2024 •

edited

Loading

github-actions bot commented Aug 21, 2024

DaanHoogland commented Aug 22, 2024

blueorangutan commented Aug 22, 2024

blueorangutan commented Aug 22, 2024

DaanHoogland commented Aug 23, 2024

blueorangutan commented Aug 23, 2024

blueorangutan commented Aug 23, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Sep 4, 2024

sureshanaparti Sep 4, 2024

sureshanaparti Sep 4, 2024

slavkap Sep 16, 2024

sureshanaparti Sep 4, 2024

slavkap Sep 16, 2024

sureshanaparti Sep 4, 2024

	private static List<String> snapshotsForRcovery(JsonArray arr) {
	private static List<String> snapshotsForRecovery(JsonArray arr) {

	this.storagePools = this.storagePools.filter(pool => pool.storagecapabilities.CAN_COPY_SNAPSHOT_BETWEEN_ZONES && pool.zoneid !== this.resource.zoneid)
	this.storagePools = this.storagePools.filter(pool => pool.storagecapabilities.CAN_COPY_SNAPSHOT_BETWEEN_ZONES_AND_SAME_POOL_TYPE && pool.zoneid !== this.resource.zoneid)

Support of snapshot copy to StorPool primary storage in different zones #9478

Are you sure you want to change the base?

Support of snapshot copy to StorPool primary storage in different zones #9478

Conversation

slavkap commented Aug 1, 2024 • edited Loading

Description

Types of changes

Feature/Enhancement Scale

Screenshots (if appropriate):

How Has This Been Tested?

codecov bot commented Aug 1, 2024 • edited Loading

Codecov Report

github-actions bot commented Aug 21, 2024

DaanHoogland commented Aug 22, 2024

blueorangutan commented Aug 22, 2024

blueorangutan commented Aug 22, 2024

DaanHoogland commented Aug 23, 2024

blueorangutan commented Aug 23, 2024

blueorangutan commented Aug 23, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Sep 4, 2024

sureshanaparti Sep 4, 2024

Choose a reason for hiding this comment

sureshanaparti Sep 4, 2024

Choose a reason for hiding this comment

slavkap Sep 16, 2024

Choose a reason for hiding this comment

sureshanaparti Sep 4, 2024

Choose a reason for hiding this comment

slavkap Sep 16, 2024

Choose a reason for hiding this comment

sureshanaparti Sep 4, 2024

Choose a reason for hiding this comment

slavkap commented Aug 1, 2024 •

edited

Loading

codecov bot commented Aug 1, 2024 •

edited

Loading