chore(refactor): add fake controller testing for cspi-controller

[mittachaitu]

Signed-off-by: mittachaitu sai.chaithanya@mayadata.io

This PR makes the following changes:

• Mock ZPOOL/ZFS commands to achieve positive and negative testing.
• Pool Provisioning with different RaidTypes and a combination of DataRaidGroup and WriteCacheRaidGroup.
• Vertical Pool Expansion by adding BlockDevices to an existing pool configuration.
• Replacing existing BlockDevices with new blockdevices.
• Pool Deprovisiong with various scenarios:
  • Delete CSPI when the pool doesn't exist.
  • Delete CSPI when the pool exists but zpool command returns error.
  • Delete CSPI when pool exist and successfully destroyed.
• Mocks the ZPOOL & ZFS utility commands.

Does It Make Changes To Production Code:
Yes

This PR fixes the part of the issue.

TODO:

• Multiple test cases need to be covered even with a negative scenario(Ex: When ZPOOL/ZFS utility returns error how CSPI-controller behaves).

[mittachaitu]

Sample output when we run go test -v inside pkg/controller/cspi-controller

sai@sai:~/gocode/src/github.com/openebs/cstor-operators/pkg/controllers/cspi-controller$ go test -v
=== RUN   TestCSPIFinalizerAdd
I0511 10:49:22.144434    5956 handler.go:381] Added Finalizer: cspi-foo, 
--- PASS: TestCSPIFinalizerAdd (0.00s)
=== RUN   TestCSPIFinalizerRemoval
=== RUN   TestCSPIFinalizerRemoval/When_pool_exist_deletion_of_CSPI_is_triggered
I0511 10:49:22.144931    5956 delete.go:29] Destroying a pool {cstor-1234}
I0511 10:49:22.144962    5956 handler.go:360] Removed Finalizer: cspi-foo-2, 
I0511 10:49:22.144971    5956 handler.go:195] Pool cspi-foo-2 deleted successfully
=== RUN   TestCSPIFinalizerRemoval/When_deletion_of_pool_is_failed
I0511 10:49:22.145063    5956 delete.go:29] Destroying a pool {cstor-1234}
Event:   ❗ Warning Validation failed No data RaidGroups exists
E0511 10:49:22.145089    5956 delete.go:43] Failed to destroy a pool {fake delete error}.. error exist status 1 fake delete error
I0511 10:49:22.145117    5956 delete.go:29] Destroying a pool {cstor-1234}
E0511 10:49:22.145135    5956 delete.go:43] Failed to destroy a pool {fake delete error}.. error exist status 1 fake delete error
=== RUN   TestCSPIFinalizerRemoval/When_pool_doesn't_exist_deletion_of_CSPI_is_triggered
Event:   ❗ Warning FailDestroy Failed to delete pool due to 'error exist status 1 fake delete error'
Event:   ❗ Warning FailDestroy Failed to delete pool due to 'error exist status 1 fake delete error'
I0511 10:49:22.145256    5956 delete.go:29] Destroying a pool {cstor-1234}
I0511 10:49:22.145276    5956 delete.go:33] Pool cstor-1234 not imported.. so, can't destroy
I0511 10:49:22.145293    5956 handler.go:360] Removed Finalizer: cspi-foo-1, 
I0511 10:49:22.145301    5956 handler.go:195] Pool cspi-foo-1 deleted successfully
--- PASS: TestCSPIFinalizerRemoval (0.00s)
    --- PASS: TestCSPIFinalizerRemoval/When_pool_exist_deletion_of_CSPI_is_triggered (0.00s)
    --- PASS: TestCSPIFinalizerRemoval/When_deletion_of_pool_is_failed (0.00s)
    --- PASS: TestCSPIFinalizerRemoval/When_pool_doesn't_exist_deletion_of_CSPI_is_triggered (0.00s)
=== RUN   TestCSPIPoolProvisioning
=== RUN   TestCSPIPoolProvisioning/Raidz_Pool_With_Multiple_Data_RaidGroups
I0511 10:49:22.146317    5956 handler.go:381] Added Finalizer: cspi-foo-raidz, 
I0511 10:49:22.146339    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:22.146355    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:22.146378    5956 create.go:41] Creating a pool for cspi-foo-raidz cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.288367    5956 exec-run.go:67] exit status 1
E0511 10:49:23.288387    5956 volumereplica.go:443] Unable to get volumes:
E0511 10:49:23.301996    5956 exec-run.go:67] exit status 1
E0511 10:49:23.302013    5956 volumereplica.go:443] Unable to get volumes:
=== RUN   TestCSPIPoolProvisioning/Raidz2_Pool_With_Multiple_Data_RaidGroups
I0511 10:49:23.304890    5956 handler.go:381] Added Finalizer: cspi-foo-raidz2, 
I0511 10:49:23.304952    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:23.304994    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:23.305029    5956 create.go:41] Creating a pool for cspi-foo-raidz2 cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.325711    5956 exec-run.go:67] exit status 1
E0511 10:49:23.325729    5956 volumereplica.go:443] Unable to get volumes:
E0511 10:49:23.339632    5956 exec-run.go:67] exit status 1
E0511 10:49:23.339650    5956 volumereplica.go:443] Unable to get volumes:
=== RUN   TestCSPIPoolProvisioning/Stripe_Pool_Provisioning_Without_WriteCache_RaidGroup
I0511 10:49:23.343992    5956 handler.go:381] Added Finalizer: cspi-foo-stripe-data, 
I0511 10:49:23.344030    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:23.344054    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:23.344077    5956 create.go:41] Creating a pool for cspi-foo-stripe-data cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.357206    5956 exec-run.go:67] exit status 1
E0511 10:49:23.357220    5956 volumereplica.go:443] Unable to get volumes:
E0511 10:49:23.365250    5956 exec-run.go:67] exit status 1
E0511 10:49:23.365263    5956 volumereplica.go:443] Unable to get volumes:
=== RUN   TestCSPIPoolProvisioning/Stripe_Pool_Provisioning_With_WriteCache_RaidGroup
I0511 10:49:23.365894    5956 handler.go:381] Added Finalizer: cspi-foo-stripe-writecache, 
I0511 10:49:23.365929    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:23.365951    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:23.365968    5956 create.go:41] Creating a pool for cspi-foo-stripe-writecache cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.373937    5956 exec-run.go:67] exit status 1
E0511 10:49:23.373950    5956 volumereplica.go:443] Unable to get volumes:
E0511 10:49:23.383358    5956 exec-run.go:67] exit status 1
E0511 10:49:23.383374    5956 volumereplica.go:443] Unable to get volumes:
=== RUN   TestCSPIPoolProvisioning/Stripe_Pool_Provisioning_With_Multiple_DataRaidGroups_&_WriteCache_RaidGroup
I0511 10:49:23.384142    5956 handler.go:381] Added Finalizer: cspi-foo-stripe-multiple-raidgroup, 
I0511 10:49:23.384168    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:23.384181    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:23.384193    5956 create.go:41] Creating a pool for cspi-foo-stripe-multiple-raidgroup cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.393036    5956 exec-run.go:67] exit status 1
E0511 10:49:23.393050    5956 volumereplica.go:443] Unable to get volumes:
E0511 10:49:23.401489    5956 exec-run.go:67] exit status 1
E0511 10:49:23.401503    5956 volumereplica.go:443] Unable to get volumes:
=== RUN   TestCSPIPoolProvisioning/Mirror_Pool_With_Multiple_RaidGroups
I0511 10:49:23.403157    5956 handler.go:381] Added Finalizer: cspi-foo-mirror, 
I0511 10:49:23.403276    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:23.403358    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:23.403426    5956 create.go:41] Creating a pool for cspi-foo-mirror cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.414234    5956 exec-run.go:67] exit status 1
E0511 10:49:23.414249    5956 volumereplica.go:443] Unable to get volumes:
E0511 10:49:23.423642    5956 exec-run.go:67] exit status 1
E0511 10:49:23.423656    5956 volumereplica.go:443] Unable to get volumes:
--- PASS: TestCSPIPoolProvisioning (1.28s)
    --- PASS: TestCSPIPoolProvisioning/Raidz_Pool_With_Multiple_Data_RaidGroups (1.16s)
    --- PASS: TestCSPIPoolProvisioning/Raidz2_Pool_With_Multiple_Data_RaidGroups (0.04s)
    --- PASS: TestCSPIPoolProvisioning/Stripe_Pool_Provisioning_Without_WriteCache_RaidGroup (0.02s)
    --- PASS: TestCSPIPoolProvisioning/Stripe_Pool_Provisioning_With_WriteCache_RaidGroup (0.02s)
    --- PASS: TestCSPIPoolProvisioning/Stripe_Pool_Provisioning_With_Multiple_DataRaidGroups_&_WriteCache_RaidGroup (0.02s)
    --- PASS: TestCSPIPoolProvisioning/Mirror_Pool_With_Multiple_RaidGroups (0.02s)
=== RUN   TestCSPIPoolExpansion
=== RUN   TestCSPIPoolExpansion/Provision_Raidz2_Pool_and_add_writecache_raidz_raid_groups
I0511 10:49:23.426465    5956 handler.go:381] Added Finalizer: cspi-add-writecache-raidz-raidgroup, 
I0511 10:49:23.426586    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:23.426666    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:23.426797    5956 create.go:41] Creating a pool for cspi-add-writecache-raidz-raidgroup cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.435723    5956 exec-run.go:67] exit status 1
E0511 10:49:23.435737    5956 volumereplica.go:443] Unable to get volumes:
Event:   🚀 Normal Pool Expansion Pool Expanded Successfully By Adding RaidGroup With BlockDevices: [blockdevice-51 blockdevice-52 blockdevice-53] device type: log pool type: raidz
E0511 10:49:23.445912    5956 exec-run.go:67] exit status 1
E0511 10:49:23.445927    5956 volumereplica.go:443] Unable to get volumes:
E0511 10:49:23.455468    5956 exec-run.go:67] exit status 1
E0511 10:49:23.455482    5956 volumereplica.go:443] Unable to get volumes:
=== RUN   TestCSPIPoolExpansion/Provision_Raidz_Pool_and_add_writecache_stripe_raid_groups
I0511 10:49:23.457327    5956 handler.go:381] Added Finalizer: cspi-add-writecache-raidgroup, 
I0511 10:49:23.457353    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:23.457374    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:23.457390    5956 create.go:41] Creating a pool for cspi-add-writecache-raidgroup cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.466171    5956 exec-run.go:67] exit status 1
E0511 10:49:23.466185    5956 volumereplica.go:443] Unable to get volumes:
Event:   🚀 Normal Pool Expansion Pool Expanded Successfully By Adding RaidGroup With BlockDevices: [blockdevice-43 blockdevice-44] device type: log pool type: stripe
E0511 10:49:23.475109    5956 exec-run.go:67] exit status 1
E0511 10:49:23.475152    5956 volumereplica.go:443] Unable to get volumes:
E0511 10:49:23.483563    5956 exec-run.go:67] exit status 1
E0511 10:49:23.483577    5956 volumereplica.go:443] Unable to get volumes:
=== RUN   TestCSPIPoolExpansion/Provision_Stripe_Pool_and_expand_data_and_writecache_raid_groups
I0511 10:49:23.484699    5956 handler.go:381] Added Finalizer: cspi-bar-stripe-expand, 
I0511 10:49:23.484734    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:23.484757    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:23.484778    5956 create.go:41] Creating a pool for cspi-bar-stripe-expand cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.493280    5956 exec-run.go:67] exit status 1
E0511 10:49:23.493295    5956 volumereplica.go:443] Unable to get volumes:
Event:   🚀 Normal Pool Expansion Pool Expanded Successfully By Adding BlockDevices: [blockdevice-8 blockdevice-9] device type: data pool type: stripe
Event:   🚀 Normal Pool Expansion Pool Expanded Successfully By Adding BlockDevices: [blockdevice-10 blockdevice-11] device type: log pool type: stripe
E0511 10:49:23.502690    5956 exec-run.go:67] exit status 1
E0511 10:49:23.502714    5956 volumereplica.go:443] Unable to get volumes:
E0511 10:49:23.511987    5956 exec-run.go:67] exit status 1
E0511 10:49:23.512000    5956 volumereplica.go:443] Unable to get volumes:
=== RUN   TestCSPIPoolExpansion/Provision_Mirror_Pool_and_expand_writecache_raid_groups
I0511 10:49:23.514306    5956 handler.go:381] Added Finalizer: cspi-bar-mirror-expand, 
I0511 10:49:23.514341    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:23.514359    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:23.514378    5956 create.go:41] Creating a pool for cspi-bar-mirror-expand cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.523042    5956 exec-run.go:67] exit status 1
E0511 10:49:23.523057    5956 volumereplica.go:443] Unable to get volumes:
Event:   🚀 Normal Pool Expansion Pool Expanded Successfully By Adding RaidGroup With BlockDevices: [blockdevice-16 blockdevice-17] device type: log pool type: mirror
Event:   🚀 Normal Pool Expansion Pool Expanded Successfully By Adding RaidGroup With BlockDevices: [blockdevice-18 blockdevice-19] device type: log pool type: mirror
E0511 10:49:23.531918    5956 exec-run.go:67] exit status 1
E0511 10:49:23.531932    5956 volumereplica.go:443] Unable to get volumes:
E0511 10:49:23.541974    5956 exec-run.go:67] exit status 1
E0511 10:49:23.541988    5956 volumereplica.go:443] Unable to get volumes:
=== RUN   TestCSPIPoolExpansion/Provision_Mirror_Pool_and_expand_data_raid_groups
I0511 10:49:23.544804    5956 handler.go:381] Added Finalizer: cspi-expand-data-raidgroup, 
I0511 10:49:23.544833    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:23.544851    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:23.544866    5956 create.go:41] Creating a pool for cspi-expand-data-raidgroup cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.553031    5956 exec-run.go:67] exit status 1
E0511 10:49:23.553045    5956 volumereplica.go:443] Unable to get volumes:
Event:   🚀 Normal Pool Expansion Pool Expanded Successfully By Adding RaidGroup With BlockDevices: [blockdevice-22 blockdevice-23] device type: data pool type: mirror
Event:   🚀 Normal Pool Expansion Pool Expanded Successfully By Adding RaidGroup With BlockDevices: [blockdevice-24 blockdevice-25] device type: data pool type: mirror
E0511 10:49:23.561376    5956 exec-run.go:67] exit status 1
E0511 10:49:23.561390    5956 volumereplica.go:443] Unable to get volumes:
E0511 10:49:23.571101    5956 exec-run.go:67] exit status 1
E0511 10:49:23.571115    5956 volumereplica.go:443] Unable to get volumes:
=== RUN   TestCSPIPoolExpansion/Provision_Mirror_Pool_and_expand_data_and_write_cache_raid_groups
I0511 10:49:23.573360    5956 handler.go:381] Added Finalizer: cspi-expand-both-mirror-raidgroups, 
I0511 10:49:23.573395    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:23.573415    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:23.573439    5956 create.go:41] Creating a pool for cspi-expand-both-mirror-raidgroups cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.581450    5956 exec-run.go:67] exit status 1
E0511 10:49:23.581465    5956 volumereplica.go:443] Unable to get volumes:
Event:   🚀 Normal Pool Expansion Pool Expanded Successfully By Adding RaidGroup With BlockDevices: [blockdevice-28 blockdevice-29] device type: data pool type: mirror
Event:   🚀 Normal Pool Expansion Pool Expanded Successfully By Adding RaidGroup With BlockDevices: [blockdevice-30 blockdevice-31] device type: log pool type: mirror
E0511 10:49:23.589881    5956 exec-run.go:67] exit status 1
E0511 10:49:23.589894    5956 volumereplica.go:443] Unable to get volumes:
E0511 10:49:23.599201    5956 exec-run.go:67] exit status 1
E0511 10:49:23.599217    5956 volumereplica.go:443] Unable to get volumes:
=== RUN   TestCSPIPoolExpansion/Provision_Raidz_Pool_and_expand_data_and_write_cache_raid_groups
I0511 10:49:23.600623    5956 handler.go:381] Added Finalizer: cspi-expand-both-raidz-raidgroups, 
I0511 10:49:23.600649    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:23.600670    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:23.600694    5956 create.go:41] Creating a pool for cspi-expand-both-raidz-raidgroups cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.609953    5956 exec-run.go:67] exit status 1
E0511 10:49:23.609966    5956 volumereplica.go:443] Unable to get volumes:
Event:   🚀 Normal Pool Expansion Pool Expanded Successfully By Adding RaidGroup With BlockDevices: [blockdevice-38 blockdevice-39] device type: log pool type: raidz
Event:   🚀 Normal Pool Expansion Pool Expanded Successfully By Adding RaidGroup With BlockDevices: [blockdevice-35 blockdevice-36 blockdevice-37] device type: data pool type: raidz
E0511 10:49:23.618543    5956 exec-run.go:67] exit status 1
E0511 10:49:23.618556    5956 volumereplica.go:443] Unable to get volumes:
E0511 10:49:23.627932    5956 exec-run.go:67] exit status 1
E0511 10:49:23.627948    5956 volumereplica.go:443] Unable to get volumes:
=== RUN   TestCSPIPoolExpansion/Provision_stripe_Pool_and_expand_data_raidgroup_by_injecting_error_in_zpool_add_command
I0511 10:49:23.629866    5956 handler.go:381] Added Finalizer: cspi-foo-stripe-expand-witherror, 
I0511 10:49:23.629902    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:23.629919    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:23.629937    5956 create.go:41] Creating a pool for cspi-foo-stripe-expand-witherror cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.638273    5956 exec-run.go:67] exit status 1
E0511 10:49:23.638287    5956 volumereplica.go:443] Unable to get volumes:
Event:   ❗ Warning Pool Expansion Failed to expand pool... Error: Failed to add devlist [/dev/disk/by-id/ata-WDC_WD10JPVX-4pvt-WXG1A-55 /dev/disk/by-id/ata-WDC_WD10JPVX-45hl-WXG1A-56].. err {fake error} {error exit status 1 fake error}
Event:   ❗ Warning FailedSync Failed to update pool due to Pool expansion... err {Failed to add devlist [/dev/disk/by-id/ata-WDC_WD10JPVX-4pvt-WXG1A-55 /dev/disk/by-id/ata-WDC_WD10JPVX-45hl-WXG1A-56].. err {fake error} {error exit status 1 fake error}}
E0511 10:49:23.646951    5956 exec-run.go:67] exit status 1
E0511 10:49:23.646965    5956 volumereplica.go:443] Unable to get volumes:
Event:   ❗ Warning Pool Expansion Failed to expand pool... Error: Failed to add devlist [/dev/disk/by-id/ata-WDC_WD10JPVX-4pvt-WXG1A-55 /dev/disk/by-id/ata-WDC_WD10JPVX-45hl-WXG1A-56].. err {fake error} {error exit status 1 fake error}
Event:   ❗ Warning FailedSync Failed to update pool due to Pool expansion... err {Failed to add devlist [/dev/disk/by-id/ata-WDC_WD10JPVX-4pvt-WXG1A-55 /dev/disk/by-id/ata-WDC_WD10JPVX-45hl-WXG1A-56].. err {fake error} {error exit status 1 fake error}}
E0511 10:49:23.655146    5956 exec-run.go:67] exit status 1
E0511 10:49:23.655162    5956 volumereplica.go:443] Unable to get volumes:
Event:   🚀 Normal Pool Expansion Pool Expanded Successfully By Adding BlockDevices: [blockdevice-55 blockdevice-56] device type: data pool type: stripe
E0511 10:49:23.664416    5956 exec-run.go:67] exit status 1
E0511 10:49:23.664430    5956 volumereplica.go:443] Unable to get volumes:
=== RUN   TestCSPIPoolExpansion/Provision_stripe_Pool_and_expand_data_raidgroup_by_injecting_error_permanently_in_zpool_add_command
I0511 10:49:23.665221    5956 handler.go:381] Added Finalizer: cspi-bar-stripe-expand-witherror, 
I0511 10:49:23.665256    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:23.665272    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:23.665287    5956 create.go:41] Creating a pool for cspi-bar-stripe-expand-witherror cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.673481    5956 exec-run.go:67] exit status 1
E0511 10:49:23.673495    5956 volumereplica.go:443] Unable to get volumes:
Event:   ❗ Warning Pool Expansion Failed to expand pool... Error: Failed to add devlist [/dev/disk/by-id/ata-WDC_WD10JPVX-rsvs-WXG1A-58 /dev/disk/by-id/ata-WDC_WD10JPVX-r5zs-WXG1A-59].. err {fake error} {error exit status 1 fake error}
Event:   ❗ Warning FailedSync Failed to update pool due to Pool expansion... err {Failed to add devlist [/dev/disk/by-id/ata-WDC_WD10JPVX-rsvs-WXG1A-58 /dev/disk/by-id/ata-WDC_WD10JPVX-r5zs-WXG1A-59].. err {fake error} {error exit status 1 fake error}}
E0511 10:49:23.681560    5956 exec-run.go:67] exit status 1
E0511 10:49:23.681587    5956 volumereplica.go:443] Unable to get volumes:
Event:   ❗ Warning Pool Expansion Failed to expand pool... Error: Failed to add devlist [/dev/disk/by-id/ata-WDC_WD10JPVX-rsvs-WXG1A-58 /dev/disk/by-id/ata-WDC_WD10JPVX-r5zs-WXG1A-59].. err {fake error} {error exit status 1 fake error}
Event:   ❗ Warning FailedSync Failed to update pool due to Pool expansion... err {Failed to add devlist [/dev/disk/by-id/ata-WDC_WD10JPVX-rsvs-WXG1A-58 /dev/disk/by-id/ata-WDC_WD10JPVX-r5zs-WXG1A-59].. err {fake error} {error exit status 1 fake error}}
E0511 10:49:23.690939    5956 exec-run.go:67] exit status 1
E0511 10:49:23.690959    5956 volumereplica.go:443] Unable to get volumes:
Event:   ❗ Warning Pool Expansion Failed to expand pool... Error: Failed to add devlist [/dev/disk/by-id/ata-WDC_WD10JPVX-rsvs-WXG1A-58 /dev/disk/by-id/ata-WDC_WD10JPVX-r5zs-WXG1A-59].. err {fake error} {error exit status 1 fake error}
Event:   ❗ Warning FailedSync Failed to update pool due to Pool expansion... err {Failed to add devlist [/dev/disk/by-id/ata-WDC_WD10JPVX-rsvs-WXG1A-58 /dev/disk/by-id/ata-WDC_WD10JPVX-r5zs-WXG1A-59].. err {fake error} {error exit status 1 fake error}}
=== RUN   TestCSPIPoolExpansion/Provision_Stripe_Pool_And_Expand_DataRaidGroup
I0511 10:49:23.691839    5956 handler.go:381] Added Finalizer: cspi-foo-stripe-expand, 
I0511 10:49:23.691880    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:23.691900    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:23.691922    5956 create.go:41] Creating a pool for cspi-foo-stripe-expand cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.700710    5956 exec-run.go:67] exit status 1
E0511 10:49:23.700724    5956 volumereplica.go:443] Unable to get volumes:
Event:   🚀 Normal Pool Expansion Pool Expanded Successfully By Adding BlockDevices: [blockdevice-2 blockdevice-3] device type: data pool type: stripe
E0511 10:49:23.709298    5956 exec-run.go:67] exit status 1
E0511 10:49:23.709313    5956 volumereplica.go:443] Unable to get volumes:
E0511 10:49:23.717481    5956 exec-run.go:67] exit status 1
E0511 10:49:23.717495    5956 volumereplica.go:443] Unable to get volumes:
--- PASS: TestCSPIPoolExpansion (0.29s)
    --- PASS: TestCSPIPoolExpansion/Provision_Raidz2_Pool_and_add_writecache_raidz_raid_groups (0.03s)
    --- PASS: TestCSPIPoolExpansion/Provision_Raidz_Pool_and_add_writecache_stripe_raid_groups (0.03s)
    --- PASS: TestCSPIPoolExpansion/Provision_Stripe_Pool_and_expand_data_and_writecache_raid_groups (0.03s)
    --- PASS: TestCSPIPoolExpansion/Provision_Mirror_Pool_and_expand_writecache_raid_groups (0.03s)
    --- PASS: TestCSPIPoolExpansion/Provision_Mirror_Pool_and_expand_data_raid_groups (0.03s)
    --- PASS: TestCSPIPoolExpansion/Provision_Mirror_Pool_and_expand_data_and_write_cache_raid_groups (0.03s)
    --- PASS: TestCSPIPoolExpansion/Provision_Raidz_Pool_and_expand_data_and_write_cache_raid_groups (0.03s)
    --- PASS: TestCSPIPoolExpansion/Provision_stripe_Pool_and_expand_data_raidgroup_by_injecting_error_in_zpool_add_command (0.04s)
    --- PASS: TestCSPIPoolExpansion/Provision_stripe_Pool_and_expand_data_raidgroup_by_injecting_error_permanently_in_zpool_add_command (0.03s)
    --- PASS: TestCSPIPoolExpansion/Provision_Stripe_Pool_And_Expand_DataRaidGroup (0.03s)
=== RUN   TestCSPIBlockDeviceReplacement
=== RUN   TestCSPIBlockDeviceReplacement/Provision_Mirror_Pool_And_Replace_BlockDevice
I0511 10:49:23.718752    5956 handler.go:381] Added Finalizer: cspi-foo-mirror-replace, 
I0511 10:49:23.718791    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:23.718809    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:23.718835    5956 create.go:41] Creating a pool for cspi-foo-mirror-replace cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.727798    5956 exec-run.go:67] exit status 1
E0511 10:49:23.727811    5956 volumereplica.go:443] Unable to get volumes:
I0511 10:49:23.728096    5956 disk_operation.go:259] Triggered replacement of /dev/disk/by-id/ata-WDC_WD10JPVX-kwxc-WXG1A-2 with /dev/disk/by-id/ata-WDC_WD10JPVX-zdgc-WXG1A-5 on pool cstor-1234
Event:   🚀 Normal BlockDevice Replacement Replacement of blockdevice-2 BlockDevice with blockdevice-5 BlockDevice is in-Progress
Event:   🚀 Normal BlockDevice Replacement Resilvering is successfull on BlockDevice blockdevice-5
I0511 10:49:23.728399    5956 disk_operation.go:259] Triggered replacement of /dev/disk/by-id/ata-WDC_WD10JPVX-jlrv-WXG1A-4 with /dev/disk/by-id/ata-WDC_WD10JPVX-ncxj-WXG1A-6 on pool cstor-1234
Event:   🚀 Normal BlockDevice Replacement Replacement of blockdevice-4 BlockDevice with blockdevice-6 BlockDevice is in-Progress
Event:   🚀 Normal BlockDevice Replacement Resilvering is successfull on BlockDevice blockdevice-6
E0511 10:49:23.736578    5956 exec-run.go:67] exit status 1
E0511 10:49:23.736593    5956 volumereplica.go:443] Unable to get volumes:
E0511 10:49:23.745415    5956 exec-run.go:67] exit status 1
E0511 10:49:23.745429    5956 volumereplica.go:443] Unable to get volumes:
=== RUN   TestCSPIBlockDeviceReplacement/Provision_Raidz_Pool_And_Replace_Both_Group_BlockDevices
I0511 10:49:23.746566    5956 handler.go:381] Added Finalizer: cspi-foo-raidz-replace, 
I0511 10:49:23.746593    5956 import.go:54] Importing pool  cstor-1234
E0511 10:49:23.746605    5956 import.go:85] Failed to import pool : no pools available to import : exit status 1
I0511 10:49:23.746617    5956 create.go:41] Creating a pool for cspi-foo-raidz-replace cstor-1234
Event:   🚀 Normal Created Pool created successfully
E0511 10:49:23.759554    5956 exec-run.go:67] exit status 1
E0511 10:49:23.759568    5956 volumereplica.go:443] Unable to get volumes:
I0511 10:49:23.760207    5956 disk_operation.go:259] Triggered replacement of /dev/disk/by-id/ata-WDC_WD10JPVX-44ts-WXG1A-9 with /dev/disk/by-id/ata-WDC_WD10JPVX-4vsz-WXG1A-19 on pool cstor-1234
Event:   🚀 Normal BlockDevice Replacement Replacement of blockdevice-9 BlockDevice with blockdevice-19 BlockDevice is in-Progress
Event:   🚀 Normal BlockDevice Replacement Resilvering is successfull on BlockDevice blockdevice-19
I0511 10:49:23.760922    5956 disk_operation.go:259] Triggered replacement of /dev/disk/by-id/ata-WDC_WD10JPVX-mg6n-WXG1A-12 with /dev/disk/by-id/ata-WDC_WD10JPVX-8xkm-WXG1A-20 on pool cstor-1234
Event:   🚀 Normal BlockDevice Replacement Replacement of blockdevice-12 BlockDevice with blockdevice-20 BlockDevice is in-Progress
Event:   🚀 Normal BlockDevice Replacement Resilvering is successfull on BlockDevice blockdevice-20
I0511 10:49:23.761484    5956 disk_operation.go:259] Triggered replacement of /dev/disk/by-id/ata-WDC_WD10JPVX-bfcr-WXG1A-14 with /dev/disk/by-id/ata-WDC_WD10JPVX-stpl-WXG1A-21 on pool cstor-1234
Event:   🚀 Normal BlockDevice Replacement Replacement of blockdevice-14 BlockDevice with blockdevice-21 BlockDevice is in-Progress
Event:   🚀 Normal BlockDevice Replacement Resilvering is successfull on BlockDevice blockdevice-21
I0511 10:49:23.761867    5956 disk_operation.go:259] Triggered replacement of /dev/disk/by-id/ata-WDC_WD10JPVX-qqds-WXG1A-15 with /dev/disk/by-id/ata-WDC_WD10JPVX-m4vl-WXG1A-22 on pool cstor-1234
Event:   🚀 Normal BlockDevice Replacement Replacement of blockdevice-15 BlockDevice with blockdevice-22 BlockDevice is in-Progress
Event:   🚀 Normal BlockDevice Replacement Resilvering is successfull on BlockDevice blockdevice-22
I0511 10:49:23.762277    5956 disk_operation.go:259] Triggered replacement of /dev/disk/by-id/ata-WDC_WD10JPVX-jv7z-WXG1A-16 with /dev/disk/by-id/ata-WDC_WD10JPVX-7kd7-WXG1A-23 on pool cstor-1234
Event:   🚀 Normal BlockDevice Replacement Replacement of blockdevice-16 BlockDevice with blockdevice-23 BlockDevice is in-Progress
Event:   🚀 Normal BlockDevice Replacement Resilvering is successfull on BlockDevice blockdevice-23
E0511 10:49:23.771883    5956 exec-run.go:67] exit status 1
E0511 10:49:23.771897    5956 volumereplica.go:443] Unable to get volumes:
E0511 10:49:23.782559    5956 exec-run.go:67] exit status 1
E0511 10:49:23.782573    5956 volumereplica.go:443] Unable to get volumes:
--- PASS: TestCSPIBlockDeviceReplacement (0.07s)
    --- PASS: TestCSPIBlockDeviceReplacement/Provision_Mirror_Pool_And_Replace_BlockDevice (0.03s)
    --- PASS: TestCSPIBlockDeviceReplacement/Provision_Raidz_Pool_And_Replace_Both_Group_BlockDevices (0.04s)
=== RUN   TestUpdateStatusConditionEventually
=== RUN   TestUpdateStatusConditionEventually/When_there_are_multiple_conditions_and_status_got_updated_to_true
=== RUN   TestUpdateStatusConditionEventually/When_there_are_no_condition_condition_new_condition_should_be_addded
=== RUN   TestUpdateStatusConditionEventually/When_existing_condition_message_in_CSPI_got_updated
=== RUN   TestUpdateStatusConditionEventually/When_new_condition_got_added
=== RUN   TestUpdateStatusConditionEventually/When_we_not_able_to_fetch_the_object_from_etcd
E0511 10:49:23.786230    5956 status.go:41] failed to update CSPI cstor-pool-4 status with condition PoolExpansion will retry 3 times at 2s interval: {failed to update cspi conditions: cstorpoolinstances.cstor.openebs.io "cstor-pool-4" not found}
--- PASS: TestUpdateStatusConditionEventually (0.00s)
    --- PASS: TestUpdateStatusConditionEventually/When_there_are_multiple_conditions_and_status_got_updated_to_true (0.00s)
    --- PASS: TestUpdateStatusConditionEventually/When_there_are_no_condition_condition_new_condition_should_be_addded (0.00s)
    --- PASS: TestUpdateStatusConditionEventually/When_existing_condition_message_in_CSPI_got_updated (0.00s)
    --- PASS: TestUpdateStatusConditionEventually/When_new_condition_got_added (0.00s)
    --- PASS: TestUpdateStatusConditionEventually/When_we_not_able_to_fetch_the_object_from_etcd (0.00s)
PASS
ok  	github.com/openebs/cstor-operators/pkg/controllers/cspi-controller	1.662s

[sonasingh46]

Few improvements as discussed will be done as part of next coming PRs.