Allow adding disks to vm provision via api and automation #13318

masayag · 2016-12-26T13:25:54Z

Restful AP:

Provisioning of RHV vm via provision restful request is extended
to support disks creation.

The request format is detailed below:
diskscsi##=<controller_#>:<controller_position_#>:<size_in_mb> - The controller number and position are ignored for RHV.
diskscsi##.datastore=<datastore name>

Optional:
diskscsi##.filename = <filename>
diskscsi##.bootable = <true/false> Default: false
diskscsi##.backing.thinProvisioned = <true/false> Default: false
diskscsi##.interface = <VIRTIO/IDE/VIRTIO_SCSI/SPAPR_VSCSI> Default: VIRTIO

An example of the disks section as part of the provision request:

"vm_fields" : {
...
"diskscsi1" : "0:0:200",
"diskscsi1.datastore" : "data-40",
"diskscsi1.backing.thinProvisioned" : "true",
"diskscsi1.bootable" : "true",
"diskscsi1.filename" : "prov_disk_1",
"diskscsi1.interface": "IDE",
}

The result is creating a bootable disk named 'prov_disk_1' on datastore 'data-40'
with size of 200MB, thinly provisioned and controlled via 'IDE' driver.

Automation:

Same support is added for the automation as well to allow adding disks
to vm.

The signature of adding disk to a vm is exposed via
MiqAeServiceManageIQ_Providers_Redhat_InfraManager_Vm#add_disk(disk_name, disk_size_mb, options = {})

Where the 'options' expects:
thin_provisioned - <true/false> Default: false
bootable - <true/false> Default: false
datastore - the storage name to create the disk on. If not specified, the VM's storage is used.
interface - <VIRTIO/IDE/VIRTIO_SCSI/SPAPR_VSCSI> Default: VIRTIO

https://bugzilla.redhat.com/show_bug.cgi?id=1123068

masayag · 2016-12-26T13:27:17Z

@miq-bot add_label providers/rhevm

masayag · 2016-12-26T13:28:58Z

@miq-bot add_label automate

masayag · 2016-12-26T13:29:08Z

@borod108 please review

borod108

I think its good, nice test coverage. Please look at the builder comment, except for that I think its good to go.

borod108 · 2016-12-26T15:16:55Z

app/models/manageiq/providers/redhat/infra_manager.rb

-    }
+  # prepare disk attachment request payload of adding disk for reconfigure vm
+  def prepare_disk(disk_spec, storage)
+    disk_attachment_builder = DiskAttachmentBuilder.new


I think you should just pass options to the attachment builder (in the initializer) and encapsulate dealing with them there

done - you're right. what's the point on having options in c'tor if not using them...

borod108 · 2016-12-26T15:17:33Z

app/models/manageiq/providers/redhat/infra_manager.rb

+    storage = options[:datastore] || vm.storage
+    raise _("Data Store does not exist, unable to add disk") unless storage
+
+    disk_attachment_builder = DiskAttachmentBuilder.new


I think you should just pass options to the attachment builder (in the initializer) and encapsulate dealing with them there

borod108 · 2016-12-26T15:18:31Z

app/models/manageiq/providers/redhat/infra_manager/provision/disk.rb

+      raise MiqException::MiqProvisionError, "Unable to find storage: <#{storage_name}> for disk: <#{disk_spec.inspect}>"
+    end
+
+    disk_attachment_builder = ManageIQ::Providers::Redhat::InfraManager::DiskAttachmentBuilder.new


masayag · 2016-12-27T10:34:04Z

@durandom @gmcculloug could you review ?

durandom

@gmcculloug can you or somebody from your team have a look at the automate/statemachine part?

Besides some minor questions this LGTM, and 👍 for adding many specs.

Maybe you want to revisit them though and check if you can use FactoryGirl.build instead of create in some places to save some DB calls

durandom · 2017-01-03T11:43:57Z

app/models/manageiq/providers/redhat/infra_manager/provision/disk.rb

+      disks = system_service.vms_service.vm_service(destination.uid_ems).disk_attachments_service.list
+      disks.each do |disk|
+        fetched_disk = system_service.disks_service.disk_service(disk.id).get
+        return true unless fetched_disk&.status == "ok"


please use fetched_disk.try(:status) until we made the switch to 2.3

durandom · 2017-01-03T11:47:01Z

app/models/manageiq/providers/redhat/infra_manager.rb

+  # prepare disk attachment request payload of adding disk for reconfigure vm
+  def prepare_disk(disk_spec, storage)
+    da_options = {
+      :size_in_mb       => disk_spec["disk_size_in_mb"],


is disk_spec containing symbols and strings as keys? Maybe call symbolize_keys on it?

durandom · 2017-01-03T11:50:14Z

app/models/manageiq/providers/redhat/infra_manager.rb

@@ -103,24 +103,44 @@ def vm_reconfigure(vm, options = {})
  end

  def add_disks(add_disks_spec, vm)
-    ems_storage_uid = add_disks_spec["ems_storage_uid"]


how about backwards compatablity of this method?
I'm not sure from where it get's called though

This method was added recently for the sake of reconfigure vm - the disk spec is created internally only from app/models/manageiq/providers/redhat/infra_manager/vm/reconfigure.rb:

def build_config_spec(task_options) { "numCoresPerSocket" => (task_options[:cores_per_socket].to_i if task_options[:cores_per_socket]), "memoryMB" => (task_options[:vm_memory].to_i if task_options[:vm_memory]), "numCPUs" => (task_options[:number_of_cpus].to_i if task_options[:number_of_cpus]), "disksRemove" => task_options[:disk_remove], "disksAdd" => (spec_for_added_disks(task_options[:disk_add]) if task_options[:disk_add]) } end

Therefore this is a legit spec change, without any risk to backward compatibility

durandom · 2017-01-03T11:58:25Z

app/models/manageiq/providers/redhat/infra_manager/disk_attachment_builder.rb

+  end
+
+  def self.true?(obj)
+    obj.to_s == "true"


is this validating text input? This feels non ruby idiomatic to me.
The ? methods are mostly inquiring the object it is called upon. So something like @thin_provisioned.true?

Would present? work here?

present? won't work since the value might be string 'true' or boolean true or false/'false'.
I'll suggest an alternative on the next patch to comply with @thin_provisioned.true? / @bootable.true?

durandom · 2017-01-03T12:03:07Z

app/models/manageiq/providers/redhat/infra_manager/disk_attachment_builder.rb

+  end
+
+  def self.mb_to_gb(size)
+    size.to_i * 1024 * 1024


maybe you can leverage http://api.rubyonrails.org/classes/Numeric.html#method-i-megabyte and inline that method?

durandom · 2017-01-03T12:03:57Z

app/models/manageiq/providers/redhat/infra_manager/disk_attachment_builder.rb

+    size.to_i * 1024 * 1024
+  end
+
+  FILE_STORAGE_TYPE = %w(NFS GLUSTERFS VMFS).freeze


if its just for lookup I tend to %w(A B).to_set.freeze

durandom · 2017-01-03T12:10:25Z

app/models/manageiq/providers/redhat/infra_manager/provision/disk.rb

+      signal :customize_guest
+    else
+      add_disks(requested_disks)
+      signal :poll_add_disks_complete


@gmcculloug not sure, but shouldnt the signals be consistent across providers? Or somehow defined at a base class?

No, this does not need to be consistent across providers. VMware supports adding disks which happens when the initial config spec is built. Here it is happening as a post-creations step. If we add support for another provider we could possible refactor this logic then.

@masayag I have two suggests:

Moving this method to the state_machine.rb file to keep the signal logic together.

Mark most of the methods in this file as private.

durandom · 2017-01-03T12:16:01Z

...ion_engine/service_methods/miq_ae_service_manageiq-providers-redhat-infra_manager-vm_spec.rb

+    let(:service_vm) { MiqAeMethodService::MiqAeServiceManageIQ_Providers_Redhat_InfraManager_Vm.find(vm.id) }
+
+    before do
+      allow_any_instance_of(Vm).to receive(:my_zone).and_return('default')


Can you get rid of allow_any_instance_of and only mock vm and service_vm if needed?

masayag

I replaced as much FactoryGirl.create with FactoryGirl.build as I could.
I've addressed the comments. Thanks for the review.

masayag · 2017-01-04T08:23:18Z

app/models/manageiq/providers/redhat/infra_manager.rb

@@ -103,24 +103,44 @@ def vm_reconfigure(vm, options = {})
  end

  def add_disks(add_disks_spec, vm)
-    ems_storage_uid = add_disks_spec["ems_storage_uid"]


This method was added recently for the sake of reconfigure vm - the disk spec is created internally only from app/models/manageiq/providers/redhat/infra_manager/vm/reconfigure.rb:

def build_config_spec(task_options) { "numCoresPerSocket" => (task_options[:cores_per_socket].to_i if task_options[:cores_per_socket]), "memoryMB" => (task_options[:vm_memory].to_i if task_options[:vm_memory]), "numCPUs" => (task_options[:number_of_cpus].to_i if task_options[:number_of_cpus]), "disksRemove" => task_options[:disk_remove], "disksAdd" => (spec_for_added_disks(task_options[:disk_add]) if task_options[:disk_add]) } end

Therefore this is a legit spec change, without any risk to backward compatibility

masayag · 2017-01-04T08:33:10Z

app/models/manageiq/providers/redhat/infra_manager.rb

+  # prepare disk attachment request payload of adding disk for reconfigure vm
+  def prepare_disk(disk_spec, storage)
+    da_options = {
+      :size_in_mb       => disk_spec["disk_size_in_mb"],


masayag · 2017-01-04T09:08:34Z

app/models/manageiq/providers/redhat/infra_manager/disk_attachment_builder.rb

+  end
+
+  def self.true?(obj)
+    obj.to_s == "true"


present? won't work since the value might be string 'true' or boolean true or false/'false'.
I'll suggest an alternative on the next patch to comply with @thin_provisioned.true? / @bootable.true?

masayag · 2017-01-04T09:16:23Z

app/models/manageiq/providers/redhat/infra_manager/disk_attachment_builder.rb

+  end
+
+  def self.mb_to_gb(size)
+    size.to_i * 1024 * 1024


masayag · 2017-01-04T09:17:04Z

app/models/manageiq/providers/redhat/infra_manager/disk_attachment_builder.rb

+    size.to_i * 1024 * 1024
+  end
+
+  FILE_STORAGE_TYPE = %w(NFS GLUSTERFS VMFS).freeze


masayag · 2017-01-04T09:18:52Z

app/models/manageiq/providers/redhat/infra_manager/provision/disk.rb

+      disks = system_service.vms_service.vm_service(destination.uid_ems).disk_attachments_service.list
+      disks.each do |disk|
+        fetched_disk = system_service.disks_service.disk_service(disk.id).get
+        return true unless fetched_disk&.status == "ok"


masayag · 2017-01-04T09:26:40Z

...ion_engine/service_methods/miq_ae_service_manageiq-providers-redhat-infra_manager-vm_spec.rb

+    let(:service_vm) { MiqAeMethodService::MiqAeServiceManageIQ_Providers_Redhat_InfraManager_Vm.find(vm.id) }
+
+    before do
+      allow_any_instance_of(Vm).to receive(:my_zone).and_return('default')


gmcculloug · 2017-01-04T13:55:40Z

app/models/manageiq/providers/redhat/infra_manager/provision/disk.rb

+      signal :customize_guest
+    else
+      add_disks(requested_disks)
+      signal :poll_add_disks_complete


No, this does not need to be consistent across providers. VMware supports adding disks which happens when the initial config spec is built. Here it is happening as a post-creations step. If we add support for another provider we could possible refactor this logic then.

@masayag I have two suggests:

Moving this method to the state_machine.rb file to keep the signal logic together.

Mark most of the methods in this file as private.

gmcculloug · 2017-01-04T13:56:23Z

app/models/manageiq/providers/redhat/infra_manager/provision/disk.rb

+    false
+  end
+
+  def poll_add_disks_complete


This method should also move to the state_machine.rb file.

Restful AP: ---------- Provisioning of RHV vm via provision restful request is extended to support disks creation. The request format is detailed below: diskscsi##=<controller_#>:<controller_position_#>:<size_in_mb> - The controller number and position are ignored for RHV. diskscsi##.datastore=<datastore name> Optional: diskscsi##.filename = <filename> diskscsi##.bootable = <true/false> Default: false diskscsi##.backing.thinProvisioned = <true/false> Default: false diskscsi##.interface = <VIRTIO/IDE/VIRTIO_SCSI/SPAPR_VSCSI> Default: VIRTIO An example of the disks section as part of the provision request: "vm_fields" : { ... "diskscsi1" : "0:0:200", "diskscsi1.datastore" : "data-40", "diskscsi1.backing.thinProvisioned" : "true", "diskscsi1.bootable" : "true", "diskscsi1.filename" : "prov_disk_1", "diskscsi1.interface": "IDE", } The result is creating a bootable disk named 'prov_disk_1' on datastore 'data-40' with size of 200MB, thinly provisioned and controlled via 'IDE' driver. Automation: ---------- Same support is added for the automation as well to allow adding disks to vm. The signature of adding disk to a vm is exposed via MiqAeServiceManageIQ_Providers_Redhat_InfraManager_Vm#add_disk(disk_name, disk_size_mb, options = {}) Where the 'options' expects: thin_provisioned - <true/false> Default: false bootable - <true/false> Default: false datastore - the storage name to create the disk on. If not specified, the VM's storage is used. interface - <VIRTIO/IDE/VIRTIO_SCSI/SPAPR_VSCSI> Default: VIRTIO

miq-bot · 2017-01-10T15:11:50Z

Checked commit masayag@c07d1d3 with ruby 2.2.5, rubocop 0.37.2, and haml-lint 0.16.1
15 files checked, 3 offenses detected

app/models/manageiq/providers/redhat/infra_manager/provision/disk.rb

❕ - Line 44, Col 121 - Metrics/LineLength - Line is too long. [121/120]

spec/lib/miq_automation_engine/service_methods/miq_ae_service_manageiq-providers-redhat-infra_manager-vm_spec.rb

❗ - Line 1, Col 1 - Style/FileName - Use snake_case for source file names.
❗ - Line 1, Col 8 - Style/ClassAndModuleCamelCase - Use CamelCase for classes and modules.

masayag · 2017-01-10T15:36:36Z

@gmcculloug I've incorporated your comments into the PR, could you have a look ?

I verified this via provision request, vm reconfigure via the UI and via automation request.

durandom

looks good from the provider side

@gmcculloug merge at your discretion

masayag · 2017-01-12T07:39:23Z

@miq-bot add_label euwe/yes

simaishi · 2017-01-18T21:44:03Z

@masayag Can you please create a BZ if it doesn't already exist?

masayag · 2017-01-19T07:22:35Z

@simaishi bug url added to commit message

https://bugzilla.redhat.com/show_bug.cgi?id=1123068

…n_vm_for_rhevm Allow adding disks to vm provision via api and automation (cherry picked from commit b1ded2f) https://bugzilla.redhat.com/show_bug.cgi?id=1414893

simaishi · 2017-01-19T17:03:14Z

Euwe backport details:

$ git log -1
commit 49322437d9d7a24735c67c944496bc8e900a9d0e
Author: Greg McCullough <gmccullo@redhat.com>
Date:   Tue Jan 17 09:53:30 2017 -0500

    Merge pull request #13318 from masayag/add_disks_via_restapi_provision_vm_for_rhevm
    
    Allow adding disks to vm provision via api and automation
    (cherry picked from commit b1ded2f1c7292627505555c8c4637c98eb67e76a)
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1414893

PR ManageIQ#13318 introduced a new class to handle disk attachments. A disk should be added to RHV as 'active' since there is no method to modify it afterward. DiskAttachmentBuilder class will set the 'active' as the default status of the new disk, so any scenario which adds disks (vm reconfigure, automation request, provision request) will share the same behavior. https://bugzilla.redhat.com/show_bug.cgi?id=1422145

psachin · 2017-03-02T14:20:36Z

Hello @masayag

What am I missing here?

irb(main):001:0> s = Vm.find_by(:name => "psachin-ext")
irb(main):005:0> s.add_disk("disk2", 2 * 1024, {:thin_provisioned => true, :datastore => s.storage_name})
NoMethodError: undefined method `vm_add_disk' for #<ManageIQ::Providers::Redhat::InfraManager:0x0000000c84d030>
	from /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activemodel-5.0.0.1/lib/active_model/attribute_methods.rb:433:in `method_missing'
	from /var/www/miq/vmdb/app/models/vm_or_template.rb:330:in `run_command_via_parent'
	from /var/www/miq/vmdb/app/models/vm_or_template/operations/configuration.rb:76:in `raw_add_disk'
	from /var/www/miq/vmdb/app/models/vm_or_template/operations/configuration.rb:81:in `add_disk'
	from (irb):5
	from /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/railties-5.0.0.1/lib/rails/commands/console.rb:65:in `start'
	from /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/railties-5.0.0.1/lib/rails/commands/console_helper.rb:9:in `start'
	from /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/railties-5.0.0.1/lib/rails/commands/commands_tasks.rb:78:in `console'
	from /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/railties-5.0.0.1/lib/rails/commands/commands_tasks.rb:49:in `run_command!'
	from /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/railties-5.0.0.1/lib/rails/commands.rb:18:in `<top (required)>'
	from bin/rails:4:in `require'
	from bin/rails:4:in `<main>'

gmcculloug · 2017-03-02T20:17:15Z

@psachin My guess is you are not running with these PR changes. The vm_add_disk method was added to the ManageIQ::Providers::Redhat::InfraManager object in this PR.

psachin · 2017-03-03T05:34:48Z

@gmcculloug Thanks. You were right, the changes were not merged. This PR has made adding additional disk to RHV so much easy using button.

👍 @masayag
👍 @gmcculloug

miq-bot added the providers/rhevm label Dec 26, 2016

miq-bot added the automate label Dec 26, 2016

borod108 reviewed Dec 26, 2016

View reviewed changes

masayag force-pushed the add_disks_via_restapi_provision_vm_for_rhevm branch from c0291cc to f6a9e41 Compare December 26, 2016 16:00

borod108 approved these changes Dec 27, 2016

View reviewed changes

durandom suggested changes Jan 3, 2017

View reviewed changes

chessbyte assigned gmcculloug Jan 3, 2017

masayag commented Jan 4, 2017

View reviewed changes

masayag force-pushed the add_disks_via_restapi_provision_vm_for_rhevm branch 2 times, most recently from eefaf91 to 41d8a9a Compare January 4, 2017 09:56

gmcculloug reviewed Jan 4, 2017

View reviewed changes

masayag force-pushed the add_disks_via_restapi_provision_vm_for_rhevm branch from 41d8a9a to c07d1d3 Compare January 10, 2017 15:06

durandom approved these changes Jan 11, 2017

View reviewed changes

miq-bot added the euwe/yes label Jan 12, 2017

gmcculloug merged commit b1ded2f into ManageIQ:master Jan 17, 2017

gmcculloug added this to the Sprint 53 Ending Jan 30, 2017 milestone Jan 17, 2017

simaishi added the bugzilla needed label Jan 18, 2017

simaishi removed the bugzilla needed label Jan 19, 2017

simaishi added euwe/backported and removed euwe/yes labels Jan 19, 2017

masayag mentioned this pull request Feb 14, 2017

Disks should be added as 'active' in RHV #13913

Merged

miq-bot added the euwe/yes label Mar 2, 2017

simaishi removed the euwe/yes label Mar 2, 2017

masayag deleted the add_disks_via_restapi_provision_vm_for_rhevm branch June 27, 2018 20:30

Fryguy added the providers/infrastructure label Jan 24, 2020

Allow adding disks to vm provision via api and automation #13318

Allow adding disks to vm provision via api and automation #13318

Conversation

masayag commented Dec 26, 2016 • edited Loading

Restful AP:

Automation:

masayag commented Dec 26, 2016

masayag commented Dec 26, 2016

masayag commented Dec 26, 2016

borod108 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

masayag commented Dec 27, 2016

durandom left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

masayag left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

miq-bot commented Jan 10, 2017

masayag commented Jan 10, 2017

durandom left a comment

Choose a reason for hiding this comment

masayag commented Jan 12, 2017

simaishi commented Jan 18, 2017

masayag commented Jan 19, 2017

simaishi commented Jan 19, 2017

psachin commented Mar 2, 2017

gmcculloug commented Mar 2, 2017

psachin commented Mar 3, 2017

masayag commented Dec 26, 2016 •

edited

Loading