Add methods to run Terraform Templates using TerraformRunner container APIs

[putmanoj]

• Add run method to run terraform templates in async mode
• Add run_async method to run terraform templates
• Response class to represent TerraformRunner-Stack API response
• ResponseAsync class with stop method to cancel a TerraformRunner-Stack API Job.
• Add spec the test terraform-runner methods

[agrare]

@putmanoj please rebase on master rather than merge master into this branch.

[putmanoj]

@putmanoj please rebase on master rather than merge master into this branch.

ok

[agrare]

@putmanoj you can add this to your ~/.gitconfig

[pull]
	rebase = true

That way git pull upstream master will "just work" (NOTE this is covered in the developer setup guide)

[putmanoj]

~/.gitconfig

ok

[jrafanie]

Question: can the response body be empty/bad in the event of a 500 error or similar? I'm wondering if .body will always be able to be JSON.parse against.

[jrafanie]

do these get automatically converted to Date/time objects or should we do this too? Can these be nil?

[jrafanie]

@putmanoj what's the thinking here? Is this debugging? is it needed?

[agrare]

Yeah we could probably drop this method entirely and just go with an attr_reader :response

[jrafanie]

it's relying on a side effect of running? doing a refresh_response to initialize the variable. I'll see if I can move that here in a followup.

[jrafanie]

@putmanoj can you test a "successful" run of create_stack_job after my changes? I'm not sure how to provide credentials and other argument correctly. Thanks!

[jrafanie]

@putmanoj here's a listing of all my changes from your original PR before I started to now:

git diff origin/terraform-runner-backup putmanoj/terraform-runner

diff --git a/lib/terraform/runner.rb b/lib/terraform/runner.rb
index 172fc0a..ae2cd37 100644
--- a/lib/terraform/runner.rb
+++ b/lib/terraform/runner.rb
@@ -1,5 +1,4 @@
-require 'rest-client'
-require 'timeout'
+require 'faraday'
 require 'tempfile'
 require 'zip'
 require 'base64'
@@ -10,8 +9,8 @@ module Terraform
       def available?
         return @available if defined?(@available)
 
-        response = terraform_runner_client['api/terraformjobs/count'].get
-        @available = response.code == 200
+        response = terraform_runner_client.get('api/terraformjobs/count')
+        @available = response.status == 200
       rescue
         @available = false
       end
@@ -38,6 +37,10 @@ module Terraform
         Terraform::Runner::ResponseAsync.new(response.stack_id)
       end
 
+      # To simplify clients who may just call run, we alias it to call
+      # run_async.  If we ever need run_sync, we'll need to revisit this.
+      alias run run_async
+
       # Stop running terraform-runner job by stack_id
       #
       # @param stack_id [String] stack_id from the terraforn-runner job
@@ -47,27 +50,6 @@ module Terraform
         cancel_stack_job(stack_id)
       end
 
-      # Runs a template, waits until it terraform-runner job completes, via terraform-runner api
-      #
-      # @param input_vars [Hash] Hash with key/value pairs that will be passed as input variables to the
-      #        terraform-runner run
-      # @param template_path [String] Path to the template we will want to run
-      # @param tags [Hash] Hash with key/values pairs that will be passed as tags to the terraform-runner run
-      # @param credentials [Array] List of Authentication object ids to provide to the terraform run
-      # @param env_vars [Hash] Hash with key/value pairs that will be passed as environment variables to the
-      #        terraform-runner run
-      # @return [Terraform::Runner::Response] Response object with final result of terraform run
-      def run(input_vars, template_path, tags: nil, credentials: [], env_vars: {})
-        _log.debug("Run template: #{template_path}")
-        create_stack_job_and_wait_until_completes(
-          template_path,
-          :input_vars  => input_vars,
-          :tags        => tags,
-          :credentials => credentials,
-          :env_vars    => env_vars
-        )
-      end
-
       # Fetch terraform-runner job result/status by stack_id
       #
       # @param stack_id [String] stack_id from the terraforn-runner job
@@ -83,7 +65,7 @@ module Terraform
       private
 
       def server_url
-        ENV['TERRAFORM_RUNNER_URL'] || 'https://opentofu-runner:27000'
+        ENV.fetch('TERRAFORM_RUNNER_URL', 'https://opentofu-runner:27000')
       end
 
       def server_token
@@ -91,15 +73,11 @@ module Terraform
       end
 
       def stack_job_interval_in_secs
-        ENV['TERRAFORM_RUNNER_STACK_JOB_CHECK_INTERVAL'].to_i
-      rescue
-        10
+        ENV.fetch('TERRAFORM_RUNNER_STACK_JOB_CHECK_INTERVAL', 10).to_i
       end
 
       def stack_job_max_time_in_secs
-        ENV['TERRAFORM_RUNNER_STACK_JOB_MAX_TIME'].to_i
-      rescue
-        120
+        ENV.fetch('TERRAFORM_RUNNER_STACK_JOB_MAX_TIME', 120).to_i
       end
 
       # Create to paramaters as used by terraform-runner api
@@ -108,33 +86,37 @@ module Terraform
       #        terraform-runner run
       # @return [Array] Array of {:name,:value}
       def convert_to_cam_parameters(vars)
-        parameters = []
-        vars&.each do |key, value|
-          parameters.push(
-            {
-              :name  => key,
-              :value => value
-            }
-          )
+        return [] if vars.nil?
+
+        vars.map do |key, value|
+          {
+            :name  => key,
+            :value => value
+          }
         end
-        parameters
       end
 
       # create http client for terraform-runner rest-api
       def terraform_runner_client
-        # TODO: verify ssl
-        verify_ssl = false
-
-        RestClient::Resource.new(
-          server_url,
-          :headers    => {:authorization => "Bearer #{server_token}"},
-          :verify_ssl => verify_ssl
-        )
+        @terraform_runner_client ||= begin
+          # TODO: verify ssl
+          verify_ssl = false
+
+          Faraday.new(
+            :url => server_url,
+            :ssl => {:verify => verify_ssl}
+          ) do |builder|
+            builder.request(:authorization, 'Bearer', -> { server_token })
+          end
+        end
       end
 
       def stack_tenant_id
-        # TODO: fix hardcoded tenant_id
-        'c158d710-d91c-11ed-9fee-d93323035b4e'
+        '00000000-0000-0000-0000-000000000000'.freeze
+      end
+
+      def json_post_arguments(payload)
+        return JSON.generate(payload), "Content-Type" => "application/json".freeze
       end
 
       # Create TerraformRunner Stack Job
@@ -148,45 +130,32 @@ module Terraform
       )
         _log.info("start stack_job for template: #{template_path}")
         tenant_id = stack_tenant_id
-
-        # Temp Zip File
-        # zip_file_path = Tempfile.new(%w/tmp .zip/).path
-        zip_file_path = Tempfile.new(%w[tmp .zip]).path
-        create_zip_file_from_directory(zip_file_path, template_path)
-        zip_file_hash = Base64.encode64(File.binread(zip_file_path))
+        encoded_zip_file = encoded_zip_from_directory(template_path)
 
         # TODO: use tags,env_vars
-        payload = JSON.generate(
-          {
-            :cloud_providers => credentials,
-            :name            => name,
-            :tenantId        => tenant_id,
-            :templateZipFile => zip_file_hash,
-            :parameters      => convert_to_cam_parameters(input_vars)
-          }
-        )
+        payload = {
+          :cloud_providers => credentials,
+          :name            => name,
+          :tenantId        => tenant_id,
+          :templateZipFile => encoded_zip_file,
+          :parameters      => convert_to_cam_parameters(input_vars)
+        }
         # _log.debug("Payload:>\n, #{payload}")
-        http_response = terraform_runner_client['api/stack/create'].post(
-          payload, :content_type => 'application/json'
+
+        http_response = terraform_runner_client.post(
+          "api/stack/create",
+          *json_post_arguments(payload)
         )
         _log.debug("==== http_response.body: \n #{http_response.body}")
         _log.info("stack_job for template: #{template_path} running ...")
         Terraform::Runner::Response.parsed_response(http_response)
-      ensure
-        # cleanup temp zip file
-        FileUtils.rm_rf(zip_file_path) if zip_file_path
-        _log.debug("Deleted #{zip_file_path}")
       end
 
       # Retrieve TerraformRunner Stack Job details
       def retrieve_stack_job(stack_id)
-        payload = JSON.generate(
-          {
-            :stack_id => stack_id
-          }
-        )
-        http_response = terraform_runner_client['api/stack/retrieve'].post(
-          payload, :content_type => 'application/json'
+        http_response = terraform_runner_client.post(
+          "api/stack/retrieve",
+          *json_post_arguments({:stack_id => stack_id})
         )
         _log.info("==== Retrieve Stack Response: \n #{http_response.body}")
         Terraform::Runner::Response.parsed_response(http_response)
@@ -194,99 +163,30 @@ module Terraform
 
       # Cancel/Stop running TerraformRunner Stack Job
       def cancel_stack_job(stack_id)
-        payload = JSON.generate(
-          {
-            :stack_id => stack_id
-          }
-        )
-        http_response = terraform_runner_client['api/stack/cancel'].post(
-          payload, :content_type => 'application/json'
+        http_response = terraform_runner_client.post(
+          "api/stack/cancel",
+          *json_post_arguments({:stack_id => stack_id})
         )
         _log.info("==== Cancel Stack Response: \n #{http_response.body}")
         Terraform::Runner::Response.parsed_response(http_response)
       end
 
-      # Wait for TerraformRunner Stack Job to complete
-      def wait_until_completes(stack_id)
-        interval_in_secs = stack_job_interval_in_secs
-        max_time_in_secs = stack_job_max_time_in_secs
-
-        response = nil
-        Timeout.timeout(max_time_in_secs) do
-          _log.debug("Starting wait for terraform-runner/stack/#{stack_id} completes ...")
-          i = 0
-          loop do
-            _log.debug("loop #{i}")
-            i += 1
-
-            response = retrieve_stack_job(stack_id)
-
-            _log.info("status: #{response.status}")
-
-            case response.status
-            when "SUCCESS"
-              _log.debug("Successful! (stack_job/:#{stack_id})")
-              break
-
-            when "FAILED"
-              _log.info("Failed! (stack_job/:#{stack_id} fails!)")
-              _log.info(response.error_message)
-              break
-
-            when nil
-              _log.info("No status! stack_job/:#{stack_id} must have failed, check response ...")
-              _log.info(response.message)
-              break
-            end
-            _log.info("============\n stack_job/:#{stack_id} status=#{response.status} \n============")
-
-            # sleep interval
-            _log.debug("Sleep for #{interval_in_secs} secs")
-            sleep interval_in_secs
-
-            break unless i < 20
-          end
-          _log.debug("loop ends: ran #{i} times")
-        end
-        response
-      end
-
-      # Create TerraformRunner Stack Job, wait until completes
-      def create_stack_job_and_wait_until_completes(
-        template_path,
-        input_vars: [],
-        tags: nil,
-        credentials: [],
-        env_vars: {},
-        name: "stack-#{rand(36**8).to_s(36)}"
-      )
-        _log.info("create_stack_job_and_wait_until_completes for #{template_path}")
-        response = create_stack_job(
-          template_path,
-          :input_vars  => input_vars,
-          :tags        => tags,
-          :credentials => credentials,
-          :env_vars    => env_vars,
-          :name        => name
-        )
-        wait_until_completes(response.stack_id)
-      end
-
-      # create zip from directory
-      def create_zip_file_from_directory(zip_file_path, template_path)
+      # encode zip of a template directory
+      def encoded_zip_from_directory(template_path)
         dir_path = template_path # directory to be zipped
         dir_path = path[0...-1] if dir_path.end_with?('/')
 
-        _log.debug("Create #{zip_file_path}")
-        Zip::File.open(zip_file_path, Zip::File::CREATE) do |zipfile|
-          Dir.chdir(dir_path)
-          Dir.glob("**/*").reject { |fn| File.directory?(fn) }.each do |file|
-            _log.debug("Adding #{file}")
-            zipfile.add(file.sub("#{dir_path}/", ''), file)
+        Tempfile.create(%w[opentofu-runner-payload .zip]) do |zip_file_path|
+          _log.debug("Create #{zip_file_path}")
+          Zip::File.open(zip_file_path, Zip::File::CREATE) do |zipfile|
+            Dir.chdir(dir_path)
+            Dir.glob("**/*").select { |fn| File.file?(fn) }.each do |file|
+              _log.debug("Adding #{file}")
+              zipfile.add(file.sub("#{dir_path}/", ''), file)
+            end
           end
+          Base64.encode64(File.binread(zip_file_path))
         end
-
-        zip_file_path
       end
     end
   end
diff --git a/lib/terraform/runner/response_async.rb b/lib/terraform/runner/response_async.rb
index 7b62fae..f8b3c9c 100644
--- a/lib/terraform/runner/response_async.rb
+++ b/lib/terraform/runner/response_async.rb
@@ -52,38 +52,15 @@ module Terraform
         @response
       end
 
-      # Dumps the Terraform::Runner::ResponseAsync into the hash
-      #
-      # @return [Hash] Dumped Terraform::Runner::ResponseAsync object
-      def dump
-        {
-          :stack_id => @stack_id
-        }
-      end
-
-      # Creates the Terraform::Runner::ResponseAsync object from hash data
-      #
-      # @param hash [Hash] Dumped Terraform::Runner::ResponseAsync object
-      #
-      # @return [Terraform::Runner::ResponseAsync] Terraform::Runner::ResponseAsync Object created from hash data
-      def self.load(hash)
-        # Dump dumps a hash and load accepts a hash, so we must expand the hash to kwargs as new expects kwargs
-        new(**hash)
-      end
-
       private
 
       def completed?(status)
-        # IF NOT SUCCESS,FAILED,CANCELLED
-        if status
-          return (
-            status.start_with?("SUCCESS", "FAILED") ||
-            # @response.status == "ERROR" ||
-            status == "CANCELLED"
-          )
+        case status.to_s.upcase
+        when "SUCCESS", "FAILED", "CANCELLED"
+          true
+        else
+          false
         end
-
-        false
       end
     end
   end
diff --git a/spec/lib/terraform/runner_spec.rb b/spec/lib/terraform/runner_spec.rb
index 2b23008..49ab330 100644
--- a/spec/lib/terraform/runner_spec.rb
+++ b/spec/lib/terraform/runner_spec.rb
@@ -21,89 +21,6 @@ RSpec.describe(Terraform::Runner) do
     end
   end
 
-  describe ".run hello-world with no input vars ( nil argument )" do
-    let(:input_vars) { nil }
-
-    before do
-      ENV["TERRAFORM_RUNNER_URL"] = "https://1.2.3.4:7000"
-
-      stub_request(:post, "https://1.2.3.4:7000/api/stack/create")
-        .with(:body => hash_including({:parameters => []}))
-        .to_return(
-          :status => 200,
-          :body   => @hello_world_create_response.to_json
-        )
-
-      stub_request(:post, "https://1.2.3.4:7000/api/stack/retrieve")
-        .with(:body => hash_including({:stack_id => @hello_world_retrieve_response['stack_id']}))
-        .to_return(
-          :status => 200,
-          :body   => @hello_world_retrieve_response.to_json
-        )
-    end
-
-    it "runs a hello-world terraform template" do
-      response = Terraform::Runner.run(input_vars, File.join(__dir__, "runner/data/hello-world"))
-
-      expect(response.status).to(eq('SUCCESS'), "terraform-runner failed with:\n#{response.status}")
-      expect(response.message).to(include('greeting = "Hello World"'))
-      expect(response.stack_id).to(eq(@hello_world_retrieve_response['stack_id']))
-      expect(response.action).to(eq('CREATE'))
-      expect(response.stack_name).to(eq(@hello_world_retrieve_response['stack_name']))
-      expect(response.details.keys).to(eq(%w[resources outputs]))
-    end
-  end
-
-  describe ".run hello-world with input_vars" do
-    let(:input_vars) { {:name => 'Mumbai'} }
-
-    def verify_request_and_respond(request)
-      body = JSON.parse(request.body)
-
-      # verify parameters
-      expect(body['parameters'].length).to(eq(1))
-      data = body['parameters'][0]
-      expect(data['name']).to(eq('name'))
-      expect(data['value']).to(eq('Mumbai'))
-
-      # verify other attributes
-      expect(body['name']).not_to(be_empty)
-      expect(body['tenantId']).not_to(be_empty)
-      expect(body['templateZipFile']).not_to(be_empty)
-
-      @hello_world_create_response.to_json
-    end
-
-    before do
-      ENV["TERRAFORM_RUNNER_URL"] = "https://1.2.3.4:7000"
-
-      stub_request(:post, "https://1.2.3.4:7000/api/stack/create")
-        .to_return(->(request) { {:body => verify_request_and_respond(request)} })
-
-      response = @hello_world_retrieve_response.clone
-      response['message'] = response['message'].gsub('World', 'Mumbai')
-      response['details']['outputs'][0]['value'] = response['details']['outputs'][0]['value'].sub('World', 'Mumbai')
-
-      stub_request(:post, "https://1.2.3.4:7000/api/stack/retrieve")
-        .with(:body => hash_including({:stack_id => @hello_world_retrieve_response['stack_id']}))
-        .to_return(
-          :status => 200,
-          :body   => response.to_json
-        )
-    end
-
-    it "runs a hello-world terraform template" do
-      response = Terraform::Runner.run(input_vars, File.join(__dir__, "runner/data/hello-world"))
-
-      expect(response.status).to(eq('SUCCESS'), "terraform-runner failed with:\n#{response.status}")
-      expect(response.message).to(include('greeting = "Hello Mumbai"'))
-      expect(response.stack_id).to(eq(@hello_world_retrieve_response['stack_id']))
-      expect(response.action).to(eq('CREATE'))
-      expect(response.stack_name).to(eq(@hello_world_retrieve_response['stack_name']))
-      expect(response.details.keys).to(eq(%w[resources outputs]))
-    end
-  end
-
   context '.run_async hello-world' do
     describe '.run_async' do
       create_stub = nil
@@ -142,7 +59,10 @@ RSpec.describe(Terraform::Runner) do
         expect(response.stack_name).to(eq(@hello_world_create_response['stack_name']))
         expect(response.message).to(be_nil)
         expect(response.details).to(be_nil)
+      end
 
+      it "is aliased as run" do
+        expect(Terraform::Runner.method(:run)).to eq(Terraform::Runner.method(:run_async))
       end
     end

[agrare]

Not for this PR, but we have a plugin logger ($embedded_terraform_log) which is typically used for api client logging we could probably use that when we're just dumping the request / response to the logs

[agrare]

Really minor but the "return" of an assignment is the value anyway so you don't need this

Suggested change

	@response

[agrare]

You could do completed?(@response&.status) since completed? would handle a nil and return false

[agrare]

Couple of minor follow-up comments but overall looks good

[Fryguy]

Super-tiny-minor, but do we really need a list of all jobs to show availability? That is, is there
a) a simpler query like maybe just hitting /api itself?
b) can we use a HEAD request?

[jrafanie]

how about ping?

irb(main):003:0> Terraform::Runner.send(:terraform_runner_client).get('api/ping').status
=> 200

[jrafanie]

opened #14 for this

[jrafanie]

I'll tackle the ping endpoint and other comments in a followup PR