Aws::SQS::Errors::AccessDenied from Shoryuken::CLI

[parov]

Hi,

I'm trying to use the new CLI amazing feature you build, but when running it from the console I get Aws::SQS::Errors::AccessDenied from Shoryuken::CLI. Looks like Aws::SQS::Client.new , but when calling cli.mv old_queue new_queuethe exception is appearing.
Shoryuken works already pretty well with AWS IAM, but for some reasons the CLI don't.
I tried also to add it in a Controller and run it from there, with the same result.

Do I need some extra setup to make it work?
Thanks for the amazing job

[phstc]

Hi @parov

I've only used shoryuken sqs commands with AWS_... keys exported into my ENV. Thinking more on it, I believe the SQS CLI works only with ENV or IAM, because of the way it's being initialized.

Do you have your AWS keys exported?

[parov]

Nope, I use IAM only.
As a workaround I added Shoryuken::EnvironmentLoader.load_for_rails_console in my controller. But anyways, also using the CLI from the command line, I still get the exception. So probably I need to setup AWS_...

[phstc]

@parov hm this Aws::SQS::Client.new should ✨ pickup IAM creds. Does the IAM user you are using have permissions to enqueue and remove messages in the source and target queues?

Could you try bundle exec shoryuken sqs ls, just to see if the issue is only with permissions with mv?

[phstc]

The required permissions: https://github.com/phstc/shoryuken/wiki/Amazon-SQS-IAM-Policy-for-Shoryuken

[parov]

Same issue with ls

bundle exec shoryuken sqs ls                                                                                                               
bundler: failed to load command: shoryuken (/usr/local/bundle/bin/shoryuken)
Aws::SQS::Errors::AccessDenied: Access to the resource https://sqs.eu-west-1.amazonaws.com/ is denied.
  /usr/local/bundle/gems/aws-sdk-core-2.8.5/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
  /usr/local/bundle/gems/aws-sdk-core-2.8.5/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
  /usr/local/bundle/gems/aws-sdk-core-2.8.5/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
  /usr/local/bundle/gems/aws-sdk-core-2.8.5/lib/aws-sdk-core/plugins/response_paging.rb:26:in `call'
  /usr/local/bundle/gems/aws-sdk-core-2.8.5/lib/seahorse/client/plugins/response_target.rb:21:in `call'
  /usr/local/bundle/gems/aws-sdk-core-2.8.5/lib/seahorse/client/request.rb:70:in `send_request'
  /usr/local/bundle/gems/aws-sdk-core-2.8.5/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
  /usr/local/bundle/bundler/gems/shoryuken-0494242fff86/bin/cli/sqs.rb:105:in `ls'
  /usr/local/bundle/gems/thor-0.19.4/lib/thor/command.rb:27:in `run'
  /usr/local/bundle/gems/thor-0.19.4/lib/thor/invocation.rb:126:in `invoke_command'
  /usr/local/bundle/gems/thor-0.19.4/lib/thor.rb:369:in `dispatch'
  /usr/local/bundle/gems/thor-0.19.4/lib/thor/invocation.rb:115:in `invoke'
  /usr/local/bundle/gems/thor-0.19.4/lib/thor.rb:242:in `block in subcommand'
  /usr/local/bundle/gems/thor-0.19.4/lib/thor/command.rb:27:in `run'
  /usr/local/bundle/gems/thor-0.19.4/lib/thor/invocation.rb:126:in `invoke_command'
  /usr/local/bundle/gems/thor-0.19.4/lib/thor.rb:369:in `dispatch'
  /usr/local/bundle/gems/thor-0.19.4/lib/thor/base.rb:444:in `start'
  /usr/local/bundle/bundler/gems/shoryuken-0494242fff86/bin/shoryuken:50:in `<top (required)>'
  /usr/local/bundle/bin/shoryuken:17:in `load'
  /usr/local/bundle/bin/shoryuken:17:in `<top (required)>'

[phstc]

hi @parov

I'm having hard to time figure this out. Both sqs CLI and shoryuken CLI initialize the SQS client in the exactly same way. But for some reason the sqs CLI isn't working. I've SSH a container which is running shoryuken just fine (IAM), and tried to use the CLI and could reproduce exactly what you are reporting.

No clue yet, but I will keep digging.

[phstc]

@parov problem solved. ls was requiring ListQueues permission, which wasn't specified here - I've just added it. mv, dump and requeue were also calling list_queues, but I updated them to use get_queue_url exactly as Shoryuken does.

All theses commands should be working fine in 3.0.3. Could you let me know if they work for you?

[parov]

@phstc It's working now. Thanks a lot for the quick fix. Would be nice to use the tool also from the Controllers. I prepared a small PR #342 that is "abusing" the CLI, requesting for a parameter limit when mv is called outside the CLI.
Please let me know what you think

[phstc]

@parov I wasn't thinking much on using it from a controller. I actually, don't even SSH for using it, I have a special user with write privileges that I use for write operations:

ce pablo_admin # login with write privileges
(pablo_admin) bundle exec shoryuken sqs mv ...
exit # logout

My setup:

# ~/.zshrc
# aws-vault https://github.com/99designs/aws-vault
alias ce='aws-vault exec -n'

function ce_prompt () {
  if [[ -n "$AWS_VAULT" ]]
  then
    echo "($AWS_VAULT) "
  fi
}

PROMPT='$(ce_prompt) ...'

But if you do want to run it from a Rails controller, we may subprocess, WDYT?

system('bundle exec shoryuken ...')

[parov]

So, the reason to have it in a controller is for me only to move back jobs from a dead letter queue.
I don't know if there's a better way to do that, but as you've implemented the mv method recently, I just hacked it in my controller:

# frozen_string_literal: true
shoryuken_path = Pathname.new(Gem.find_files("shoryuken.rb").first).dirname
require "thor"
require "aws-sdk-core"
require "#{shoryuken_path}/../bin/cli/base"
require "#{shoryuken_path}/../bin/cli/sqs"

class DeadLetterQueuesController < ApplicationController
  before_action :shoryuken_runner, only: :update

  def update
    queue = params[:id]
    cli.mv queue, queue.chomp("_dead"), limit: 5000

    render status: :accepted
  end

  private

  def cli
    @cli ||= Shoryuken::CLI::SQS.new
  end

  def shoryuken_runner
    Shoryuken::EnvironmentLoader.load_for_rails_console
  end
end

Subprocessing looks cleaner, but probably it's not needed here

[phstc]

@parov I replied you in the PR https://github.com/phstc/shoryuken/pull/342/files#r107178275