Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exception at startup in Celluloid actor on aws-sdk (2.0.47) #833

Closed
tmornini opened this issue Jun 4, 2015 · 13 comments
Closed

Exception at startup in Celluloid actor on aws-sdk (2.0.47) #833

tmornini opened this issue Jun 4, 2015 · 13 comments
Labels
guidance Question that needs advice or information.

Comments

@tmornini
Copy link
Contributor

tmornini commented Jun 4, 2015

Hey there.

Reporting some odd behavior we're seeing during startup of a Celluloid based app.

NoMethodError: undefined method `new' for nil:NilClass
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/aws-sdk-core/xml/parser.rb:28:in `parse'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/aws-sdk-core/query/handler.rb:36:in `parse_xml'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/aws-sdk-core/query/handler.rb:13:in `block in call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/response.rb:43:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/response.rb:43:in `block in on'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/http/response.rb:137:in `block in on_success'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/http/response.rb:164:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/http/response.rb:164:in `block in listener'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/http/response.rb:128:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/http/response.rb:128:in `on_done'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/http/response.rb:135:in `on_success'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/response.rb:42:in `on'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/response.rb:51:in `on_success'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/aws-sdk-core/query/handler.rb:11:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/plugins/restful_bindings.rb:13:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/plugins/endpoint.rb:31:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/plugins/param_validation.rb:22:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/plugins/param_conversion.rb:22:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/plugins/response_target.rb:18:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/request.rb:70:in `send_request'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/base.rb:216:in `block (2 levels) in define_operation_methods'

... through our code ...

  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `public_send'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `dispatch'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/calls.rb:63:in `dispatch'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/cell.rb:60:in `block in invoke'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/cell.rb:71:in `block in task'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'

This issue is mostly non-deterministic:

  1. doesn't always happen
  2. sometimes happens more than once before resolving
  3. always resolves automatically as Actors are re-spawed

We've been upgrading for v1 -> v2 and this reminds me of AWS SDK v1 issues solved with #eager_autoload! and this same code triggered a common exception experienced in threads in ActiveSupport (undefined method `encode' for ActiveSupport::JSON:Module)
@tmornini
Copy link
Contributor Author

tmornini commented Jun 4, 2015

Here's another we just noticed, exact same situation.

NoMethodError: undefined method `search' for #<JMESPath::Runtime:0x007ffd5c334ad0>
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/jmespath-1.0.2/lib/jmespath.rb:32:in `search'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-resources-2.0.47/lib/aws-sdk-resources/builder_sources.rb:71:in `extract'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-resources-2.0.47/lib/aws-sdk-resources/builder.rb:37:in `block in build'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-resources-2.0.47/lib/aws-sdk-resources/builder.rb:36:in `each'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-resources-2.0.47/lib/aws-sdk-resources/builder.rb:36:in `with_object'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-resources-2.0.47/lib/aws-sdk-resources/builder.rb:36:in `build'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-resources-2.0.47/lib/aws-sdk-resources/operations.rb:87:in `call'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-resources-2.0.47/lib/aws-sdk-resources/operation_methods.rb:19:in `block in add_operation'

... through our code ...

    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `public_send'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `dispatch'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/calls.rb:63:in `dispatch'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/cell.rb:60:in `block in invoke'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/cell.rb:71:in `block in task'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'
    (celluloid):0:in `remote procedure call'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/calls.rb:92:in `value'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/proxies/sync_proxy.rb:33:in `method_missing'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/proxies/cell_proxy.rb:17:in `_send_'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/pool_manager.rb:41:in `_send_'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/pool_manager.rb:140:in `method_missing'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `public_send'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `dispatch'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/calls.rb:122:in `dispatch'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/cell.rb:60:in `block in invoke'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/cell.rb:71:in `block in task'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'

@trevorrowe
Copy link
Member

Do you have the ability to run this in Ruby 2.0 or newer? While it certainly is possible autoload is problematic, my understanding is that was resolve, but maybe not in Ruby 1.9.3.

@tmornini
Copy link
Contributor Author

tmornini commented Jun 5, 2015

@trevorrowe This code base has severe memory growth issues on 2.0 or later.

@trevorrowe
Copy link
Member

@tmornini Ruby 2.0+ has a bug in StringIO wich leaks memory. See this issue for more information and possible workarounds:

#785

I'm considering creating a compatible replacement for StringIO and using that by default in the SDK to avoid this, but I have not done so yet.

@tmornini
Copy link
Contributor Author

tmornini commented Jun 6, 2015

@trevorrowe Thanks. Does that exist in 2.2.2 as well? Because we have the same issue in both, and I'm pretty confident the problem is in our code, and we leak with the AWS SDK v1 as well...

Here's another spurious issue that we've only seen once. Including it here to be helpful, not pedantic. :-)

NameError: uninitialized constant Aws::S3::BUCKET_REGIONS
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/aws-sdk-core/plugins/s3_request_signer.rb:130:in `use_regional_endpoint_when_known'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/aws-sdk-core/plugins/s3_request_signer.rb:122:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/aws-sdk-core/plugins/s3_redirects.rb:15:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/aws-sdk-core/plugins/retry_errors.rb:88:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/aws-sdk-core/plugins/s3_md5s.rb:33:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:21:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/aws-sdk-core/rest_body_handler.rb:9:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/plugins/restful_bindings.rb:13:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/plugins/endpoint.rb:31:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/plugins/param_validation.rb:22:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:18:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/plugins/param_conversion.rb:22:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/plugins/response_target.rb:18:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/request.rb:70:in `send_request'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.0.47/lib/seahorse/client/base.rb:216:in `block (2 levels) in define_operation_methods'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-resources-2.0.47/lib/aws-sdk-resources/request.rb:24:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-resources-2.0.47/lib/aws-sdk-resources/operations.rb:41:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-resources-2.0.47/lib/aws-sdk-resources/operations.rb:87:in `call'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-resources-2.0.47/lib/aws-sdk-resources/operation_methods.rb:19:in `block in add_operation'

... through our code ...

  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `public_send'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `dispatch'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/calls.rb:63:in `dispatch'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/cell.rb:60:in `block in invoke'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/cell.rb:71:in `block in task'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
  /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'

@trevorrowe
Copy link
Member

Yes, the memory leak exists in the latest version of Ruby.

All of these initialized constant errors are going to be caused by the same issue with autoloading. I'll see about porting the eager autoload utility.

@tmornini
Copy link
Contributor Author

tmornini commented Jun 8, 2015

@trevorrowe Thanks

@trevorrowe
Copy link
Member

I've pushed a commit to the version-2.1-branch of the SDK that adds Aws.eager_autoload!

79f8170

I'm hoping to release 2.1 later this week or early next week.

@trevorrowe trevorrowe mentioned this issue Jun 11, 2015
Merged
@trevorrowe
Copy link
Member

This is now part of the 2.1 public release, available now.

@tmornini
Copy link
Contributor Author

This seems to be working better, but not yet perfect.

Using 2.1.0, and calling:

require 'aws-sdk'
Aws.eager_autoload!

we're still seeing (at least) this one exception:

NoMethodError: undefined method `new' for nil:NilClass
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/aws-sdk-core/xml/parser.rb:28:in `parse'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/aws-sdk-core/query/handler.rb:53:in `parse_xml'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/aws-sdk-core/query/handler.rb:29:in `block in call'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/seahorse/client/response.rb:43:in `call'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/seahorse/client/response.rb:43:in `block in on'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/seahorse/client/http/response.rb:137:in `block in on_success'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/seahorse/client/http/response.rb:164:in `call'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/seahorse/client/http/response.rb:164:in `block in listener'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/seahorse/client/http/response.rb:128:in `call'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/seahorse/client/http/response.rb:128:in `on_done'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/seahorse/client/http/response.rb:135:in `on_success'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/seahorse/client/response.rb:42:in `on'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/seahorse/client/response.rb:51:in `on_success'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/aws-sdk-core/query/handler.rb:27:in `call'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/seahorse/client/plugins/endpoint.rb:34:in `call'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/aws-sdk-core/plugins/param_validator.rb:21:in `call'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/aws-sdk-core/plugins/param_converter.rb:21:in `call'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/seahorse/client/plugins/response_target.rb:18:in `call'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/seahorse/client/request.rb:70:in `send_request'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/aws-sdk-core-2.1.0/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'

... through our code ...

    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `public_send'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `dispatch'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/calls.rb:63:in `dispatch'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/cell.rb:60:in `block in invoke'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/cell.rb:71:in `block in task'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
    /Users/tmornini/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'

I say "at least" because this behavior has always been non-deterministic.

trevorrowe added a commit that referenced this issue Jun 12, 2015
@trevorrowe
Aws.eager_autoload! now loads non-service utility classes.
465695a 
Fixes #833
@tmornini
Copy link
Contributor Author

@trevorrowe After a few hours of testing, I can confirm that the exception above is very repeatable in my case, and is the only such issue we've seen in many iterations.

trevorrowe added a commit that referenced this issue Jun 12, 2015
@trevorrowe
Test failure fixes.
878272d 
The fix for eager_autoload! exposed an issue with autoloading optional
dependencies eagerly. Travis caught these errors when testing against
Ruby installations that did not have these optional dependencies
present.

See #833
@trevorrowe
Copy link
Member

@tmornini The eager_autoload! method is only loading constants under the service modules, which means utility classes are getting missed. That was an oversight. I've corrected it in the commits above. My apologies. This will go out with the next release.

@tmornini
Copy link
Contributor Author

@trevorrowe Thanks, looking forward to next release.

@cheeseplus cheeseplus mentioned this issue Feb 26, 2016
Closed
hfinucane added a commit to hfinucane/kitchen-ec2 that referenced this issue May 24, 2016
@hfinucane
Turn on eager loading for AWS resources
e93d0ad 
See aws/aws-sdk-ruby#833
@hfinucane hfinucane mentioned this issue May 24, 2016
Merged
@srchase srchase added the guidance Question that needs advice or information. label Dec 24, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
guidance Question that needs advice or information.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tmornini @trevorrowe @srchase