Parsing exception raised when ServiceUnavailableException returned

[adammw]

Issue description

Parsing exception raised when ServiceUnavailableException returned

Gem versions

    aws-eventstream (1.0.1)
    aws-partitions (1.94.0)
    aws-sdk-core (3.22.0)
      aws-eventstream (~> 1.0)
      aws-partitions (~> 1.0)
      aws-sigv4 (~> 1.0)
      jmespath (~> 1.0)
    aws-sdk-firehose (1.4.0)
      aws-sdk-core (~> 3)
      aws-sigv4 (~> 1.0)
    aws-sdk-kinesis (1.3.0)
      aws-sdk-core (~> 3)
      aws-sigv4 (~> 1.0)
    aws-sigv4 (1.0.2)

Version of Ruby, OS environment

Ruby MRI 2.3.3
Debian Linux

Code snippets / steps to reproduce

Calling PutRecords on a Kinesis client occasionally fails due to AWS API returning <ServiceUnavailableException/> which the gem fails to parse as it attempts to decode the exception as JSON.

Backtrace:

EncodingError: Empty input at line 1, column 1 [parse.c:928] in '<ServiceUnavailableException/>
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/aws-sdk-core/json.rb:25:in `load'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/aws-sdk-core/json.rb:25:in `load'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/aws-sdk-core/json/error_handler.rb:17:in `extract_error'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/aws-sdk-core/xml/error_handler.rb:22:in `error'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/aws-sdk-core/json/error_handler.rb:9:in `block in call'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/seahorse/client/response.rb:45:in `block in on'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/seahorse/client/http/response.rb:139:in `block in on_success'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/seahorse/client/http/response.rb:166:in `block in listener'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/seahorse/client/http/response.rb:130:in `on_done'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/seahorse/client/http/response.rb:137:in `on_success'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/seahorse/client/response.rb:44:in `on'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/aws-sdk-core/json/error_handler.rb:8:in `call'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/aws-sdk-core/plugins/signature_v4.rb:66:in `call'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/aws-sdk-core/plugins/retry_errors.rb:138:in `call'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/aws-sdk-core/json/handler.rb:11:in `call'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/seahorse/client/plugins/endpoint.rb:45:in `call'
/fluentd/vendor/bundle/ruby/2.3.0/gems/ddtrace-0.13.0/lib/ddtrace/contrib/aws/instrumentation.rb:19:in `block in call'
/fluentd/vendor/bundle/ruby/2.3.0/gems/ddtrace-0.13.0/lib/ddtrace/tracer.rb:287:in `trace'
/fluentd/vendor/bundle/ruby/2.3.0/gems/ddtrace-0.13.0/lib/ddtrace/contrib/aws/instrumentation.rb:18:in `call'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/aws-sdk-core/plugins/param_validator.rb:24:in `call'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/seahorse/client/plugins/response_target.rb:23:in `call'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.0/lib/seahorse/client/request.rb:70:in `send_request'
/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-kinesis-1.3.0/lib/aws-sdk-kinesis/client.rb:1413:in `put_records'

[awood45]

Thanks for the report - we'll engage with the service team here and take a look at possible options to move forward on this.

[adammw]

@awood45 any update on this? saw an increase of them around 2018-08-06 13:06:48 UTC which triggered a massive exponential backoff in our fluentd instance.

[stiak]

I'm also seeing this now with aws-sdk-core (3.54.2). It looks like when using the OJ gem, OJ returns EncodingError rather than the expected Oj::ParseError on strings like "<ServiceUnavailableException/>". This then bubbles out rather than being rescued and transformed into a ParseError.

[stiak]

This seems like the related change in OJ: ohler55/oj#441