[META 408] Instrumentation for AWS SQS

[AlexanderWert]

Provide instrumentation for AWS SQS.

Meta issue: elastic/apm#408
SPEC: elastic/apm#409

[astorm]

Some notes on the implementation details of the AWS SDK that will be helpful in instrumenting API methods for the SQS API as well as the other APIs we'll want to instrument.

The AWS SDK provides an internal system for building constructor-functions that can inherit from one another and provide other features that an end-user-programmer might expect to find in a class based system. This has been a popular pattern in javascript frameworks since the language's inception.

Here's an example of using this system to create a simple ClassA extends BaseClass hierarchy.

    const AWS = require('aws-sdk')
    BaseClass = AWS.util.inherit({
      constructor: function BaseClass() {
        console.log("BaseClass constructor")
      },

      test: function(value) {
        console.log(`In BaseClass: ${value}`)
        this.science = 'science'
        return true
      }
    })

    ClassA = AWS.util.inherit(BaseClass, {
      constructor: function ClassA() {
        console.log("ClassA constructor")
        BaseClass.call(this)
        this.science = null
      },

      test: function(value) {
        console.log(`Before: this.science = ${this.science}`)
        console.log(`In ClassA: ${value}`)
        BaseClass.prototype.test.call(this,"goodbye")
        console.log(`After: this.science = ${this.science}`)
      }
    })
    const a = new ClassA()
    a.test("hello")

While AWS.util.inherit is marked as a private API, this system has been in place since the 1.0 release of the framework and has remained remarkably stable. If we need to extend any constructor-functions that aren't service-constructor-functions (see below) this seems like a reasonable API to use.

Service Constructor Functions

In addition to these AWS.util.inherit objects, the aws-sdk also features special service-constructor-functions. These service-constructor-functions allow end-user-programmer to create objects that allow them to use the aws-sdk's public APIs.

For example, in this code

    var sqs = new AWS.SQS({apiVersion: '2012-11-05'});
    var params = {/* ... */ }
    sqs.sendMessage(params, function(err, data) {
      if (err) {
        console.log("Error", err);
      } else {
        console.log("Success", data.MessageId);
      }
    });    

The AWS.SQS object is one of these service-constructor-functions.

Amazon's programmers create their service-constructor-functions via calls to the AWS.Service.defineService method.

One interesting thing about these service objects are that their public API methods (like sendMessage above) are not normally defined functions. Instead, when a service object is instantiated, its parent Service class will read a configuration file which defines the public API, and dynamically assigns each public API method to the object.

This means that every call to a method that the aws-sdk considers as part of its official API is actually calling one of two methods: AWS.Service.makeRequest or AWS.Service.makeUnauthenticatedRequest.

You can see the creation/assignment of these methods here.

// File: lib/service.js

defineMethods: function defineMethods(svc) {
  AWS.util.each(svc.prototype.api.operations, function iterator(method) {
    if (svc.prototype[method]) return;
    var operation = svc.prototype.api.operations[method];
    if (operation.authtype === 'none') {
      svc.prototype[method] = function (params, callback) {
        return this.makeUnauthenticatedRequest(method, params, callback);
      };
    } else {
      svc.prototype[method] = function (params, callback) {
        return this.makeRequest(method, params, callback);
      };
    }
  });
},
//...

In this code block svc is the service-constructor-function object and will eventually be assigned to the AWS object (ex. AWS.SQS). This each is looping over every method (also called operations) configured for a service (example of a service configuration) and creating a method on the service object that's a simple function containing a call to either Service.makeRequest or Service.makeUnauthenticatedRequest. The makeUnauthenticatedRequest method eventually calls the makeRequest object itself.

All this means every call to a public API method in the aws-sdk is handled here

// File: lib/service.js

var request = new AWS.Request(this, operation, params);
//...
if (callback) request.send(callback);
return request;    

The makeRequest function will instantiate an AWS.Request object which accepts the the service object, the method name (operation above) and the params argument as constructor arguments. These arguments are eventually assigned as properties to the Request object. Calling send on this AWS.Request object will initiate the HTTPS request to the AWS backend that does the thing for any particular public API.

For the purposes of instrumentation, this means every call to a public API on a service object can be identified, (with it's params argument), via patching AWS.Request.send. This is fortune, because these service-constructor-functions can't be extended, which means we can't just replace service-constructor-functions like AWS.SQS with out own versions.

Can't Extend Service Constructor Functions

If you try extending a service-constructor-function using AWS.util.inherit, you'll end up with a disappointing result. Code like this

```
const SQS2 = AWS.util.inherit(AWS.SQS, {})
sqs = new SQS2({apiVersion: '2012-11-05'})
//...
sqs.sendMessage(params, callback)
```

Will give us an error like this.

sqs.sendMessage(params, function(err, data) {
    ^

TypeError: sqs.sendMessage is not a function
    at Object.<anonymous> (/Users/astorm/Documents/test-sqs/sqs_sendmessage.js:33:5)
    at Module._compile (internal/modules/cjs/loader.js:1063:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
    at Module.load (internal/modules/cjs/loader.js:928:32)
    at Function.Module._load (internal/modules/cjs/loader.js:769:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
    at internal/main/run_main_module.js:17:47

It's unclear if this is a quirk of this object system or a deliberate decision to disable inheritance on service objects. Without getting too into the weeds, the call to Service.defineService defines a number of properties on the a service-constructor-function object, and AWS.util.inherit will not inherit these properties. Without these properties, the new constructor function will not behave correctly, and fails to load the public API method definitions.

[trentm]

Re-opening. #2008 was only part of the work for this.

[astorm]

Priliminary version of this feature is complete, with additional issues opened for possible expansions to our functionality. Closing this one out.