feature: cross language integration tests (#417)

* feat: very first pass at spinning up a mock metrics server to write tests

The theory is that we'll use this, pass in the command to spin up the
child server, and provide the METRICS_SERVER as an env variable that
each of the sdks can look for. Once we can pass that in we can make
HTTP level assertions about what the SDKs can and can't do!

Lets try and write a test first against the node server and see if we
can make it work 🤞

* test: add a first integration test via the new testing method

Now i'm gunna try and put this into a dockerfile like httpsnippet, then
I can try and add another language 🤞

* test: add docker-compose file for running node/express integration tests

* test: try and get new integration tests working on gh actions

* test: attempting to get integration tests working on actions

* refactor: attempt to bubble up errors better

* refactor: toString potential errors before rejecting

* test(integration): build the node module before running tests

* test(integration): try adding it as a dependent job

* test(integration): remove integration test from `npm test`

It's gunna be run separately via another action, so don't do it twice.

* test(integration): fix incorrect eslint-config reference

* test(integration): fix eslint package reference

* test(integration): linty linty

* test(integration): attempt to get it working

* test(integration): ignore lint error in express example

This dependency will be there when we run the integration tests, but it
won't be when the linter gets run because the TS files haven't been built

* test(integration): turning off a lint rule

* feat(integration/node): move more setup to Dockerfile instead of gh action

* feature: cross language integration tests (dotnet edition) (#435)

* test(integration/dotnet): add super simple dotnet 6.0 test server

Generated using:

```
dotnet new web --no-https
```

* test(integration/dotnet): add very basic .net 6.0 app for metrics

Still outstanding:
- make the sdk respect `METRICS_SERVER` env variable
- try and get `readme__apiKey` better as an env variable
- wire it up to an integration test

* test(integration/dotnet): use regular environment variable name

This is slightly confusing since if they actually wanted to set this up
themselves then they would probably use appsettings.json instead.

This is fine for a demo app though, and will mean it works from the
integration test layer 🤞

* test(integration/node): update tests to work with new port number stuff

* refactor(dotnet): stylistic and whitespace changes

I installed the C# vs code plugin and it did this 🤷‍♂️

* refactor(dotnet): fix warning to do with null variable in test app

* refactor(integration/dotnet): allow the metrics server to be set from the env

Use UriBuilder to construct a URI that always has the correct path on the end

* refactor(dotnet): remove the await from the metrics call so it doesnt block

Right now if metrics is down, this will cause our user's server to stop
responding to requests. This is probably what you'd want in the vast
majority of cases - since it's actually kinda rare you'd wanna fire and
forget an HTTP request.

* chore(integration/dotnet): make host header lookup case insensitive

* test(integration/node): fix up child process usage to use spawn

When using child_process.exec, the process could not reliably be killed
when the test ended, which meant the docker container never closed.

This caused timeouts on the github action:
https://github.com/readmeio/metrics-sdks/runs/6215430863?check_suite_focus=true

* test(integration/node): this no longer needs to be executable

* test(integration): remove the snapshot test for now

There's way too many inconsistencies between even these two web servers
to do jest snapshot testing, so opting to be much more explicit about
assertions for the time being. Would rather get something up and passing
than not running these tests going forward.

* test(integration/dotnet): add docker file for .net test

Added to docker-compose and a new dotnet.yml to be run on gh actions

* refactor: rename incorrect naming of job in gh actions

* test(integration/dotnet): ugh, stupid case insensitive macOS

This failed on github cos it's using a real linux env:

https://github.com/readmeio/metrics-sdks/runs/6226930070?check_suite_focus=true

Lets see if this does the trick 🤞

.dockerignore

@@ -0,0 +1 @@
+node_modules

.eslintignore

@@ -0,0 +1,5 @@
+coverage/
+dist/
+node_modules/
+# We just want to lint the very top level
+packages/

.eslintrc

@@ -0,0 +1,5 @@
+{
+  "extends": [
+    "@readme/eslint-config"
+  ]
+}

.github/workflows/dotnet.yml

@@ -0,0 +1,16 @@
+name: dotnet
+
+on: [push]
+
+jobs:
+  integration:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v3
+
+    - run: docker-compose run integration_dotnet_v6.0
+
+    - name: Cleanup
+      if: always()
+      run: docker-compose down

.github/workflows/nodejs.yml

@@ -33,3 +33,18 @@ jobs:
 
       - run: npm ci --ignore-scripts
       - run: npm test
+
+  integration:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: packages/node/examples/express
+
+    steps:
+    - uses: actions/checkout@v3
+
+    - run: docker-compose run integration_node_express
+
+    - name: Cleanup
+      if: always()
+      run: docker-compose down

__tests__/.eslintrc

@@ -0,0 +1,3 @@
+{
+  "extends": "@readme/eslint-config/testing"
+}

__tests__/__snapshots__/integration.test.js.snap

@@ -0,0 +1,9 @@
+// Jest Snapshot v1, https://goo.gl/fbAQLP
+
+exports[`Metrics SDK Integration Tests should make a request to a metrics backend with a har file 1`] = `
+Object {
+  "email": "owlbert@example.com",
+  "id": "owlbert-api-key",
+  "label": "Owlbert",
+}
+`;

__tests__/integration.test.js

@@ -0,0 +1,158 @@
+import http from 'http';
+import { cwd } from 'process';
+import { spawn } from 'child_process';
+import { promisify } from 'util';
+import { once } from 'events';
+import getPort from 'get-port';
+import caseless from 'caseless';
+
+if (!process.env.EXAMPLE_SERVER) {
+  // eslint-disable-next-line no-console
+  console.error('Missing `EXAMPLE_SERVER` environment variable');
+  process.exit(1);
+}
+
+// https://gist.github.com/krnlde/797e5e0a6f12cc9bd563123756fc101f
+http.get[promisify.custom] = function getAsync(options) {
+  return new Promise((resolve, reject) => {
+    http
+      .get(options, response => {
+        response.end = new Promise(res => {
+          response.on('end', res);
+        });
+        resolve(response);
+      })
+      .on('error', reject);
+  });
+};
+
+const get = promisify(http.get);
+
+const randomApiKey = 'a-random-readme-api-key';
+
+// Converts an array of headers like this:
+// [
+//   { name: 'host', value: 'localhost:49914' },
+//   { name: 'connection', value: 'close' },
+// ];
+//
+// To an object that can be passed in to caseless:
+// {
+//    host: 'localhost:49914',
+//    connection: 'close'
+// }
+function arrayToObject(array) {
+  return array.reduce((prev, next) => {
+    return Object.assign(prev, { [next.name]: next.value });
+  }, {});
+}
+
+describe('Metrics SDK Integration Tests', () => {
+  let metricsServer;
+  let httpServer;
+  let PORT;
+
+  beforeAll(async () => {
+    metricsServer = http.createServer().listen(0, 'localhost');
+
+    await once(metricsServer, 'listening');
+    const { address, port } = metricsServer.address();
+    PORT = await getPort();
+
+    // In order to use child_process.spawn, we have to provide a
+    // command along with an array of arguments. So this is a very
+    // rudimental way of splitting the two values provided to us
+    // from the environment variable.
+    //
+    // I tried refactoring this to use child_process.exec, which just
+    // takes in a single string to run, but that creates it's own
+    // shell so we can't do `cp.kill()` on it later on (because that
+    // just kills the shell, not the actual command we're running).
+    //
+    // Annoyingly this works under macOS, so it must be a platform
+    // difference when running under docker/linux.
+    const [command, ...args] = process.env.EXAMPLE_SERVER.split(' ');
+
+    httpServer = spawn(command, args, {
+      cwd: cwd(),
+      env: {
+        PORT,
+        METRICS_SERVER: new URL(`http://${address}:${port}`).toString(),
+        README_API_KEY: randomApiKey,
+        ...process.env,
+      },
+    });
+    return new Promise((resolve, reject) => {
+      httpServer.stderr.on('data', data => {
+        // eslint-disable-next-line no-console
+        console.error(`stderr: ${data}`);
+        return reject(data.toString());
+      });
+      httpServer.on('error', err => {
+        // eslint-disable-next-line no-console
+        console.error('error', err);
+        return reject(err.toString());
+      });
+      // eslint-disable-next-line consistent-return
+      httpServer.stdout.on('data', data => {
+        if (data.toString().match(/listening/)) return resolve();
+        // eslint-disable-next-line no-console
+        console.log(`stdout: ${data}`);
+      });
+    });
+  });
+
+  afterAll(() => {
+    httpServer.kill();
+    return new Promise((resolve, reject) => {
+      metricsServer.close(err => {
+        if (err) return reject(err);
+        return resolve();
+      });
+    });
+  });
+
+  // TODO this needs fleshing out more with more assertions and complex
+  // test cases, along with more servers in different languages too!
+  it('should make a request to a metrics backend with a har file', async () => {
+    await get(`http://localhost:${PORT}`);
+
+    const [req] = await once(metricsServer, 'request');
+    expect(req.url).toBe('/v1/request');
+    expect(req.headers.authorization).toBe('Basic YS1yYW5kb20tcmVhZG1lLWFwaS1rZXk6');
+
+    let body = '';
+    // eslint-disable-next-line no-restricted-syntax
+    for await (const chunk of req) {
+      body += chunk;
+    }
+    body = JSON.parse(body);
+    const [har] = body;
+
+    // Check for a uuid
+    // https://uibakery.io/regex-library/uuid
+    // eslint-disable-next-line no-underscore-dangle
+    expect(har._id).toMatch(/^[0-9a-f]{8}-[0-9a-f]{4}-[0-5][0-9a-f]{3}-[089ab][0-9a-f]{3}-[0-9a-f]{12}$/i);
+    expect(har.group).toMatchSnapshot();
+    expect(har.clientIPAddress).toBe('127.0.0.1');
+
+    const { request, response } = har.request.log.entries[0];
+
+    expect(request.url).toBe(`http://localhost:${PORT}/`);
+    expect(request.method).toBe('GET');
+    expect(request.httpVersion).toBe('HTTP/1.1');
+
+    const requestHeaders = caseless(arrayToObject(request.headers));
+    expect(requestHeaders.get('connection')).toBe('close');
+    expect(requestHeaders.get('host')).toBe(`localhost:${PORT}`);
+
+    expect(response.status).toBe(200);
+    expect(response.statusText).toBe('OK');
+    expect(response.content.text).toBe(JSON.stringify({ message: 'hello world' }));
+    expect(response.content.size).toBe(25);
+    expect(response.content.mimeType).toBe('application/json; charset=utf-8');
+
+    const responseHeaders = caseless(arrayToObject(response.headers));
+    expect(responseHeaders.get('content-type')).toBe('application/json; charset=utf-8');
+  });
+});

__tests__/integrations/dotnet.Dockerfile

@@ -0,0 +1,21 @@
+FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build-env 
+
+ADD packages/dotnet /src
+
+WORKDIR /src/examples/net6.0
+RUN dotnet publish -o out
+
+# Build runtime image
+# TODO add this to base.Dockerfile?
+FROM node:16
+WORKDIR /src
+ADD package*.json /src/
+RUN npm ci
+ADD __tests__ /src/__tests__
+
+COPY --from=build-env /src /src
+COPY --from=build-env /usr/share/dotnet /usr/share/dotnet
+
+# Put the dotnet executable in the path
+ENV PATH /usr/share/dotnet:$PATH
+ENV DOTNET_CLI_TELEMETRY_OPTOUT=true

__tests__/integrations/node.Dockerfile

@@ -0,0 +1,14 @@
+FROM node:16
+
+ADD packages/node /src/packages/node
+
+# Build node sdk
+WORKDIR /src/packages/node
+RUN npm ci --ignore-scripts
+RUN npm run build
+
+# Install top level dependencies
+WORKDIR /src
+ADD package*.json /src/
+RUN npm ci
+ADD __tests__ /src/__tests__

docker-compose.yml

@@ -0,0 +1,17 @@
+version: '3'
+services:
+  integration_node_express:
+    build:
+      context: .
+      dockerfile: ./__tests__/integrations/node.Dockerfile
+    command: npm run test:integration
+    environment:
+      - EXAMPLE_SERVER=node ./packages/node/examples/express/index.js
+
+  integration_dotnet_v6.0:
+    build:
+      context: .
+      dockerfile: ./__tests__/integrations/dotnet.Dockerfile
+    command: npm run test:integration
+    environment:
+      - EXAMPLE_SERVER=dotnet examples/net6.0/out/net6.0.dll