Merge branch 'master' into bug1525030

clients/client/README.md

@@ -8,32 +8,51 @@ references from which Client classes are already constructed.
 ## Calling API End-Points
 To invoke an API end-point instantiate a taskcluster Client class, these are
 classes can be created from a JSON reference object, but a number of them are
-also built-in to this library. In the following example we instantiate an
-instance of the `Queue` Client class and use to to create a task.
+also built-in to this library. The following example instantiates an
+instance of the `Queue` Client class, showing all available options, and
+uses it to to create a task.  Note that only the `rootUrl` option is required.
 
 ```js
 var taskcluster = require('taskcluster-client');
 
 // Instantiate the Queue Client class
 var queue = new taskcluster.Queue({
-  // rootUrl for this Taskcluster instance
+  // rootUrl for this Taskcluster instance (required)
   rootUrl: 'https://taskcluster.myproject.org',
 
-  timeout: 30 * 1000, // timeout for _each_ invidual http request
-
-  // By default we share a global agent if you specify your instance
-  // will have it's own agent with the given options...
-  agent: {
-    // https://nodejs.org/api/http.html#http_new_agent_options
-  },
-
+  // Taskcluster credentials (required only for API methods that require scopes)
   credentials: {
     clientId:     '...',
     accessToken:  '...',
     // Certificate must also be provided if using temporary credentials,
     // this can be either a JSON object or a JSON string.
     certificate:  {...}   // Only applicable for temporary credentials
   }
+
+  // timeout for _each_ invidual http request
+  timeout: 30 * 1000,
+
+  // maximum number of retries for transient errors (default 5)
+  retries: 5,
+
+  // Multiplier for computation of retry delay: 2 ^ retry * delayFactor,
+  // 100 ms is solid for servers, and 500ms - 1s is suitable for background
+  // processes
+  delayFactor: 100,
+
+  // Randomization factor added as.
+  // delay = delay * random([1 - randomizationFactor; 1 + randomizationFactor])
+  randomizationFactor: 0.25,
+
+  // Maximum retry delay (defaults to 30 seconds)
+  maxDelay: 30 * 1000,
+
+  // By default we share a global HTTP agent. If you specify one, your instance
+  // will have its own agent with the given options...
+  agent: undefined,
+
+  // Fake methods, for testing
+  fake: null,
 });
 
 // Create task using the queue client
@@ -65,7 +84,7 @@ This replaces any given options with new values.
 
 **NOTE** `PulseListener` is no longer included in `taskcluster-client`;
 instead, use `PulseConsumer` from
-[taskcluster-lib-pulse](https://github.com/taskcluster/taskcluster-lib-pulse).
+[taskcluster-lib-pulse](../../libraries/pulse).
 
 However, this library helpfully includes bindings for exchanges declared by
 various Taskcluster services.  To use these with `taskcluster-lib-pulse`,
@@ -90,7 +109,7 @@ let pc = await pulse.consume({
 
 The set of API entries listed below is generated from the built-in references.
 Detailed documentation with description, payload and result format details is
-available in the [docs reference section](https://docs.taskcluster.net/docs/reference).
+available in the reference section of the Taskcluster documentation.
 
 On the documentation site, entries have a
 _signature_.  You'll find that it matches the signatures below. Notice that all
@@ -632,75 +651,6 @@ queue.defineTask(taskId taskDefinition).then(function(result) {
 ```
 
 
-## Using the Listener
-Taskcluster relies on pulse for exchange of messages. You'll need an pulse
-credentials for using `taskcluster.PulseListener`.
-An outline of how to create an instance and use is given below. Note, you
-must call `resume()` before message starts arriving.
-
-```js
-var listener = new taskcluster.PulseListener({
-  prefetch:             5,          // Number of tasks to process in parallel
-  credentials: {                    // If not instance of PulseConnection
-    username:           '...',      // Pulse username from pulse guardian
-    password:           '...',      // Pulse password from pulse guardian
-    hostname:           '...',      // hostname to connect to using username/password
-    vhost   :           '...'       // virtual host to use on the AMQP host   
-  },
-  connection:           connection, // If credentials isn't provided
-  // If no queue name is given, the queue is:
-  //    exclusive, autodeleted and non-durable
-  // If a queue name is given, the queue is:
-  //    durable, not auto-deleted and non-exclusive
-  queueName:          'my-queue',   // Queue name, undefined if none
-  maxLength:          0,            // Max allowed queue size
-});
-
-listener.bind({exchange, routingKeyPattern}).then(...);
-                                    // bind to an exchange; note that for
-                                    // Taskcluster components the argument
-                                    // can be created by Client; see above.
-listener.connect().then(...);       // Setup listener and bind queue
-listener.resume().then(...);        // Start getting new messages
-listener.pause().then(...);         // Pause retrieval of new messages
-listener.deleteQueue();             // Delete named queue and disconnect
-listener.close();                   // Disconnect from pulse
-```
-
-To actually receive messages, subscribe to the listener's `message` event:
-
-```js
-listener.on('message', (message) => async {
-  message.exchange
-  message.payload
-  .. etc. (see "Listening for Events", above)
-});
-```
-
-**Using `PulseConnection`**, instead of giving a `username` and `password` it
-is possible to give the `Listener` the key `connection` which must then be a
-`taskcluster.PulseConnection` object. Using a `PulseConnection` object it's
-possible to have multiple listeners using the same AMQP TCP connection, which
-is the recommended way of using AMQP. Notice, that the `PulseConnection` will
-not be closed with the `Listener`s, so you must `close()` it manually.
-
-```js
-var connection = new taskcluster.PulseConnection({
-  username:           '...',        // Pulse username from pulse guardian
-  password:           '...',        // Pulse password from pulse guardian
-  hostname:           '...',        // hostname to connect to using username/password
-  vhost   :           '...'         // virtual host to use on the AMQP host   
-});
-
-// Create listener
-var listener = new taskcluster.PulseListener({
-  connection:         connection,   // AMQP connection object
-});
-
-
-connection.close();                 // Disconnect from AMQP/pulse
-```
-
 ## Relative Date-time Utilities
 A lot of taskcluster APIs requires ISO 8601 time stamps offset into the future
 as way of providing expiration, deadlines, etc. These can be easily created

clients/client/test/client_test.js

@@ -1,10 +1,11 @@
-suite('client requests/responses', function() {
-  let taskcluster = require('../');
-  let assert = require('assert');
-  let path = require('path');
-  let nock = require('nock');
-  let MonitorManager = require('taskcluster-lib-monitor');
-
+const taskcluster = require('../');
+const assert = require('assert');
+const path = require('path');
+const nock = require('nock');
+const MonitorManager = require('taskcluster-lib-monitor');
+const testing = require('taskcluster-lib-testing');
+
+suite(testing.suiteName(), function() {
   // This suite exercises the request and response functionality of
   // the client against a totally fake service defined by this reference
   // and implemented via Nock.

clients/client/test/credinfo_test.js

@@ -1,8 +1,9 @@
-suite('taskcluster.credentialInfo', function() {
-  let taskcluster = require('../');
-  let assert = require('assert');
-  let nock = require('nock');
+const taskcluster = require('../');
+const assert = require('assert');
+const nock = require('nock');
+const testing = require('taskcluster-lib-testing');
 
+suite(testing.suiteName(), function() {
   teardown(function() {
     assert(nock.isDone());
     nock.cleanAll();

clients/client/test/creds_test.js

@@ -1,9 +1,10 @@
-suite('client credential handling', function() {
-  let taskcluster = require('../');
-  let assert = require('assert');
-  let request = require('superagent');
-  let _ = require('lodash');
+const taskcluster = require('../');
+const assert = require('assert');
+const request = require('superagent');
+const _ = require('lodash');
+const testing = require('taskcluster-lib-testing');
 
+suite(testing.suiteName(), function() {
   // This suite exercises the credential-handling functionality of the client
   // against a the auth service's testAuthenticate endpoint.
 

clients/client/test/retry_test.js

@@ -1,16 +1,14 @@
-suite('retry-test', function() {
-  const taskcluster = require('../');
-  const assert = require('assert');
-  const SchemaSet = require('taskcluster-lib-validate');
-  const MonitorManager = require('taskcluster-lib-monitor');
-  const APIBuilder = require('taskcluster-lib-api');
-  const testing = require('taskcluster-lib-testing');
-  const App = require('taskcluster-lib-app');
-  const http = require('http');
-  const httpProxy = require('http-proxy');
+const taskcluster = require('../');
+const assert = require('assert');
+const SchemaSet = require('taskcluster-lib-validate');
+const MonitorManager = require('taskcluster-lib-monitor');
+const APIBuilder = require('taskcluster-lib-api');
+const testing = require('taskcluster-lib-testing');
+const App = require('taskcluster-lib-app');
 
-  const PROXY_PORT = 60551;
-  const rootUrl = `http://localhost:${PROXY_PORT}`;
+const rootUrl = `http://localhost:60526`;
+
+suite(testing.suiteName(), function() {
   let proxier;
 
   // Construct API
@@ -144,24 +142,6 @@ suite('retry-test', function() {
       trustProxy: false,
       apis: [api],
     });
-
-    // Finally, we set up a proxy that runs on rootUrl
-    // and sends requests to either of the services based on path.
-
-    const proxy = httpProxy.createProxyServer({proxyTimeout: 0});
-    proxy.on('error', (err, req, res) => {
-      req.connection.end(); // Nasty hack to make httpProxy pass along the connection close
-    });
-    proxier = http.createServer(function(req, res) {
-      if (req.url.startsWith('/api/auth/')) {
-        proxy.web(req, res, {target: 'http://localhost:60552'});
-      } else if (req.url.startsWith('/api/retrytest/')) {
-        proxy.web(req, res, {target: 'http://localhost:60526'});
-      } else {
-        throw new Error(`Unknown service request: ${req.url}`);
-      }
-    });
-    proxier.listen(PROXY_PORT);
   });
 
   // Close server

clients/client/test/utils_test.js

@@ -1,8 +1,9 @@
-suite('taskcluster utilities', function() {
-  let taskcluster = require('../');
-  let parseTime = require('../src/parsetime');
-  let assert = require('assert');
+const taskcluster = require('../');
+const parseTime = require('../src/parsetime');
+const assert = require('assert');
+const testing = require('taskcluster-lib-testing');
 
+suite(testing.suiteName(), function() {
   test('parseTime 1 year', function() {
     assert.equal(parseTime('1y').years, 1);
     assert.equal(parseTime('1 yr').years, 1);

dev-docs/best-practices/libraries.md

@@ -12,3 +12,4 @@ This file will automatically be linked from the `README.md` in the root of the r
 
 Library source code should be in a `src` subdirectory.
 No transpilation should be used: write JS that can be interpreted directly by the Node version in use in the repository.
+The `main` property in `package.json` should point to `src/index.js`, which may then load other parts of the library.

dev-docs/best-practices/testing.md

@@ -92,10 +92,13 @@ Import all required modules at the top of a test file, then begin defining suite
 For all `suite` and `test` calls, use the `function() { .. }` notation instead of an arrow function.
 Doing so allows use of `this` to access the Mocha object.
 
+Each file should have a top-level suite (or mockSuite) with its title generated by `testing.suiteName()`.
+
 ```javascript
 const frobs = require('../src/frobs.js');
+const testing = require('taskcluster-lib-testing');
 
-suite('frobs', function() {
+suite(testing.suiteName(), function() {
   test('frobnicates', async function() {
     // ...
   });

infrastructure/builder/README.md

@@ -46,7 +46,7 @@ docker run -ti --rm -e PORT=80 -e .. <image> auth/web
 To run an interactive shell in the image, use
 
 ```shell
-docker run -ti --rm -e PORT=80 -e .. <image> sh
+docker run -ti --rm -e PORT=80 -e .. <image> bash
 ```
 
 ## Skipping Tasks

infrastructure/builder/src/build/monoimage.js

@@ -367,7 +367,10 @@ const generateMonoimageTasks = ({tasks, baseDir, cfg, cmdOptions}) => {
 
       const dockerfile = [
         `FROM ${nodeAlpineImage}`,
-        'RUN apk update && apk add nginx && mkdir /run/nginx',
+        `RUN apk update && \
+          apk add nginx && \
+          mkdir /run/nginx && \
+          apk add bash`,
         'COPY app /app',
         'ENV HOME=/app',
         'WORKDIR /app',