Bug 1522992 - generate TOC's in READMEs

README.md

@@ -1,4 +1,3 @@
-
 <h1 align="center">
   <br>
   <img src="https://media.taskcluster.net/logo/logo.png" alt="Taskcluster" width="80">
@@ -28,6 +27,50 @@
 This repository is used to develop, build, and release the Taskcluster services.
 It is not possible to run a full Taskcluster deployment directly from this repository, although individual services can be run for development purposes.
 
+## Table of Contents
+
+<!-- TOC BEGIN -->
+* [Infrastructure](infrastructure)
+    * [References](infrastructure/references)
+    * [Terraform](infrastructure/terraform)
+    * [Taskcluster Builder](infrastructure/builder)
+* [Libraries](libraries)
+    * [Typed-Env-Config Library](libraries/typed-env-config)
+    * [References Library](libraries/references)
+    * [Validate Library](libraries/validate)
+    * [Testing Library](libraries/testing)
+    * [Iterate Library](libraries/iterate)
+    * [Monitor Library](libraries/monitor)
+    * [Scopes Library](libraries/scopes)
+    * [Taskcluster Client](libraries/client)
+    * [Loader Library](libraries/loader)
+    * [Azure Library](libraries/azure)
+    * [Pulse Library](libraries/pulse)
+    * [Docs Library](libraries/docs)
+    * [API Library](libraries/api)
+    * [App Library](libraries/app)
+* [Services](services)
+    * [Built-In Workers Service](services/built-in-workers)
+    * [Purge-Cache Service](services/purge-cache)
+    * [Web-Server Service](services/web-server)
+    * [Treeherder Service](services/treeherder)
+    * [Secrets Service](services/secrets)
+    * [Notify Service](services/notify)
+    * [Github Service](services/github)
+    * [Queue Service](services/queue)
+    * [Login Service](services/login)
+    * [Index Service](services/index)
+    * [Hooks Service](services/hooks)
+    * [Auth Service](services/auth)
+* [Taskcluster Web](ui)
+    * [ui/src/components/DateDistance](ui/src/components/DateDistance)
+    * [ui/src/components/StatusLabel](ui/src/components/StatusLabel)
+    * [ui/src/components/SpeedDial](ui/src/components/SpeedDial)
+    * [ui/src/components/Snackbar](ui/src/components/Snackbar)
+    * [ui/src/components/Markdown](ui/src/components/Markdown)
+    * [ui/src/components/Search](ui/src/components/Search)
+<!-- TOC END -->
+
 ### Setup
 
 To set up the repository, run `yarn` in the root directory.

generated/docs/auth/README.md

@@ -1,5 +1,4 @@
-Taskcluster - Authentication Server
------------------------------------
+# Auth Service
 
 The taskcluster authentication server manages permissions and credentials
 in the taskcluster eco-system. Identifiers, credentials and authorized

generated/docs/built-in-workers/README.md

@@ -1,4 +1,4 @@
-# Taskcluster Built-In Workers
+# Built-In Workers Service
 
 This service implements the `built-in/succeed` and `built-in/fail` workerTypes, which simply succeed or fail immediately.
 Such workerTypes are useful the same way the `true` and `false` commands are useful in UNIX.

generated/docs/github/README.md

@@ -1,5 +1,4 @@
-Taskcluster Github Service
-==========================
+# Github Service
 
 This service monitors all of the repositories associated with an organization for changes and schedules Taskcluster tasks for any repository which contains a `.taskcluster.yml` configuration file. The goal of this project is to provide project owners a method for scheduling jobs in Taskcluster which is quick and straight forward.
 

generated/docs/hooks/README.md

@@ -1,5 +1,4 @@
-TaskCluster Hooks
-=================
+# Hooks Service
 
 A hooks service for triggering tasks from events.
 

generated/docs/index/README.md

@@ -1,5 +1,4 @@
-TaskCluster - Task Index
-========================
+# Index Service
 
 The task _index_ provides a service that indexes successfully completed tasks.
 To get a task indexed you must add routes on the form `index.<namespace>`, where

generated/docs/login/README.md

@@ -1,5 +1,4 @@
-Taskcluster User Login Service
-==============================
+# Login Service
 
 This service supports the generation of Taskcluster credentials appropriate to
 a user.

generated/docs/notify/README.md

@@ -1,5 +1,4 @@
-TaskCluster Notifications Service
-=================================
+# Notify Service
 
 No longer will you need to keep going back to the task-inspector page to know if your task is complete! Merely add some routes and we will tell you when your task is done! Note: You'll need to have the appropriate scopes to add these routes.
 

generated/docs/purge-cache/README.md

@@ -1,5 +1,4 @@
-Taskcluster Purge Worker Cache Service
-======================================
+# Purge-Cache Service
 
 Many taskcluster workers implements some generic form cache folders.
 These cache often have a `name` that identifies them, for example a task

generated/docs/queue/README.md

@@ -1,4 +1,4 @@
-# Taskcluster Queue
+# Queue Service
 
 This is the central queue coordinating execution of tasks in the Taskcluster setup.
 

generated/docs/secrets/README.md

@@ -1,4 +1,4 @@
-# TaskCluster Secrets Service
+# Secrets Service
 
 The secrets service allows task cluster clients with appropriate scopes to write secrets securely, and in such a way that each secret is tied to a scope. Further, scopes are used to limit the operations a particular client may perform on any secret they have been granted access to.
 

generated/docs/treeherder/README.md

@@ -1,5 +1,4 @@
-TaskCluster - Treeherder Integration
-====================================
+# Treeherder Service
 
 taskcluster-treeherder is a service that will respond to TaskCluster task events
 (e.g. task completed, task failed, etc) and compose a Treeherder job pulse message

infrastructure/README.md

@@ -1,3 +1,3 @@
-# Taskcluster Infrastructure
+# Infrastructure
 
 Tools and packages that are used to run a deployment of Taskcluster.

infrastructure/builder/README.md

@@ -1,5 +1,4 @@
-Taskcluster Builder
-===================
+# Taskcluster Builder
 
 This tool builds Taskcluster images and performs code generation.
 

infrastructure/builder/src/generate/generators/readme-tocs.js

@@ -0,0 +1,75 @@
+const path = require('path');
+const fs = require('fs');
+const {gitLsFiles, readFile, writeFile} = require('../util');
+
+exports.tasks = [{
+  title: 'README TOCs',
+  provides: ['target-readme-tocs'],
+  run: async (requirements, utils) => {
+    utils.status({message: 'gathering READMEs'});
+    let readmes = (await gitLsFiles())
+      .filter(file => file.endsWith('README.md'))
+      // ignore generated output
+      .filter(file => !file.startsWith('generated/'))
+      // some test directories have READMEs
+      .filter(file => !file.match(/\/test\//))
+      .map(file => ({dir: path.dirname(file).replace(/^.$/, ''), children: []}));
+
+    // read each README and extract titles where available
+    const firstLine = /^# (.*)\n/;
+    for (let readme of readmes) {
+      readme.content = await readFile(path.join(readme.dir, 'README.md'));
+      const match = firstLine.exec(readme.content);
+      if (match) {
+        readme.title = match[1];
+      }
+    }
+
+    // organize readmes into a hierarchy
+    readmes.sort((a, b) => b.dir.length - a.dir.length);
+    for (let readme of [...readmes]) {
+      const remainder = [];
+      for (let child of readmes) {
+        if (child.dir.length > readme.dir.length && child.dir.startsWith(readme.dir)) {
+          readme.children.push(child);
+        } else {
+          remainder.push(child);
+        }
+      }
+      readmes = remainder;
+    }
+
+    // generate the lines of a table of contents for a particular README
+    const tocLines = (lines, indent, dir, children) => {
+      for (let child of children) {
+        const relative = path.relative(dir, child.dir);
+        const title = child.title || relative;
+        lines.push(`${indent}* [${title}](${relative})`);
+        tocLines(lines, `${indent}    `, dir, child.children);
+      }
+      return lines;
+    };
+
+    // recursively write out TOC's
+    const rewrite = async ({content, dir, title, children}) => {
+      const lines = tocLines([], '', dir, children);
+      if (lines.length > 0) {
+        utils.status({message: `rewriting ${path.join(dir, 'README.md')}`});
+        const newContent = content.replace(
+          /(<!-- TOC BEGIN -->)(?:.|\n)*(<!-- TOC END -->)/m,
+          `$1\n${lines.join('\n')}\n$2`);
+        if (content !== newContent) {
+          await writeFile(path.join(dir, 'README.md'), newContent);
+        }
+      }
+
+      for (let child of children) {
+        await rewrite(child);
+      }
+    };
+
+    for (let readme of readmes) {
+      await rewrite(readme);
+    }
+  },
+}];

infrastructure/builder/src/generate/util.js

@@ -2,6 +2,7 @@ const {promisify} = require('util');
 const path = require('path');
 const fs = require('fs');
 const stringify = require('json-stable-stringify');
+const exec = promisify(require('child_process').execFile);
 
 const readFile = promisify(fs.readFile);
 const writeFile = promisify(fs.writeFile);
@@ -64,3 +65,13 @@ exports.modifyJSON = async (filename, modifier) => {
     return JSON.stringify(data, null, 2) + '\n';
   });
 };
+
+/**
+ * Call `git ls-files`
+ */
+exports.gitLsFiles = async () => {
+  const opts = {cwd: REPO_ROOT};
+  const files = (await exec('git', ['ls-files', '-z'], opts))
+    .stdout.split(/\0/);
+  return files;
+};

infrastructure/terraform/README.md

@@ -1,4 +1,4 @@
-# Taskcluster Terraform
+# Terraform
 
 A simple module to set up some requirements for a Taskcluster deployment.
 

libraries/README.md

@@ -1,3 +1,22 @@
-# Taskcluster Libraries
+# Libraries
 
 These are any packages that are used by Taskcluster services that aren't intended to be used widely externally.
+
+## Table of Contents
+
+<!-- TOC BEGIN -->
+* [Typed-Env-Config Library](typed-env-config)
+* [References Library](references)
+* [Validate Library](validate)
+* [Testing Library](testing)
+* [Iterate Library](iterate)
+* [Monitor Library](monitor)
+* [Scopes Library](scopes)
+* [Taskcluster Client](client)
+* [Loader Library](loader)
+* [Azure Library](azure)
+* [Pulse Library](pulse)
+* [Docs Library](docs)
+* [API Library](api)
+* [App Library](app)
+<!-- TOC END -->

libraries/api/README.md

@@ -1,5 +1,4 @@
-TaskCluster API
-===============
+# API Library
 
 This library provides support for building an API for a TaskCluster
 microservice.  It consists of some abstractions over `express` for declaring

libraries/app/README.md

@@ -1,5 +1,4 @@
-TaskCluster-Lib-App
-===================
+# App Library
 
 This library supports TaskCluster microservices, providing a pre-built Express
 server based on a common configuration format.

libraries/azure/README.md

@@ -1,4 +1,4 @@
-# Taskcluster Azure Shim
+# Azure Library
 
 A simple library to allow using `azure-entities` and the like, with Taskcluster credentials.
 

libraries/client/README.md

@@ -1,4 +1,4 @@
-# TaskCluster Client
+# Taskcluster Client
 
 This client library is generated from the auto-generated API reference.
 You can create a Client class from a JSON reference object at runtime using

libraries/docs/README.md

@@ -1,4 +1,4 @@
-# Taskcluster Documentation Tool
+# Docs Library
 
 A simple library to support generation of metadata for Taskcluster libraries, services, and applications.
 

libraries/iterate/README.md

@@ -1,4 +1,4 @@
-# taskcluster-lib-iterate
+# Iterate Library
 
 The motivation for this library is to provide a common framework for the running
 of code many times in a robust and fail-safe manner.  At its core, this library

libraries/loader/README.md

@@ -1,4 +1,4 @@
-# Taskcluster Component Loader
+# Loader Library
 
 This library provides a means of loading application "components", each of
 which can depend on other components.  This makes application startup more

libraries/monitor/README.md

@@ -1,5 +1,4 @@
-TaskCluster Metrics and Monitoring Library
-==========================================
+# Monitor Library
 
 A convenient library to wrap up all of the pieces needed for a Taskcluster service to record metrics with Statsum and report errors with Sentry.
 By default it will report any errors that cause the process to exit, and report as warnings any errors that cause stats writing to not work. To

libraries/pulse/README.md

@@ -1,4 +1,4 @@
-# taskcluster-lib-pulse
+# Pulse Library
 
 Library for interacting with Pulse and Taskcluster-Pulse.  See [the
 docs](https://docs.taskcluster.net/manual/design/apis/pulse) for more

libraries/references/README.md

@@ -1,4 +1,4 @@
-# taskcluster-lib-references
+# References Library
 
 Taskcluster-lib-references is responsible for handling the API reference data,
 including manifests, API references, exchange references, and JSON schemas.

libraries/scopes/README.md

@@ -1,5 +1,4 @@
-Taskcluster Scopes Utilities
-============================
+# Scopes Library
 
 Simple utilities to validate scopes, scope-sets, and scope-expression satisfiability.
 

libraries/testing/README.md

@@ -1,5 +1,4 @@
-Taskcluster-lib-testing
-=======================
+# Testing Library
 
 Support for testing TaskCluster components.
 

libraries/typed-env-config/README.md

@@ -1,5 +1,4 @@
-YAML Configuration Loader
-=========================
+# Typed-Env-Config Library
 
 This module makes it easy to load configuration from YAML files, and allows
 these YAML files to specify environment variables to substitute into the

libraries/validate/README.md

@@ -1,5 +1,4 @@
-TaskCluster Validation Library
-==============================
+# Validate Library
 
 A single purpose library to wrap up all of the logic for ensuring that
 content matches established schemas. This is a replacement for

services/README.md

@@ -1,3 +1,20 @@
-# Taskcluster Services
+# Services
 
 These are any packages that are run as part of a deployment of Taskcluster.
+
+## Table of Contents
+
+<!-- TOC BEGIN -->
+* [Built-In Workers Service](built-in-workers)
+* [Purge-Cache Service](purge-cache)
+* [Web-Server Service](web-server)
+* [Treeherder Service](treeherder)
+* [Secrets Service](secrets)
+* [Notify Service](notify)
+* [Github Service](github)
+* [Queue Service](queue)
+* [Login Service](login)
+* [Index Service](index)
+* [Hooks Service](hooks)
+* [Auth Service](auth)
+<!-- TOC END -->

services/auth/README.md

@@ -1,5 +1,4 @@
-Taskcluster - Authentication Server
------------------------------------
+# Auth Service
 
 The taskcluster authentication server manages permissions and credentials
 in the taskcluster eco-system. Identifiers, credentials and authorized

services/built-in-workers/README.md

@@ -1,4 +1,4 @@
-# Taskcluster Built-In Workers
+# Built-In Workers Service
 
 This service implements the `built-in/succeed` and `built-in/fail` workerTypes, which simply succeed or fail immediately.
 Such workerTypes are useful the same way the `true` and `false` commands are useful in UNIX.