diff --git a/README.md b/README.md index 2b17167fb54..0bb2dcf61a8 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,3 @@ -


Taskcluster @@ -28,6 +27,50 @@ This repository is used to develop, build, and release the Taskcluster services. It is not possible to run a full Taskcluster deployment directly from this repository, although individual services can be run for development purposes. +## Table of Contents + + +* [Infrastructure](infrastructure) + * [References](infrastructure/references) + * [Terraform](infrastructure/terraform) + * [Taskcluster Builder](infrastructure/builder) +* [Libraries](libraries) + * [Typed-Env-Config Library](libraries/typed-env-config) + * [References Library](libraries/references) + * [Validate Library](libraries/validate) + * [Testing Library](libraries/testing) + * [Iterate Library](libraries/iterate) + * [Monitor Library](libraries/monitor) + * [Scopes Library](libraries/scopes) + * [Taskcluster Client](libraries/client) + * [Loader Library](libraries/loader) + * [Azure Library](libraries/azure) + * [Pulse Library](libraries/pulse) + * [Docs Library](libraries/docs) + * [API Library](libraries/api) + * [App Library](libraries/app) +* [Services](services) + * [Built-In Workers Service](services/built-in-workers) + * [Purge-Cache Service](services/purge-cache) + * [Web-Server Service](services/web-server) + * [Treeherder Service](services/treeherder) + * [Secrets Service](services/secrets) + * [Notify Service](services/notify) + * [Github Service](services/github) + * [Queue Service](services/queue) + * [Login Service](services/login) + * [Index Service](services/index) + * [Hooks Service](services/hooks) + * [Auth Service](services/auth) +* [Taskcluster Web](ui) + * [ui/src/components/DateDistance](ui/src/components/DateDistance) + * [ui/src/components/StatusLabel](ui/src/components/StatusLabel) + * [ui/src/components/SpeedDial](ui/src/components/SpeedDial) + * [ui/src/components/Snackbar](ui/src/components/Snackbar) + * [ui/src/components/Markdown](ui/src/components/Markdown) + * [ui/src/components/Search](ui/src/components/Search) + + ### Setup To set up the repository, run `yarn` in the root directory. diff --git a/generated/docs/auth/README.md b/generated/docs/auth/README.md index a636bb1d151..1c76d315391 100644 --- a/generated/docs/auth/README.md +++ b/generated/docs/auth/README.md @@ -1,5 +1,4 @@ -Taskcluster - Authentication Server ------------------------------------ +# Auth Service The taskcluster authentication server manages permissions and credentials in the taskcluster eco-system. Identifiers, credentials and authorized diff --git a/generated/docs/built-in-workers/README.md b/generated/docs/built-in-workers/README.md index 1781dadcc99..baf60c81b21 100644 --- a/generated/docs/built-in-workers/README.md +++ b/generated/docs/built-in-workers/README.md @@ -1,4 +1,4 @@ -# Taskcluster Built-In Workers +# Built-In Workers Service This service implements the `built-in/succeed` and `built-in/fail` workerTypes, which simply succeed or fail immediately. Such workerTypes are useful the same way the `true` and `false` commands are useful in UNIX. diff --git a/generated/docs/github/README.md b/generated/docs/github/README.md index 436ec1d017d..5b3bd88c83c 100644 --- a/generated/docs/github/README.md +++ b/generated/docs/github/README.md @@ -1,5 +1,4 @@ -Taskcluster Github Service -========================== +# Github Service This service monitors all of the repositories associated with an organization for changes and schedules Taskcluster tasks for any repository which contains a `.taskcluster.yml` configuration file. The goal of this project is to provide project owners a method for scheduling jobs in Taskcluster which is quick and straight forward. diff --git a/generated/docs/hooks/README.md b/generated/docs/hooks/README.md index 4e7a0bd2d0a..397dce671d7 100644 --- a/generated/docs/hooks/README.md +++ b/generated/docs/hooks/README.md @@ -1,5 +1,4 @@ -TaskCluster Hooks -================= +# Hooks Service A hooks service for triggering tasks from events. diff --git a/generated/docs/index/README.md b/generated/docs/index/README.md index 1c85f9c41d0..fb64b7b30e6 100644 --- a/generated/docs/index/README.md +++ b/generated/docs/index/README.md @@ -1,5 +1,4 @@ -TaskCluster - Task Index -======================== +# Index Service The task _index_ provides a service that indexes successfully completed tasks. To get a task indexed you must add routes on the form `index.`, where diff --git a/generated/docs/login/README.md b/generated/docs/login/README.md index 8e59bc64d7e..215c45829d7 100644 --- a/generated/docs/login/README.md +++ b/generated/docs/login/README.md @@ -1,5 +1,4 @@ -Taskcluster User Login Service -============================== +# Login Service This service supports the generation of Taskcluster credentials appropriate to a user. diff --git a/generated/docs/notify/README.md b/generated/docs/notify/README.md index 27edecb992f..15e3eff0988 100644 --- a/generated/docs/notify/README.md +++ b/generated/docs/notify/README.md @@ -1,5 +1,4 @@ -TaskCluster Notifications Service -================================= +# Notify Service No longer will you need to keep going back to the task-inspector page to know if your task is complete! Merely add some routes and we will tell you when your task is done! Note: You'll need to have the appropriate scopes to add these routes. diff --git a/generated/docs/purge-cache/README.md b/generated/docs/purge-cache/README.md index 637d9370bd7..3158fceeaa4 100644 --- a/generated/docs/purge-cache/README.md +++ b/generated/docs/purge-cache/README.md @@ -1,5 +1,4 @@ -Taskcluster Purge Worker Cache Service -====================================== +# Purge-Cache Service Many taskcluster workers implements some generic form cache folders. These cache often have a `name` that identifies them, for example a task diff --git a/generated/docs/queue/README.md b/generated/docs/queue/README.md index 30ebc27f389..bdf942a1cd9 100644 --- a/generated/docs/queue/README.md +++ b/generated/docs/queue/README.md @@ -1,4 +1,4 @@ -# Taskcluster Queue +# Queue Service This is the central queue coordinating execution of tasks in the Taskcluster setup. diff --git a/generated/docs/secrets/README.md b/generated/docs/secrets/README.md index 45d64dc070a..fa524de4218 100644 --- a/generated/docs/secrets/README.md +++ b/generated/docs/secrets/README.md @@ -1,4 +1,4 @@ -# TaskCluster Secrets Service +# Secrets Service The secrets service allows task cluster clients with appropriate scopes to write secrets securely, and in such a way that each secret is tied to a scope. Further, scopes are used to limit the operations a particular client may perform on any secret they have been granted access to. diff --git a/generated/docs/treeherder/README.md b/generated/docs/treeherder/README.md index 7730f981f9b..f0f158fd0cf 100644 --- a/generated/docs/treeherder/README.md +++ b/generated/docs/treeherder/README.md @@ -1,5 +1,4 @@ -TaskCluster - Treeherder Integration -==================================== +# Treeherder Service taskcluster-treeherder is a service that will respond to TaskCluster task events (e.g. task completed, task failed, etc) and compose a Treeherder job pulse message diff --git a/infrastructure/README.md b/infrastructure/README.md index 8d0a9e1ea74..0452f950b2d 100644 --- a/infrastructure/README.md +++ b/infrastructure/README.md @@ -1,3 +1,3 @@ -# Taskcluster Infrastructure +# Infrastructure Tools and packages that are used to run a deployment of Taskcluster. diff --git a/infrastructure/builder/README.md b/infrastructure/builder/README.md index f5a8e69b5d8..d8c8c55d6ca 100644 --- a/infrastructure/builder/README.md +++ b/infrastructure/builder/README.md @@ -1,5 +1,4 @@ -Taskcluster Builder -=================== +# Taskcluster Builder This tool builds Taskcluster images and performs code generation. diff --git a/infrastructure/builder/src/generate/generators/readme-tocs.js b/infrastructure/builder/src/generate/generators/readme-tocs.js new file mode 100644 index 00000000000..9d09dd8ae1f --- /dev/null +++ b/infrastructure/builder/src/generate/generators/readme-tocs.js @@ -0,0 +1,75 @@ +const path = require('path'); +const fs = require('fs'); +const {gitLsFiles, readFile, writeFile} = require('../util'); + +exports.tasks = [{ + title: 'README TOCs', + provides: ['target-readme-tocs'], + run: async (requirements, utils) => { + utils.status({message: 'gathering READMEs'}); + let readmes = (await gitLsFiles()) + .filter(file => file.endsWith('README.md')) + // ignore generated output + .filter(file => !file.startsWith('generated/')) + // some test directories have READMEs + .filter(file => !file.match(/\/test\//)) + .map(file => ({dir: path.dirname(file).replace(/^.$/, ''), children: []})); + + // read each README and extract titles where available + const firstLine = /^# (.*)\n/; + for (let readme of readmes) { + readme.content = await readFile(path.join(readme.dir, 'README.md')); + const match = firstLine.exec(readme.content); + if (match) { + readme.title = match[1]; + } + } + + // organize readmes into a hierarchy + readmes.sort((a, b) => b.dir.length - a.dir.length); + for (let readme of [...readmes]) { + const remainder = []; + for (let child of readmes) { + if (child.dir.length > readme.dir.length && child.dir.startsWith(readme.dir)) { + readme.children.push(child); + } else { + remainder.push(child); + } + } + readmes = remainder; + } + + // generate the lines of a table of contents for a particular README + const tocLines = (lines, indent, dir, children) => { + for (let child of children) { + const relative = path.relative(dir, child.dir); + const title = child.title || relative; + lines.push(`${indent}* [${title}](${relative})`); + tocLines(lines, `${indent} `, dir, child.children); + } + return lines; + }; + + // recursively write out TOC's + const rewrite = async ({content, dir, title, children}) => { + const lines = tocLines([], '', dir, children); + if (lines.length > 0) { + utils.status({message: `rewriting ${path.join(dir, 'README.md')}`}); + const newContent = content.replace( + /()(?:.|\n)*()/m, + `$1\n${lines.join('\n')}\n$2`); + if (content !== newContent) { + await writeFile(path.join(dir, 'README.md'), newContent); + } + } + + for (let child of children) { + await rewrite(child); + } + }; + + for (let readme of readmes) { + await rewrite(readme); + } + }, +}]; diff --git a/infrastructure/builder/src/generate/util.js b/infrastructure/builder/src/generate/util.js index 5886d04413d..675415a46ae 100644 --- a/infrastructure/builder/src/generate/util.js +++ b/infrastructure/builder/src/generate/util.js @@ -2,6 +2,7 @@ const {promisify} = require('util'); const path = require('path'); const fs = require('fs'); const stringify = require('json-stable-stringify'); +const exec = promisify(require('child_process').execFile); const readFile = promisify(fs.readFile); const writeFile = promisify(fs.writeFile); @@ -64,3 +65,13 @@ exports.modifyJSON = async (filename, modifier) => { return JSON.stringify(data, null, 2) + '\n'; }); }; + +/** + * Call `git ls-files` + */ +exports.gitLsFiles = async () => { + const opts = {cwd: REPO_ROOT}; + const files = (await exec('git', ['ls-files', '-z'], opts)) + .stdout.split(/\0/); + return files; +}; diff --git a/infrastructure/terraform/README.md b/infrastructure/terraform/README.md index 87da7324c8b..488f07fe386 100644 --- a/infrastructure/terraform/README.md +++ b/infrastructure/terraform/README.md @@ -1,4 +1,4 @@ -# Taskcluster Terraform +# Terraform A simple module to set up some requirements for a Taskcluster deployment. diff --git a/libraries/README.md b/libraries/README.md index d2c7cf179c6..b0d5c9680b8 100644 --- a/libraries/README.md +++ b/libraries/README.md @@ -1,3 +1,22 @@ -# Taskcluster Libraries +# Libraries These are any packages that are used by Taskcluster services that aren't intended to be used widely externally. + +## Table of Contents + + +* [Typed-Env-Config Library](typed-env-config) +* [References Library](references) +* [Validate Library](validate) +* [Testing Library](testing) +* [Iterate Library](iterate) +* [Monitor Library](monitor) +* [Scopes Library](scopes) +* [Taskcluster Client](client) +* [Loader Library](loader) +* [Azure Library](azure) +* [Pulse Library](pulse) +* [Docs Library](docs) +* [API Library](api) +* [App Library](app) + diff --git a/libraries/api/README.md b/libraries/api/README.md index 8143b6754f4..1d706407e6f 100644 --- a/libraries/api/README.md +++ b/libraries/api/README.md @@ -1,5 +1,4 @@ -TaskCluster API -=============== +# API Library This library provides support for building an API for a TaskCluster microservice. It consists of some abstractions over `express` for declaring diff --git a/libraries/app/README.md b/libraries/app/README.md index 83521bce215..ae28d097976 100644 --- a/libraries/app/README.md +++ b/libraries/app/README.md @@ -1,5 +1,4 @@ -TaskCluster-Lib-App -=================== +# App Library This library supports TaskCluster microservices, providing a pre-built Express server based on a common configuration format. diff --git a/libraries/azure/README.md b/libraries/azure/README.md index 941de6bfb69..55b2ce39937 100644 --- a/libraries/azure/README.md +++ b/libraries/azure/README.md @@ -1,4 +1,4 @@ -# Taskcluster Azure Shim +# Azure Library A simple library to allow using `azure-entities` and the like, with Taskcluster credentials. diff --git a/libraries/client/README.md b/libraries/client/README.md index 3d8f1a18e04..fc52889db18 100644 --- a/libraries/client/README.md +++ b/libraries/client/README.md @@ -1,4 +1,4 @@ -# TaskCluster Client +# Taskcluster Client This client library is generated from the auto-generated API reference. You can create a Client class from a JSON reference object at runtime using diff --git a/libraries/docs/README.md b/libraries/docs/README.md index b1f19ee77c5..7cafab87f71 100644 --- a/libraries/docs/README.md +++ b/libraries/docs/README.md @@ -1,4 +1,4 @@ -# Taskcluster Documentation Tool +# Docs Library A simple library to support generation of metadata for Taskcluster libraries, services, and applications. diff --git a/libraries/iterate/README.md b/libraries/iterate/README.md index 41abd874579..21a9af08f73 100644 --- a/libraries/iterate/README.md +++ b/libraries/iterate/README.md @@ -1,4 +1,4 @@ -# taskcluster-lib-iterate +# Iterate Library The motivation for this library is to provide a common framework for the running of code many times in a robust and fail-safe manner. At its core, this library diff --git a/libraries/loader/README.md b/libraries/loader/README.md index d8ab3ae93eb..df812b5ee76 100644 --- a/libraries/loader/README.md +++ b/libraries/loader/README.md @@ -1,4 +1,4 @@ -# Taskcluster Component Loader +# Loader Library This library provides a means of loading application "components", each of which can depend on other components. This makes application startup more diff --git a/libraries/monitor/README.md b/libraries/monitor/README.md index afb1d2f77c6..35e02c31734 100644 --- a/libraries/monitor/README.md +++ b/libraries/monitor/README.md @@ -1,5 +1,4 @@ -TaskCluster Metrics and Monitoring Library -========================================== +# Monitor Library A convenient library to wrap up all of the pieces needed for a Taskcluster service to record metrics with Statsum and report errors with Sentry. By default it will report any errors that cause the process to exit, and report as warnings any errors that cause stats writing to not work. To diff --git a/libraries/pulse/README.md b/libraries/pulse/README.md index 3c8e3a166a6..74533e121ef 100644 --- a/libraries/pulse/README.md +++ b/libraries/pulse/README.md @@ -1,4 +1,4 @@ -# taskcluster-lib-pulse +# Pulse Library Library for interacting with Pulse and Taskcluster-Pulse. See [the docs](https://docs.taskcluster.net/manual/design/apis/pulse) for more diff --git a/libraries/references/README.md b/libraries/references/README.md index 9c03569f382..ce02b4a5116 100644 --- a/libraries/references/README.md +++ b/libraries/references/README.md @@ -1,4 +1,4 @@ -# taskcluster-lib-references +# References Library Taskcluster-lib-references is responsible for handling the API reference data, including manifests, API references, exchange references, and JSON schemas. diff --git a/libraries/scopes/README.md b/libraries/scopes/README.md index 30367f3e9ca..49bd83ead3f 100644 --- a/libraries/scopes/README.md +++ b/libraries/scopes/README.md @@ -1,5 +1,4 @@ -Taskcluster Scopes Utilities -============================ +# Scopes Library Simple utilities to validate scopes, scope-sets, and scope-expression satisfiability. diff --git a/libraries/testing/README.md b/libraries/testing/README.md index c345b7c17a4..64dd31aafaf 100644 --- a/libraries/testing/README.md +++ b/libraries/testing/README.md @@ -1,5 +1,4 @@ -Taskcluster-lib-testing -======================= +# Testing Library Support for testing TaskCluster components. diff --git a/libraries/typed-env-config/README.md b/libraries/typed-env-config/README.md index e6808078def..ce354c6d50d 100644 --- a/libraries/typed-env-config/README.md +++ b/libraries/typed-env-config/README.md @@ -1,5 +1,4 @@ -YAML Configuration Loader -========================= +# Typed-Env-Config Library This module makes it easy to load configuration from YAML files, and allows these YAML files to specify environment variables to substitute into the diff --git a/libraries/validate/README.md b/libraries/validate/README.md index 3f8415993e8..2431bdf30a4 100644 --- a/libraries/validate/README.md +++ b/libraries/validate/README.md @@ -1,5 +1,4 @@ -TaskCluster Validation Library -============================== +# Validate Library A single purpose library to wrap up all of the logic for ensuring that content matches established schemas. This is a replacement for diff --git a/services/README.md b/services/README.md index 32543134591..9763678b277 100644 --- a/services/README.md +++ b/services/README.md @@ -1,3 +1,20 @@ -# Taskcluster Services +# Services These are any packages that are run as part of a deployment of Taskcluster. + +## Table of Contents + + +* [Built-In Workers Service](built-in-workers) +* [Purge-Cache Service](purge-cache) +* [Web-Server Service](web-server) +* [Treeherder Service](treeherder) +* [Secrets Service](secrets) +* [Notify Service](notify) +* [Github Service](github) +* [Queue Service](queue) +* [Login Service](login) +* [Index Service](index) +* [Hooks Service](hooks) +* [Auth Service](auth) + diff --git a/services/auth/README.md b/services/auth/README.md index a636bb1d151..1c76d315391 100644 --- a/services/auth/README.md +++ b/services/auth/README.md @@ -1,5 +1,4 @@ -Taskcluster - Authentication Server ------------------------------------ +# Auth Service The taskcluster authentication server manages permissions and credentials in the taskcluster eco-system. Identifiers, credentials and authorized diff --git a/services/built-in-workers/README.md b/services/built-in-workers/README.md index 1781dadcc99..baf60c81b21 100644 --- a/services/built-in-workers/README.md +++ b/services/built-in-workers/README.md @@ -1,4 +1,4 @@ -# Taskcluster Built-In Workers +# Built-In Workers Service This service implements the `built-in/succeed` and `built-in/fail` workerTypes, which simply succeed or fail immediately. Such workerTypes are useful the same way the `true` and `false` commands are useful in UNIX. diff --git a/services/github/README.md b/services/github/README.md index 436ec1d017d..5b3bd88c83c 100644 --- a/services/github/README.md +++ b/services/github/README.md @@ -1,5 +1,4 @@ -Taskcluster Github Service -========================== +# Github Service This service monitors all of the repositories associated with an organization for changes and schedules Taskcluster tasks for any repository which contains a `.taskcluster.yml` configuration file. The goal of this project is to provide project owners a method for scheduling jobs in Taskcluster which is quick and straight forward. diff --git a/services/hooks/README.md b/services/hooks/README.md index 4e7a0bd2d0a..397dce671d7 100644 --- a/services/hooks/README.md +++ b/services/hooks/README.md @@ -1,5 +1,4 @@ -TaskCluster Hooks -================= +# Hooks Service A hooks service for triggering tasks from events. diff --git a/services/index/README.md b/services/index/README.md index 1c85f9c41d0..fb64b7b30e6 100644 --- a/services/index/README.md +++ b/services/index/README.md @@ -1,5 +1,4 @@ -TaskCluster - Task Index -======================== +# Index Service The task _index_ provides a service that indexes successfully completed tasks. To get a task indexed you must add routes on the form `index.`, where diff --git a/services/login/README.md b/services/login/README.md index 8e59bc64d7e..215c45829d7 100644 --- a/services/login/README.md +++ b/services/login/README.md @@ -1,5 +1,4 @@ -Taskcluster User Login Service -============================== +# Login Service This service supports the generation of Taskcluster credentials appropriate to a user. diff --git a/services/notify/README.md b/services/notify/README.md index 27edecb992f..15e3eff0988 100644 --- a/services/notify/README.md +++ b/services/notify/README.md @@ -1,5 +1,4 @@ -TaskCluster Notifications Service -================================= +# Notify Service No longer will you need to keep going back to the task-inspector page to know if your task is complete! Merely add some routes and we will tell you when your task is done! Note: You'll need to have the appropriate scopes to add these routes. diff --git a/services/purge-cache/README.md b/services/purge-cache/README.md index 637d9370bd7..3158fceeaa4 100644 --- a/services/purge-cache/README.md +++ b/services/purge-cache/README.md @@ -1,5 +1,4 @@ -Taskcluster Purge Worker Cache Service -====================================== +# Purge-Cache Service Many taskcluster workers implements some generic form cache folders. These cache often have a `name` that identifies them, for example a task diff --git a/services/queue/README.md b/services/queue/README.md index 30ebc27f389..bdf942a1cd9 100644 --- a/services/queue/README.md +++ b/services/queue/README.md @@ -1,4 +1,4 @@ -# Taskcluster Queue +# Queue Service This is the central queue coordinating execution of tasks in the Taskcluster setup. diff --git a/services/secrets/README.md b/services/secrets/README.md index 45d64dc070a..fa524de4218 100644 --- a/services/secrets/README.md +++ b/services/secrets/README.md @@ -1,4 +1,4 @@ -# TaskCluster Secrets Service +# Secrets Service The secrets service allows task cluster clients with appropriate scopes to write secrets securely, and in such a way that each secret is tied to a scope. Further, scopes are used to limit the operations a particular client may perform on any secret they have been granted access to. diff --git a/services/treeherder/README.md b/services/treeherder/README.md index 7730f981f9b..f0f158fd0cf 100644 --- a/services/treeherder/README.md +++ b/services/treeherder/README.md @@ -1,5 +1,4 @@ -TaskCluster - Treeherder Integration -==================================== +# Treeherder Service taskcluster-treeherder is a service that will respond to TaskCluster task events (e.g. task completed, task failed, etc) and compose a Treeherder job pulse message diff --git a/services/web-server/README.md b/services/web-server/README.md index cc8c84ebcb8..800ca48c0a8 100644 --- a/services/web-server/README.md +++ b/services/web-server/README.md @@ -1,4 +1,4 @@ -# Taskcluster Web Server +# Web-Server Service A web server for supporting the taskcluster-web UI repository. Serves as a GraphQL gateway to the Taskcluster REST APIs, and should eventually support and diff --git a/ui/README.md b/ui/README.md index ca17839a598..10f9ee2846a 100644 --- a/ui/README.md +++ b/ui/README.md @@ -101,3 +101,14 @@ import BookOpenPageVariantIcon from 'mdi-react/BookOpenPageVariantIcon'; ``` * We use this library because it provides substantially more icons with minimal file-system headaches. + +## Table of Contents + + +* [src/components/DateDistance](src/components/DateDistance) +* [src/components/StatusLabel](src/components/StatusLabel) +* [src/components/SpeedDial](src/components/SpeedDial) +* [src/components/Snackbar](src/components/Snackbar) +* [src/components/Markdown](src/components/Markdown) +* [src/components/Search](src/components/Search) +