The Elasticsearch API Specification provides the contract for communication between client and server components within the Elasticsearch stack. With almost 500 API endpoints and around 3000 data types across the entire API surface, this project is a vitally important part of sustaining our engineering efforts at scale.
The repository has the following structure:
| Path | Description |
|---|---|
api-design-guidelines/ |
Knowledge base of best practices for API design. |
compiler/ |
TypeScript compiler for specification definition to JSON. |
compiler-rs/ |
|
docs/ |
|
output/ |
|
specification/ |
Elasticsearch request/response definitions in TypeScript. |
typescript-generator/ |
This JSON representation is formally defined by a set of TypeScript definitions (a meta-model) that also explains the various properties and their values.
For generating the JSON representation and running the validation code you need to install and configure Node.js in your development environment.
You can install Node.js with nvm:
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bashOnce the installation is completed, install Node.js with nvm:
# this command will install the version configured in .nvmrc
nvm install# clone the project
$ git clone https://github.com/elastic/elasticsearch-specification.git
# install the dependencies
$ make setup
# generate the JSON representation
$ make generate
# the generated output can be found in ./output/schema/schema.json
$ cat output/schema/schema.json
Usage:
make <target>
validate Validate a given endpoint request or response
validate-no-cache Validate a given endpoint request or response without local cache
generate Generate the output spec
compile Compile the specification
license-check Add the license headers to the files
license-add Add the license headers to the files
spec-format-check Check specification formatting rules
spec-format-fix Format/fix the specification according to the formatting rules
spec-dangling-types Generate the dangling types rreport
setup Install dependencies for contrib target
clean-dep Clean npm dependencies
transform-expand-generics Create a new schema with all generics expanded
transform-to-openapi Generate the OpenAPI definition from the compiled schema
filter-for-serverless Generate the serverless version from the compiled schema
dump-routes Create a new schema with all generics expanded
contrib Pre contribution target
lint-docs Lint the OpenAPI documents
lint-docs-serverless Lint only the serverless OpenAPI document
help Display help
The JSON representation is formally defined as TypeScript definitions. Refer to them for the full details. It is an object with two top level keys:
{
"types": [...],
"endpoints": [...]
}The first one, types, contains all the type definitions from the specification, such as
IndexRequest or MainError, while the second one, endpoints, contains every
endpoint of Elasticsearch and the respective type mapping. For example:
{
"types": [ {
"attachedBehaviors": [
"CommonQueryParameters"
],
"body": {
"kind": "value",
"value": {
"kind": "instance_of",
"type": {
"name": "TDocument",
"namespace": "_global.index"
}
}
},
"generics": [
{
"name": "TDocument",
"namespace": "_global.index"
}
],
"inherits": {
"type": {
"name": "RequestBase",
"namespace": "_types"
}
},
"kind": "request",
"name": {
"name": "Request",
"namespace": "_global.index"
},
"path": [...],
"query": [...]
}, {
"inherits": {
"type": {
"name": "WriteResponseBase",
"namespace": "_types"
}
},
"kind": "response",
"name": {
"name": "Response",
"namespace": "_global.index"
}
}],
"endpoints": [{
"accept": [
"application/json"
],
"contentType": [
"application/json"
],
"description": "Creates or updates a document in an index.",
"docUrl": "https://www.elastic.co/guide/en/elasticsearch/reference/master/docs-index_.html",
"name": "index",
"request": {
"name": "Request",
"namespace": "_global.index"
},
"requestBodyRequired": true,
"response": {
"name": "Response",
"namespace": "_global.index"
},
"since": "0.0.0",
"stability": "stable",
"urls": [...],
"visibility": "public"
}]
}The example above represents the index
request, inside the endpoints array you can find the API name and the type mappings under request.name
and response.name. The respective type definitons can be found inside the types array.
In some cases an endpoint might be defined, but there is no a type definition yet, in such case
the request and response value will be null.
The specification is validated daily by the client-flight-recorder project. The validation result can be found here.
The following step only apply if you don't have ~/.elastic/github.token in place.
Create GitHub token to allow authentication with Vault.
- Go to https://github.com/settings/tokens.
- Click
Generate new token. - Give your token a name and make sure to click the
repoandread:orgscopes. - Create a file at
~/.elastic/github.tokenand paste the GitHub token into it. - Change permissions on the file allow access only from the user.
chmod 600 ~/.elastic/github.token
You can see here how to generate a token.
Once you have configured the environment, run the following commands:
git clone https://github.com/elastic/elasticsearch-specification.git
git clone https://github.com/elastic/clients-flight-recorder.git
cd elasticsearch-specification
# this will validate the xpack.info request type against the main branch of Elasticsearch
make validate api=xpack.info type=request branch=main
# this will validate the xpack.info request and response types against the 8.15 branch
make validate api=xpack.info branch=8.15The last command above will install all the dependencies and run, download
the test recordings and finally validate the specification.
If you need to download the recordings again, run make validate-no-cache api=xpack.info type=request branch=main.
Once you see the errors, you can fix the original definition in /specification
and then run the command again until the types validator does not trigger any new error.
Finally open a pull request with your changes.
- How to add a new API
- Behaviors
- Compiler
- Documenting the API specification
- Known issues
- Modeling Guide
- Schema structure
- Specification structure
- Style Guide
- Fixing a defintion, a complete story
You can find a report of the main branch here.
When you define a property the syntax is propertyName: propertyType.
By default a property is required to exist. If you know that a property will not
always be there, you can add a question mark just before the column:
propertyRequired: string
propertyOptional?: stringSee here.
All the definitons are inside /specification folder, search the bad defintion and update it,
you can find above how to run the validation of the spec.
See here.
All the endpoint definitons are inside /specification/_json_spec folder, which contains a series of
JSON files taken directly from the Elasticsearch rest-api-spec.
You should copy from there the updated endpoint defintion and change it here.
Very likely the recordings on your machine are stale, rerun the validation with the validate-no-cache make target.
You should pull the latest change from the client-flight-recorder as well.
cd client-flight-recorder
git pullEverytime you run make validate script, a series of test will be generated and dumped on disk.
You can find the failed tests in clients-flight-recorder/scripts/types-validator/workbench.
The content of this folder is a series of recorded responses from Elasticsearch wrapped inside an helper
that verifies if the type definiton is correct.
Any editor is fine, but to have a better development experience it should be configured to work with TypeScript. Visual Studio Code and IntelliJ IDEA come with TypeScript support out of the box.
Yes, take a look here.
The validation script uses realpath which may be not present in your system. If you are using MacOS, run the following command to fix the issue:
brew install coreutilsTake a look at the compiler documentation.
The work of several repositories come together in this repository. This diagram aims to sketch an overview of how different pieces connect
