diff --git a/_sass/friendly_code.scss b/_sass/friendly_code.scss index 4339b108e..796a8844d 100644 --- a/_sass/friendly_code.scss +++ b/_sass/friendly_code.scss @@ -61,4 +61,7 @@ .highlight .vc { color: #bb60d5 } /* Name.Variable.Class */ .highlight .vg { color: #bb60d5 } /* Name.Variable.Global */ .highlight .vi { color: #bb60d5 } /* Name.Variable.Instance */ -.highlight .il { color: #40a070 } /* Literal.Number.Integer.Long */ \ No newline at end of file +.highlight .il { color: #40a070 } /* Literal.Number.Integer.Long */ + +.language-json-doc .err { border: none; font-style: italic } +.language-js .nx { font-style: italic; color: #007020; } \ No newline at end of file diff --git a/api/index.md b/api/index.md index 14a3981e0..1b7f548bf 100644 --- a/api/index.md +++ b/api/index.md @@ -4,6 +4,7 @@ title: API nav_order: 20 has_children: true --- + # Overview The [Data Commons Graph](https://datacommons.org) aggregates data from many @@ -11,32 +12,22 @@ different [data sources](https://datacommons.org/datasets) into a single database. Data Commons is based on the data model used by [schema.org](https://schema.org); for more information, see [the guide to the data model](/data_model.html). -The **Data Commons API** allows developers to programmatically access the data in Data Commons. -Data Commons provides several different ways to access its API's resources: +The Data Commons APIs allow developers to programmatically access the data in Data Commons. +Data Commons provides several different ways to access its resources: -1. A [REST API](/api/rest/v2) that can be used on the command line as well as in any language with an HTTP library. -1. A lightweight [Python](/api/python) wrapper. -1. A heavier [Pandas](/api/pandas) wrapper. -1. A convenient [Google Sheets](/api/sheets) add-on. +* A [REST API](/api/rest/v2) that can be used on the command line as well as in any language with an HTTP library. +* [Python](/api/python) and [Pandas](/api/pandas) wrappers. -The endpoints can be roughly grouped into four categories. +The endpoints can be roughly grouped into four categories: -- **Local Node Exploration**: Given a set of nodes, explore the - graph around those nodes. +- **Statistical data**: Given a set of statistical variables, dates and entities, get observations. -- **Domain specific APIs**: These are groups of APIs, specific to particular - domains. +- **Graph exploration**: Given a set of nodes, explore the + graph around those nodes. -- **Graph Query/SPARQL**: Given a subgraph where some of the nodes are +- **Graph query/SPARQL**: Given a subgraph where some of the nodes are variables, retrieve possible matches. This corresponds to a subset of the - graph query language [SPARQL](https://www.w3.org/TR/rdf-sparql-query/). - -- **Utilities**: These are Python notebook specific APIs for helping with - Pandas DataFrames, etc. - -Most of the provided endpoints take references to nodes and properties as arguments. Every -node or property has a `Data Commons ID (DCID)`, which is used -to pass nodes as arguments to API calls. - -**Note:** The DCID of schema.org terms used in Data Commons is their schema.org ID. + graph query language [SPARQL](https://www.w3.org/TR/rdf-sparql-query/). This is useful for getting very specific observations which would require multiple API calls (e.g. "hate crimes motivated by disability status in Californian cities"). +- **Utilities**: These are Python notebook-specific APIs for helping with + Pandas DataFrames, etc. \ No newline at end of file diff --git a/api/python/query.md b/api/python/query.md index ad9b75dc2..7fdfc217b 100644 --- a/api/python/query.md +++ b/api/python/query.md @@ -12,7 +12,9 @@ Returns the results of running a graph query on the Data Commons knowledge graph using [SPARQL](https://www.w3.org/TR/rdf-sparql-query/). Note that Data Commons is only able to support a limited subsection of SPARQL functionality at this time: specifically only the keywords `ORDER BY`, `DISTINCT`, and `LIMIT`. -## General information about this method +Note: The Python SPARQL library currently only supports the [v1](/api/v1/query.html) version of the API. + +## General information about the query() method **Signature**: @@ -24,7 +26,7 @@ datacommons.query(query_string, select=None) * `query_string`: A SPARQL query string. -## How to construct a call to the query method +## How to construct a call to the query() method This method makes it possible to query the Data Commons knowledge graph using SPARQL. SPARQL is a query language developed to retrieve data from websites. It leverages the graph structure innate in the data it queries to return specific information to an end user. For more information on assembling SPARQL queries, check out [the Wikipedia page about SPARQL](https://en.wikipedia.org/wiki/SPARQL) and [the W3C specification information](https://www.w3.org/TR/sparql11-query/). @@ -49,11 +51,11 @@ A correct response will always look like this: The response contains an array of dictionaries, each corresponding to one node matching the conditions of the query. Each dictionary's keys match the variables in the query SELECT clause, and the values in the dictionaries are those associated to the given node's query-specified properties. -## Examples and error returns +## Examples and error responses -### Examples +The following examples and error responses, along with explanations and fixes for the errors, are available in this [Python notebook](https://colab.research.google.com/drive/1Jd0IDHnMdtxhsmXhL5Ib5tL0zgJud1k5?usp=sharing). -#### Example 1. Retrieve the name of the state associated with DCID geoId/06. +### Example 1: Retrieve the name of the state associated with DCID geoId/06. ```python >>> geoId06_name_query = 'SELECT ?name ?dcid WHERE { ?a typeOf Place . ?a name ?name . ?a dcid ("geoId/06" "geoId/21" "geoId/24") . ?a dcid ?dcid }' @@ -61,7 +63,7 @@ The response contains an array of dictionaries, each corresponding to one node m [{'?name': 'Kentucky', '?dcid': 'geoId/21'}, {'?name': 'California', '?dcid': 'geoId/06'}, {'?name': 'Maryland', '?dcid': 'geoId/24'}] ``` -#### Example 2. Retrieve a list of ten biological specimens in reverse alphabetical order. +### Example 2: Retrieve a list of ten biological specimens in reverse alphabetical order. ```python >>> bio_specimens_reverse_alphabetical_order_query = 'SELECT ?name WHERE { ?biologicalSpecimen typeOf BiologicalSpecimen . ?biologicalSpecimen name ?name } ORDER BY DESC(?name) LIMIT 10' @@ -69,7 +71,7 @@ The response contains an array of dictionaries, each corresponding to one node m [{'?name': 'x Triticosecale'}, {'?name': 'x Silene'}, {'?name': 'x Silene'}, {'?name': 'x Silene'}, {'?name': 'x Pseudelymus saxicola (Scribn. & J.G.Sm.) Barkworth & D.R.Dewey'}, {'?name': 'x Pseudelymus saxicola (Scribn. & J.G.Sm.) Barkworth & D.R.Dewey'}, {'?name': 'x Pseudelymus saxicola (Scribn. & J.G.Sm.) Barkworth & D.R.Dewey'}, {'?name': 'x Pseudelymus saxicola (Scribn. & J.G.Sm.) Barkworth & D.R.Dewey'}, {'?name': 'x Pseudelymus saxicola (Scribn. & J.G.Sm.) Barkworth & D.R.Dewey'}, {'?name': 'x Pseudelymus saxicola (Scribn. & J.G.Sm.) Barkworth & D.R.Dewey'}] ``` -#### Example 3. Retrieve a list of GNI observations by country. +### Example 3: Retrieve a list of GNI observations by country. ```python >>> gni_by_country_query = 'SELECT ?observation ?place WHERE { ?observation typeOf StatVarObservation . ?observation variableMeasured Amount_EconomicActivity_GrossNationalIncome_PurchasingPowerParity_PerCapita . ?observation observationAbout ?place . ?place typeOf Country . } ORDER BY ASC (?place) LIMIT 10' @@ -77,7 +79,7 @@ The response contains an array of dictionaries, each corresponding to one node m [{'?observation': 'dc/o/syrpc3m8q34z7', '?place': 'country/ABW'}, {'?observation': 'dc/o/bqtfmc351v0f2', '?place': 'country/ABW'}, {'?observation': 'dc/o/md36fx6ty4d64', '?place': 'country/ABW'}, {'?observation': 'dc/o/bm28zvchsyf4b', '?place': 'country/ABW'}, {'?observation': 'dc/o/3nleez1feevw6', '?place': 'country/ABW'}, {'?observation': 'dc/o/x2yg38d0xecnf', '?place': 'country/ABW'}, {'?observation': 'dc/o/7swdqf6yjdyw8', '?place': 'country/ABW'}, {'?observation': 'dc/o/yqmsmbx1qskfg', '?place': 'country/ABW'}, {'?observation': 'dc/o/6hlhrz3k8p5wf', '?place': 'country/ABW'}, {'?observation': 'dc/o/txfw505ydg629', '?place': 'country/ABW'}] ``` -#### Example 4. Retrieve a sample list of observations with the unit InternationalDollar. +### Example 4: Retrieve a sample list of observations with the unit InternationalDollar. ```python >>> internationalDollar_obs_query = 'SELECT ?observation WHERE { ?observation typeOf StatVarObservation . ?observation unit InternationalDollar } LIMIT 10' @@ -85,7 +87,7 @@ The response contains an array of dictionaries, each corresponding to one node m [{'?observation': 'dc/o/s3gzszzvj34f1'}, {'?observation': 'dc/o/gd41m7qym86d4'}, {'?observation': 'dc/o/wq62twxx902p4'}, {'?observation': 'dc/o/d93kzvns8sq4c'}, {'?observation': 'dc/o/6s741lstdqrg4'}, {'?observation': 'dc/o/2kcq1xjkmrzmd'}, {'?observation': 'dc/o/ced6jejwv224f'}, {'?observation': 'dc/o/q31my0dmcryzd'}, {'?observation': 'dc/o/96frt9w0yjwxf'}, {'?observation': 'dc/o/rvjz5xn9mlg73'}] ``` -#### Example 5. Retrieve a list of ten distinct annual estimates of life expectancy, along with the year of estimation, for forty-seven-year-old Hungarians. +### Example 5: Retrieve a list of ten distinct annual estimates of life expectancy, along with the year of estimation, for forty-seven-year-old Hungarians. ```python >>> life_expectancy_query = 'SELECT DISTINCT ?LifeExpectancy ?year WHERE { ?o typeOf StatVarObservation . ?o variableMeasured LifeExpectancy_Person_47Years . ?o observationAbout country/HUN . ?o value ?LifeExpectancy . ?o observationDate ?year } ORDER BY ASC(?LifeExpectancy) LIMIT 10' @@ -93,7 +95,7 @@ The response contains an array of dictionaries, each corresponding to one node m [{'?LifeExpectancy': '26.4', '?year': '1993'}, {'?LifeExpectancy': '26.5', '?year': '1992'}, {'?LifeExpectancy': '26.7', '?year': '1990'}, {'?LifeExpectancy': '26.7', '?year': '1994'}, {'?LifeExpectancy': '26.8', '?year': '1991'}, {'?LifeExpectancy': '26.9', '?year': '1995'}, {'?LifeExpectancy': '27.2', '?year': '1996'}, {'?LifeExpectancy': '27.4', '?year': '1999'}, {'?LifeExpectancy': '27.5', '?year': '1997'}, {'?LifeExpectancy': '27.5', '?year': '1998'}] ``` -#### Example 6: Use the `select` function to filter returns based on name. +### Example 6: Use the `select` function to filter returns based on name. ```python >>> names_for_places_query = 'SELECT ?name ?dcid WHERE { ?a typeOf Place . ?a name ?name . ?a dcid ("geoId/06" "geoId/21" "geoId/24") . ?a dcid ?dcid }' @@ -105,9 +107,7 @@ The response contains an array of dictionaries, each corresponding to one node m {'?name': 'Maryland', '?dcid': 'geoId/24'} ``` -### Error returns - -#### Error return 1: Malformed SPARQL query. +### Error response 1: Malformed SPARQL query ```python >>> gni_by_country_query = 'SELECT ?observation WHERE { ?observation typeOf StatVarObservation . ?observation variableMeasured Amount_EconomicActivity_GrossNationalIncome_PurchasingPowerParity_PerCapita . ?observation observationAbout ?place . ?place typeOf Country . } ORDER BY ASC (?place) LIMIT 10' @@ -139,7 +139,7 @@ ValueError: Response error 500: b'{\n "code": 2,\n "message": "googleapi: Error 400: Unrecognized name: place; Did you mean name? at [1:802], invalidQuery",\n "details": [\n {\n "@type": "type.googleapis.com/google.rpc.DebugInfo",\n "stackEntries": [],\n "detail": "internal"\n }\n ]\n}\n' ``` -#### Error return 2: Malformed SPARQL query string. +### Error response 2: Malformed SPARQL query string ```python >>> gni_by_country_query = 'SELECT ?observation WHERE { ?observation typeOf StatVarObservation . ?observation variableMeasured Amount_EconomicActivity_GrossNationalIncome_PurchasingPowerParity_PerCapita . ?observation observationAbout ?place . ?place typeOf Country . } ORDER BY ASC (?place) LIMIT 10' @@ -172,7 +172,7 @@ b'{\n "code": 2,\n "message": "googleapi: Error 400: Unrecognized name: place; D >>> gni_by_country_query = 'SELECT ?observation WHERE { ?observation typeOf StatVarObservation . \\\\\ ?observation variableMeasured Amount_EconomicActivity_GrossNationalIncome_PurchasingPowerParity_PerCapita . ?observation observationAbout ?place . ?place typeOf Country . } ORDER BY ASC (?place) LIMIT 10' ``` -#### Error return 3: Bad selector. +### Error response 3: Bad selector ```python >>> names_for_places_query = 'SELECT ?name ?dcid WHERE { ?a typeOf Place . ?a name ?name . ?a dcid ("geoId/06" "geoId/21" "geoId/24") . ?a dcid ?dcid }' @@ -186,4 +186,4 @@ Traceback (most recent call last): KeyError: '?earthquake' ``` -These examples and errors, along with explanations and fixes for the errors, are available in this [Python notebook](https://colab.research.google.com/drive/1Jd0IDHnMdtxhsmXhL5Ib5tL0zgJud1k5?usp=sharing). + diff --git a/api/rest/v2/getting_started.md b/api/rest/v2/getting_started.md index 37fd10be3..ac183808a 100644 --- a/api/rest/v2/getting_started.md +++ b/api/rest/v2/getting_started.md @@ -1,273 +1,8 @@ --- -layout: default +layout: redirect title: Getting Started Guide -nav_order: 0 parent: REST (v2) grand_parent: API -published: true permalink: /api/rest/v2/getting_started ---- - -# Getting Started Guide - -Welcome! Whether you're new to Data Commons or are just looking for a refresher, this guide gives an overview of what you need to know to get started using our REST API. - -Use the links below to jump to any section: - -* TOC -{:toc} - -## What is a REST API? - -Our REST API follows the [RESTful architectural style](https://en.wikipedia.org/wiki/Representational_state_transfer) to allow you to query the Data Commons Knowledge Graph via [HTTP](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods). This allows you to explore the local graph and query data from specific variables and entities programatically. - -## How to Use the REST API - -Our REST API can be used with any tool or language that supports HTTP. You can make queries on the command line (e.g. using [cURL](https://curl.se/)), by scripting HTTP requests in another language like javascript, or even by entering an endpoint into your web browser! - -Following HTTP, a REST API call consists of a **request** that you provide, and a **response** from our servers with the data you requested in [JSON](https://json.org) format. The next section details how to assemble a request. - -### Assembling a Request - -#### Endpoints - -Requests are made through [API endpoints](https://en.wikipedia.org/wiki/Web_API#Endpoints). We provide endpoints for many different queries (see the list of available endpoints [here](/api/rest/v2)). - -Each endpoint can be acessed using its unique URL, which is a combination of a base URL and the endpoint's [URI](https://en.wikipedia.org/wiki/Uniform_Resource_Identifier). - -The base URL for all REST endpoints is: - -```bash -https://api.datacommons.org -``` - -And a URI looks like [`/v2/node`](/api/rest/v2/node). To access a particular endpoint, append the URI to the base URL (e.g. `https://api.datacommons.org/v2/node` ). - -#### Parameters - -Endpoints take a set of **parameters** which allow you to specify which entities, variables, timescales, etc. you are interested in. The V2 APIs only use query parameters. - -##### Query Parameters - -Query parameters are chained at the end of a URL behind a `?` symbol. Separate multiple parameter entries with an `&` symbol. For example, this would look like: - -```bash -https://api.datacommons.org/v2/node?key=your_api_key&nodes=dcid1&nodes=dcid2&property=<-* -``` - -Still confused? Each endpoint's documentation page has examples at the bottom tailored to the endpoint you're trying to use. - -#### POST requests - -All V2 endpoints allow for `POST` requests. For `POST` requests, feed all parameters in JSON format. For example, in cURL, this would look like: - -```bash -curl -X POST \ --H "X-API-Key: AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI" \ ---url https://api.datacommons.org/v2/node \ ---data '{ - "nodes": [ - "geoId/06085", - "geoId/06086" - ], - "property": "->[name, latitude, longitude]" -}' -``` - -#### Finding Available Entities, Variables, and their DCIDs - -Most requests require the [DCID](/glossary.html#dcid) of the entity or variable you wish to query. Curious what entities and variables are available? Want to find a DCID? Take a look at our explorer tools: - -- [Search](https://datacommons.org/search) Search Data Commons -- [Knowledge Graph](https://datacommons.org/browser/) Click through nodes in the knowledge graph -- [Place Browser](https://datacommons.org/place) Summaries of data available for entities that are geographic locations -- [Statistical Variable Explorer](https://datacommons.org/tools/statvar) See metadata for variables - -#### Finding Datetimes for Observations - -Many endpoints allow the user to filter their results to specific dates. When querying for data at a specific date, the string passed for the date queried must match the date format (in [ISO-8601](https://en.wikipedia.org/wiki/ISO_8601)) used by the target variable. An easy way to see what date format a variable uses is to look up your variable of interest in the [Statistical Variable Explorer](https://datacommons.org/tools/statvar). - -{: #authentication} -### Authentication - - - -#### Using API Keys - -API keys are required in any REST API request. To include an API key, add your API key to the URL as a query parameter by appending `?key=`. - -For GET requests, this looks like: - -```bash -https://api.datacommons.org/v2/endpoint?key= -``` - -If the key is not the first query parameter, use `&key=` instead. This looks like: - -```bash -https://api.datacommons.org/v2/endpoint?query=value&key= -``` - -For POST requests, pass the key as a header. For example, in cURL, this looks like: - -```bash -curl -X POST \ ---url https://api.datacommons.org/v2/node \ ---header 'X-API-Key: ' \ ---data '{ - "nodes": [ - "entity_dcid_1", - "entity_dcid_2", - ... - ], - "property: "relation_expression" -}' -``` - -#### Getting API Keys - -We've provided a trial API key for general public use. This key will let you try the API and make single requests. - - - -The trial key is capped with a limited quota for requests. If you are planning on using our APIs more rigorously (e.g. for personal or school projects, developing applications, etc.) please request one by -[filling out this form](https://docs.google.com/forms/d/e/1FAIpQLSeVCR95YOZ56ABsPwdH1tPAjjIeVDtisLF-8oDYlOxYmNZ7LQ/viewform) and selecting "API access" to request an official key without any quota limits. We'll be happy to hear from you! - -{: #pagination} -### Pagination - -When the response to a request is too long, the returned payload is -**paginated**. Only a subset of the response is returned, along with a long string -of characters called a **token**. To get the next set of entries, repeat the -request with `nextToken` as an query parameter, with the token as its value. - -For example, the request: - -```bash -curl --request GET \ - 'https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=geoId/06&property=<-*' -``` - -will return something like: - -```json -{ - "data": { - "geoId/06": { - "arcs": < ... output truncated for brevity ...> - }, - }, - "nextToken": "SoME_veRy_L0ng_S+rIng" -} -``` - -To get the next set of entries, repeat the previous command and append the `nextToken`: - -```bash -curl --request GET \ - 'https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=geoId/06&property=<-*&nextToken=SoME_veRy_L0ng_S+rIng' -``` - -Similarly for POST requests, this would look like: - -```bash -curl -X POST \ --H "X-API-Key: AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI" \ ---url https://api.datacommons.org/v2/node \ ---data '{ - "nodes": "geoId/06", - "property": "<-*", - "nextToken": "SoME_veRy_L0ng_S+rIng" -}' -``` - -## Troubleshooting - -### Common Error Responses - -#### "Method does not exist" - -```json -{ - "code": 5, - "message": "Method does not exist.", - "details": [ - { - "@type": "type.googleapis.com/google.rpc.DebugInfo", - "stackEntries": [], - "detail": "service_control" - } - ] -} -``` - -This is most commonly seen when the endpoint is misspelled or otherwise malformed. Check the spelling of your endpoint and that all required path parameters are provided in the right order. - -#### Missing API key - -```json -{ - "code": 16, - "message": "Method doesn't allow unregistered callers (callers without established identity). Please use API Key or other form of API consumer identity to call this API.", - "details": [ - { - "@type": "type.googleapis.com/google.rpc.DebugInfo", - "stackEntries": [], - "detail": "service_control" - } - ] -} -``` - -This is seen when your request is missing an API key. Please use the trial key provided above, or request your own API key. - - -#### "Invalid request URI" - -```json -{ - "code": 3, - "message": "Invalid request URI", - "details": [ - { - "@type": "type.googleapis.com/google.rpc.DebugInfo", - "stackEntries": [], - "detail": "internal" - } - ] -} -``` - -This is most commonly seen when your request is missing a required path parameter. Make sure endpoints and parameters are both spelled correctly and provided in the right order. - -#### Empty Response - -```json -{} -``` - -Sometimes your query might return an empty result. This is most commonly seen when the value provided for a parameter is misspelled or doesn't exist. Make sure the values you are passing for parameters are spelled correctly. - -#### Marshaling errors - -```json -{ - "code": 13, - "message": "grpc: error while marshaling: proto: Marshal called with nil", - "details": [ - { - "@type": "type.googleapis.com/google.rpc.DebugInfo", - "stackEntries": [], - "detail": "internal" - } - ] -} -``` - -This is most commonly seen when a query parameter is missing, misspelled or incorrect. Check the spelling of query parameters and ensure all required parameters are sent in the request. +redirect: index.html +--- \ No newline at end of file diff --git a/api/rest/v2/index.md b/api/rest/v2/index.md index 23293004a..5ffae9328 100644 --- a/api/rest/v2/index.md +++ b/api/rest/v2/index.md @@ -1,48 +1,200 @@ --- layout: default title: REST (v2) -nav_order: 0 +nav_order: 1 parent: API has_children: true published: true -permalink: /api/rest/v2 --- -# Data Commons REST API +{:.no_toc} +# Data Commons REST API + +* TOC +{:toc} + +## Overview The Data Commons REST API is a [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) library that enables developers to programmatically access data in the Data Commons -knowledge graph. This package allows users to explore the structure of the +knowledge graph, using [HTTP](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods). This allows you to explore the structure of the graph, integrate statistics from the graph into data analysis applications and much more. +You can use the REST API with any tool or language that supports HTTP. You can make queries on the command line (e.g. using [cURL](https://curl.se/)), by scripting HTTP requests in another language like Javascript, or even by entering an endpoint into your web browser! + + ## What's new in V2 -The V2 API collapses functionality from [V1 API](/api/rest/v1) into a smaller number of endpoints. We do this by introducing a syntax for "Relation Expressions", [described below](#relation-expressions). Each API endpoint can also handle both single and bulk requests. +The V2 API collapses functionality from [V1 API](/api/rest/v1) into a smaller number of endpoints, by introducing a syntax for _relation expressions_, [described below](#relation-expressions). Each API endpoint can also handle both single and bulk requests. -## Getting Started +## Assemble a request -First time using the Data Commons API, or just need a refresher? Take a look at -our [Getting Started Guide](/api/rest/v2/getting_started). +Following HTTP, a REST API call consists of a _request_ that you provide, and a _response_ from the Data Commons servers with the data you requested, in [JSON](https://json.org) format. The following sections detail how to assemble a request. -## Service Endpoints +### Service endpoints -The base URL for all endpoints below is: +You make requests through [API endpoints](https://en.wikipedia.org/wiki/Web_API#Endpoints). You access each endpoint using its unique URL, which is a combination of a base URL and the endpoint's [URI](https://en.wikipedia.org/wiki/Uniform_Resource_Identifier). -```bash -https://api.datacommons.org/v2 -``` +The base URL for all REST endpoints is: + +
+https://api.datacommons.org/VERSION
+
+ +The current version is `v2`. + +To access a particular endpoint, append the URI to the base URL (e.g. `https://api.datacommons.org/v2/node` ). +The URIs for the V2 API are below: | API | URI path | Description | | --- | --- | ----------- | | Node | [/v2/node](/api/rest/v2/node) | Fetches information about edges and neighboring nodes | | Observation | [/v2/observation](/api/rest/v2/observation) | Fetches statistical observations | -| Resolve Entities | [/v2/resolve](/api/rest/v2/resolve) | Returns a Data Commons ID ([`DCID`](/glossary.html#dcid)) for entities in the graph | +| Resolve entities | [/v2/resolve](/api/rest/v2/resolve) | Returns a Data Commons ID ([`DCID`](/glossary.html#dcid)) for entities in the graph | +| SPARQL | [/v2/sparql](/api/rest/v2/sparql) | Returns matches to a [SPARQL](https://www.w3.org/TR/rdf-sparql-query/) graph query | + +### Parameters + +Endpoints take a set of parameters which allow you to specify the entities, variables, timescales, etc. you are interested in. The V2 APIs only use query parameters. + +### Query parameters + +Query parameters are chained at the end of a URL behind a `?` symbol. Separate multiple parameter entries with an `&` symbol. For example, this would look like: + +
+https://api.datacommons.org/v2/node?key=API_KEY&nodes=DCID1&nodes=DCID2&property=<-*
+
+ +Still confused? Each endpoint's documentation page has examples at the bottom tailored to the endpoint you're trying to use. + +### POST requests + +All V2 endpoints allow for `POST` requests. For `POST` requests, feed all parameters in JSON format. For example, in cURL, this would look like: + +
+curl -X POST \
+-H "X-API-Key: API_KEY" \
+--url https://api.datacommons.org/v2/node \
+--data '{
+  "nodes": [
+    "geoId/06085",
+    "geoId/06086"
+  ],
+  "property": "->[name, latitude, longitude]"
+}'
+
+ +### Find available entities, variables, and their DCIDs + +Most requests require the [DCID](/glossary.html#dcid) of the entity or variable you wish to query. Curious what entities and variables are available? Want to find a DCID? Take a look at our explorer tools: + +- [Search](https://datacommons.org/search) Search Data Commons +- [Knowledge Graph](https://datacommons.org/browser/) Click through nodes in the knowledge graph +- [Place Browser](https://datacommons.org/place) Summaries of data available for entities that are geographic locations +- [Statistical Variable Explorer](https://datacommons.org/tools/statvar) See metadata for variables + + +### Find date-times for observations + +Many endpoints allow the user to filter their results to specific dates. When querying for data at a specific date, the string passed for the date queried must match the date format (in [ISO-8601](https://en.wikipedia.org/wiki/ISO_8601)) used by the target variable. An easy way to see what date format a variable uses is to look up your variable of interest in the [Statistical Variable Explorer](https://datacommons.org/tools/statvar). + +{: #authentication} +## Authentication + +API keys are required in any REST API request. To include an API key, add your API key to the URL as a query parameter by appending ?key=API_KEY. + +For GET requests, this looks like: +
+https://api.datacommons.org/v2/ENDPOINT?key=API_KEY
+
+ +If the key is not the first query parameter, use &key=API_KEY instead. This looks like: + +
+https://api.datacommons.org/v2/ENDPOINT?QUERY=VALUE&key=API_KEY
+
+ +For POST requests, pass the key as a header. For example, in cURL, this looks like: + +
+curl -X POST \
+--url https://api.datacommons.org/v2/node \
+--header 'X-API-Key: API_KEY' \
+--data '{
+  "nodes": [
+    "ENTITY_DCID1",
+    "ENTITY_DCID2",
+    ...
+  ],
+  "property: "RELATION_EXPRESSION"
+}'
+
+ +{: #get-key} +### Get API keys + +We provide a trial API key for general public use. This key will let you try the API and make single requests. + + + +_The trial key is capped with a limited quota for requests._ If you are planning on using our APIs more rigorously (e.g. for personal or school projects, developing applications, etc.) please request one by +[filling out this form](https://docs.google.com/forms/d/e/1FAIpQLSeVCR95YOZ56ABsPwdH1tPAjjIeVDtisLF-8oDYlOxYmNZ7LQ/viewform) and selecting "API access" to request an official key without any quota limits. + +{: #pagination} +## Pagination + +When the response to a request is too long, the returned payload is +_paginated_. Only a subset of the response is returned, along with a long string +of characters called a _token_. To get the next set of entries, repeat the +request with `nextToken` as an query parameter, with the token as its value. + +For example, the request: + +``` +curl --request GET \ + 'https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=geoId/06&property=<-*' +``` + +will return something like: + +```json +{ + "data": { + "geoId/06": { + "arcs": < ... output truncated for brevity ...> + }, + }, + "nextToken": "SoME_veRy_L0ng_S+rIng" +} +``` + +To get the next set of entries, repeat the previous command and append the `nextToken`: + +``` +curl --request GET \ + 'https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=geoId/06&property=<-*&nextToken=SoME_veRy_L0ng_S+rIng' +``` + +Similarly for POST requests, this would look like: + +```json +curl -X POST \ +-H "X-API-Key: AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI" \ +--url https://api.datacommons.org/v2/node \ +--data '{ + "nodes": "geoId/06", + "property": "<-*", + "nextToken": "SoME_veRy_L0ng_S+rIng" +``` {: #relation-expressions} -## Introducing Relation Expressions +## Relation expressions Data Commons represents real world entities and data as nodes. These nodes are connected by directed edges, or arcs, to form a knowledge graph. The @@ -56,49 +208,43 @@ expressions in the `property` or `expression` parameter. The following table describes symbols in the V2 API relation expressions: -| Symbol | Represents | | ------ | ---------- | | `->` | An `out` arc | | `<-` | An `in` arc | -| `{property:value}` | Filtering; identifies the property and associated value | +| {PROPERTY:VALUE} | Filtering; identifies the property and associated value | | `[]` | Multiple properties, separated by commas | | `*` | All properties linked to this node | | `+` | One or more expressions chained together for indirect relationships, like `containedInPlace+{typeOf:City}` | -### In and out arcs: `<-` & `->` +### In and out arcs -Note that arcs in the Data Commons Graph have directions. In the case of the [Argentina](https://datacommons.org/browser/country/ARG), the property `containedInPlace` exists in both `in` and `out` directions, illustrated in Figure 1: +Arcs in the Data Commons Graph have directions. In the case of the [Argentina](https://datacommons.org/browser/country/ARG), the property `containedInPlace` exists in both `in` and `out` directions, illustrated in the following figure: ![](/assets/images/rest/property_value_direction_example.png) -*Figure 1. Relationship diagram for the property `containedInPlace` of the country Argentina. Note the directionality of the property `containedInPlace`: for the node "Argentina", the `in` arc represents "Argentina contains Buenos Aires", while the `out` arc represents "Argentina in South America".* - +Note the directionality of the property `containedInPlace`: for the node "Argentina", the `in` arc represents "Argentina contains Buenos Aires", while the `out` arc represents "Argentina in South America".* -For example, nodes from `out` arcs are represented by `->`, while nodes from +Nodes from `out` arcs are represented by `->`, while nodes from `in` arcs are represented by `<-`. To illustrate using the above example: -- Regions that include Argentina (dcid: `country/ARG`): `country/ARG->containedInPlace` -- All cities directly contained in Argentina (dcid: `country/ARG`): `country/ARG<-containedInPlace{typeOf:City}` +- Regions that include Argentina (DCID: `country/ARG`): `country/ARG->containedInPlace` +- All cities directly contained in Argentina (DCID: `country/ARG`): `country/ARG<-containedInPlace{typeOf:City}` -### Filters: `{property:value}` +### Filters -Filters can be used to reduce results to only match nodes with a specified property and value. Using the same example, `country/ARG<-containedInPlace+{typeOf:City}` will only return nodes with the `typeOf:City`, filtering out `typeOf:AdministrativeArea1` and so on. +You can use filters to reduce results to only match nodes with a specified property and value. Use {} to specify property:value pairs to define the filter. Using the same example, `country/ARG<-containedInPlace+{typeOf:City}` only returns nodes with the `typeOf:City`, filtering out `typeOf:AdministrativeArea1` and so on. -### Specifying multiple properties: `[property1, property2]` +### Specify multiple properties -Multiple properties can be combined together within `[]`. For example, in order to request a few `out` arcs for a node, use -`->[name, latitude, longitude]` (this example is [fully described in this Node API example](/api/rest/v2/node.html#multiple-properties)). +You can combine multiple properties together within `[]`. For example, in order to request a few `out` arcs for a node, use +`->[name, latitude, longitude]` (this example is fully described in this [Node API example](/api/rest/v2/node.html#multiple-properties)). -### Wildcard: `*` +### Wildcard -In order to retrieve all properties linked to a node, use the `*`, e.g. `<-*`. -This example is [fully described in this Node API example](/api/rest/v2/node.html#wildcard). +To retrieve all properties linked to a node, use the `*`, e.g. `<-*`. +This example is fully described in this [Node API example](/api/rest/v2/node.html#wildcard). -### Chaining properties: `+` +### Chain properties A property chain expression represents requests for information about nodes -which are connected by the same property, but are a few hops away. This is supported only for the `containedInPlace` property. - -To illustrate again using the Argentina example: -- All cities directly contained in Argentina (dcid: `country/ARG`): `country/ARG<-containedInPlace{typeOf:City}` -- All cities indirectly contained in Argentina (dcid: `country/ARG`): `country/ARG<-containedInPlace+{typeOf:City}` \ No newline at end of file +which are connected by the same property, but are a few hops away. This is supported only for the `containedInPlace` property. \ No newline at end of file diff --git a/api/rest/v2/node.md b/api/rest/v2/node.md index 8f3dba300..ac2dc0680 100644 --- a/api/rest/v2/node.md +++ b/api/rest/v2/node.md @@ -1,33 +1,29 @@ --- layout: default title: Node -nav_order: 1 +nav_order: 3 parent: REST (v2) grand_parent: API published: true -permalink: /api/rest/v2/node --- # /v2/node -Fetches node information for edges and neighboring nodes. This is useful for +The Node API fetches node information for edges and neighboring nodes. This is useful for finding local connections between nodes of the Data Commons knowledge graph. More specifically, this API can perform the following tasks: - - Get all property labels associated with individual or multiple nodes. - Get the values of a property for individual or multiple nodes. These can also be chained for multiple degrees in the graph. - Get all connected nodes that are linked with invidiual or mutiple nodes. - Data Commons represents node relations as directed edges between nodes, or property. The name of the property is label, while the target node is a value of the property. This endpoint returns the property labels and values that are connected to the queried node. - The REST (v2) API introduces [relation expressions](/api/rest/v2/#relation-expressions) in the API syntax to represent neighboring nodes, and to support chaining and filtering. For more information -see [Data Commons REST (v2) API Overview](/api/rest/v2/#relation-expressions). +see [Data Commons REST (v2) API overview](/api/rest/v2/index.html#relation-expressions).