Data Commons Website

This repo hosts code for the Data Commons website.

About Data Commons

Data Commons is an open knowledge graph that provides a unified view across multiple public data sets and statistics. We've bootstrapped the graph with lots of data from US Census, CDC, NOAA, etc., and through collaborations with the New York Botanical Garden, Opportunity Insights, and more. However, Data Commons is meant to be for community, by the community. We are excited to work with you to make public data accessible to everyone.

To see the extent of data we have today, browse the graph using our browser.

Contributing

GitHub Workflow

In https://github.com/datacommonsorg/website, click on "Fork" button to fork the repo.

Clone your forked repo to your desktop.

Add datacommonsorg/website repo as a remote:

git remote add dc https://github.com/datacommonsorg/website.git

Every time when you want to send a Pull Request, do the following steps:

git checkout master
git pull dc master
git checkout -b new_branch_name
# Make some code change
git add .
# Run tests
./run_test.sh -a
git commit -m "commit message"
# If tests pass
git push -u origin new_branch_name

Then in your forked repo, you can send a Pull Request. If this is your first time contributing to a Google Open Source project, you may need to follow the steps in CONTRIBUTING.md.

Wait for approval of the Pull Request and merge the change.

Local Development

Get permission to use API KEY

Contact Data Commons team to get permission for access of secret manager.

Set Google Application Credential

Contact Data Commons team to get permission for GCP application credential. Then run the following command once

gcloud config set project datcom-browser-staging && \
gcloud auth application-default login

This will generate a json file in your local machine, record the path and set the environment variable in your ~/.bash_profile or ~/.bashrc file

export GOOGLE_APPLICATION_CREDENTIALS=<JSON_CREDENTIAL_PATH>

Install node.js

Before running client-side tests and packages, install nodejs

Install web browser and webdriver

Before running the tests, install the browser and webdriver. Here we recommend you use Google Chrome browser and ChromeDriver.

• Chrome browser can be downloaded here.

• ChromeDriver can be downloaded here, or you can download it using package manager directly:

  npm install chromedriver

If using Linux system, you can run the following commands to download Chrome browser and ChromeDriver, this will also include the path setup:

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo dpkg -i google-chrome-stable_current_amd64.deb; sudo apt-get -fy install
wget https://chromedriver.storage.googleapis.com/84.0.4147.30/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
sudo mv chromedriver /usr/bin/chromedriver
sudo chown root:root /usr/bin/chromedriver
sudo chmod +x /usr/bin/chromedriver

Note: You can change 84.0.4147.30(version number) to the lastet version here.

Run all tests

./run_test.sh -a

Update React test snapshots

cd static
npm test testfilename -- -u

Package javascript and static assets

./run_npm.sh

This will watch static files change and re-build on code edit.

Start the Flask Server

Start the flask webserver locally at localhost:8080

./run_server.sh

Start the server using Docker

Another way to start the server locally is to use Docker.

Make sure you have installed Docker Desktop.

Build Docker image

docker build -t datacommonsorg-website .

Run it locally

docker run \
-p 8080:8080 \
-e GOOGLE_APPLICATION_CREDENTIALS=<JSON_CREDENTIAL_PATH> \
-v $GOOGLE_APPLICATION_CREDENTIALS:<JSON_CREDENTIAL_PATH>:ro \
datacommonsorg-website

Update resource files

placeid2dcid.json

This file is stored in GCS bucket: datcom-browser-prod.appspot.com (for prod) and datcom-browser-staging.appspot.com (for local and staging). To update this file, please contact Data Commons team.

Redis memcache

Check Redis instance information.

gcloud config set core/project datcom-browser-staging(datcom-browser-prod)
gcloud redis instances describe redis-instance --region=us-west2

License

Apache 2.0

Support

For general questions or issues, please open an issue on our issues page. For all other questions, please send an email to support@datacommons.org.

Note - This is not an officially supported Google product.