Skip to content

Commit

Permalink
Merge branch 'main' into bump-echarts
Browse files Browse the repository at this point in the history
  • Loading branch information
archiewood authored May 16, 2023
2 parents b333b0e + b768cf9 commit c76e4ad
Show file tree
Hide file tree
Showing 9 changed files with 130 additions and 52 deletions.
5 changes: 5 additions & 0 deletions .changeset/afraid-planets-sip.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@evidence-dev/components': patch
---

Error icon in value component is now vertically centered and color is white
5 changes: 5 additions & 0 deletions .changeset/metal-clouds-confess.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@evidence-dev/components': patch
---

Update docs link
121 changes: 98 additions & 23 deletions sites/docs/docs/core-concepts/data-sources/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,59 +16,141 @@ To connect your local development environment to a database:

Evidence will save your credentials locally, and run a test query to confirm that it can connect.

Connections to databases in production are managed via [environment variables](/cli#environment-variables)

### Supported data sources

Evidence supports:

- [BigQuery](/guides/bigquery)
- Snowflake
- Redshift
- PostgreSQL
- MySQL
- SQLite
- DuckDB
- [CSV files](#local-data-files)
- [BigQuery](#bigquery)
- [Snowflake](#snowflake)
- [Redshift](#redshift)
- [PostgreSQL](#postgresql)
- [MySQL](#mysql)
- [SQLite](#sqlite)
- [DuckDB](#duckdb)
- [CSV and Parquet files](#local-data-files)
- & More

We're adding new connectors regularly. [Create a GitHub issue](https://github.com/evidence-dev/evidence/issues) or [send us a message in Slack](https://join.slack.com/t/evidencedev/shared_invite/zt-uda6wp6a-hP6Qyz0LUOddwpXW5qG03Q) if you'd like to use Evidence with a database that isn't currently supported.

## Local Data Files
The source code for Evidence's connectors is available [on GitHub](https://github.com/evidence-dev/evidence/tree/main/packages)


## Data source specific info

All databases can be connected via the UI settings page as described above. Where relevant, additional information is provided below.

### BigQuery

Evidence supports connecting to Google BigQuery by using a [service account](https://cloud.google.com/iam/docs/service-accounts) and a JSON key.

OAuth is not currently supported (though PRs are welcome).

Follow the instructions below to set up your service account and get a JSON key.

#### Create a Service Account Key

1. [Go to the Service Account Page](https://console.cloud.google.com/projectselector/iam-admin/serviceaccounts/create?supportedpurview=project) and click on your project
2. Add a name for your service account, then click Create
3. Assign your service account a role for BigQuery (scroll down the role dropdown to find BigQuery roles).
1. **BigQuery User** should work for most use cases.
1. **BigQuery Data Viewer** may be required (depending on your organization's permissions settings in Google Cloud).
1. Reach out to us if you run into issues or need help with BigQuery permissions.
4. Click Continue, then click Done. You should see a table of users.
5. Click on the email address for the service account you just created, then click the **Keys** tab
6. Click Add Key, then Create New Key, then Create
7. Google will download a JSON Key File to your computer



### Snowflake

Evidence supports connecting to Snowflake using [Basic Authentication](https://docs.snowflake.com/en/user-guide/api-authentication), i.e. username and password.

OAuth and Key-Pair authentication are not currently supported, though PRs are welcome.

### Redshift

The Redshift connector uses the Postgres connector under the hood, so configuration options are similar.
### PostgreSQL

#### SSL

To connect to a Postgres database using SSL, you may need to modify the SSL settings used. Once you have selected a PostgreSQL data connection type, you can set the SSL value as follows:
- `false`: Don't connect using SSL (default)
- `true`: Connect using SSL, validating the SSL certificates. Self-signed certificates will fail using this approach.
- `no-verify`: Connect using SSL, but don't validate the certificates.

Other SSL options will require the use of a custom connection string. Evidence uses the node-postgres package to manage these connections, and the details of additional SSL options via the connection string can be found at the [package documentation](https://node-postgres.com/features/ssl).

One scenario might be a Postgres platform that issues a self-signed certificate for the database connection, but provides a CA certificate to validate that self-signed certificate. In this scenario you could use a CONNECTION STRING value as follows:

```
postgresql://{user}:{password}@{host}:{port}/{database}?sslmode=require&sslrootcert=/path/to/file/ca-certificate.crt
```

Replace the various `{properties}` as needed, and replace `/path/to/file/ca-certificate.crt` with the path and filename of your certificate.

### MySQL

#### SSL

SSL options are:

In Evidence, you can query local CSV files directly in SQL.
- `false` (default)
- `true`
- `Amazon RDS`
- A credentials object


### SQLite

SQLite is a local file-based database. It should be stored in the root of your Evidence project.

### DuckDB

DuckDB is a local file-based database. It should be stored in the root of your Evidence project.

See the [DuckDB docs](https://duckdb.org/docs/guides/index) for more information.

### CSV and Parquet files

In Evidence, you can query local CSV or Parquet files directly in SQL.

Get started by selecting the `CSV` connector on the Settings page in your project.

### How to Query a CSV File
#### How to Query a CSV File

#### Inside your Evidence Project
##### Inside your Evidence Project

Evidence looks for CSV files stored in a `sources` folder in the root of your Evidence project. You can then query them using this syntax:

```sql
select * from 'sources/myfile.csv'
```

#### Absolute Filepaths
##### Absolute Filepaths

You can pass in an absolute filepath:

```sql
select * from 'Users/myname/Downloads/myfile.csv'
```

#### Relative Filepaths
##### Relative Filepaths

Paths are **relative to two files deep** in your Evidence project. For example, to query a CSV in the root of an Evidence project, you would use this syntax:

```sql
select * from '../../myfile.csv'
```

### SQL Syntax for Querying CSVs
#### SQL Syntax for Querying CSVs

Evidence uses DuckDB to run SQL against a CSVs. For query syntax, see the [DuckDB docs](https://duckdb.org/docs/sql/query_syntax/select).

### Parsing Headers
#### Parsing Headers

When parsing headers in CSV files, the `read_csv_auto` helper function provided by DuckDB can be helpful.

Expand All @@ -80,16 +162,9 @@ In addition to the `HEADER` argument, this function can also accept changes to t

Additional information about CSV helper functions can be found in the [DuckDB docs](https://duckdb.org/docs/data/csv).

## SSL with PostgreSQL

To connect to a Postgres database using SSL, you may need to modify the SSL settings used. Once you have selected a PostgreSQL data connection type, you can set the SSL value as follows:
- false: Don't connect using SSL (default)
- true: Connect using SSL, validating the SSL certificates. Self-signed certificates will fail using this approach.
- no-verify: Connect using SSL, but don't validate the certificates.

Other SSL options will require the use of a custom connection string. Evidence uses the node-postgres package to manage these connections, and the details of additional SSL options via the connection string can be found at the [package documentation](https://node-postgres.com/features/ssl).

One scenario might be a Postgres platform that issues a self-signed certificate for the database connection, but provides a CA certificate to validate that self-signed certificate. In this scenario you could use a CONNECTION STRING value as follows: `postgresql://{user}:{password}@{host}:{port}/{database}?sslmode=require&sslrootcert=/path/to/file/ca-certificate.crt`. Replace the various `{properties}` as needed, and replace `/path/to/file/ca-certificate.crt` with the path and filename of your certificate.

## Troubleshooting

Expand Down
16 changes: 14 additions & 2 deletions sites/docs/docs/core-concepts/queries/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,18 @@ my-evidence-project/

These queries can then be used on `my_page.md` with the following [frontmatter](/markdown/#frontmatter)

```yaml
---
sources:
- q4_data: my_query.sql
- q4_sales_reps: some_category/my_category_query.sql
---
```

In your evidence file, you can now reference `q4_data` and `q4_sales_reps` the same way you would use any other query.

Optionally, you can omit the query name, and the filename will be used instead; these queries will be available at `my_query` and `some_category_my_category_query` (note that `/` became `_`).

```yaml
---
sources:
Expand All @@ -113,10 +125,10 @@ sources:
---
```

In your evidence file, you can now reference `my_query` and `some_category_my_category_query` (note that `/` became `_`) the same way you would use any other query.

### Advanced Usage

#### File Query Chaining

SQL file queries can [depend on other query files](/core-concepts/queries/#query-chaining), but they will all need to be referenced in the files you use them in. For example, if `my_query` depends on `some_category_my_category_query`, then you will have to have them both in your [frontmatter](/markdown/#frontmatter), as shown above.

## Query Cache
Expand Down
2 changes: 1 addition & 1 deletion sites/docs/docs/deployment/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Deployment Overview

In production, Evidence is a [static site generator](https://www.netlify.com/blog/2020/04/14/what-is-a-static-site-generator-and-3-ways-to-find-the-best-one/). This means it doesn't run queries when someone visits your site, but pre-builds all possible pages as HTML beforehand.

You can host your Evidence project using Evidence Cloud, cloud services like Netlify or Vercel, or your own infrastructure.
You can host your Evidence project using Evidence Cloud, cloud services like Netlify or Vercel, or your own infrastructure. Evidence does not currently support Github Pages, there is more information on [GitHub](https://github.com/evidence-dev/evidence/issues/603).

## Evidence Cloud

Expand Down
23 changes: 0 additions & 23 deletions sites/docs/docs/guides/bigquery.md

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -33,5 +33,6 @@
svg {
margin: auto;
display: block;
height: 100%;
}
</style>
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
id: 'bigquery',
name: 'BigQuery',
formComponent: BigqueryForm,
docsHref: 'https://docs.evidence.dev/guides/bigquery'
docsHref: 'https://docs.evidence.dev/core-concepts/data-sources/#bigquery'
},
{ id: 'postgres', name: 'PostgreSQL', formComponent: PostgresForm },
{ id: 'mysql', name: 'MySQL', formComponent: MysqlForm },
Expand Down
7 changes: 5 additions & 2 deletions sites/example-project/src/components/viz/Value.svelte
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@
<span class="error">
<span class="error-label">Error</span>
<span class="additional-info-icon">
<HelpCircleIcon height="18" width="18" verticalOffset="2" color="--grey-000" />
<HelpCircleIcon height="18" width="18" verticalOffset="1" color="--grey-100" />
</span>
<span class="error-msg">{error}</span>
</span>
Expand Down Expand Up @@ -117,7 +117,10 @@
}
.additional-info-icon {
display: inline;
display: flex;
align-items: center;
height: 100%;
width: 100%;
vertical-align: middle;
width: 14px;
color: white;
Expand Down

0 comments on commit c76e4ad

Please sign in to comment.