Skip to content

Commit

Permalink
Update integration docs with URL info
Browse files Browse the repository at this point in the history
  • Loading branch information
criccomini committed Oct 10, 2023
1 parent 182e60a commit f27920c
Show file tree
Hide file tree
Showing 7 changed files with 42 additions and 112 deletions.
26 changes: 5 additions & 21 deletions docs/integrations/bigquery.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,8 @@ parent: "Integrations"
### CLI

```bash
recap add my_bq bigquery://
```

### Environment Variables

```bash
export RECAP_SYSTEM__MY_BQ=bigquery://
recap ls bigquery://
recap schema bigquery://some-project/some-dataset/some-table
```

### Python API
Expand All @@ -31,28 +26,17 @@ from recap.clients import create_client

with create_client("bigquery://") as client:
client.ls("my_project")
client.schema("some-project", "some-dataset", "some-table")
```

## Format

### URLs
## URLs

Recap's BigQuery URL format:

```
bigquery://<project>
bigquery://[project]/[dataset]/[table]
```

### Paths

Recap's BigQuery paths are formatted as:

```
[system]/[project]/[dataset]/[table]
```

The `BigQueryClient` class is used to read BigQuery table schemas as Recap schemas.

## Type Conversion

This table shows the corresponding Recap types for each BigQuery type, along with the associated attributes:
Expand Down
32 changes: 11 additions & 21 deletions docs/integrations/confluent-schema-registry.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,42 +15,32 @@ parent: "Integrations"
### CLI

```bash
recap add my_csr http+csr://my-registry:8081
```

### Environment Variables

```bash
export RECAP_SYSTEM__MY_CSR=http+csr://my-registry:8081
recap ls http+csr://my-registry:8081/some/root/path
recap schema http+csr://my-registry:8081/some/root/path/my-topic
```

### Python API

```python
from recap.clients import create_client

with create_client("http+csr://my-registry:8081") as client:
with create_client("http+csr://my-registry:8081/some/root/path") as client:
client.ls()
client.schema("my-topic")
```

## Format
## URLs

### URLs

Recap's [Confluent Schema Registry](https://docs.confluent.io/platform/current/schema-registry/index.html) client takes a URL pointing to the Confluent Schema Registry HTTP server.

{: .note }
The scheme must be `http+csr` or `https+csr`. The `+csr` suffix is required to distinguish this client from other clients that also use HTTP connections (similar to [SQLAlchemy's `dialect+driver` format](https://docs.sqlalchemy.org/en/20/core/engines.html#database-urls))

### Paths

Recap's Confluent Schema Registry paths are formatted as:
Recap's [Confluent Schema Registry](https://docs.confluent.io/platform/current/schema-registry/index.html) client takes a URL pointing to the Confluent Schema Registry HTTP server and an optional subject path.

```
[system]/[topic]
http+csr://[host]:[port]/[path*]/[subject]
```

You may optionally include `-key` or `-value` at the end of the path to specify the key or value schema, respectively. If no suffix is supplied `-value` is assumed.
You may optionally include `-key` or `-value` at the end of the subject name to specify the key or value schema, respectively. If no suffix is supplied `-value` is assumed.

{: .note }
The scheme must be `http+csr` or `https+csr`. The `+csr` suffix is required to distinguish this client from other clients that also use HTTP connections (similar to [SQLAlchemy's `dialect+driver` format](https://docs.sqlalchemy.org/en/20/core/engines.html#database-urls))

## Type Conversion

Expand Down
26 changes: 8 additions & 18 deletions docs/integrations/hive-metastore.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,8 @@ parent: "Integrations"
### CLI

```bash
recap add my_hms thrift+hms://hive:password@localhost:9083
```

### Environment Variables

```bash
export RECAP_SYSTEM__MY_HMS=thrift+hms://hive:password@localhost:9083
recap ls thrift+hms://hive:password@localhost:9083
recap schema thrift+hms://hive:password@localhost:9083/testdb/testtable
```

### Python API
Expand All @@ -31,25 +26,20 @@ from recap.clients import create_client

with create_client("thrift+hms://hive:password@localhost:9083") as client:
client.ls("testdb")
client.schema("testdb", "testtable")
```

## Format

### URLs
## URLs

Recap's Hive Metastore client takes the Thrift URL to the Hive Metastore.

{: .note }
The scheme must be `thrift+hms`. The `+hms` suffix is required to distinguish this client from other clients that also use Thrift connections (similar to [SQLAlchemy's `dialect+driver` format](https://docs.sqlalchemy.org/en/20/core/engines.html#database-urls))

### Paths

Recap's Confluent Schema Registry paths are formatted as:

```
[system]/[database]/[table]
thrift+hms://[username]:[password]@[host]:[port]/[database]/[table]
```

{: .note }
The scheme must be `thrift+hms`. The `+hms` suffix is required to distinguish this client from other clients that also use Thrift connections (similar to [SQLAlchemy's `dialect+driver` format](https://docs.sqlalchemy.org/en/20/core/engines.html#database-urls))

## Type Conversion

| Hive Type | Recap Type |
Expand Down
22 changes: 6 additions & 16 deletions docs/integrations/mysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,36 +15,26 @@ parent: "Integrations"
### CLI

```bash
recap add my_sql mysql://mysql:password@localhost:3306
```

### Environment Variables

```bash
export RECAP_SYSTEM__MY_SQL=mysql://mysql:password@localhost:3306
recap ls mysql://user:pass@localhost:3306/testdb
recap schema mysql://user:pass@localhost:3306/testdb/testtable
```

### Python API

```python
from recap.clients import create_client

with create_client("mysql://mysql:password@localhost:3306") as client:
with create_client("mysql://user:pass@localhost:3306") as client:
client.ls("testdb")
client.schema("testdb", "testtable")
```

## Format

### URLs
## URLs

Recap's MySQL client takes a MySQL URL with an optional DB in the path.

### Paths

Recap's MySQL paths are formatted as:

```
[system]/[database]/[table]
mysql://[username]:[password]@[host]:[port]/[database]/[table]
```

## Type Conversion
Expand Down
22 changes: 6 additions & 16 deletions docs/integrations/postgresql.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,36 +15,26 @@ parent: "Integrations"
### CLI

```bash
recap add my_pg postgresql://postgres:password@localhost:5432
```

### Environment Variables

```bash
export RECAP_SYSTEM__MY_PG=postgresql://postgres:password@localhost:5432
recap ls postgresql://user:pass@localhost:5432
recap schema postgresql://user:pass@localhost:5432/testdb/public/test_types
```

### Python API

```python
from recap.clients import create_client

with create_client("postgresql://postgres:password@localhost:5432") as client:
with create_client("postgresql://user:pass@localhost:5432") as client:
client.ls("testdb")
client.schema("testdb", "public", "test_types")
```

## Format

### URLs
## URLs

Recap's PostgreSQL client takes a PostgreSQL URL with an optional DB in the path.

### Paths

Recap's PostgreSQL paths are formatted as:

```
[system]/[database]/[schema]/[table]
postgresql://[username]:[password]@[host]:[port]/[database]/[schema]/[table]
```

{: .note }
Expand Down
24 changes: 5 additions & 19 deletions docs/integrations/snowflake.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,8 @@ parent: "Integrations"
### CLI

```bash
recap add my_snowflake snowflake://user:pass@my-account
```

### Environment Variables

```bash
export RECAP_SYSTEM__MY_SNOWFLAKE=snowflake://user:pass@my-account
recap ls snowflake://user:pass@my-account/testdb?warehouse=some_dwh&role=some_role
recap schema snowflake://user:pass@my-account/testdb/public/testtable?warehouse=some_dwh&role=some_role
```

### Python API
Expand All @@ -31,24 +26,15 @@ from recap.clients import create_client

with create_client("snowflake://user:pass@my-account") as client:
client.ls("testdb")
client.schema("testdb", "public", "testtable")
```

## Format

### URLs
## URLs

Recap's Snowflake client uses [Snowflake's SQLAlchemy URL format](https://github.com/snowflakedb/snowflake-sqlalchemy#connection-parameters):

```
snowflake://[user_login_name]:[password]@[account_name]/[database_name]/[schema_name]?warehouse=[warehouse_name]&role=[role_name]
```

### Paths

Recap's PostgreSQL paths are formatted as:

```
[system]/[database]/[schema]/[table]
snowflake://[username]:[password]@[account]/[database]/[schema]/[table]?warehouse=[warehose]&role=[role]
```

{: .note }
Expand Down
2 changes: 1 addition & 1 deletion docs/python.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ with create_client('postgresql://user:pass@host:port/dbname') as client:
struct = client.get_schema("testdb", "public", "users")
```

See the [integrations](/docs/integrations) page for each systme's URL format.
See the [integrations](/docs/integrations) page for each system's URL format.

## Converters

Expand Down

0 comments on commit f27920c

Please sign in to comment.