Skip to content

Commit

Permalink
docs(file support): add badge and docstrings for read_* methods
Browse files Browse the repository at this point in the history
  • Loading branch information
gforsyth committed Jun 1, 2023
1 parent b414d86 commit 0767b7c
Show file tree
Hide file tree
Showing 9 changed files with 278 additions and 15 deletions.
5 changes: 4 additions & 1 deletion docs/backends/BigQuery.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
---
backend_name: Google BigQuery
backend_url: https://cloud.google.com/bigquery
exports: ["PyArrow", "Parquet", "CSV", "Pandas"]
---

{% include 'backends/badges.md' %}

## Install

Install `ibis` and dependencies for the BigQuery backend:
Expand All @@ -22,7 +25,7 @@ Install `ibis` and dependencies for the BigQuery backend:

{% endfor %}

## Usage
## Connect

### `ibis.bigquery.connect`

Expand Down
64 changes: 62 additions & 2 deletions docs/backends/Datafusion.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,68 @@
backend_name: Datafusion
backend_url: https://arrow.apache.org/datafusion/
backend_module: datafusion
backend_param_style: a dictionary of paths
version_added: "2.1"
exports: ["PyArrow", "Parquet", "CSV", "Pandas"]
imports: ["CSV", "Parquet"]
---

{% include 'backends/template.md' %}
{% include 'backends/badges.md' %}

## Install

Install `ibis` and dependencies for the Apache Datafusion backend:

=== "pip"

```sh
pip install 'ibis-framework[datafusion]`
```

{% for mgr in ["conda", "mamba"] %}
=== "{{ mgr }}"

```sh
{{ mgr }} install -c conda-forge ibis-datafusion
```

{% endfor %}

## Connect

### `ibis.datafusion.connect`

```python
con = ibis.datafusion.connect()
```

```python
con = ibis.datafusion.connect(
config={"table1": "path/to/file.parquet", "table2": "path/to/file.csv"}
)
```

<!-- prettier-ignore-start -->
!!! info "`ibis.datafusion.connect` is a thin wrapper around [`ibis.backends.datafusion.Backend.do_connect`][ibis.backends.datafusion.Backend.do_connect]."
<!-- prettier-ignore-end -->

### Connection Parameters

<!-- prettier-ignore-start -->
::: ibis.backends.datafusion.Backend.do_connect
options:
heading_level: 4
show_docstring_examples: false
<!-- prettier-ignore-end -->

## File Support

<!-- prettier-ignore-start -->
::: ibis.backends.datafusion.Backend.read_csv
options:
heading_level: 4
show_docstring_returns: false
::: ibis.backends.datafusion.Backend.read_parquet
options:
heading_level: 4
show_docstring_returns: false
<!-- prettier-ignore-end -->
90 changes: 83 additions & 7 deletions docs/backends/DuckDB.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,93 @@
backend_name: DuckDB
backend_url: https://duckdb.org/
backend_module: duckdb
backend_param_style: a path to a DuckDB database
backend_connection_example: ibis.duckdb.connect("path/to/my.duckdb")
version_added: "3.0"
intro: |
!!! danger "`duckdb` >= 0.5.0 requires `duckdb-engine` >= 0.6.2"
exports: ["PyArrow", "Parquet", "CSV", "Pandas"]
imports: ["CSV", "Parquet", "JSON", "PyArrow", "Pandas", "SQLite", "Postgres"]
---

{% include 'backends/badges.md' %}

??? danger "`duckdb` >= 0.5.0 requires `duckdb-engine` >= 0.6.2"

If you encounter problems when using `duckdb` >= **0.5.0** you may need to
upgrade `duckdb-engine` to at least version **0.6.2**.

See [this issue](https://github.com/ibis-project/ibis/issues/4503) for
more details.
---

{% include 'backends/template.md' %}
## Install

Install `ibis` and dependencies for the DuckDB backend:

=== "pip"

```sh
pip install 'ibis-framework[duckdb]`
```

{% for mgr in ["conda", "mamba"] %}
=== "{{ mgr }}"

```sh
{{ mgr }} install -c conda-forge ibis-duckdb
```

{% endfor %}

## Connect

### `ibis.duckdb.connect`

```python
con = ibis.duckdb.connect() # (1)
```

1. Use an ephemeral, in-memory database

```python
con = ibis.duckdb.connect("mydb.duckdb") # (1)
```

1. Connect to, or create, a local DuckDB file

<!-- prettier-ignore-start -->
!!! info "`ibis.duckdb.connect` is a thin wrapper around [`ibis.backends.duckdb.Backend.do_connect`][ibis.backends.duckdb.Backend.do_connect]."
<!-- prettier-ignore-end -->

### Connection Parameters

<!-- prettier-ignore-start -->
::: ibis.backends.duckdb.Backend.do_connect
options:
heading_level: 4
<!-- prettier-ignore-end -->

## File Support

<!-- prettier-ignore-start -->
::: ibis.backends.duckdb.Backend.read_csv
options:
heading_level: 4
show_docstring_returns: false
::: ibis.backends.duckdb.Backend.read_parquet
options:
heading_level: 4
show_docstring_returns: false
::: ibis.backends.duckdb.Backend.read_json
options:
heading_level: 4
show_docstring_returns: false
::: ibis.backends.duckdb.Backend.read_in_memory
options:
heading_level: 4
show_docstring_returns: false
::: ibis.backends.duckdb.Backend.read_sqlite
options:
heading_level: 4
show_docstring_examples: false
show_docstring_returns: false
::: ibis.backends.duckdb.Backend.read_postgres
options:
heading_level: 4
show_docstring_returns: false
<!-- prettier-ignore-end -->
61 changes: 59 additions & 2 deletions docs/backends/Polars.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,66 @@
backend_name: Polars
backend_url: https://pola-rs.github.io/polars-book/user-guide/index.html
backend_module: polars
backend_param_style: connection parameters
is_experimental: true
version_added: "4.0"
exports: ["PyArrow", "Parquet", "CSV", "Pandas"]
imports: ["CSV", "Parquet", "Pandas"]
---

{% include 'backends/template.md' %}
{% include 'backends/badges.md' %}

!!! experimental "Introduced in v4.0"

The Polars backend is experimental and is subject to backwards incompatible changes.

## Install

Install `ibis` and dependencies for the Polars backend:

=== "pip"

```sh
pip install 'ibis-framework[polars]'
```

{% for mgr in ["conda", "mamba"] %}
=== "{{ mgr }}"

```sh
{{ mgr }} install -c conda-forge ibis-polars
```

{% endfor %}

## Connect

### `ibis.polars.connect`

```python
con = ibis.polars.connect()
```

<!-- prettier-ignore-start -->
!!! info "`ibis.polars.connect` is a thin wrapper around [`ibis.backends.polars.Backend.do_connect`][ibis.backends.polars.Backend.do_connect]."
<!-- prettier-ignore-end -->

### Connection Parameters

<!-- prettier-ignore-start -->
::: ibis.backends.polars.Backend.do_connect
options:
heading_level: 4
<!-- prettier-ignore-end -->

## File Support

<!-- prettier-ignore-start -->
::: ibis.backends.polars.Backend.read_csv
options:
heading_level: 4
show_docstring_returns: false
::: ibis.backends.polars.Backend.read_parquet
options:
heading_level: 4
show_docstring_returns: false
<!-- prettier-ignore-end -->
60 changes: 59 additions & 1 deletion docs/backends/PySpark.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,64 @@ backend_name: PySpark
backend_url: https://spark.apache.org/docs/latest/api/python/
backend_module: pyspark
backend_param_style: PySpark things
exports: ["PyArrow", "Parquet", "CSV", "Pandas"]
imports: ["CSV", "Parquet"]
---

{% include 'backends/template.md' %}
{% include 'backends/badges.md' %}

## Install

Install `ibis` and dependencies for the PySpark backend:

=== "pip"

```sh
pip install 'ibis-framework[pyspark]`
```

{% for mgr in ["conda", "mamba"] %}
=== "{{ mgr }}"

```sh
{{ mgr }} install -c conda-forge ibis-pyspark
```

{% endfor %}

## Connect

### `ibis.pyspark.connect`

```python
con = ibis.pyspark.connect(session=session)
```

<!-- prettier-ignore-start -->
!!! info "`ibis.pyspark.connect` is a thin wrapper around [`ibis.backends.pyspark.Backend.do_connect`][ibis.backends.pyspark.Backend.do_connect]."
<!-- prettier-ignore-end -->

<!-- prettier-ignore-start -->
!!! info "The `pyspark` backend does not create `SparkSession` objects, you must create a `SparkSession` and pass that to `ibis.pyspark.connect`."
<!-- prettier-ignore-end -->

### Connection Parameters

<!-- prettier-ignore-start -->
::: ibis.backends.pyspark.Backend.do_connect
options:
heading_level: 4
<!-- prettier-ignore-end -->

## File Support

<!-- prettier-ignore-start -->
::: ibis.backends.pyspark.Backend.read_csv
options:
heading_level: 4
show_docstring_returns: false
::: ibis.backends.pyspark.Backend.read_parquet
options:
heading_level: 4
show_docstring_returns: false
<!-- prettier-ignore-end -->
7 changes: 6 additions & 1 deletion docs/backends/Snowflake.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
---
backend_name: Snowflake
backend_url: https://snowflake.com/
exports: ["PyArrow", "Parquet", "CSV", "Pandas"]
---

{% include 'backends/badges.md' %}

!!! experimental "Introduced in v4.0"

The Snowflake backend is experimental and is subject to backwards incompatible changes.
Expand All @@ -26,7 +29,7 @@ Install `ibis` and dependencies for the Snowflake backend:

{% endfor %}

## Usage
## Connect

### `ibis.snowflake.connect`

Expand All @@ -43,6 +46,8 @@ con = ibis.snowflake.connect(
!!! info "`ibis.snowflake.connect` is a thin wrapper around [`ibis.backends.snowflake.Backend.do_connect`][ibis.backends.snowflake.Backend.do_connect]."
<!-- prettier-ignore-end -->

### Connection Parameters

<!-- prettier-ignore-start -->
::: ibis.backends.snowflake.Backend.do_connect
options:
Expand Down
3 changes: 3 additions & 0 deletions docs/backends/badges.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{% if imports %} ![filebadge](https://img.shields.io/badge/Reads-{{ "%20|%20".join(imports) }}-blue?style=flat-square) {% endif %}

{% if exports %} ![exportbadge](https://img.shields.io/badge/Exports-{{ "%20|%20".join(exports) }}-orange?style=flat-square) {% endif %}
2 changes: 1 addition & 1 deletion docs/how_to/duckdb_register.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# How to Use `register` to load external data files with the DuckDB backend

<!-- prettier-ignore-start -->
Here we use the [`register`][ibis.backends.duckdb.Backend.register] method to load external data files and join them.
Here we use the `register` method to load external data files and join them.
<!-- prettier-ignore-end -->

We're going to download one month of [NYC Taxi
Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ plugins:
- exclude:
glob:
- backends/template.md
- backends/badges.md
- backends/*_support_matrix.csv
- backends/app/*
- CONTRIBUTING.md
Expand Down

0 comments on commit 0767b7c

Please sign in to comment.