Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Start on dev docs #107

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions DEVELOP.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
# Contributing Documentation
6 changes: 6 additions & 0 deletions dev-docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Developer Documentation

This folder contains documentation on the architecture of `geoarrow-rs` and how to contribute.

- [Install and set up Rust](./rust-setup.md)
- [Project architecture](./architecture/README.md)
4 changes: 4 additions & 0 deletions dev-docs/architecture/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Project Architecture

- [Rust core](./rust-core.md)
- [JavaScript bindings](./js-bindings.md)
1 change: 1 addition & 0 deletions dev-docs/architecture/geoarrow-spec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
#
49 changes: 49 additions & 0 deletions dev-docs/architecture/js-bindings.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# JavaScript Bindings Architecture

The goal of the bindings is to be as slim as possible. As much as possible should be included in the core Rust code.

The JS bindings use [wasm-bindgen](https://rustwasm.github.io/docs/wasm-bindgen/) to generate a WebAssembly binary plus JavaScript functions (plus TypeScript types!) to interact with the Wasm.

## Arrays

...

One thing to keep an eye on is that the bindings have _another set_ of struct names that coincide with the core rust binding! So for example, there's a `LineStringArray` in the JS bindings. It needs to have that name so that JS gets that name, but this is a _different_ struct than the Rust core. So you can't use them interchangeably; you need to convert from one to the other (this conversion is `O(1)`).

## Scalars

Scalars are not yet implemented for JS. We need to figure out the right way to abstract over a scalar

## Algorithms

Algorithms are defined in individual modules which implement algorithms onto the existing wasm-bindgen structs.

Ideally the algorithm binding should be extremely minimal. For example, implementing `euclidean_length` in JS happens in [`euclidean_length.rs`](../../js/src/algorithm/geo/euclidean_length.rs). The _entire_ binding for this function is

```rs
#[wasm_bindgen]
impl LineStringArray {
/// Calculation of the length of a Line
#[wasm_bindgen(js_name = euclideanLength)]
pub fn euclidean_length(&self) -> FloatArray {
use geoarrow::algorithm::geo::EuclideanLength;
FloatArray(EuclideanLength::euclidean_length(&self.0))
}
}
```

Since the implementation is exactly the same for multiple geometry types, we use a macro to deduplicate the binding for multiple geometry array types.

In this example, you can see you need to apply `#[wasm_bindgen]` on the `impl`, as well as another `#[wasm_bindgen]` on the function itself. We rename the function for JS so that the Rust side can have idiomatic snake case naming, while the JS side has idiomatic camel case naming.

## Arrow FFI

Arrow defines a single memory specification for every implementation. This means that the way Arrow memory is laid out in WebAssembly's memory space is the same as in JavaScript's own memory. This means we can use the JS Arrow implementation to interpret Arrow memory from inside Wasm memory.

I [wrote more on this in a blog post](https://observablehq.com/@kylebarron/zero-copy-apache-arrow-with-webassembly), and have a [JS library here](https://github.com/kylebarron/arrow-js-ffi) that implements the reading across the Wasm boundary.

All that's needed on the Rust side is to create the structs that fulfill the [C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html). The code in the [`ffi` mod](../../js/src/ffi/) contains the `FFIArrowArray`, which stores pointers to both the [`ArrowSchema`](https://arrow.apache.org/docs/format/CDataInterface.html#the-arrowschema-structure) struct and the [`ArrowArray`](https://arrow.apache.org/docs/format/CDataInterface.html#the-arrowarray-structure) struct. Those can be read from JS using `arrow-js-ffi`.

## API Documentation

Anything documented with `///` in Rust gets converted to a [JSDoc comment](https://github.com/jsdoc/jsdoc) within the exported TypeScript `.d.ts` type file. We then use [TypeDoc](https://github.com/TypeStrong/typedoc) to generate an API documentation website from these types and docstrings.
19 changes: 19 additions & 0 deletions dev-docs/architecture/rust-core.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Rust Core Architecture

## Layout

- `array`. The `array` module contains all the

```
.
├── algorithm
├── array
├── error.rs
├── geo_traits
├── io
├── lib.rs
├── scalar
├── test
├── trait_.rs
└── util.rs
```
53 changes: 53 additions & 0 deletions dev-docs/rust-setup.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Rust Setup

## Install Rust

Follow the [official instructions](https://www.rust-lang.org/tools/install) to install Rust on your system.

## Editor environment

I use and recommend VSCode with the stellar [Rust Analyzer](https://marketplace.visualstudio.com/items?itemName=rust-lang.rust-analyzer) extension.

Note that Rust Analyzer works with the _currently active Cargo workspace_. This means that if you open this repository from the root, Rust Analyzer completions will work for the pure-Rust library, but **not for the JavaScript bindings** because the JS bindings are a _separate Cargo project_. You need to open VSCode from the `/js` folder in order for the completions to work for the JS bindings.

## System dependencies

One of the examples uses GDAL, and so you might need to have GDAL (3.6+) installed on your system, even if you're not running that example (I'm not sure).

No other system dependencies are required to my knowledge.

## Run tests

We use the default Cargo test runner, so just run:

```bash
cargo test --all-features
```

## Run linter

We use the default Cargo linter, so just run:

```bash
cargo clippy --all-features
```

## Format code

Cargo includes a default code formatter. If the code hasn't been formatted, it won't pass CI.

```bash
cargo fmt
```

## View crate documentation

Any object documented with `///` will be automatically documented.

To see the current crate documentation locally, run

```bash
cargo doc --open
```

See the sections in the [Rust Book](https://doc.rust-lang.org/book/ch14-02-publishing-to-crates-io.html#making-useful-documentation-comments) and the [Rustdoc guide](https://doc.rust-lang.org/rustdoc/index.html) for all the syntax supported in code comments.
2 changes: 2 additions & 0 deletions js/src/ffi/to_ffi.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ use arrow2::datatypes::Field;
use geoarrow::GeometryArrayTrait;
use wasm_bindgen::prelude::*;

/// Implement exporting to an FFI struct for GeoArrow arrays
macro_rules! impl_to_ffi {
($struct_name:ident) => {
#[wasm_bindgen]
Expand All @@ -26,6 +27,7 @@ impl_to_ffi!(MultiLineStringArray);
impl_to_ffi!(MultiPolygonArray);
impl_to_ffi!(GeometryArray);

/// Implement exporting to an FFI struct for native Arrow2 arrays
macro_rules! impl_to_ffi_arrow2 {
($struct_name:ident) => {
#[wasm_bindgen]
Expand Down