Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Documentation - developer and helper functionality documentation for xtdb-cli tool #3023

Merged
merged 9 commits into from
Jun 12, 2024
Merged

Documentation - developer and helper functionality documentation for xtdb-cli tool #3023

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
4e4cba3
adding helper texts xtdb-cli
stephanie0x00 Jun 3, 2024
3772d93
Fix pre-commit issues
stephanie0x00 Jun 3, 2024
5411978
Adding developer documentation for xtdb-cli tool.
stephanie0x00 Jun 4, 2024
dbd523a
Delete octopoes/tools/xtdb-cli.py
stephanie0x00 Jun 5, 2024
9e43f0d
Revert "Delete octopoes/tools/xtdb-cli.py"
ammar92 Jun 5, 2024
26ef0eb
Merge branch 'main' into docs/xtdb-cli-developer-docs
stephanie0x00 Jun 5, 2024
81725f7
Merge branch 'main' into docs/xtdb-cli-developer-docs
originalsouth Jun 10, 2024
8fd6f80
Apply suggestions from code review
stephanie0x00 Jun 10, 2024
f18d36f
Merge branch 'main' into docs/xtdb-cli-developer-docs
underdarknl Jun 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions docs/source/developer_documentation/octopoes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,92 @@ Good to know: XTDB tracks the history of each object by its **primary key**.

[Read more about XTDB bitemporality](https://v1-docs.xtdb.com/concepts/bitemporality/)

### XTDB-cli tool

The XTDB-cli tool is a script that can be used to query the XTDB database (v1) directly. This can be useful to query for transactions, queries, attributes, keys, etc. It can be used to directly manipulate XTDB, without the interference of Octopoes. This can be useful for experimenting, bug hunting, optimisations, etc.
stephanie0x00 marked this conversation as resolved.
Show resolved Hide resolved

Please note: the XTDB version within OpenKAT in not vanilla XTDB, but the version found at: [OpenKAT XTDB](https://github.com/dekkers/xtdb-http-multinode)
stephanie0x00 marked this conversation as resolved.
Show resolved Hide resolved

The XTDB-cli tool can be found in the following folder:

```
https://github.com/minvws/nl-kat-coordination/tree/main/octopoes/tools
```

It is recommended to create a virtual environment and install the developer-requirements (in the octopoes folder) within this virtual environment.

The XTDB-cli tool can be queried as shown below.

```
$ ./xtdb-cli.py -h
Usage: xtdb-cli.py [OPTIONS] COMMAND [ARGS]...

This help functionality explains how to query XTDB using the xtdb-cli tool. The help functionality for all default XTDB commands was copied from the official XTDB docs for the HTTP implementation. Not all optional parameters as available on the HTTP docs may be implemented.

Options:
 -n, --node TEXT        XTDB node  [default: 0]
 -u, --url TEXT         XTDB server base url  [default: http://localhost:3000]
 -t, --timeout INTEGER  XTDB request timeout (in ms)  [default: 5000]
 -v, --verbosity        Increase the verbosity level  [default: 0]
 -h, --help             Show this message and exit.

Commands:
 active-queries       Returns a list of currently running queries.
 attribute-stats      Returns frequencies of indexed attributes
 await-tx             Waits until the node has indexed a transaction that is at or past the supplied tx-id.
 await-tx-time        Blocks until the node has indexed a transaction that is past the supplied tx-time.
 entity               Returns the document map for a particular entity.
 entity-tx            Returns the transaction details for an entity - returns a map containing the tx-id and...
 history              Returns the history of a particular entity.
 latest-completed-tx  Returns the latest transaction to have been indexed by this node.
 latest-submitted-tx  Returns the latest transaction to have been submitted to this cluster.
 list-keys            List all keys in node
 list-values          List all values in node
 query                EDN Query (default: "{:query {:find [ ?var ] :where [[?var :xt/id ]]}}")
 recent-queries       Returns a list of recently completed/failed queries.
 slowest-queries      Returns a list of slowest completed/failed queries ran on the node.
 status               Returns the current status information of the node
 submit-tx            Takes a vector of transactions (any combination of put, delete, match, evict and fn) and...
 sync                 Wait until the consumer’s lag is back to 0 (i.e.
 tx-committed         Checks if a submitted tx was successfully committed, returning a map with tx-committed and...
 tx-log               Returns a list of all transactions, from oldest to newest transaction time - optionally...
 txs                  Show all document transactions
```

The help file for the commands can be queried as shown below.

```
$ ./xtdb-cli.py history -h
Usage: xtdb-cli.py history [OPTIONS] KEY

 Returns the history of a particular entity.

Options:
 --with-docs         Includes the documents in the response sequence, under the doc key (boolean, default: false)
 --with-corrections  Includes bitemporal corrections in the response, inline, sorted by valid-time then tx-id
                     (boolean, default: false)
 -h, --help          Show this message and exit.
```

The output below gives an example for querying the XTDB database for an organisation called 'MyOrganisationName'. This is the organisation name, not the organisation code and can be found in the organisation overview at: `http://127.0.0.1:8000/en/organizations/`.

```
$ ./xtdb-cli.py -n MyOrganisationName attribute-stats |jq .
{
 "DNSSOARecord/serial": 14,
 "Website/ip_service": 10,
 "HTTPResource/website": 10,
 "Hostname/primary_key": 8,
 "DNSNSRecord/name_server_hostname": 3,
 "DNSSOARecord/hostname": 14,
 "HostnameHTTPURL/primary_key": 2,
 "DNSTXTRecord/dns_record_type": 2,
 "TLSCipher/ip_service": 4,
 "DNSMXRecord/value":
 ....
 }
```

## OOI

OOI objects are instances of relatively simple classes, which inherit from `OOIBase`.
Expand Down
91 changes: 62 additions & 29 deletions octopoes/tools/xtdb-cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@
@click.option("-v", "--verbosity", count=True, help="Increase the verbosity level")
@click.pass_context
def cli(ctx: click.Context, url: str, node: str, timeout: int, verbosity: int):
"""This help functionality explains how to query XTDB using the xtdb-cli tool.
The help functionality for all default XTDB commands was copied from the official
XTDB docs for the HTTP implementation. Not all optional parameters as available
on the HTTP docs may be implemented."""
verbosities = [logging.WARN, logging.INFO, logging.DEBUG]
try:
if verbosity:
Expand All @@ -48,7 +52,7 @@ def cli(ctx: click.Context, url: str, node: str, timeout: int, verbosity: int):
ctx.obj["client"] = client


@cli.command
@cli.command(help="Returns the current status information of the node")
@click.pass_context
def status(ctx: click.Context):
client: XTDBClient = ctx.obj["client"]
Expand Down Expand Up @@ -84,10 +88,10 @@ def list_values(ctx: click.Context):
click.echo(json.dumps(client.query("{:query {:find [(pull ?var [*])] :where [[?var :xt/id]]}}")))


@cli.command
@click.option("--tx-id", type=int)
@click.option("--tx-time", type=click.DateTime())
@click.option("--valid-time", type=click.DateTime())
@cli.command(help="Returns the document map for a particular entity.")
@click.option("--tx-id", type=int, help="Defaulting to latest transaction id (integer)")
@click.option("--tx-time", type=click.DateTime(), help="Defaulting to latest transaction time (date)")
@click.option("--valid-time", type=click.DateTime(), help="Defaulting to now (date)")
@click.argument("key")
@click.pass_context
def entity(
Expand All @@ -102,9 +106,18 @@ def entity(
click.echo(json.dumps(client.entity(key, valid_time, tx_time, tx_id)))


@cli.command
@click.option("--with-docs", is_flag=True)
@click.option("--with-corrections", is_flag=True)
@cli.command(help="Returns the history of a particular entity.")
@click.option(
"--with-docs",
is_flag=True,
help="Includes the documents in the response sequence, under the doc key (boolean, default: false)",
)
@click.option(
"--with-corrections",
is_flag=True,
help="""Includes bitemporal corrections in the response, inline,
sorted by valid-time then tx-id (boolean, default: false)""",
)
@click.argument("key")
@click.pass_context
def history(ctx: click.Context, key: str, with_corrections: bool, with_docs: bool):
Expand All @@ -113,10 +126,10 @@ def history(ctx: click.Context, key: str, with_corrections: bool, with_docs: boo
click.echo(json.dumps(client.history(key, with_corrections, with_docs)))


@cli.command
@click.option("--tx-id", type=int)
@click.option("--tx-time", type=click.DateTime())
@click.option("--valid-time", type=click.DateTime())
@cli.command(help="Returns the transaction details for an entity - returns a map containing the tx-id and tx-time.")
@click.option("--tx-id", type=int, help="Defaulting to the latest transaction id (integer)")
@click.option("--tx-time", type=click.DateTime(), help="Defaulting to the latest transaction time (date)")
@click.option("--valid-time", type=click.DateTime(), help="Defaulting to now (date)")
@click.argument("key")
@click.pass_context
def entity_tx(
Expand All @@ -131,25 +144,31 @@ def entity_tx(
click.echo(json.dumps(client.entity_tx(key, valid_time, tx_time, tx_id)))


@cli.command
@cli.command(help="Returns frequencies of indexed attributes")
@click.pass_context
def attribute_stats(ctx: click.Context):
client: XTDBClient = ctx.obj["client"]

click.echo(json.dumps(client.attribute_stats()))


@cli.command
@click.option("--timeout", type=int)
@cli.command(
help="""Wait until the Kafka consumer’s lag is back to 0 (i.e. when it no longer has
pending transactions to write). Returns the transaction time of the most recent transaction."""
)
@click.option("--timeout", type=int, help="Specified in milliseconds (integer)")
@click.pass_context
def sync(ctx: click.Context, timeout: int | None):
client: XTDBClient = ctx.obj["client"]

click.echo(json.dumps(client.sync(timeout)))


@cli.command
@click.option("--timeout", type=int)
@cli.command(
help="""Waits until the node has indexed a transaction that is at or past the
supplied tx-id. Returns the most recent tx indexed by the node."""
)
@click.option("--timeout", type=int, help="Specified in milliseconds, defaulting to 10 seconds (integer)")
@click.argument("tx-id", type=int)
@click.pass_context
def await_tx(ctx: click.Context, tx_id: int, timeout: int | None):
Expand All @@ -158,8 +177,11 @@ def await_tx(ctx: click.Context, tx_id: int, timeout: int | None):
click.echo(json.dumps(client.await_tx(tx_id, timeout)))


@cli.command
@click.option("--timeout", type=int)
@cli.command(
help="""Blocks until the node has indexed a transaction that is past the supplied tx-time.
The returned date is the latest index time when this node has caught up as of this call."""
)
@click.option("--timeout", type=int, help="Specified in milliseconds, defaulting to 10 seconds (integer)")
@click.argument("tx-time", type=click.DateTime())
@click.pass_context
def await_tx_time(
Expand All @@ -172,9 +194,13 @@ def await_tx_time(
click.echo(json.dumps(client.await_tx_time(tx_time, timeout)))


@cli.command
@click.option("--with-ops", is_flag=True)
@click.option("--after-tx-id", type=int)
@cli.command(
help="Returns a list of all transactions, from oldest to newest transaction time - optionally including documents."
)
@click.option(
"--with-ops", is_flag=True, help="Should the operations with documents be included? (boolean, default: false)"
)
@click.option("--after-tx-id", type=int, help="Transaction id to start after (integer, default: unbounded)")
@click.pass_context
def tx_log(ctx: click.Context, after_tx_id: int | None, with_ops: bool):
client: XTDBClient = ctx.obj["client"]
Expand All @@ -190,7 +216,10 @@ def txs(ctx: click.Context):
click.echo(json.dumps(client.tx_log(None, True)))


@cli.command
@cli.command(
help="""Takes a vector of transactions (any combination of put, delete, match, evict and fn)
and executes them in order. This is the only 'write' endpoint."""
)
@click.argument("txs", nargs=-1)
@click.pass_context
def submit_tx(ctx: click.Context, txs):
Expand All @@ -199,7 +228,11 @@ def submit_tx(ctx: click.Context, txs):
click.echo(json.dumps(client.submit_tx(txs)))


@cli.command
@cli.command(
help="""Checks if a submitted tx was successfully committed, returning a map with tx-committed and
either true or false (or a NodeOutOfSyncException exception response if the node has not yet indexed
the transaction)."""
)
@click.argument("tx-id", type=int)
@click.pass_context
def tx_committed(ctx: click.Context, tx_id: int) -> None:
Expand All @@ -208,39 +241,39 @@ def tx_committed(ctx: click.Context, tx_id: int) -> None:
click.echo(json.dumps(client.tx_committed(tx_id)))


@cli.command
@cli.command(help="Returns the latest transaction to have been indexed by this node.")
@click.pass_context
def latest_completed_tx(ctx: click.Context):
client: XTDBClient = ctx.obj["client"]

click.echo(json.dumps(client.latest_completed_tx()))


@cli.command
@cli.command(help="Returns the latest transaction to have been submitted to this cluster.")
@click.pass_context
def latest_submitted_tx(ctx: click.Context):
client: XTDBClient = ctx.obj["client"]

click.echo(json.dumps(client.latest_submitted_tx()))


@cli.command
@cli.command(help="Returns a list of currently running queries.")
@click.pass_context
def active_queries(ctx: click.Context):
client: XTDBClient = ctx.obj["client"]

click.echo(json.dumps(client.active_queries()))


@cli.command
@cli.command(help="Returns a list of recently completed/failed queries.")
@click.pass_context
def recent_queries(ctx: click.Context):
client: XTDBClient = ctx.obj["client"]

click.echo(json.dumps(client.recent_queries()))


@cli.command
@cli.command(help="Returns a list of slowest completed/failed queries ran on the node.")
@click.pass_context
def slowest_queries(ctx: click.Context):
client: XTDBClient = ctx.obj["client"]
Expand Down