CAVE is short for Connectome Annotation Versioning Engine. CAVE is a set of microservices that provide a framework for storing and versioning connectomics data and large sets of dynamic annotations, metadata, and segmentations. This repository supplies client-side code to easily interact with the microservices in CAVE.
"},{"location":"#documentation-layout","title":"Documentation Layout","text":"To learn how to install caveclient, visit Installation.
To see hands-on examples of using caveclient in a notebook, visit the Tutorials.
To see API reference documentation for interacting with a caveclient.CAVEclient object, visit the Client API.
To see a glossary of terms used in the documentation, visit the Glossary (work in progress).
For information about how to contribute to the documentation or the package, visit the Contributing page. Feedback on the documentation is welcome! Please open an issue or use the \"Edit this page\" button at the top right of any page to suggest changes.
To read a full description of the Connectome Annotation Versioning Engine, including a description of the various services please read this paper.
"},{"location":"changelog/","title":"Changelog","text":""},{"location":"changelog/#743-december-20-2024","title":"7.4.3 (December 20, 2024)","text":"annotation module: Added support for \"None\", \"none\", etc. to remove notice_text and improved documentation on method
"},{"location":"changelog/#732-december-10-2024","title":"7.3.2 (December 10, 2024)","text":"Dropped oldest supported numpy requirements and declared python 3.8 support
"},{"location":"changelog/#731-december-5-2024","title":"7.3.1 (December 5, 2024)","text":"Improved documentation to annotation.update_table_metadata
"},{"location":"changelog/#730-december-5-2024","title":"7.3.0 (December 5, 2024)","text":"Added get_minimal_covering_nodes to chunkedgraph (#274)
"},{"location":"changelog/#721-december-3-2024","title":"7.2.1 (December 3, 2024)","text":"Added raw strings to avoid deprecation warning (#275)
"},{"location":"changelog/#720-december-3-2024","title":"7.2.0 (December 3, 2024)","text":"Allow get_state_json to accept direct URLs (#271)
"},{"location":"changelog/#710-december-3-2024","title":"7.1.0 (December 3, 2024)","text":"Added get_leaves_multiple to chunkedgraph module (#273)
"},{"location":"changelog/#700-november-20-2024","title":"7.0.0 (November 20, 2024)","text":"log_warnings=FalseCache is saved on a local file ~/.cloudvolume/secrets/cave_datastack_to_server_map.json
Cache will populate the first time caveclient.CAVEclient('my_datastack', server_address=\"https://my_server.com\") is called. Subsequent calls can then just be caveclient.CAVEclient('my_datastack').
"},{"location":"changelog/#510","title":"5.1.0","text":"Change all select_column calls to pass dictionaries rather than lists. Change all suffix calls to pass dictionaries rather than lists. Advocate for your server administrator to upgrade to MaterializationEngine 4.7.0 or later, so you can use the new MaterializationEngine 3.0 API and client.
"},{"location":"contributing/","title":"Contributing","text":"Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.
You can contribute in many ways:
"},{"location":"contributing/#types-of-contributions","title":"Types of Contributions","text":""},{"location":"contributing/#report-bugs","title":"Report Bugs","text":"Report bugs to our issues page.
If you are reporting a bug, please include:
Look through the GitHub issues for bugs. Anything tagged with \"bug\" and \"help wanted\" is open to whoever wants to implement it.
"},{"location":"contributing/#implement-features","title":"Implement Features","text":"Look through the GitHub issues for features. Anything tagged with \"enhancement\" and \"help wanted\" is open to whoever wants to implement it.
"},{"location":"contributing/#write-documentation","title":"Write Documentation","text":"caveclient could always use more documentation, whether as part of the official caveclient docs, in docstrings, or even on the web in blog posts, articles, and such.
The best way to send feedback is to create an issue on GitHub.
If you are proposing a feature:
Ready to contribute? Here's how to set up caveclient for local development.
git clone git@github.com:your_name_here/CAVEclient.git\nuv for various developer tasks. Ensure you have uv installed according to the installation instructions.Note
While we recommend using uv as described here, these tasks could also be achieved using pip to install and run the various required tools. You can view the development requirements and build/check commands in our pyproject.toml, so we avoid duplicating them here.
cd CAVEclient\nuv sync --python 3.12\ngit checkout -b name-of-your-bugfix-or-feature\nMake your changes locally
If you have added code that should be tested, add tests.
If you have modified dependencies in any way, make sure to run
uv sync\nuvx --from poethepoet poe doc-build\nuvx --from poethepoet poe lint-fix\nuvx --from poethepoet poe checks\nYou may be interested in running some of these checks individually, such as:
uvx --from poethepoet poe test\nuvx --from poethepoet poe doc-build\nuvx --from poethepoet poe lint\ngit add .\ngit commit -m \"Your detailed description of your changes.\"\ngit push origin name-of-your-bugfix-or-feature\nBefore you submit a pull request, check that it meets these guidelines:
The docs should be updated with whatever changes you have made. Put your new functionality into a function with a docstring, and make sure the new functionality is documented after building the documentation (described above).
Once you submit a pull request, automated checks will run. You may require administrator approval before running these checks if this is your first time contributing to the repo.
Warning
This glossary is a work in progress; for now we are documenting the commonly used terms that we need to define. Please feel free to contribute definitions or additional terms.
To install caveclient, run this command in your terminal:
pip install caveclient\nThis is the preferred method to install caveclient, as it will always install the most recent stable release.
You can also specify a particular version, e.g.
pip install caveclient==5.0.0\nIf you don't have pip installed, this Python installation guide can guide you through the process.
"},{"location":"installation/#from-source","title":"From source","text":"The source for caveclient can be downloaded from the Github repo.
You can either clone the public repository:
git clone git://github.com/CAVEconnectome/CAVEclient\nOr download the tarball:
curl -OJL https://github.com/CAVEconnectome/CAVEclient/tarball/master\nOnce you have a copy of the source, you can install it with:
pip install .\nOr in editable mode, it can be installed with:
pip install -e .\nThe most common method of interacting with the CAVE Framework is by instantiating a client (caveclient.CAVEclient) and then using that client to interact with various services. Under the hood, the CAVEclient is a collection of individual clients, which can be accessed via properties. For example, to access the materialization client, you can use client.materialize, which (up to the exact version) will actually return a MaterializationClient object.
These pages describe the functionality of each of the individual clients, but note that some features may not be available if the server you are accessing is using older versions of particular services.
"},{"location":"api/annotation/","title":"client.annotation","text":"Client for interacting with the annotation engine.
Methods:
Name Descriptioncreate_table Creates a new data table based on an existing schema
delete_annotation Delete one or more annotations in a table. Annotations that are
delete_table Marks a table for deletion requires super admin privileges
get_annotation Retrieve an annotation or annotations by id(s) and table name.
get_annotation_count Get number of annotations in a table
get_table_metadata Get metadata about a table
get_tables Gets a list of table names for a aligned_volume_name
post_annotation Post one or more new annotations to a table in the AnnotationEngine.
post_annotation_df Post one or more new annotations to a table in the AnnotationEngine.
process_position_columns Process a dataframe into a list of dictionaries
raise_for_status Raises requests.HTTPError, if one occurred.
stage_annotations Get a StagedAnnotations object to help produce correctly formatted annotations for a given table or schema.
update_annotation Update one or more new annotations to a table in the AnnotationEngine.
update_annotation_df Update one or more annotations to a table in the AnnotationEngine using a
update_metadata Update the metadata on an existing table
upload_staged_annotations Upload annotations directly from an Annotation Guide object.
Attributes:
Name Type Descriptionserver_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.create_table","title":"create_table(table_name, schema_name, description, voxel_resolution, reference_table=None, track_target_id_updates=None, flat_segmentation_source=None, user_id=None, aligned_volume_name=None, write_permission='PRIVATE', read_permission='PUBLIC', notice_text=None)","text":"Creates a new data table based on an existing schema
Parameters:
Name Type Description Defaulttable_name str Name of the new table. Cannot be the same as an existing table
requiredschema_name str Name of the schema for the new table.
requireddescription str Human readable description for what is in the table. Should include information about who generated the table What data it covers, and how it should be interpreted. And who should you talk to if you want to use it. An Example: a manual synapse table to detect chandelier synapses on 81 PyC cells with complete AISs [created by Agnes - agnesb@alleninstitute.org, uploaded by Forrest]
requiredvoxel_resolution List[float] voxel resolution points will be uploaded in, typically nm, i.e [1,1,1] means nanometers [4,4,40] would be 4nm, 4nm, 40nm voxels
requiredreference_table str If the schema you are using is a reference schema Meaning it is an annotation of another annotation. Then you need to specify what the target table those annotations are in.
None track_target_id_updates bool Indicates whether to automatically update reference table's foreign key if target annotation table row is updated.
None flat_segmentation_source str the source to a flat segmentation that corresponds to this table i.e. precomputed:\\gs:\\mybucket his_tables_annotation
None user_id int If you are uploading this schema on someone else's behalf and you want to link this table with their ID, you can specify it here Otherwise, the table will be created with your userID in the user_id column.
None aligned_volume_name str Name of the aligned_volume. If None, uses the one specified in the client.
None write_permission str What permissions to give the table for writing. One of PRIVATE: only you can write to this table (DEFAULT) GROUP: only members that share a group with you can write (excluding some groups) PUBLIC: Anyone can write to this table. Note all data is logged, and deletes are done by marking rows as deleted, so all data is always recoverable
'PRIVATE' read_permission str What permissions to give the table for reading. One of PRIVATE: only you can read this table. Intended to be used for sorting out bugs. GROUP: only members that share a group with you can read (intended for within group vetting) PUBLIC: anyone with permissions to read this datastack can read this data (DEFAULT)
'PUBLIC' notice_text str Text the user will see when querying this table. Can be used to warn users of flaws, and uncertainty in the data, or to advertise citations that should be used with this table. Defaults to None, no text. If you want to remove text, send empty string.
None Returns:
Type Descriptionjson Response JSON
Examples:
Basic annotation table:
description = \"Some description about the table\"\nvoxel_res = [4,4,40]\nclient.create_table(\"some_synapse_table\", \"synapse\", description, voxel_res)\ndelete_annotation(table_name, annotation_ids, aligned_volume_name=None)","text":"Delete one or more annotations in a table. Annotations that are deleted are recorded as 'non-valid' but are not physically removed from the table.
Parameters:
Name Type Description Defaulttable_name str Name of the table where annotations will be added
requiredannotation_ids (dict or list) A list of (or a single) dict of schematized annotation data matching the target table. each dict must contain an \"id\" field which is the ID of the annotation to update
requiredaligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionjson Response JSON: a list of new annotation IDs.
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.delete_table","title":"delete_table(table_name, aligned_volume_name=None)","text":"Marks a table for deletion requires super admin privileges
Parameters:
Name Type Description Defaulttable_name str name of table to mark for deletion
requiredaligned_volume_name str Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionjson Response JSON
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_annotation","title":"get_annotation(table_name, annotation_ids, aligned_volume_name=None)","text":"Retrieve an annotation or annotations by id(s) and table name.
Parameters:
Name Type Description Defaulttable_name str Name of the table
requiredannotation_ids int or iterable ID or IDS of the annotation to retreive
requiredaligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionlist Annotation data
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_annotation_count","title":"get_annotation_count(table_name, aligned_volume_name=None)","text":"Get number of annotations in a table
Parameters:
Name Type Description Defaulttable_name str name of table to mark for deletion
requiredaligned_volume_name str Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionint number of annotations
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_table_metadata","title":"get_table_metadata(table_name, aligned_volume_name=None)","text":"Get metadata about a table
Parameters:
Name Type Description Defaulttable_name str name of table to mark for deletion
requiredaligned_volume_name str Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionjson metadata about table
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_tables","title":"get_tables(aligned_volume_name=None)","text":"Gets a list of table names for a aligned_volume_name
Parameters:
Name Type Description Defaultaligned_volume_name str or None Name of the aligned_volume, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client
None Returns:
Type Descriptionlist List of table names
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.post_annotation","title":"post_annotation(table_name, data, aligned_volume_name=None)","text":"Post one or more new annotations to a table in the AnnotationEngine. All inserted annotations will be marked as 'valid'. To invalidate annotations refer to 'update_annotation', 'update_annotation_df' and 'delete_annotation' methods.
Parameters:
Name Type Description Defaulttable_name str Name of the table where annotations will be added
requireddata (dict or list) A list of (or a single) dict of schematized annotation data matching the target table.
requiredaligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionjson Response JSON
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.post_annotation_df","title":"post_annotation_df(table_name, df, position_columns, aligned_volume_name=None)","text":"Post one or more new annotations to a table in the AnnotationEngine. All inserted annotations will be marked as 'valid'. To invalidate annotations see 'update_annotation', 'update_annotation_df' and 'delete_annotation' methods.
Parameters:
Name Type Description Defaulttable_name str Name of the table where annotations will be added
requireddf DataFrame A pandas dataframe containing the annotations. Columns should be fields in schema, position columns need to be called out in position_columns argument.
requiredposition_columns Optional[Union[Iterable[str], Mapping[str, str]]] if None, will look for all columns with 'X_position' in the name and assume they go in fields called \"X\". if Iterable assumes each column given ends in _position. (i.e. ['pt_position'] if 'pt' is the name of the position field in schema) if Mapping, keys are names of columns in dataframe, values are the names of the fields (i.e. {'pt_column': 'pt'} would be correct if you had one column named 'pt_column' which needed to go into a schema with a position column called 'pt')
requiredaligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionjson Response JSON
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.process_position_columns","title":"process_position_columns(df, position_columns) staticmethod","text":"Process a dataframe into a list of dictionaries
Parameters:
Name Type Description Defaultdf DataFrame Dataframe to process
requiredposition_columns Optional[Union[Iterable[str], Mapping[str, str]]] See .post_annotation_df
Returns:
Type Descriptiondict Annotations ready for posting
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.raise_for_status","title":"raise_for_status(r, log_warning=True) staticmethod","text":"Raises requests.HTTPError, if one occurred.
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.stage_annotations","title":"stage_annotations(table_name=None, schema_name=None, update=False, id_field=False, table_resolution=None, annotation_resolution=None)","text":"Get a StagedAnnotations object to help produce correctly formatted annotations for a given table or schema. StagedAnnotation objects can be uploaded directly with upload_staged_annotations.
Parameters:
Name Type Description Defaulttable_name str Table name to stage annotations for, by default None.
None schema_name str Schema name to use to make annotations. Only needed if the table_name is not set, by default None
None update bool Set to True if individual annotations are going to be updated, by default False.
False id_field bool Set to True if id fields are to be specified. Not needed if update is True, which always needs id fields. Optional, by default False
False table_resolution list - like or None Voxel resolution of spatial points in the table in nanometers. This is found automatically from the info service if a table name is provided, by default None. If annotation_resolution is also set, this allows points to be scaled correctly for the table.
None annotation_resolution list - like Voxel resolution of spatial points provided by the user when creating annotations. If the table resolution is also available (manually or from the info service), annotations are correctly rescaled for the volume. By default, None.
None"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_annotation","title":"update_annotation(table_name, data, aligned_volume_name=None)","text":"Update one or more new annotations to a table in the AnnotationEngine. Updating is implemented by invalidating the old annotation and inserting a new annotation row, which will receive a new primary key ID.
NotesIf annotations ids were user provided upon insertion the database will autoincrement from the current max id in the table.
Parameters:
Name Type Description Defaulttable_name str Name of the table where annotations will be added
requireddata (dict or list) A list of (or a single) dict of schematized annotation data matching the target table. each dict must contain an \"id\" field which is the ID of the annotation to update
requiredaligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionjson Response JSON: a list of new annotation IDs.
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_annotation_df","title":"update_annotation_df(table_name, df, position_columns, aligned_volume_name=None)","text":"Update one or more annotations to a table in the AnnotationEngine using a dataframe as format. Updating is implemented by invalidating the old annotation and inserting a new annotation row, which will receive a new primary key ID.
NotesIf annotations ids were user provided upon insertion the database will autoincrement from the current max id in the table.
Parameters:
Name Type Description Defaulttable_name str Name of the table where annotations will be added
requireddf DataFrame A pandas dataframe containing the annotations. Columns should be fields in schema, position columns need to be called out in position_columns argument.
requiredposition_columns Iterable[str] or Mapping[str, str] or None if None, will look for all columns with 'X_position' in the name and assume they go in fields called \"X\". if Iterable assumes each column given ends in _position. (i.e. ['pt_position'] if 'pt' is the name of the position field in schema) if Mapping, keys are names of columns in dataframe, values are the names of the fields (i.e. {'pt_column': 'pt'} would be correct if you had one column named 'pt_column' which needed to go into a schema with a position column called 'pt')
requiredaligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionjson Response JSON
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_metadata","title":"update_metadata(table_name, description=None, flat_segmentation_source=None, read_permission=None, write_permission=None, user_id=None, notice_text=None, aligned_volume_name=None)","text":"Update the metadata on an existing table
Parameters:
Name Type Description Defaulttable_name str required description str Defaults to the None type, in which case no change will be made to this metadata field.
None flat_segmentation_source str Defaults to the None type, in which case no change will be made to this metadata field.
None read_permission str What permissions to give the table for reading. One of PRIVATE: only you can read this table. Intended to be used for sorting out bugs. GROUP: only members that share a group with you can read (intended for within group vetting) PUBLIC: anyone with permissions to read this datastack can read this data Defaults to the None type, in which case no change will be made to this metadata field.
None write_permission str What permissions to give the table for writing. One of PRIVATE: only you can write to this table GROUP: only members that share a group with you can write (excluding some groups) PUBLIC: Anyone can write to this table. Note all data is logged, and deletes are done by marking rows as deleted, so all data is always recoverable Defaults to the None type, in which case no change will be made to this metadata field.
None user_id int Note, if you use this you will not be able to update the metadata on this table any longer and depending on permissions may not be able to read or write to it Defaults to the None type, in which case no change will be made to this metadata field.
None notice_text str Text the user will see when querying this table. Can be used to warn users of flaws, and uncertainty in the data, or to advertise citations that should be used with this table. If you wish to remove the notice_text pass an empty string, or \"none\", \"None\", \"NONE\", or any other capitaliztion of the word \"none\". Defaults to the None type, in which case no change will be made to this metadata field.
None aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.upload_staged_annotations","title":"upload_staged_annotations(staged_annos, aligned_volume_name=None)","text":"Upload annotations directly from an Annotation Guide object. This method uses the options specified in the object, including table name and if the annotation is an update or not.
Parameters:
Name Type Description Defaultstaged_annos AnnotationGuide AnnotationGuide object with a specified table name and a collection of annotations already filled in.
requiredaligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type DescriptionList or dict If new annotations are posted, a list of ids. If annotations are being updated, a dictionary with the mapping from old ids to new ids.
"},{"location":"api/auth/","title":"client.auth","text":"Client to find and use auth tokens to access the other services.
Methods:
Name Descriptionget_group_users Get users in a group
get_new_token Currently, returns instructions for getting a new token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.
get_token Load a token with a given key the specified token file
get_tokens Get the tokens setup for this users
get_user_information Get user data.
save_token Conveniently save a token in the correct format.
setup_token Currently, returns instructions for getting your auth token based on the current settings and saving it to the local environment.
Attributes:
Name Type Descriptionrequest_header Formatted request header with the specified token
token Secret token used to authenticate yourself to the Connectome Annotation Versioning Engine services.
"},{"location":"api/auth/#caveclient.auth.AuthClient.request_header","title":"request_header property","text":"Formatted request header with the specified token
"},{"location":"api/auth/#caveclient.auth.AuthClient.token","title":"token property writable","text":"Secret token used to authenticate yourself to the Connectome Annotation Versioning Engine services.
"},{"location":"api/auth/#caveclient.auth.AuthClient.get_group_users","title":"get_group_users(group_id)","text":"Get users in a group
Parameters:
Name Type Description Defaultgroup_id int ID value for a given group
requiredReturns:
Type Descriptionlist List of dicts of user ids. Returns empty list if group does not exist.
"},{"location":"api/auth/#caveclient.auth.AuthClient.get_new_token","title":"get_new_token(open=False, no_text=False)","text":"Currently, returns instructions for getting a new token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.
Parameters:
Name Type Description Defaultopen bool If True, opens a web browser to the web page where you can generate a new token.
False"},{"location":"api/auth/#caveclient.auth.AuthClient.get_token","title":"get_token(token_key=None)","text":"Load a token with a given key the specified token file
Parameters:
Name Type Description Defaulttoken_key str or None key in the token file JSON, by default None. If None, uses 'token'.
None"},{"location":"api/auth/#caveclient.auth.AuthClient.get_tokens","title":"get_tokens()","text":"Get the tokens setup for this users
Returns:
Type Descriptionlist[dict]: a list of dictionary of tokens, each with the keys \"id\": the id of this token \"token\": the token (str) \"user_id\": the users id (should be your ID)
"},{"location":"api/auth/#caveclient.auth.AuthClient.get_user_information","title":"get_user_information(user_ids)","text":"Get user data.
Parameters:
Name Type Description Defaultuser_ids list of int user_ids to look up
required"},{"location":"api/auth/#caveclient.auth.AuthClient.save_token","title":"save_token(token=None, token_key=default_token_key, overwrite=False, token_file=None, switch_token=True, write_to_server_file=True, ignore_readonly=True, local_server=True)","text":"Conveniently save a token in the correct format.
After getting a new token by following the instructions in authclient.get_new_token(), you can save it with a fully default configuration by running:
token = 'my_shiny_new_token'
authclient.save_token(token=token)
Now on next load, authclient=AuthClient() will make an authclient instance using this token. If you would like to specify more information about the json file where the token will be stored, see the parameters below.
Parameters:
Name Type Description Defaulttoken str New token to save, by default None
None token_key str Key for the token in the token_file json, by default \"token\"
default_token_key overwrite bool Allow an existing token to be changed, by default False
False token_file str Path to the token file, by default None. If None, uses the default file location specified above.
None switch_token bool If True, switch the auth client over into using the new token, by default True
True write_to_server_file bool If True, will write token to a server specific file to support this machine interacting with multiple auth servers.
True ignore_readonly bool If True, will only attempt to save a token if the directory is writeable.
True local_server bool If True, saves the token to the local server file as well.
True"},{"location":"api/auth/#caveclient.auth.AuthClient.setup_token","title":"setup_token(make_new=True, open=True)","text":"Currently, returns instructions for getting your auth token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.
Parameters:
Name Type Description Defaultmake_new bool If True, will make a new token, else prompt you to open a page to retrieve an existing token.
True open bool If True, opens a web browser to the web page where you can retrieve a token.
True"},{"location":"api/chunkedgraph/","title":"client.chunkedgraph","text":"Client for interacting with the chunkedgraph.
Methods:
Name Descriptionget_roots Get the root ID for a list of supervoxels.
get_root_id Get the root ID for a specified supervoxel.
get_minimal_covering_nodes Get the minimal covering nodes for a list of root IDs.
get_merge_log Get the merge log (splits and merges) for an object.
get_change_log Get the change log (splits and merges) for an object.
get_user_operations Get operation details for a user ID. Currently, this is only available to
get_tabular_change_log Get a detailed changelog for neurons.
get_leaves_many Get all supervoxels for a list of root IDs.
get_leaves Get all supervoxels for a root ID.
do_merge Perform a merge on the chunked graph.
undo_operation Undo an operation.
execute_split Execute a multicut split based on points or supervoxels.
preview_split Get supervoxel connected components from a preview multicut split.
get_children Get the children of a node in the chunked graph hierarchy.
get_contact_sites Get contacts for a root ID.
find_path Find a path between two locations on a root ID using the level 2 chunked
get_subgraph Get subgraph of root id within a bounding box.
level2_chunk_graph Get graph of level 2 chunks, the smallest agglomeration level above supervoxels.
remesh_level2_chunks Submit specific level 2 chunks to be remeshed in case of a problem.
get_operation_details Get the details of a list of operations.
get_lineage_graph Returns the lineage graph for a root ID, optionally cut off in the past or
get_latest_roots Returns root IDs that are related to the given root_id at a given
get_original_roots Returns root IDs that are the latest successors of a given root ID.
is_latest_roots Check whether these root IDs are still a root at this timestamp.
suggest_latest_roots Suggest latest roots for a given root id, based on overlap of component
is_valid_nodes Check whether nodes are valid for given timestamp range.
get_root_timestamps Retrieves timestamps when roots where created.
get_past_ids For a set of root IDs, get the list of IDs at a past or future time point
get_delta_roots Get the list of roots that have changed between timetamp_past and
get_oldest_timestamp Get the oldest timestamp in the database.
Attributes:
Name Type Descriptionserver_version Optional[Version] The version of the service running on the remote server. Note that this
timestamp Optional[datetime] The default timestamp for queries which expect a timestamp. If None, uses the
segmentation_info Complete segmentation metadata
base_resolution MIP 0 resolution for voxels assumed by the ChunkedGraph
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.timestamp","title":"timestamp: Optional[datetime.datetime] property writable","text":"The default timestamp for queries which expect a timestamp. If None, uses the current time.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.segmentation_info","title":"segmentation_info property","text":"Complete segmentation metadata
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.base_resolution","title":"base_resolution property","text":"MIP 0 resolution for voxels assumed by the ChunkedGraph
Returns:
Type Descriptionlist 3-long list of x/y/z voxel dimensions in nm
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_roots","title":"get_roots(supervoxel_ids, timestamp=None, stop_layer=None)","text":"Get the root ID for a list of supervoxels.
Parameters:
Name Type Description Defaultsupervoxel_ids list or np.array of int Supervoxel IDs to look up.
requiredtimestamp datetime UTC datetime to specify the state of the chunkedgraph at which to query, by default None. If None, uses the timestamp property for this client, which defaults to the current time.
None stop_layer int or None If True, looks up IDs only up to a given stop layer. Default is None.
None Returns:
Type Descriptionnp.array of np.uint64 Root IDs containing each supervoxel.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_root_id","title":"get_root_id(supervoxel_id, timestamp=None, level2=False)","text":"Get the root ID for a specified supervoxel.
Parameters:
Name Type Description Defaultsupervoxel_id int Supervoxel id value
requiredtimestamp datetime UTC datetime to specify the state of the chunkedgraph at which to query, by default None. If None, uses the timestamp property for this client, which defaults to the current time.
None Returns:
Type Descriptionint64 Root ID containing the supervoxel.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_minimal_covering_nodes","title":"get_minimal_covering_nodes(node_ids)","text":"Get the minimal covering nodes for a list of root IDs.
Parameters:
Name Type Description Defaultnodes_ids Iterable of int or np.int64 List of root IDs to query.
requiredReturns:
Type Descriptionnp.array of np.int64: List of PCG node_ids that minimally and exactly cover the input nodes
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_merge_log","title":"get_merge_log(root_id)","text":"Get the merge log (splits and merges) for an object.
Parameters:
Name Type Description Defaultroot_id int Object root ID to look up.
requiredReturns:
Type Descriptionlist List of merge events in the history of the object.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_change_log","title":"get_change_log(root_id, filtered=True)","text":"Get the change log (splits and merges) for an object.
Parameters:
Name Type Description Defaultroot_id int Object root ID to look up.
requiredfiltered bool Whether to filter the change log to only include splits and merges which affect the final state of the object (filtered=True), as opposed to including edit history for objects which as some point were split from the query object root_id (filtered=False). Defaults to True.
True Returns:
Type Descriptiondict Dictionary summarizing split and merge events in the object history, containing the following keys:
get_user_operations(user_id, timestamp_start, include_undo=True, timestamp_end=None)","text":"Get operation details for a user ID. Currently, this is only available to admins.
Parameters:
Name Type Description Defaultuser_id int User ID to query (use 0 for all users (admin only)).
requiredtimestamp_start datetime Timestamp to start filter (UTC).
requiredinclude_undo bool Whether to include undos. Defaults to True.
True timestamp_end datetime Timestamp to end filter (UTC). Defaults to now.
None Returns:
Type DescriptionDataFrame DataFrame including the following columns:
get_tabular_change_log(root_ids, filtered=True)","text":"Get a detailed changelog for neurons.
Parameters:
Name Type Description Defaultroot_ids list of int Object root IDs to look up.
requiredfiltered bool Whether to filter the change log to only include splits and merges which affect the final state of the object (filtered=True), as opposed to including edit history for objects which as some point were split from the query objects in root_ids (filtered=False). Defaults to True.
True Returns:
Type Descriptiondict of pd.DataFrame The keys are the root IDs, and the values are DataFrames with the following columns and datatypes:
datetime.datetime.utcfromtimestamp(timestamp/1000).get_leaves_many(root_ids, bounds=None, stop_layer=None)","text":"Get all supervoxels for a list of root IDs.
Parameters:
Name Type Description Defaultroot_ids Iterable Root IDs to query.
requiredbounds If specified, returns supervoxels within a 3x2 numpy array of bounds [[minx,maxx],[miny,maxy],[minz,maxz]]. If None, finds all supervoxels.
None stop_layer int If specified, returns chunkedgraph nodes at layer stop_layer default will be stop_layer=1 (supervoxels).
None Returns:
Type Descriptiondict Dict relating ids to contacts
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_leaves","title":"get_leaves(root_id, bounds=None, stop_layer=None)","text":"Get all supervoxels for a root ID.
Parameters:
Name Type Description Defaultroot_id int Root ID to query.
requiredbounds If specified, returns supervoxels within a 3x2 numpy array of bounds [[minx,maxx],[miny,maxy],[minz,maxz]]. If None, finds all supervoxels.
None stop_layer int If specified, returns chunkedgraph nodes at layer stop_layer default will be stop_layer=1 (supervoxels).
None Returns:
Type Descriptionnp.array of np.int64 Array of supervoxel IDs (or node ids if stop_layer>1).
do_merge(supervoxels, coords, resolution=(4, 4, 40))","text":"Perform a merge on the chunked graph.
Parameters:
Name Type Description Defaultsupervoxels iterable An N-long list of supervoxels to merge.
requiredcoords array An Nx3 array of coordinates of the supervoxels in units of resolution.
resolution tuple What to multiply coords by to get nanometers. Defaults to (4,4,40).
(4, 4, 40)"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.undo_operation","title":"undo_operation(operation_id)","text":"Undo an operation.
Parameters:
Name Type Description Defaultoperation_id int Operation ID to undo.
requiredReturns:
Type Descriptiondict"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.execute_split","title":"execute_split(source_points, sink_points, root_id, source_supervoxels=None, sink_supervoxels=None)","text":"Execute a multicut split based on points or supervoxels.
Parameters:
Name Type Description Defaultsource_points array or list Nx3 list or array of 3d points in nm coordinates for source points (red).
requiredsink_points array or list Mx3 list or array of 3d points in nm coordinates for sink points (blue).
requiredroot_id int Root ID of object to do split preview.
requiredsource_supervoxels (array, list or None) If providing source supervoxels, an N-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.
None sink_supervoxels (array, list or None) If providing sink supervoxels, an M-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.
None Returns:
Name Type Descriptionoperation_id int Unique ID of the split operation
new_root_ids list of int List of new root IDs resulting from the split operation.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.preview_split","title":"preview_split(source_points, sink_points, root_id, source_supervoxels=None, sink_supervoxels=None, return_additional_ccs=False)","text":"Get supervoxel connected components from a preview multicut split.
Parameters:
Name Type Description Defaultsource_points array or list Nx3 list or array of 3d points in nm coordinates for source points (red).
requiredsink_points array or list Mx3 list or array of 3d points in nm coordinates for sink points (blue).
requiredroot_id int Root ID of object to do split preview.
requiredsource_supervoxels (array, list or None) If providing source supervoxels, an N-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.
None sink_supervoxels (array, list or None) If providing sink supervoxels, an M-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.
None return_additional_ccs bool If True, returns any additional connected components beyond the ones with source and sink points. In most situations, this can be ignored. By default, False.
False Returns:
Name Type Descriptionsource_connected_component list Supervoxel IDs in the component with the most source points.
sink_connected_component list Supervoxel IDs in the component with the most sink points.
successful_split bool True if the split worked.
other_connected_components (optional) : list of lists of int List of lists of supervoxel IDs for any other resulting connected components. Only returned if return_additional_ccs is True.
get_children(node_id)","text":"Get the children of a node in the chunked graph hierarchy.
Parameters:
Name Type Description Defaultnode_id int Node ID to query.
requiredReturns:
Type Descriptionnp.array of np.int64 IDs of child nodes.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_contact_sites","title":"get_contact_sites(root_id, bounds, calc_partners=False)","text":"Get contacts for a root ID.
Parameters:
Name Type Description Defaultroot_id int Root ID to query.
requiredbounds Bounds within a 3x2 numpy array of bounds [[minx,maxx],[miny,maxy],[minz,maxz]] for which to find contacts. Running this query without bounds is too slow.
calc_partners bool If True, get partner root IDs. By default, False.
False Returns:
Type Descriptiondict Dict relating ids to contacts
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.find_path","title":"find_path(root_id, src_pt, dst_pt, precision_mode=False)","text":"Find a path between two locations on a root ID using the level 2 chunked graph.
Parameters:
Name Type Description Defaultroot_id int Root ID to query.
requiredsrc_pt array 3-element array of xyz coordinates in nm for the source point.
requireddst_pt array 3-element array of xyz coordinates in nm for the destination point.
requiredprecision_mode bool Whether to perform the search in precision mode. Defaults to False.
False Returns:
Name Type Descriptioncentroids_list array Array of centroids along the path.
l2_path np.array of int Array of level 2 chunk IDs along the path.
failed_l2_ids np.array of int Array of level 2 chunk IDs that failed to find a path.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_subgraph","title":"get_subgraph(root_id, bounds)","text":"Get subgraph of root id within a bounding box.
Parameters:
Name Type Description Defaultroot_id int Root (or any node ID) of chunked graph to query.
requiredbounds array 3x2 bounding box (x,y,z) x (min,max) in chunked graph coordinates.
requiredReturns:
Type Descriptionnp.array of np.int64 Node IDs in the subgraph.
np.array of np.double Affinities of edges in the subgraph.
np.array of np.int32 Areas of nodes in the subgraph.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.level2_chunk_graph","title":"level2_chunk_graph(root_id, bounds=None)","text":"Get graph of level 2 chunks, the smallest agglomeration level above supervoxels.
Parameters:
Name Type Description Defaultroot_id int Root id of object
requiredbounds array 3x2 bounding box (x,y,z) x (min,max) in chunked graph coordinates (use client.chunkedgraph.base_resolution to view this default resolution for your chunkedgraph client). Note that the result will include any level 2 nodes which have chunk boundaries within some part of this bounding box, meaning that the representative point for a given level 2 node could still be slightly outside of these bounds. If None, returns all level 2 chunks for the root ID.
None Returns:
Type Descriptionlist of list Edge list for level 2 chunked graph. Each element of the list is an edge, and each edge is a list of two node IDs (source and target).
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.remesh_level2_chunks","title":"remesh_level2_chunks(chunk_ids)","text":"Submit specific level 2 chunks to be remeshed in case of a problem.
Parameters:
Name Type Description Defaultchunk_ids list List of level 2 chunk IDs.
required"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_operation_details","title":"get_operation_details(operation_ids)","text":"Get the details of a list of operations.
Parameters:
Name Type Description Defaultoperation_ids Iterable[int] List/array of operation IDs.
requiredReturns:
Type Descriptiondict of str to dict A dict of dicts of operation info, keys are operation IDs (as strings), values are a dictionary of operation info for the operation. These dictionaries contain the following keys:
get_lineage_graph(root_id, timestamp_past=None, timestamp_future=None, as_nx_graph=False, exclude_links_to_future=False, exclude_links_to_past=False)","text":"Returns the lineage graph for a root ID, optionally cut off in the past or the future.
Each change in the chunked graph creates a new root ID for the object after that change. This function returns a graph of all root IDs for a given object, tracing the history of the object in terms of merges and splits.
Parameters:
Name Type Description Defaultroot_id int Object root ID.
requiredtimestamp_past datetime or None Cutoff for the lineage graph backwards in time. By default, None.
None timestamp_future datetime or None Cutoff for the lineage graph going forwards in time. By default, uses the timestamp property for this client, which defaults to the current time.
None as_nx_graph If True, a NetworkX graph is returned.
False exclude_links_to_future If True, links from nodes before timestamp_future to after timestamp_future are removed. If False, the link(s) which has one node before timestamp and one node after timestamp is kept.
False exclude_links_to_past If True, links from nodes before timestamp_past to after timestamp_past are removed. If False, the link(s) which has one node before timestamp and one node after timestamp is kept.
False Returns:
Type Descriptiondict Dictionary describing the lineage graph and operations for the root ID. Not returned if as_nx_graph is True. The dictionary contains the following keys:
DiGraph NetworkX directed graph of the lineage graph. Only returned if as_nx_graph is True.
get_latest_roots(root_id, timestamp=None, timestamp_future=None)","text":"Returns root IDs that are related to the given root_id at a given timestamp. Can be used to find the \"latest\" root IDs associated with an object.
Parameters:
Name Type Description Defaultroot_id int Object root ID.
requiredtimestamp datetime or None Timestamp of where to query IDs from. If None, uses the timestamp property for this client, which defaults to the current time.
None timestamp_future datetime or None DEPRECATED name, use timestamp instead. Timestamp to suggest IDs from (note can be in the past relative to the root). By default, None.
None Returns:
Type Descriptionndarray 1d array with all latest successors.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_original_roots","title":"get_original_roots(root_id, timestamp_past=None)","text":"Returns root IDs that are the latest successors of a given root ID.
Parameters:
Name Type Description Defaultroot_id int Object root ID.
requiredtimestamp_past datetime or None Cutoff for the search going backwards in time. By default, None.
None Returns:
Type Descriptionndarray 1d array with all latest successors.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.is_latest_roots","title":"is_latest_roots(root_ids, timestamp=None)","text":"Check whether these root IDs are still a root at this timestamp.
Parameters:
Name Type Description Defaultroot_ids array-like of int Root IDs to check.
requiredtimestamp datetime Timestamp to check whether these IDs are valid root IDs in the chunked graph. If None, uses the timestamp property for this client, which defaults to the current time.
None Returns:
Type Descriptionnp.array of bool Array of whether these are valid root IDs.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.suggest_latest_roots","title":"suggest_latest_roots(root_id, timestamp=None, stop_layer=None, return_all=False, return_fraction_overlap=False)","text":"Suggest latest roots for a given root id, based on overlap of component chunk IDs. Note that edits change chunk IDs, and so this effectively measures the fraction of unchanged chunks at a given chunk layer, which sets the size scale of chunks. Higher layers are coarser.
Parameters:
Name Type Description Defaultroot_id int Root ID of the potentially outdated object.
requiredtimestamp datetime Datetime at which \"latest\" roots are being computed, by default None. If None, uses the timestamp property for this client, which defaults to the current time. Note that this has to be a timestamp after the creation of the root_id.
None stop_layer int Chunk level at which to compute overlap, by default None. No value will take the 4th from the top layer, which emphasizes speed and works well for larger objects. Lower values are slower but more fine-grained. Values under 2 (i.e. supervoxels) are not recommended except in extremely fine grained scenarios.
None return_all bool If True, return all current IDs sorted from most overlap to least, by default False. If False, only the top is returned.
False return_fraction_overlap bool If True, return all fractions sorted by most overlap to least, by default False. If False, only the top value is returned.
False"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.is_valid_nodes","title":"is_valid_nodes(node_ids, start_timestamp=None, end_timestamp=None)","text":"Check whether nodes are valid for given timestamp range.
Valid is defined as existing in the chunked graph. This makes no statement about these IDs being roots, supervoxel or anything in-between. It also does not take into account whether a root ID has since been edited.
Parameters:
Name Type Description Defaultnode_ids array-like of int Node IDs to check.
requiredstart_timestamp datetime Timestamp to check whether these IDs were valid after this timestamp. Defaults to None (assumes now).
None end_timestamp datetime Timestamp to check whether these IDs were valid before this timestamp. If None, uses the timestamp property for this client, which defaults to the current time.
None Returns:
Type Descriptionnp.array of bool Array of whether these are valid IDs.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_root_timestamps","title":"get_root_timestamps(root_ids, latest=False, timestamp=None)","text":"Retrieves timestamps when roots where created.
Parameters:
Name Type Description Defaultroot_ids Iterable of root IDs to query.
requiredlatest bool If False, returns the first timestamp that the root_id was valid for each root ID. If True, returns the newest/latest timestamp for each root ID. Note, this will return the timestamp at which the query was run when the root is currently valid. This means that you will get a different answer if you make this same query at a later time if you don't specify a timestamp parameter.
False timestamp datetime Timestamp to query when using latest=True. Use this to provide consistent results for a particular timestamp. If an ID is still valid at a point in the future past this timestamp, the query will still return this timestamp as the latest moment in time. An error will occur if you provide a timestamp for which the root ID is not valid. If None, uses the timestamp property for this client, which defaults to the current time.
None Returns:
Type Descriptionnp.array of datetime.datetime Array of timestamps when root_ids were created.
get_past_ids(root_ids, timestamp_past=None, timestamp_future=None)","text":"For a set of root IDs, get the list of IDs at a past or future time point that could contain parts of the same object.
Parameters:
Name Type Description Defaultroot_ids Iterable of int Iterable of root IDs to query.
requiredtimestamp_past datetime or None Time of a point in the past for which to look up root ids. Default is None.
None timestamp_future datetime or None Time of a point in the future for which to look up root ids. Not implemented on the server currently. Default is None.
None Returns:
Type Descriptiondict Dict with keys \"future_id_map\" and \"past_id_map\". Each is a dict whose keys are the supplied root_ids and whose values are the list of related root IDs at timestamp_past/timestamp_future.
get_delta_roots(timestamp_past, timestamp_future=datetime.datetime.now(datetime.timezone.utc))","text":"Get the list of roots that have changed between timetamp_past and timestamp_future.
Parameters:
Name Type Description Defaulttimestamp_past datetime Past timepoint to query
requiredtimestamp_future datetime Future timepoint to query. Defaults to datetime.datetime.now(datetime.timezone.utc).
now(utc) Returns:
Name Type Descriptionold_roots np.ndarray of np.int64 Roots that have expired in that interval.
new_roots np.ndarray of np.int64 Roots that are new in that interval.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_oldest_timestamp","title":"get_oldest_timestamp()","text":"Get the oldest timestamp in the database.
Returns:
Type Descriptiondatetime Oldest timestamp in the database.
"},{"location":"api/client/","title":"caveclient.CAVEclient","text":"A manager for all clients sharing common datastack and authentication information.
This client wraps all the other clients and keeps track of the things that need to be consistent across them. To instantiate a client:
from caveclient import CAVEclient\n\nclient = CAVEclient(datastack_name='my_datastack',\n server_address='www.myserver.com',\n auth_token_file='~/.mysecrets/secrets.json')\nThen
client.annotation is an AnnotationClient (see client.annotation)client.auth is an AuthClient (see client.auth)client.chunkedgraph is a ChunkedGraphClient (see client.chunkedgraph)client.info is an InfoServiceClient (see client.info)client.l2cache is an L2CacheClient (see client.l2cache)client.materialize is a MaterializationClient (see client.materialize)client.skeleton is a SkeletonClient (see client.skeleton)client.schema is a SchemaClient (see client.schema)client.state is a neuroglancer JSONService (see client.state)All subclients are loaded lazily and share the same datastack name, server address, and auth tokens where used.
Parameters:
Name Type Description Defaultdatastack_name str Datastack name for the services. Almost all services need this and will not work if it is not passed.
None server_address str or None URL of the framework server. If None, chooses the default server global.daf-apis.com. Optional, defaults to None.
None auth_token_file str or None Path to a json file containing the auth token. If None, uses the default location. See Auth client documentation. Optional, defaults to None.
default_token_file auth_token_key str Dictionary key for the token in the the JSON file. Optional, default is 'token'.
'token' auth_token str or None Direct entry of an auth token. If None, uses the file arguments to find the token. Optional, default is None.
None max_retries int or None Sets the default number of retries on failed requests. Optional, by default 2.
None pool_maxsize int or None Sets the max number of threads in a requests pool, although this value will be exceeded if pool_block is set to False. Optional, uses requests defaults if None.
None pool_block If True, prevents the number of threads in a requests pool from exceeding the max size. Optional, uses requests defaults (False) if None.
None desired_resolution Iterable[float] or None If given, should be a list or array of the desired resolution you want queries returned in useful for materialization queries.
None info_cache Pre-computed info cache, bypassing the lookup of datastack info from the info service. Should only be used in cases where this information is cached and thus repetitive lookups can be avoided.
None version Optional[int] The default materialization version of the datastack to use. If None, the latest version is used. Optional, defaults to None.
None See Also set_session_defaults
get_session_defaults
Methods:
Name Descriptionchange_auth Change the authentication token and reset services.
Attributes:
Name Type Descriptionannotation AnnotationClient A client for the annotation service. See client.annotation
auth AuthClient A client for the auth service. See client.auth for more information.
chunkedgraph ChunkedGraphClient A client for the chunkedgraph service. See client.chunkedgraph
datastack_name str The name of the datastack for the client.
info InfoServiceClient A client for the info service. See client.info for more information.
l2cache L2CacheClient A client for the L2 cache service. See client.l2cache
materialize MaterializationClient A client for the materialization service. See client.materialize
schema SchemaClient A client for the EM Annotation Schemas service. See client.schema
server_address The server address for the client.
skeleton SkeletonClient A client for the skeleton service. See client.skeleton
state JSONService A client for the neuroglancer state service. See client.state
timestamp Optional[datetime] The default timestamp to use for queries which rely on a timestamp.
version Optional[int] The default materialization version of the datastack to use for queries which
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.annotation","title":"annotation: AnnotationClient property","text":"A client for the annotation service. See client.annotation for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.auth","title":"auth: AuthClient property","text":"A client for the auth service. See client.auth for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.chunkedgraph","title":"chunkedgraph: ChunkedGraphClient property","text":"A client for the chunkedgraph service. See client.chunkedgraph for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.datastack_name","title":"datastack_name: str property","text":"The name of the datastack for the client.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.info","title":"info: InfoServiceClient property","text":"A client for the info service. See client.info for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.l2cache","title":"l2cache: L2CacheClient property","text":"A client for the L2 cache service. See client.l2cache for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.materialize","title":"materialize: MaterializationClient property","text":"A client for the materialization service. See client.materialize for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.schema","title":"schema: SchemaClient property","text":"A client for the EM Annotation Schemas service. See client.schema for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.server_address","title":"server_address property","text":"The server address for the client.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.skeleton","title":"skeleton: SkeletonClient property","text":"A client for the skeleton service. See client.skeleton for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.state","title":"state: JSONService property","text":"A client for the neuroglancer state service. See client.state for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.timestamp","title":"timestamp: Optional[datetime] property","text":"The default timestamp to use for queries which rely on a timestamp.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.version","title":"version: Optional[int] property writable","text":"The default materialization version of the datastack to use for queries which expect a version. Also sets the timestamp to the corresponding timestamp of the version for queries which rely on a timestamp.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.change_auth","title":"change_auth(auth_token_file=None, auth_token_key=None, auth_token=None)","text":"Change the authentication token and reset services.
Parameters:
Name Type Description Defaultauth_token_file str New auth token json file path, by default None, which defaults to the existing state.
None auth_token_key str New dictionary key under which the token is stored in the json file, by default None, which defaults to the existing state.
None auth_token str Direct entry of a new token, by default None.
None"},{"location":"api/config/","title":"Configuration","text":""},{"location":"api/config/#caveclient.set_session_defaults","title":"caveclient.set_session_defaults(max_retries=3, pool_block=False, pool_maxsize=10, backoff_factor=0.1, backoff_max=120, status_forcelist=(502, 503, 504))","text":"Set global default values to configure how all clients will communicate with servers. Should be done prior to initializing a client.
Note that these values are only used when not set at the client level.
Parameters:
Name Type Description Defaultmax_retries int The maximum number of retries each connection should attempt. Set to 0 to fail on the first retry.
3 pool_block bool Whether the connection pool should block for connections.
False pool_maxsize int The maximum number of connections to save in the pool.
10 backoff_factor Union[float, int] A backoff factor to apply between attempts after the second try (most errors are resolved immediately by a second try without a delay). The query will sleep for: {backoff factor} * (2 ^ ({number of total retries} - 1)) seconds. For example, if the backoff_factor is 0.1, then will sleep for [0.0s, 0.2s, 0.4s, 0.8s, \u2026] between retries. No backoff will ever be longer than backoff_max.
0.1 backoff_max Union[float, int] The maximum backoff time.
120 status_forcelist Optional[Collection] A set of integer HTTP status codes that we should force a retry on.
(502, 503, 504) Usage from caveclient import set_session_defaults\n\nset_session_defaults(\n max_retries=5, # would increase the default number of retries\n backoff_factor=0.5, # would increase the default backoff factor between retries\n backoff_max=240, # would increase the default maximum backoff time\n status_forcelist=(502, 503, 504, 505), # would add 505 to the default list\n)\n\nset_session_defaults() # would revert all defaults to their original values\nCalling this function will set the default values for all clients created after the call.
Calling this function with any arguments missing will reset that value to the default value.
See Also:urllib3.util.Retry
requests.adapters.HTTPAdapter
"},{"location":"api/config/#caveclient.get_session_defaults","title":"caveclient.get_session_defaults()","text":"Get the current default values for session configuration.
Returns:
Type Descriptiondict Dictionary of current default values for session configuration.
"},{"location":"api/datastack_lookup/","title":"Datastack lookup","text":""},{"location":"api/datastack_lookup/#caveclient.datastack_lookup.reset_server_address_cache","title":"caveclient.datastack_lookup.reset_server_address_cache(datastack, filename=None)","text":"Remove one or more datastacks from the datastack-to-server cache.
Parameters:
Name Type Description Defaultdatastack str or list of str Datastack names to remove from the cache, by default None
requiredfilename str Name of the cache file, by default None
None"},{"location":"api/info/","title":"client.info","text":"Client for interacting with the info service.
Methods:
Name Descriptionannotation_endpoint AnnotationEngine endpoint for a dataset.
get_aligned_volume_info Gets the info record for a aligned_volume
get_datastack_info Gets the info record for a datastack
get_datastacks Query which datastacks are available at the info service
get_datastacks_by_aligned_volume Lookup what datastacks are associated with this aligned volume
image_cloudvolume Generate a cloudvolume instance based on the image source, using authentication if needed and
image_source Cloud path to the imagery for the dataset
refresh_stored_data Reload the stored info values from the server.
segmentation_cloudvolume Generate a cloudvolume instance based on the segmentation source, using authentication if needed and
segmentation_source Cloud path to the chunkgraph-backed Graphene segmentation for a dataset
synapse_segmentation_source Cloud path to the synapse segmentation for a dataset
viewer_resolution Get the viewer resolution metadata for this datastack
viewer_site Get the base Neuroglancer URL for the dataset
Attributes:
Name Type Descriptionserver_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.annotation_endpoint","title":"annotation_endpoint(datastack_name=None, use_stored=True)","text":"AnnotationEngine endpoint for a dataset.
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the datastack to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True Returns:
Type Descriptionstr Location of the AnnotationEngine
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_aligned_volume_info","title":"get_aligned_volume_info(datastack_name=None, use_stored=True)","text":"Gets the info record for a aligned_volume
Parameters:
Name Type Description Defaultdatastack_name str datastack_name to look up. If None, uses the one specified by the client. By default None
None use_stored bool If True and the information has already been queried for that dataset, then uses the cached version. If False, re-queries the infromation. By default True
True Returns:
Type Descriptiondict or None The complete info record for the aligned_volume
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastack_info","title":"get_datastack_info(datastack_name=None, use_stored=True)","text":"Gets the info record for a datastack
Parameters:
Name Type Description Defaultdatastack_name str datastack to look up. If None, uses the one specified by the client. By default None
None use_stored bool If True and the information has already been queried for that datastack, then uses the cached version. If False, re-queries the infromation. By default True
True Returns:
Type Descriptiondict or None The complete info record for the datastack
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastacks","title":"get_datastacks()","text":"Query which datastacks are available at the info service
Returns:
Type Descriptionlist List of datastack names
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastacks_by_aligned_volume","title":"get_datastacks_by_aligned_volume(aligned_volume=None)","text":"Lookup what datastacks are associated with this aligned volume
Args: aligned_volume (str, optional): aligned volume to lookup. Defaults to None.
Raises: ValueError: if no aligned volume is specified
Returns: list: a list of datastack string
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.image_cloudvolume","title":"image_cloudvolume(**kwargs)","text":"Generate a cloudvolume instance based on the image source, using authentication if needed and sensible default values for reading CAVE resources. By default, fill_missing is True and bounded is False. All keyword arguments are passed onto the CloudVolume initialization function, and defaults can be overridden.
Requires cloudvolume to be installed, which is not included by default.
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.image_source","title":"image_source(datastack_name=None, use_stored=True, format_for='raw')","text":"Cloud path to the imagery for the dataset
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the datastack to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True format_for 'raw', 'cloudvolume', or 'neuroglancer' Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"precomputed://gs://\" type path is converted to a full https URL. If 'neuroglancer', a full https URL is converted to a \"precomputed://gs://\" type path.
'raw' Returns:
Type Descriptionstr Formatted cloud path to the flat segmentation
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.refresh_stored_data","title":"refresh_stored_data()","text":"Reload the stored info values from the server.
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.segmentation_cloudvolume","title":"segmentation_cloudvolume(use_client_secret=True, **kwargs)","text":"Generate a cloudvolume instance based on the segmentation source, using authentication if needed and sensible default values for reading CAVE resources. By default, fill_missing is True and bounded is False. All keyword arguments are passed onto the CloudVolume initialization function, and defaults can be overridden.
Requires cloudvolume to be installed, which is not included by default.
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.segmentation_source","title":"segmentation_source(datastack_name=None, format_for='raw', use_stored=True)","text":"Cloud path to the chunkgraph-backed Graphene segmentation for a dataset
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the datastack to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True format_for 'raw', 'cloudvolume', or 'neuroglancer' Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"graphene://https://\" type path is used If 'neuroglancer', a \"graphene://https://\" type path is used, as needed by Neuroglancer.
'raw' Returns:
Type Descriptionstr Formatted cloud path to the Graphene segmentation
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.synapse_segmentation_source","title":"synapse_segmentation_source(datastack_name=None, use_stored=True, format_for='raw')","text":"Cloud path to the synapse segmentation for a dataset
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the dataset to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True format_for 'raw', 'cloudvolume', or 'neuroglancer' Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"precomputed://gs://\" type path is converted to a full https URL. If 'neuroglancer', a full https URL is converted to a \"precomputed://gs://\" type path.
'raw' Returns:
Type Descriptionstr Formatted cloud path to the synapse segmentation
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.viewer_resolution","title":"viewer_resolution(datastack_name=None, use_stored=True)","text":"Get the viewer resolution metadata for this datastack
Parameters:
Name Type Description Defaultdatastack_name If None use the default one configured in the client
None use_stored Use the cached value, if False go get a new value from server
True Returns:
Type Descriptionarray Voxel resolution as a len(3) np.array
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.viewer_site","title":"viewer_site(datastack_name=None, use_stored=True)","text":"Get the base Neuroglancer URL for the dataset
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the datastack to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True Returns:
Type Descriptionstr Base URL for the Neuroglancer viewer
"},{"location":"api/l2cache/","title":"client.l2cache","text":"Client for interacting with the level2 cache service.
Methods:
Name Descriptioncache_metadata Retrieves the meta data for the cache
get_l2data Gets the attributed statistics data for L2 ids.
get_l2data_table Gets the attributed statistics data for L2 ids, returned as a dataframe.
has_cache Checks if the l2 cache is available for the dataset
table_mapping Retrieves table mappings for l2 cache.
Attributes:
Name Type Descriptionserver_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.cache_metadata","title":"cache_metadata()","text":"Retrieves the meta data for the cache
Returns:
Type Descriptiondict keys are attribute names, values are datatypes
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.get_l2data","title":"get_l2data(l2_ids, attributes=None)","text":"Gets the attributed statistics data for L2 ids.
Parameters:
Name Type Description Defaultl2_ids list or ndarray A list of level 2 ids.
requiredattributes list A list of attributes to retrieve. Defaults to None, which will return all that are available. Available stats are:
area_nm2chunk_intersect_countmax_dt_nmmean_dt_nmpcapca_valrep_coord_nmsize_nm3See the tutorial for more description.
None Returns:
Type Descriptiondict Keys are l2 ids, values are data.
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.get_l2data_table","title":"get_l2data_table(l2_ids, attributes=None, split_columns=True)","text":"Gets the attributed statistics data for L2 ids, returned as a dataframe.
Parameters:
Name Type Description Defaultl2_ids list or ndarray A list of level 2 ids.
requiredattributes list A list of attributes to retrieve. Defaults to None, which will return all that are available. Available stats are:
area_nm2chunk_intersect_countmax_dt_nmmean_dt_nmpcapca_valrep_coord_nmsize_nm3See the tutorial for more description.
None split_columns bool Whether to split columns with multiple values into separate columns.
True Returns:
Type DescriptionDataFrame A pandas dataframe with the requested attributes as columns and indexed by l2_id.
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.has_cache","title":"has_cache(datastack_name=None)","text":"Checks if the l2 cache is available for the dataset
Parameters:
Name Type Description Defaultdatastack_name str The name of the datastack to check, by default None (if None, uses the client's datastack)
None Returns:
Type Descriptionbool True if the l2 cache is available, False otherwise
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.table_mapping","title":"table_mapping()","text":"Retrieves table mappings for l2 cache.
Returns:
Type Descriptiondict keys are pcg table names, values are dicts with fields l2cache_id and cv_path.
Client for interacting with the materialization engine.
Methods:
Name Descriptionget_table_metadata Get metadata about a table
get_tables Gets a list of table names for a datastack
get_tables_metadata Get metadata about tables
get_timestamp Get datetime.datetime timestamp for a materialization version.
get_unique_string_values Get unique string values for a table
get_version_metadata Get metadata about a version
get_versions Get the versions available
get_versions_metadata Get the metadata for all the versions that are presently available and valid
get_view_metadata Get metadata for a view
get_view_schema Get schema for a view
get_view_schemas Get schema for a view
get_views Get all available views for a version
ingest_annotation_table Trigger supervoxel lookup and root ID lookup of new annotations in a table.
join_query Generic query on materialization tables
live_live_query Beta method for querying cave annotation tables with root IDs and annotations
live_query Generic query on materialization tables
lookup_supervoxel_ids Trigger supervoxel lookups of new annotations in a table.
map_filters Translate a list of filter dictionaries from a point in the
most_recent_version Get the most recent version of materialization for this datastack name
query_table Generic query on materialization tables
query_view Generic query on a view
synapse_query Convenience method for querying synapses.
Attributes:
Name Type Descriptioncg_client The chunked graph client.
datastack_name The name of the datastack.
homepage HTML The homepage for the materialization engine.
server_version Optional[Version] The version of the service running on the remote server. Note that this
tables TableManager The table manager for the materialization engine.
version int The version of the materialization. Can be used to set up the
views ViewManager The view manager for the materialization engine.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.cg_client","title":"cg_client property","text":"The chunked graph client.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.datastack_name","title":"datastack_name property","text":"The name of the datastack.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.homepage","title":"homepage: HTML property","text":"The homepage for the materialization engine.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.tables","title":"tables: TableManager property","text":"The table manager for the materialization engine.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.version","title":"version: int property writable","text":"The version of the materialization. Can be used to set up the client to default to a specific version when timestamps or versions are not specified in queries. If not set, defaults to the most recent version.
Note that if this materialization client is attached to a CAVEclient, the version must be set at the CAVEclient level.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.views","title":"views: ViewManager property","text":"The view manager for the materialization engine.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_table_metadata","title":"get_table_metadata(table_name, datastack_name=None, version=None, log_warning=True)","text":"Get metadata about a table
Parameters:
Name Type Description Defaulttable_name str name of table to mark for deletion
requireddatastack_name str or None Name of the datastack_name. If None, uses the one specified in the client.
None version int The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None log_warning bool Whether to print out warnings to the logger. Defaults to True.
True Returns:
Type Descriptiondict Metadata dictionary for table
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_tables","title":"get_tables(datastack_name=None, version=None)","text":"Gets a list of table names for a datastack
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the datastack, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client
None version int or None The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None Returns:
Type Descriptionlist List of table names
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_tables_metadata","title":"get_tables_metadata(datastack_name=None, version=None, log_warning=True)","text":"Get metadata about tables
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the datastack_name. If None, uses the one specified in the client.
None version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None log_warning bool Whether to print out warnings to the logger. Defaults to True.
True Returns:
Type Descriptiondict Metadata dictionary for table
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_timestamp","title":"get_timestamp(version=None, datastack_name=None)","text":"Get datetime.datetime timestamp for a materialization version.
Parameters:
Name Type Description Defaultversion int or None The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None datastack_name str or None Datastack name, by default None. If None, defaults to the value set in the client.
None Returns:
Type Descriptiondatetime Datetime when the materialization version was frozen.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_unique_string_values","title":"get_unique_string_values(table, datastack_name=None)","text":"Get unique string values for a table
Parameters:
Name Type Description Defaulttable str Table to query
requireddatastack_name Optional[str] Datastack to query. If None, uses the one specified in the client.
None Returns:
Type Descriptiondict[str] A dictionary of column names and their unique values
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_version_metadata","title":"get_version_metadata(version=None, datastack_name=None)","text":"Get metadata about a version
Parameters:
Name Type Description Defaultversion int or None The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None datastack_name str or None Datastack name, by default None. If None, defaults to the value set in the client.
None Returns:
Type Descriptiondict Dictionary of metadata about the version
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_versions","title":"get_versions(datastack_name=None, expired=False)","text":"Get the versions available
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the datastack, by default None. If None, uses the one specified in the client.
None expired bool Whether to include expired versions, by default False.
False Returns:
Type Descriptiondict Dictionary of versions available
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_versions_metadata","title":"get_versions_metadata(datastack_name=None, expired=False)","text":"Get the metadata for all the versions that are presently available and valid
Parameters:
Name Type Description Defaultdatastack_name str or None Datastack name, by default None. If None, defaults to the value set in the client.
None expired bool Whether to include expired versions, by default False.
False Returns:
Type Descriptionlist[dict] List of metadata dictionaries
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_metadata","title":"get_view_metadata(view_name, materialization_version=None, datastack_name=None, log_warning=True)","text":"Get metadata for a view
Parameters:
Name Type Description Defaultview_name str Name of view to query.
requiredmaterialization_version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None log_warning bool Whether to log warnings.
True Returns:
Type Descriptiondict Metadata of view
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_schema","title":"get_view_schema(view_name, materialization_version=None, datastack_name=None, log_warning=True)","text":"Get schema for a view
Parameters:
Name Type Description Defaultview_name str Name of view to query.
requiredmaterialization_version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None log_warning bool Whether to log warnings.
True Returns:
Type Descriptiondict Schema of view.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_schemas","title":"get_view_schemas(materialization_version=None, datastack_name=None, log_warning=True)","text":"Get schema for a view
Parameters:
Name Type Description Defaultmaterialization_version Optional[int] Version to query. If None, will use version set by client.
None log_warning bool Whether to log warnings.
True Returns:
Type Descriptiondict Schema of view.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_views","title":"get_views(version=None, datastack_name=None)","text":"Get all available views for a version
Parameters:
Name Type Description Defaultversion Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None datastack_name str Datastack to query. If None, uses the one specified in the client.
None Returns:
Type Descriptionlist List of views
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.ingest_annotation_table","title":"ingest_annotation_table(table_name, datastack_name=None)","text":"Trigger supervoxel lookup and root ID lookup of new annotations in a table.
Parameters:
Name Type Description Defaulttable_name str Table to trigger
requireddatastack_name str Datastack to trigger it. Defaults to what is set in client.
None Returns:
Type Descriptiondict Status code of response from server
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.join_query","title":"join_query(tables, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, suffixes=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, metadata=True, desired_resolution=None, random_sample=None, log_warning=True)","text":"Generic query on materialization tables
Parameters:
Name Type Description Defaulttables list of lists with length 2 or 'str' list of two lists: first entries are table names, second entries are the columns used for the join.
requiredfilter_in_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are allowed entries, by default None
None filter_out_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are not allowed entries, by default None
None filter_equal_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are specified entry, by default None
None filter_greater_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are exclusive upper-bound, by default None
None filter_less_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are exclusive lower-bound, by default None
None filter_greater_equal_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are inclusive upper-bound, by default None
None filter_less_equal_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are inclusive lower-bound, by default None
None filter_spatial_dict dict of dicts outer layer: keys are table names, inner layer: keys are column names. Values are bounding boxes as [[min_x, min_y,min_z],[max_x, max_y, max_z]], expressed in units of the voxel_resolution of this dataset. Defaults to None.
None filter_regex_dict dict of dicts outer layer: keys are table names. inner layer: keys are column names, values are regex strings. Defaults to None
None select_columns dict of lists of str keys are table names,values are the list of columns from that table. Defaults to None, which will select all tables. Will be passed to server as select_column_maps. Passing a list will be passed as select_columns which is deprecated.
None offset int result offset to use. Defaults to None. Will only return top K results.
None limit int maximum results to return (server will set upper limit, see get_server_config)
None suffixes dict suffixes to use for duplicate columns, keys are table names, values are the suffix
None datastack_name str datastack to query. If None defaults to one specified in client.
None return_df bool whether to return as a dataframe default True, if False, data is returned as json (slower)
True split_positions bool whether to break position columns into x,y,z columns default False, if False data is returned as one column with [x,y,z] array (slower)
False materialization_version int The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None metadata bool toggle to return metadata If True (and return_df is also True), return table and query metadata in the df.attr dictionary.
True desired_resolution Iterable What resolution to convert position columns to. Defaults to None will use defaults.
None random_sample int if given, will do a tablesample of the table to return that many annotations
None log_warning bool Whether to log warnings, by default True
True Returns:
Type DescriptionDataFrame a pandas dataframe of results of query
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.live_live_query","title":"live_live_query(table, timestamp, joins=None, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, split_positions=False, metadata=True, suffixes=None, desired_resolution=None, allow_missing_lookups=False, allow_invalid_root_ids=False, random_sample=None, log_warning=True)","text":"Beta method for querying cave annotation tables with root IDs and annotations at a particular timestamp. Note: this method requires more explicit mapping of filters and selection to table as its designed to test a more general endpoint that should eventually support complex joins.
Parameters:
Name Type Description Defaulttable str Principle table to query
requiredtimestamp datetime Timestamp to query
requiredjoins List of joins, where each join is a list of [table1,column1, table2, column2]
None filter_in_dict A dictionary with tables as keys, values are dicts with column keys and list values to accept.
None filter_out_dict A dictionary with tables as keys, values are dicts with column keys and list values to reject.
None filter_equal_dict A dictionary with tables as keys, values are dicts with column keys and values to equate.
None filter_greater_dict A dictionary with tables as keys, values are dicts with column keys and values as exclusive upper-bound.
None filter_less_dict A dictionary with tables as keys, values are dicts with column keys and values as exclusive lower-bound.
None filter_greater_equal_dict A dictionary with tables as keys, values are dicts with column keys and values as inclusive upper-bound.
None filter_less_equal_dict A dictionary with tables as keys, values are dicts with column keys and values as inclusive lower-bound.
None filter_spatial_dict A dictionary with tables as keys, values are dicts with column keys and values of 2x3 list of bounds.
None filter_regex_dict A dictionary with tables as keys, values are dicts with column keys and values of regex strings.
None select_columns A dictionary with tables as keys, values are lists of columns to select.
None offset int Value to offset query by.
None limit int Limit of query.
None datastack_name str Datastack to query. Defaults to set by client.
None split_positions bool Whether to split positions into separate columns, True is faster.
False metadata bool Whether to attach metadata to dataframe.
True suffixes dict What suffixes to use on joins, keys are table_names, values are suffixes.
None desired_resolution Iterable What resolution to convert position columns to.
None allow_missing_lookups bool If there are annotations without supervoxels and root IDs yet, allow results.
False allow_invalid_root_ids bool If True, ignore root ids not valid at the given timestamp, otherwise raise an error.
False random_sample int If given, will do a table sample of the table to return that many annotations.
None log_warning bool Whether to log warnings.
True Returns:
Type DescriptionResults of query
Examples:
>>> from caveclient import CAVEclient\n>>> client = CAVEclient('minnie65_public_v117')\n>>> live_live_query(\"table_name\", datetime.datetime.now(datetime.timezone.utc),\n>>> joins=[[table_name, table_column, joined_table, joined_column],\n>>> [joined_table, joincol2, third_table, joincol_third]]\n>>> suffixes={\n>>> \"table_name\":\"suffix1\",\n>>> \"joined_table\":\"suffix2\",\n>>> \"third_table\":\"suffix3\"\n>>> },\n>>> select_columns= {\n>>> \"table_name\":[ \"column\",\"names\"],\n>>> \"joined_table\":[\"joined_colum\"]\n>>> },\n>>> filter_in_dict= {\n>>> \"table_name\":{\n>>> \"column_name\":[included,values]\n>>> }\n>>> },\n>>> filter_out_dict= {\n>>> \"table_name\":{\n>>> \"column_name\":[excluded,values]\n>>> }\n>>> },\n>>> filter_equal_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_greater_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_less_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_greater_equal_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_less_equal_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_spatial_dict\"= {\n>>> \"table_name\": {\n>>> \"column_name\": [[min_x, min_y, min_z], [max_x, max_y, max_z]]\n>>> }\n>>> filter_regex_dict\"= {\n>>> \"table_name\": {\n>>> \"column_name\": \"regex_string\"\n>>> }\nlive_query(table, timestamp, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, split_positions=False, post_filter=True, metadata=True, merge_reference=True, desired_resolution=None, random_sample=None, log_warning=True)","text":"Generic query on materialization tables
Parameters:
Name Type Description Defaulttable str Table to query
requiredtimestamp datetime Time to materialize (in utc). Pass datetime.datetime.now(datetime.timezone.utc) for present time.
requiredfilter_in_dict dict Keys are column names, values are allowed entries.
None filter_out_dict dict Keys are column names, values are not allowed entries.
None filter_equal_dict dict Keys are column names, values are specified entry.
None filter_greater_dict dict Keys are column names, values are exclusive upper-bounds.
None filter_less_dict dict Keys are column names, values are exclusive lower-bounds.
None filter_greater_equal_dict dict Keys are column names, values are inclusive upper-bounds.
None filter_less_equal_dict dict Keys are column names, values are inclusive lower-bounds.
None filter_spatial_dict dict Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]].
None filter_regex_dict dict Keys are column names, values are regex strings.
None select_columns list of str Columns to select.
None offset int Offset in query result.
None limit int Maximum results to return (server will set upper limit, see get_server_config).
None datastack_name str Datastack to query. If None, defaults to one specified in client.
None split_positions bool Whether to break position columns into x,y,z columns. If False data is returned as one column with [x,y,z] array (slower).
False post_filter bool Whether to filter down the result based upon the filters specified. If False, it will return the query with present root_ids in the root_id columns, but the rows will reflect the filters translated into their past IDs. So if, for example, a cell had a false merger split off since the last materialization, those annotations on that incorrect portion of the cell will be included if this is False, but will be filtered down if this is True.
True metadata bool Toggle to return metadata. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.
True merge_reference bool Toggle to automatically join reference table. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that table.
True desired_resolution Iterable Desired resolution you want all spatial points returned in. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata.
None random_sample int If given, will do a tablesample of the table to return that many annotations.
None log_warning bool Whether to log warnings.
True Returns:
Type DescriptionDataFrame A pandas dataframe of results of query
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.lookup_supervoxel_ids","title":"lookup_supervoxel_ids(table_name, annotation_ids=None, datastack_name=None)","text":"Trigger supervoxel lookups of new annotations in a table.
Parameters:
Name Type Description Defaulttable_name str Table to trigger
requiredannotation_ids list List of annotation ids to lookup. Default is None, which will trigger lookup of entire table.
None datastack_name str Datastack to trigger it. Defaults to what is set in client.
None Returns:
Type Descriptiondict Status code of response from server
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.map_filters","title":"map_filters(filters, timestamp, timestamp_past)","text":"Translate a list of filter dictionaries from a point in the future to a point in the past
Parameters:
Name Type Description Defaultfilters list[dict] filter dictionaries with root_ids
requiredtimestamp datetime timestamp to query
requiredtimestamp_past datetime timestamp to query from
requiredReturns:
Type Descriptionlist[dict] filter dictionaries with past root_ids
dict mapping of future root_ids to past root_ids
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.most_recent_version","title":"most_recent_version(datastack_name=None)","text":"Get the most recent version of materialization for this datastack name
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the datastack, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client
None Returns:
Type Descriptionint Most recent version of materialization for this datastack name
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.query_table","title":"query_table(table, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, timestamp=None, metadata=True, merge_reference=True, desired_resolution=None, get_counts=False, random_sample=None, log_warning=True)","text":"Generic query on materialization tables
Parameters:
Name Type Description Defaulttable str Table to query
requiredfilter_in_dict dict Keys are column names, values are allowed entries, by default None
None filter_out_dict dict Keys are column names, values are not allowed entries, by default None
None filter_equal_dict dict Keys are column names, values are specified entry, by default None
None filter_greater_dict dict Keys are column names, values are exclusive upper-bound value, by default None
None filter_less_dict dict Keys are column names, values are exclusive lower-bound value, by default None
None filter_greater_equal_dict dict Keys are column names, values are inclusive upper-bound value, by default None
None filter_less_equal_dict dict Keys are column names, values are inclusive lower-bound value, by default None
None filter_spatial_dict dict Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]], by default None
None filter_regex_dict dict Keys are column names, values are regex strings, by default None
None select_columns list of str Columns to select, by default None
None offset int Result offset to use, by default None. Will only return top K results.
None limit int Maximum results to return (server will set upper limit, see get_server_config), by default None
None datastack_name str Datastack to query, by default None. If None, defaults to one specified in client.
None return_df bool Whether to return as a dataframe, by default True. If False, data is returned as json (slower).
True split_positions bool Whether to break position columns into x,y,z columns, by default False. If False data is returned as one column with [x,y,z] array (slower)
False materialization_version int The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None timestamp datetime Timestamp to query, by default None. If passsed will do a live query. Error if also passing a materialization version
None metadata bool Toggle to return metadata (default True), by default True. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.
True merge_reference bool Toggle to automatically join reference table, by default True. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that
True desired_resolution Iterable[float] Desired resolution you want all spatial points returned in, by default None. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata
None get_counts bool Whether to get counts of the query, by default False
False random_sample int If given, will do a tablesample of the of the table to return that many annotations
None log_warning bool Whether to log warnings, by default True
True Returns:
Type DescriptionDataFrame A pandas dataframe of results of query
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.query_view","title":"query_view(view_name, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, metadata=True, merge_reference=True, desired_resolution=None, get_counts=False, random_sample=None)","text":"Generic query on a view
Parameters:
Name Type Description Defaultview_name str View to query
requiredfilter_in_dict dict Keys are column names, values are allowed entries, by default None
None filter_out_dict dict Keys are column names, values are not allowed entries, by default None
None filter_equal_dict dict Keys are column names, values are specified entry, by default None
None filter_greater_dict dict Keys are column names, values are exclusive upper-bound, by default None
None filter_less_dict dict Keys are column names, values are exclusive lower-bound, by default None
None filter_greater_equal_dict dict Keys are column names, values are inclusive upper-bound, by default None
None filter_less_equal_dict dict Keys are column names, values are inclusive lower-bound, by default None
None filter_spatial_dict dict Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]], by default None
None filter_regex_dict dict Keys are column names, values are regex strings, by default None
None select_columns list of str Columns to select, by default None
None offset int Result offset to use, by default None. Will only return top K results.
None limit int Maximum results to return (server will set upper limit, see get_server_config), by default None
None datastack_name str Datastack to query, by default None. If None, defaults to one specified in client.
None return_df bool Whether to return as a dataframe, by default True. If False, data is returned as json (slower).
True split_positions bool Whether to break position columns into x,y,z columns, by default False. If False data is returned as one column with [x,y,z] array (slower)
False materialization_version int The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None metadata bool Toggle to return metadata (default True), by default True. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.
True merge_reference bool Toggle to automatically join reference table, by default True. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that
True desired_resolution Iterable[float] Desired resolution you want all spatial points returned in, by default None. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata
None get_counts bool Whether to get counts of the query, by default False
False random_sample int If given, will do a tablesample of the of the table to return that many annotations
None Returns:
Type DescriptionDataFrame A pandas dataframe of results of query
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.synapse_query","title":"synapse_query(pre_ids=None, post_ids=None, bounding_box=None, bounding_box_column='post_pt_position', timestamp=None, remove_autapses=True, include_zeros=True, limit=None, offset=None, split_positions=False, desired_resolution=None, materialization_version=None, synapse_table=None, datastack_name=None, metadata=True)","text":"Convenience method for querying synapses.
Will use the synapse table specified in the info service by default. It will also remove autapses by default. NOTE: This is not designed to allow querying of the entire synapse table. A query with no filters will return only a limited number of rows (configured by the server) and will do so in a non-deterministic fashion. Please contact your dataset administrator if you want access to the entire table.
Parameters:
Name Type Description Defaultpre_ids Union[int, Iterable, ndarray] Pre-synaptic cell(s) to query.
None post_ids Union[int, Iterable, ndarray] Post-synaptic cell(s) to query.
None bounding_box Optional[Union[list, ndarray]] [[min_x, min_y, min_z],[max_x, max_y, max_z]] bounding box to filter synapse locations. Expressed in units of the voxel_resolution of this dataset.
None bounding_box_column str Which synapse location column to filter by.
'post_pt_position' timestamp datetime Timestamp to query. If passed recalculate query at timestamp, do not pass with materialization_version.
None remove_autapses bool Whether to remove autapses from query results.
True include_zeros bool Whether to include synapses to/from id=0 (out of segmentation).
True limit int Number of synapses to limit. Server-side limit still applies.
None offset int Number of synapses to offset query.
None split_positions bool Whether to split positions into separate columns, True is faster.
False desired_resolution Iterable[float] List or array of the desired resolution you want queries returned in useful for materialization queries.
None materialization_version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None metadata bool Whether to attach metadata to dataframe in the df.attr dictionary.
True Returns:
Type DescriptionDataFrame Results of query.
"},{"location":"api/schema/","title":"client.schema","text":"Client for interacting with the schema service.
Methods:
Name Descriptionget_schemas Get the available schema types
schema_definition Get the definition of a specified schema_type
schema_definition_all Get the definition of all schema_types
schema_definition_multi Get the definition of multiple schema_types
Attributes:
Name Type Descriptionserver_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.get_schemas","title":"get_schemas()","text":"Get the available schema types
Returns:
Type Descriptionlist List of schema types available on the Schema service.
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition","title":"schema_definition(schema_type)","text":"Get the definition of a specified schema_type
Parameters:
Name Type Description Defaultschema_type str Name of a schema_type
requiredReturns:
Type Descriptionjson Schema definition
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition_all","title":"schema_definition_all()","text":"Get the definition of all schema_types
Returns:
Type Descriptiondict Dictionary of schema definitions. Keys are schema names, values are definitions.
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition_multi","title":"schema_definition_multi(schema_types)","text":"Get the definition of multiple schema_types
Parameters:
Name Type Description Defaultschema_types list List of schema names
requiredReturns:
Type Descriptiondict Dictionary of schema definitions. Keys are schema names, values are definitions.
"},{"location":"api/skeleton/","title":"client.skeleton","text":"Client for interacting with the skeleton service.
Methods:
Name Descriptionget_skeleton Gets basic skeleton information for a datastack
get_cache_contents Mirror CloudFiles.list() for skeletons as a pass-through interface to the underlying service and bucket.
skeletons_exist Confirm or deny that a set of root ids have H5 skeletons in the cache.
get_bulk_skeletons Generates skeletons for a list of root ids without retrieving them.
generate_bulk_skeletons_async Generates skeletons for a list of root ids without retrieving them.
Attributes:
Name Type Descriptionserver_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.get_skeleton","title":"get_skeleton(root_id, datastack_name=None, skeleton_version=3, output_format='dict', log_warning=True, verbose_level=0)","text":"Gets basic skeleton information for a datastack
Parameters:
Name Type Description Defaultroot_id int The root id of the skeleton to retrieve
requireddatastack_name str The name of the datastack to check
None skeleton_version int The skeleton version to generate and retrieve. Options are documented in SkeletonService. Use 0 for Neuroglancer-compatibility. Use -1 for latest.
3 output_format string The format to retrieve. Options are: - 'dict': A dictionary - 'swc': A pandas DataFrame
'dict' Returns:
Type DescriptionSkeleton of the requested type. See output_format for details.
get_cache_contents(datastack_name=None, skeleton_version=3, root_id_prefixes=0, limit=0, log_warning=True)","text":"Mirror CloudFiles.list() for skeletons as a pass-through interface to the underlying service and bucket.
"},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.skeletons_exist","title":"skeletons_exist(datastack_name=None, skeleton_version=3, root_ids=0, log_warning=True)","text":"Confirm or deny that a set of root ids have H5 skeletons in the cache.
"},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.get_bulk_skeletons","title":"get_bulk_skeletons(root_ids, datastack_name=None, skeleton_version=3, output_format='dict', generate_missing_skeletons=False, log_warning=True, verbose_level=0)","text":"Generates skeletons for a list of root ids without retrieving them.
Parameters:
Name Type Description Defaultroot_ids List A list of root ids of the skeletons to generate
requireddatastack_name str The name of the datastack to check
None skeleton_version int The skeleton version to generate. Use 0 for Neuroglancer-compatibility. Use -1 for latest.
3"},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.generate_bulk_skeletons_async","title":"generate_bulk_skeletons_async(root_ids, datastack_name=None, skeleton_version=None, log_warning=True, verbose_level=0)","text":"Generates skeletons for a list of root ids without retrieving them.
Parameters:
Name Type Description Defaultroot_ids List A list of root ids of the skeletons to generate
requireddatastack_name str The name of the datastack to check
None skeleton_version int The skeleton version to generate. Use 0 for Neuroglancer-compatibility. Use -1 for latest.
None Returns:
Type Descriptionfloat The estimated time in seconds to generate all skeletons (a comparable message will be output to the console prior to return).
"},{"location":"api/state/","title":"client.state","text":"Client to interface with the JSON state service.
Methods:
Name Descriptionbuild_neuroglancer_url Build a URL for a Neuroglancer deployment that will automatically retrieve specified state.
get_neuroglancer_info Get the info field from a Neuroglancer deployment
get_property_json Download a Neuroglancer JSON state
get_state_json Download a Neuroglancer JSON state
save_state_json_local Save a Neuroglancer JSON state to a JSON file locally.
upload_property_json Upload a Neuroglancer JSON state
upload_state_json Upload a Neuroglancer JSON state
Attributes:
Name Type Descriptionserver_version Optional[Version] The version of the service running on the remote server. Note that this
state_service_endpoint str Endpoint URL for posting JSON state
"},{"location":"api/state/#caveclient.jsonservice.JSONService.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/state/#caveclient.jsonservice.JSONService.state_service_endpoint","title":"state_service_endpoint: str property","text":"Endpoint URL for posting JSON state
"},{"location":"api/state/#caveclient.jsonservice.JSONService.build_neuroglancer_url","title":"build_neuroglancer_url(state_id, ngl_url=None, target_site=None, static_url=False, format_properties=False)","text":"Build a URL for a Neuroglancer deployment that will automatically retrieve specified state. If the datastack is specified, this is prepopulated from the info file field \"viewer_site\". If no ngl_url is specified in either the function or the client, a fallback neuroglancer deployment is used.
Parameters:
Name Type Description Defaultstate_id int State id to retrieve
requiredngl_url str Base url of a neuroglancer deployment. If None, defaults to the value for the datastack or the client. As a fallback, a default deployment is used.
None target_site seunglab or cave - explorer or mainline or None Set this to 'seunglab' for a seunglab deployment, or either 'cave-explorer'/'mainline' for a google main branch deployment. If None, checks the info field of the neuroglancer endpoint to determine which to use. Default is None.
None static_url bool If True, treats \"state_id\" as a static URL directly to the JSON and does not use the state service.
False format_properties bool If True, formats the url as a segment_properties info file
False Returns:
Type Descriptionstr The full URL requested
"},{"location":"api/state/#caveclient.jsonservice.JSONService.get_neuroglancer_info","title":"get_neuroglancer_info(ngl_url=None)","text":"Get the info field from a Neuroglancer deployment
Parameters:
Name Type Description Defaultngl_url str(optional) URL to a Neuroglancer deployment. If None, defaults to the value for the datastack or the client.
None Returns:
Type Descriptiondict JSON-formatted info field from the Neuroglancer deployment
"},{"location":"api/state/#caveclient.jsonservice.JSONService.get_property_json","title":"get_property_json(state_id)","text":"Download a Neuroglancer JSON state
Parameters:
Name Type Description Defaultstate_id int ID of a JSON state uploaded to the state service.
requiredReturns:
Type Descriptiondict JSON specifying a Neuroglancer state.
"},{"location":"api/state/#caveclient.jsonservice.JSONService.get_state_json","title":"get_state_json(state_id)","text":"Download a Neuroglancer JSON state
Parameters:
Name Type Description Defaultstate_id int or string ID of a JSON state uploaded to the state service. If a string, treats this as the url to the json state.
requiredReturns:
Type Descriptiondict JSON specifying a Neuroglancer state.
"},{"location":"api/state/#caveclient.jsonservice.JSONService.save_state_json_local","title":"save_state_json_local(json_state, filename, overwrite=False)","text":"Save a Neuroglancer JSON state to a JSON file locally.
Parameters:
Name Type Description Defaultjson_state dict Dict representation of a neuroglancer state
requiredfilename str Filename to save the state to
requiredoverwrite bool Whether to overwrite the file if it exists. Default False.
False Returns:
Type DescriptionNone"},{"location":"api/state/#caveclient.jsonservice.JSONService.upload_property_json","title":"upload_property_json(property_json, state_id=None, max_size=2500000)","text":"Upload a Neuroglancer JSON state
Parameters:
Name Type Description Defaultpropery_json dict Dict representation of a neuroglancer segment properties json
requiredstate_id int ID of a JSON state uploaded to the state service. Using a state_id is an admin feature.
None max_size Optional[int] Maximum size in bytes for the data to upload. Default is 2.5MB. Set to None for no limit.
2500000 Returns:
Type Descriptionint state_id of the uploaded JSON state
"},{"location":"api/state/#caveclient.jsonservice.JSONService.upload_state_json","title":"upload_state_json(json_state, state_id=None, timestamp=None)","text":"Upload a Neuroglancer JSON state
Parameters:
Name Type Description Defaultjson_state dict Dict representation of a neuroglancer state
requiredstate_id int ID of a JSON state uploaded to the state service. Using a state_id is an admin feature.
None timestamp Optional[time] Timestamp for json state date. Requires state_id.
None Returns:
Type Descriptionint state_id of the uploaded JSON state
"},{"location":"api/testing/","title":"caveclient.tools.testing","text":"Functions:
Name DescriptionCAVEclientMock Created a mocked CAVEclient function for testing using the responses library to mock
get_materialiation_info Get the materialization versions and version metadata for the materialization service.
get_server_versions Get the server versions for the services used in testing.
get_server_information Generate the datastack name and server locations used in testing.
get_api_versions Get the API versions for the services used in testing.
default_info Generate a info service info file for testing
"},{"location":"api/testing/#caveclient.tools.testing.CAVEclientMock","title":"CAVEclientMock(datastack_name=None, global_server=None, local_server=None, info_file=None, chunkedgraph=False, chunkedgraph_server_version=DEFAULT_CHUNKEDGRAPH_SERVER_VERSION, chunkedgraph_api_versions=None, materialization=False, materialization_server_version=DEFAULT_MATERIALIZATION_SERVER_VERSON, materialization_api_versions=None, available_materialization_versions=None, set_version=None, set_version_metadata=None, json_service=False, json_service_server_version=DEFAULT_JSON_SERVICE_SERVER_VERSION, skeleton_service=False, skeleton_service_server_version=DEFAULT_SKELETON_SERVICE_SERVER_VERSION, schema_api_versions=None, l2cache=False, l2cache_disabled=False, global_only=False)","text":"Created a mocked CAVEclient function for testing using the responses library to mock the server responses. This function returns a drop-in replacement for the CAVEclient function that will be able to initialize itself and selected individual service clients with the selected options.
Note that the test configuration is intended to be purely for pytest purposes and should not actually result in calls to active endpoints.
Parameters:
Name Type Description Defaultdatastack_name str Name of the test datastack, by default None
None global_server str Test global server address, by default None
None local_server str Test local server address, by default None
None info_file dictionary Info service dictionary for the datastack, by default None
None chunkedgraph bool If True, configures the client to initialize a chunkedgraph subclient, by default False
False chunkedgraph_server_version str Sets the value of the chunkedgraph server version as a three-element semanatic version (e.g \"2.3.4\"), by default the value in DEFAULT_CHUNKEDGRAPH_SERVER_VERSION.
DEFAULT_CHUNKEDGRAPH_SERVER_VERSION chunkedgraph_api_versions list List of chunkedgraph API versions that the chunkedgraph client thinks exists, by default None. If None, returns the value in CHUNKEDGRAPH_API_VERSIONS.
None materialization bool If True, configures the client to initalize a materialization subclient, by default False Note that materialization being set to True will also configure the chunkedgraph client.
False materialization_server_version str Sets the value of the materialization server version as a three-element semanatic version (e.g \"2.3.4\"), by default the value in DEFAULT_MATERIALIZATION_SERVER_VERSON.
DEFAULT_MATERIALIZATION_SERVER_VERSON available_materialization_versions list List of materialization database versions that the materialization client thinks exists, by default None. If None, returns the value in DEFAULT_MATERIALIZATION_VERSONS.
None materialization_api_versions list List of materialization API versions that the materialization client thinks exists, by default None. If None, returns the value in MATERIALIZATION_API_VERSIONS.
None set_version Optional[int] If set, will set the version of the materialization server to the value of set_version, by default None. To work, this version must be in the list of available materialization versions.
None set_version_metadata Optional[dict] If set, will set the version metadata of the materialization server to the value of set_version_metadata. Default value is in DEFAULT_MATERIALIZATION_VERSION_METADATA.
None json_service bool If True, configures the client to initalize a materialization subclient, by default False
False json_service_server_version _type_ Sets the value of the json state server version as a three-element semanatic version (e.g \"2.3.4\"), by default the value in DEFAULT_JSON_SERVICE_SERVER_VERSION.
DEFAULT_JSON_SERVICE_SERVER_VERSION skeleton_service bool If True, configures the client to initalize a skeleton service subclient, by default False
False skeleton_service_server_version _type_ Sets the value of the skeleton service version as a three-element semanatic version (e.g \"2.3.4\"), by default the value in DEFAULT_SKELETON_SERVICE_SERVER_VERSION.
DEFAULT_SKELETON_SERVICE_SERVER_VERSION l2cache bool If True, configures the client to initialize an l2cache subclient, by default False Note that l2cache being set to True will also configure the chunkedgraph client.
False l2cache_disabled bool If True, allows a subclient to be initialized, but emulates a situation without an L2 cache, by default False Only used if l2cache is True.
False global_only bool If True, only initializes the global services and does not use a datastack, by default False.
False Returns:
Type DescriptionCAVEclient A mocked and initialized CAVEclient object for testing
Examples:
To make a basic pytest fixture to test chunkedgraph features with an initialized CAVEclient object in your pytest conftest.py file:
import pytest\nfrom caveclient.tools.testing import CAVEclientMock\n\ntest_datastack = \"test_stack\"\ntest_global_server = \"https://test.cave.com\"\ntest_local_server = \"https://local.cave.com\"\n\n@pytest.fixture()\ndef test_client():\n return CAVEclientMock(\n datastack_name=test_datastack,\n global_server=test_global_server,\n local_server=test_local_server,\n chunkedgraph=True,\n )\n\nYou can also create more complex fixtures with multiple services initialized and specific server versions:\n\n```python\n@pytest.fixture()\ndef fancier_test_client():\n return CAVEclientMock(\n datastack_name=test_datastack,\n global_server=test_global_server,\n local_server=test_local_server,\n chunkedgraph=True,\n chunkedgraph_server_version=\"3.0.2\",\n materialization=True,\n materialization_server_version=\"4.21.4\",\n l2cache=True,\n )\nget_materialiation_info(materialization_versions=DEFAULT_MATERIALIZATION_VERSONS, version_metadata=DEFAULT_MATERIALIZATION_VERSION_METADATA)","text":"Get the materialization versions and version metadata for the materialization service.
Parameters:
Name Type Description Defaultmaterialization_versions list List of materialization database versions that the materialization client thinks exists, by default DEFAULT_MATERIALIZATION_VERSONS.
DEFAULT_MATERIALIZATION_VERSONS version_metadata dict Version metadata for the materialization service, by default DEFAULT_MATERIALIZATION_VERSION_METADATA.
DEFAULT_MATERIALIZATION_VERSION_METADATA Returns:
Type Descriptiondict Dictionary with keys: \"materialization_versions\", \"version_metadata\".
"},{"location":"api/testing/#caveclient.tools.testing.get_server_versions","title":"get_server_versions(chunkedgraph_version=DEFAULT_CHUNKEDGRAPH_SERVER_VERSION, materialization_version=DEFAULT_MATERIALIZATION_SERVER_VERSON, skeleton_service_version=DEFAULT_SKELETON_SERVICE_SERVER_VERSION, json_service_version=DEFAULT_JSON_SERVICE_SERVER_VERSION)","text":"Get the server versions for the services used in testing.
Parameters:
Name Type Description Defaultchunkedgraph_version str Version of the chunkedgraph server, by default DEFAULT_CHUNKEDGRAPH_SERVER_VERSION.
DEFAULT_CHUNKEDGRAPH_SERVER_VERSION materialization_version str Version of the materialization server, by default DEFAULT_MATERIALIZATION_SERVER_VERSON.
DEFAULT_MATERIALIZATION_SERVER_VERSON skeleton_service_version str Version of the skeleton service server, by default DEFAULT_SKELETON_SERVICE_SERVER_VERSION.
DEFAULT_SKELETON_SERVICE_SERVER_VERSION json_service_version str Version of the json service server, by default DEFAULT_JSON_SERVICE_SERVER_VERSION.
DEFAULT_JSON_SERVICE_SERVER_VERSION Returns:
Type Descriptiondict Dictionary with keys: \"chunkedgraph_version\", \"materialization_version\", \"skeleton_service_version\", \"json_service_version\". Values are Version objects from packaging.versions.
"},{"location":"api/testing/#caveclient.tools.testing.get_server_information","title":"get_server_information(datastack_name=TEST_DATASTACK, global_server=TEST_GLOBAL_SERVER, local_server=TEST_LOCAL_SERVER)","text":"Generate the datastack name and server locations used in testing.
Parameters:
Name Type Description Defaultdatastack_name str Datastack value, by default the value in TEST_DATASTACK.
TEST_DATASTACK global_server str Server for global services, by default TEST_GLOBAL_SERVER.
TEST_GLOBAL_SERVER local_server str Server for local services, by default TEST_LOCAL_SERVER.
TEST_LOCAL_SERVER Returns:
Type Descriptiondict Dictionary with keys: \"datastack_name\", \"local_server\", \"global_server\".
"},{"location":"api/testing/#caveclient.tools.testing.get_api_versions","title":"get_api_versions(chunkedgraph_api_versions=CHUNKEDGRAPH_API_VERSIONS, materialization_api_versions=MATERIALIZATION_API_VERSIONS, schema_api_versions=SCHEMA_API_VERSIONS)","text":"Get the API versions for the services used in testing.
Parameters:
Name Type Description Defaultchunkedgraph_api_versions list List of chunkedgraph API versions that the chunkedgraph client thinks exists, by default CHUNKEDGRAPH_API_VERSIONS.
CHUNKEDGRAPH_API_VERSIONS materialization_api_versions list List of materialization API versions that the materialization client thinks exists, by default MATERIALIZATION_API_VERSIONS.
MATERIALIZATION_API_VERSIONS schema_api_versions list List of schema API versions that the schema client thinks exists, by default SCHEMA_API_VERSIONS.
SCHEMA_API_VERSIONS Returns:
Type Descriptiondict Dictionary with keys: \"chunkedgraph_api_versions\", \"materialization_api_versions\", \"schema_api_versions\".
"},{"location":"api/testing/#caveclient.tools.testing.default_info","title":"default_info(local_server=TEST_LOCAL_SERVER)","text":"Generate a info service info file for testing
Parameters:
Name Type Description Defaultlocal_server str Name of the local service, by default the value in TEST_LOCAL_SERVER.
TEST_LOCAL_SERVER Returns:
Type Descriptiondict Info file for the datastack.
"},{"location":"tutorials/","title":"Getting Started","text":"AnnotationFramework client is a package for simplifying interactions with HTML services associated with the CAVE (Connectome Annotation Versioning Engine).
For a larger introduction to CAVE and its services, see the main GitHub organization site: https://github.com/CAVEconnectome
"},{"location":"tutorials/#installation","title":"Installation","text":"The CAVEclient can be installed with pip:
$ pip install caveclient\nThe code is setup to work flexibly with any deployment of these services, but you need to specify the server_address if that address is not https://globalv1.daf-apis.com/ for each client when initializing it. Similarly, the clients can query the info service for metadata to simplify the interaction with a datastack, but you have to specify a datastack name.
"},{"location":"tutorials/advanced/","title":"Advanced Usage","text":""},{"location":"tutorials/advanced/#changing-session-configurations","title":"Changing session configurations","text":"It is possible to change details of how a client talks to the various servers it needs to interface with. For instance, the CAVEclient will attempt to retry specific failed requests to the server, but will only try a specific number of times, and will only wait specific amounts of time between retries. These values can be changed via the set_session_defaults method. For instance, to change the number of retries to 5, and to increase the delay between subsequent retries, you could do:
from caveclient import set_session_defaults\n\nset_session_defaults(max_retries=5, backoff_factor=0.5)\nNote that this needs to happen before initializing the client for this to work properly. Some of these parameters are also adjustable at the client level.
To view the current session defaults, you can use the get_session_defaults method:
from caveclient import get_session_defaults\n\nclient.get_session_defaults()\nMore information on the available parameters can be found in the API documentation.
"},{"location":"tutorials/advanced/#writing-unit-tests-using-caveclient","title":"Writing unit tests using caveclient","text":"If you have a module that relies on caveclient and you want to write unit tests for it, you can use the caveclient.testing module to help you create an initialized, mocked client with user-specified options. This helps deal with the complex conversation that caveclient has with various services when initializing itself so that you get the appropriate functionality. The function CAVEclientMock offers a number of options for how to build a client, including server versions for various services, available API versions, and more. Importantly, the module provides sensible defaults but also allows you to override them as needed. Note that using testing requires the responses to be installed, which is not a dependency of caveclient.
The CAVEclientMock function requires you to specify which services you expect to call on in order to ensure that your code is working as expected. For example, to create a pytest fixture that would utilize the chunkedgraph module, but otherwise use default nonsense test values, you could include this in your conftest.py:
import pytest\nfrom caveclient.tools.testing import CAVEclientMock\n\n@pytest.fixture()\ndef test_client():\n return CAVEclientMock(\n chunkedgraph=True,\n )\nThen, in your test module, you can use the test_client fixture to get a client. Note that after mocked initialization, the client will attempt normal network requests and thus you should mock the responses. If you only care to get a specific value back from a given function, you can use pytest-mock to mock the response to a given function call. For example, if you have a function do_something_with_roots that takes a caveclient that uses the get_roots function, you could mock the get_roots function so that it return a specific value:
from pytest_mock import mocker\nfrom conftest import test_client\nfrom my_module import do_something_with_roots\n\ndef test_get_roots(mocker, test_client):\n mocker.patch.object(test_client.chunkedgraph, 'get_roots', return_value=[1, 2, 3])\n test_data = # Something appropriate\n test_output = do_something_with_roots(test_client, test_data)\n assert test_output == # The expected answer\nNote that if you your own datastack info_file that has a different local_server address than the default value (TEST_LOCAL_SERVER variable, which defaults to https://local.cave.com) you will also need to specify a local_server with the same value as in the local_server field of your info file.
While sensible defaults are provided, you can also specify things like server versions to make sure your code works with the versions of the services you expect. For example, let's make a richer mock client that specifies the server versions for the chunkedgraph, materailization, and l2cache services:
@pytest.fixture()\ndef version_specified_client():\n return CAVEclientMock(\n chunkedgraph=True,\n chunkedgraph_server_version='3.0.1',\n materialization=True,\n materialization_server_version='2.0.0',\n l2cache=True,\n )\nNote that some services like l2cache do not currently use a server version to offer different functionality, and this value is not exposed for them currently. See the API documentation for more information.
You can also override default values like server_address or datastack_name:
@pytest.fixture()\ndef server_specified_client():\n return CAVEclientMock(\n datastack_name='my_datastack',\n server_address='http://my.server.com',\n chunkedgraph=True,\n materialization=True,\n )\nIf you want to get access to the various default values of server version, datastack name, datastack info file, and api versions, you can use functions get_server_versions, get_server_information, default_info, get_api_versions respectively. Each of these functions will return a dictionary of the values that can be used as a kwargs input into CAVEclientMock. If you specify your own override values, it will take precedence over the default value and you can just use the dictionary in your tests. See the caveclient tests for an example of how to use these functions.
The AnnotationClient is used to interact with the AnnotationEngine service to create tables from existing schema, upload new data, and download existing annotations. Note that annotations in the AnnotationEngine are not linked to any particular segmentation, and thus do not include any root ids. An annotation client is accessed with client.annotation.
A list of the existing tables for the datastack can be found with get_tables().
all_tables = client.annotation.get_tables()\nall_tables[0]\nEach table has three main properties that can be useful to know:
table_name : The table name, used to refer to it when uploading or downloading annotations. This is also passed through to the table in the Materialized database.schema_name : The name of the table's schema from EMAnnotationSchemas (see below).max_annotation_id : An upper limit on the number of annotations already contained in the table.You can download the JSON representation of a data point through the get_annotation() method. This can be useful if you need to look up information on unmaterialized data, or to see what a properly templated annotation looks like.
table_name = all_tables[0]['table_name'] # 'ais_analysis_soma'\nannotation_id = 100\nclient.annotation.get_annotation(annotation_ids=annotation_id, table_name=table_name)\nOne can create a new table with a specified schema with the create_table() method:
client.annotation.create_table(table_name='test_table',\n schema_name='microns_func_coreg',\n voxel_resolution = [1,1,1],\n description=\"some text to describe your table\")\nThe voxel resolution is the units your position columns will be uploaded in [1,1,1] would imply a nm location, where as [4,4,40] would correspond to voxels of that size. If you are uploading points from a neuroglancer session, you want this to match the units of that neuroglancer view.
Note there are some optional metadata parameters to create_table()
notice_text : This is text that will show up to users who access this data as a warning. This could be used to warn users that the data is not complete or checked yet, or to advertise that a particular publication should be cited when using this table.read_permission : one of \"PRIVATE\" which means only you can read data in this table. \"PUBLIC\" (default) which means anyone can read this table that has read permissions to this dataset. So if and only if you can read the segmentation results of this data, you can read this table. \"GROUP\" which means that you must share a common group with this user for them to be able to read. We need to make a way to discover what groups you are in and who you share groups with.write_permission: one of \"PRIVATE\" (default), which means only you can write to this table. \"PUBLIC\" which means anyone can write to this table that has write permissions to this dataset. Note although this means anyone can add data, no annotations are ever truly overwritten. \"GROUP\" which means that you must share a common group with this user for them to be able to write. We need to make a way to discover what groups you are in and who you share groups with.If you change your mind about what you want for metadata, some but not all fields can be updated with update_metadata(). This includes the description, the notice_text, and the permissions, but not the name, schema or voxel resolution.
# to update description\nclient.annotation.update_metadata(table_name='test_table',\n description=\"a new description for my table\")\n\n# to make your table readable by anybody who can read this dataset\nclient.annotation.update_metadata(table_name='test_table',\n notice_text=\"This table isn't done yet, don't trust it. Contact me\")\n\n# to make your table readable by anybody who can read this dataset\nclient.annotation.update_metadata(table_name='test_table',\n read_permisison=\"PUBLIC\")\nNew data can be generated as a dict or list of dicts following the schema and uploaded with post_annotation. For example, a microns_func_coreg point needs to have:
type set to microns_func_coregpt set to a dict with position as a key and the xyz location as a value.func_id set to an integer.The following could would create a new annotation and then upload it to the service. Note that you get back the annotation id(s) of what you uploaded.
new_data = {'type': 'microns_func_coreg',\n 'pt': {'position': [1,2,3]},\n 'func_id': 0}\nclient.annotation.post_annotation(table_name='test_table', data=[new_data])\nThere are methods to simplify annotation uploads if you have a pandas dataframe whose structure mirrors the struction of the annotation schema you want to upload
import pandas as pd\n\ndf = pd.DataFrame([{'id':0,\n 'type': 'microns_func_coreg',\n 'pt_position': [1,2,3]},\n 'func_id': 0},\n {'id':1,\n 'type': 'microns_func_coreg',\n 'pt_position': [3,2,1]},\n 'func_id': 2}])\nclient.annotation.post_annotation_df('test_table', df)\nNote that here I specified the IDs of my annotations, which you can do, but then its up to you to assure that the IDs don't collide with other IDs. If you leave them blank then the service will assign the IDs for you.
There is a similar method for updating update_annotation_df()
"},{"location":"tutorials/annotation/#staged-annotations","title":"Staged Annotations","text":"Staged annotations help ensure that the annotations you post follow the appropriate schema, both by providing guides to the field names and locally validating against a schema before uploading. The most common use case for staged annotations is to create a StagedAnnotation object for a given table, then add annotations to it individually or as a group, and finally upload to the annotation table.
To get a StagedAnnotation object, you can start with either a table name or a schema name. Here, we'll assume that there's already a table called \"my_table\" that is running a \"cell_type_local\" schema. If we want to add new annotations to the table, we simply use the table name with stage_annotations().
stage = client.annotation.stage_annotations(\"my_table\")\nThis stage object retrieves the schema for the table and hosts a local collection of annotations. Every time you add an annotation, it is immediately validated against the schema. To add an annotation, use the add method:
stage.add(\n cell_type = \"pyramidal_cell\",\n classification_system=\"excitatory\",\n pt_position=[100,100,10],\n)\nThe argument names derive from fields in the schema and you must provide all required fields. Any number of annotations can be added to the stage. A dataframe of annotations can also be added with stage.add_dataframe, and requires an exact match between column names and schema fields. The key difference between this and posting a dataframe directly is that annotations added to a StagedAnnotations are validated locally, allowing any issues to be caught before uploading.
You can see the annotations as a list of dictionary records with stage.annotation_list or as a Pandas dataframe with stage.annotation_dataframe. Finally, if you initialized the stage with a table name, this information is stored in the stage and you can simply upload it from the client.
client.annotation.upload_staged_annotations(stage)\nUpdating annotations requires knowing the annotation id of the annotation you are updating, which is not required in the schema otherwise. In order to stage updated annotations, set the update parameter to True when creating the stage.
update_stage = client.annotation.stage_annotations(\"my_table\", update=True)\nupdate_stage.add(\n id=1,\n cell_type = \"stellate_cell\",\n classification_system=\"excitatory\",\n pt_position=[100,100,10],\n)\nThe update also informs the framework client to treat the annotations as an update and it will use the appropriate methods automatically when uploading client.annotation.upload_staged_annotations.
If you want to specify ids when posting new annotations, id_field can be set to True when creating the StagedAnnotation object. This will enforce an id column but still post the data as new annotations.
If you might be adding spatial data in coordinates that might be different than the resolution for the table, you can also set the annotation_resolution when creating the stage. The stage will convert between the resolution you specify for your own annotations and the resolution that the table expects.
stage = client.annotation.stage_annotations(\"my_table\", annotation_resolution=[8,8,40])\nstage.add(\n cell_type='pyramidal_cell',\n classification_system=\"excitatory\",\n pt_position=[50,50,10],\n)\nAuthentication tokens are generally needed for programmatic access to our services. The AuthClient handles storing and loading your token or tokens and inserting it into requests in other clients.
We can access the auth client from client.auth. Once you have saved a token, you probably won't interact with this client very often, however it has some convenient features for saving new tokens the first time. Let's see if you have a token already. Probably not.
client = CAVEclient()\nauth = client.auth\nprint(f\"My current token is: {auth.token}\")\nTo get a new token, you will need to manually acquire it. For convenience, the function client.auth.get_new_token() provides instructions for how to get and save the token.
By default, the token is saved to ~/.cloudvolume/secrets/cave-secret.json as a string under the key token. This makes it compatible by default with Cloudvolume projects, which can come in handy. The following steps will save a token to the default location.
auth.get_new_token()\nnew_token = 'abcdef1234567890' #This is the text you see after you visit the website.\nauth.save_token(token=new_token)\nprint(f\"My token is now: {auth.token}\")\nNote that requesting a new token will invalidate your previous token on the same project. If you want to use the same token across different computers, you will need to share the same token information.
"},{"location":"tutorials/authentication/#loading-saved-tokens","title":"Loading saved tokens","text":"Try opening ~/.cloudvolume/secrets/cave-secret.json to see what we just created.
If we had wanted to use a different file or a different json key, we could have specified that in auth.save_token.
Because we used the default values, this token is used automatically when we initialize a new CAVEclient. If we wanted to use a different token file, token key, or even directly specify a token we could do so here.
client = CAVEclient(datastack_name)\nprint(f\"Now my basic token is: {client.auth.token}\")\n\nclient_direct = CAVEclient(datastack_name, auth_token='another_fake_token_678')\nprint(f\"A directly specified token is: {client_direct.auth.token}\")\nIf you use a CAVEclient, the AuthClient and its token will be automatically applied to any other services without further use.
"},{"location":"tutorials/chunkedgraph/","title":"Chunked Graph","text":"The chunkedgraph is a dynamic oct-tree connected components supervoxel graph.
A visual representation of an oct-tree (Wikipedia (WhiteTimberwolf) CC BY-SA 3.0)
As with any oct-tree, it is organized in hierarchical levels, with the bottom level 1 corresponding to the supervoxels of the segmentations, and the top level being the unique connected components of the supervoxel graph.
A figure illustrating the spatial chunking and editing of a the chunkedgraph. From Dorkenwald et. al 2021
The ChunkedGraph client allows one to interact with the ChunkedGraph service, which stores and updates the supervoxel agglomeration graph. This is most often useful for looking up an object root id of a supervoxel or looking up supervoxels belonging to a root id. The ChunkedGraph client is at client.chunkedgraph.
Usually in Neuroglancer, one never notices supervoxel ids, but they are important for programmatic work. In order to look up the root id for a location in space, one needs to use the supervoxel segmentation to get the associated supervoxel id. The ChunkedGraph client makes this easy using the client.chunkedgraph.get_root_id() method.
sv_id = 104200755619042523\nclient.chunkedgraph.get_root_id(supervoxel_id=sv_id)\nHowever, as proofreading occurs, the root id that a supervoxel belongs to can change. By default, this function returns the current state, however one can also provide a UTC timestamp to get the root id at a particular moment in history. This can be useful for reproducible analysis. Note below that the root id for the same supervoxel is different than it is now.
import datetime\n\n# I looked up the UTC POSIX timestamp from a day in early 2019.\ntimestamp = datetime.datetime.utcfromtimestamp(1546595253)\n\nsv_id = 104200755619042523\nclient.chunkedgraph.get_root_id(supervoxel_id=sv_id, timestamp=timestamp)\nIf you are doing this across lots of supervoxels (or any nodes) then you can do it more efficiently in one request with client.chunkedgraph.get_roots()
node_ids = [104200755619042523, 104200755619042524,104200755619042525]\nroot_ids = client.chunkedgraph.get_roots(node_ids)\nA root id is associated with a particular agglomeration of supervoxels, which can be found with the client.chunkedgraph.get_leaves() method. A new root id is generated for every new change in the chunkedgraph, so time stamps do not apply.
root_id = 648518346349541252\nclient.chunkedgraph.get_leaves(root_id)\nYou can also query the chunkedgraph not all the way to the bottom, using the stop_layer option
root_id = 648518346349541252\nclient.chunkedgraph.get_leaves(root_id,stop_layer=2)\nThis will get all the level 2 IDs for this root, which correspond to the lowest chunk of the hierarchy. An analogous option exists for client.chunkedgraph.get_roots(). This is useful to help find nodes to query within the l2cache, amongst other things.
There are a variety of other interesting functions to explore in client.chunkedgraph.
"},{"location":"tutorials/framework/","title":"Introduction","text":""},{"location":"tutorials/framework/#caveclient-one-client-for-all-services","title":"CAVEclient: one client for all services","text":"The CAVE Framework consists of a number of different services, each with a specific set of tasks that it can perform through REST endpoints. The CAVEclient is designed to ease programmatic interaction with all of the various endpoints. In addition, most programmatic access requires the use of authentication tokens. In order to collect a given server, datastack name, and user token together into a coherent package that can be used on multiple endpoints, the CAVEclient builds appropriately configured clients for each of the specific services. Each of the individual services has their own specific documentation as well.
"},{"location":"tutorials/framework/#global-and-local-services","title":"Global and Local Services","text":"There are two categories of data in CAVE: Global and local. Local services are associated with a single so-called datastack, which refers to a precise collection of imagery and segmentation data that function together. For example, EM imagery and a specific pychunkedgraph segmentation would be one datastack, while the same EM imagery but an initial static segmentation would be another. Datastacks are referred to by a short name, for instance pinky100_public_flat_v185.
Global services are those that are potentially shared across multiple different specific datastacks. These include the info service, which can describe the properties of all available datastacks, the authentication service, and the state service that hosts neuroglancer states. Global services are associated with a particular URL (by default http://globalv1.daf-apis.com), but not a single datastack.
Assuming that the services are on http://globalv1.daf-apis.com and authentication tokens are either not being used or set up with default values (see Authentication), a simple CAVEclient that can only access global services can be initialized:
from caveclient import CAVEclient\n\nclient = CAVEclient()\nJust to confirm that this works, let's see if we can get the EM image source from the InfoService. If you get a list of names of datastacks, all is good. If you have not yet set up an authentication token or you get an authentication error, look at Getting a new token for information about how to set up your auth token.
client.info.get_datastacks()\nIf you have a specific datastack you want to use, you can initialize your CAVEclient with it. This gives you access to the full range of client functions.
client = CAVEclient(datastack_name='my_datastack')\nIf your data is hosted by a different global server, you specify its address when initializing the client.
client = CAVEclient(datastack_name='my_datastack', server_address='http://global.myserver.com')\nBy default, if you pass both a server address and a datastack, the client will store the mapping from datastack to server address in the same location as the default for authentication tokens. Once stored, the client will automatically use the correct server address for the datastack if none is provided. You can override storing the server address by passing write_server_address=False. Datastacks can be removed from the cache using
caveclient.datastack_lookup.reset_server_address_cache(datastack_name).
"},{"location":"tutorials/framework/#accessing-specific-clients","title":"Accessing specific clients","text":"Each client can be accessed as a property of the main client. See the documentation at left for the capabilities of each. Assuming your client is named client, the subclients for each service are:
client.authclient.annotationclient.chunkedgraphclient.infoclient.schemasclient.stateclient.skeletonserviceA datastack has a number of complex paths to various data sources that together comprise a datastack. Rather than hardcode these paths, the InfoService allows one to query the location of each data source. This is also convenient in case data sources change.
An InfoClient is accessed at client.info.
client = CAVEclient(datastack_name)\nprint(f\"This is an info client for {client.info.datastack_name} on {client.info.server_address}\")\nAll of the information accessible for the datastack can be seen as a dict using get_datastack_info().
info.get_datastack_info()\nIndividual entries can be found as well. Use tab autocomplete to see the various possibilities.
info.graphene_source()\nBecause of the way neuroglancer looks up data versus cloudvolume, sometimes one needs to convert between gs:// style paths to https://storage.googleapis.com/ stype paths. All of the path sources in the info client accept a format_for argument that can handle this, and correctly adapts to graphene vs precomputed data sources.
neuroglancer_style_source = info.image_source(format_for='neuroglancer')\nprint(f\"With gs-style: { neuroglancer_style_source }\")\n\ncloudvolume_style_source = info.image_source(format_for='cloudvolume')\nprint(f\"With https-style: { cloudvolume_style_source }\")\nTo understand the level 2 cache, you must understand the structure of the chunkedgraph so see the chunkedgraph tutorial.
Nodes on the second level or layer of the graph, corresponds to all the supervoxels that are locally connected to one another within a single level 2 spatial \"chunk\" of the data. The Level 2 Cache, is a service whose job it is to track and update relevant statistics about every level 2 node within the a chunkedgraph. The source code of this service can be found here.
"},{"location":"tutorials/l2cache/#finding-level-2-nodes","title":"Finding Level 2 Nodes","text":"The chunkedgraph can be used to find the level2 nodes of a rootID using a stop_layer=2 keyword argument on the client.chunkedgraph.get_leaves(). Conversely the level 2 node of a supervoxel can be found using the same keyword argument of client.chunkedgraph.get_roots(). Note if you don't specify a timestamp it will give you the level2 node that is presently associated with the object.
The statistics that are available are:
Level 2 stats about nodes can be retreived using the client.l2cache.get_l2data() method. It simply takes a list of level 2 nodes you want to retrieve. Optionally you can specify only the attributes that you are interested in retrieving which will speed up the request.
"},{"location":"tutorials/l2cache/#missing-data","title":"Missing Data","text":"The service is constantly watching for changes made to objects and recalculating stats on new level2 nodes that are created, in order to keep its database of statistics current. This however takes some time, and is subject to sporadic rare failures. If you request stats on a level 2 node which are not in the database, you will receive an empty dictionary for that node. This will immediately trigger the system to recalculate the statistics of that missing data, and so it should be available shortly (on the order of seconds) if systems are operational. Please note that PCA is not calculated for very small objects because it is not meaningful. So if you are interested in differentiating whether PCA is not available because it hasn't been calculated, vs when its not available because it is not possible to calculate, you should ask for at least one other non PCA statistic as well. You will see that its availability correlates strongly with size_nm3.
Say you want to calculate the total surface area and volume of a object in the dataset. The areas and volume of each component can simply be added together to do this.
import pandas as pd\nroot_id = 648518346349541252\nlvl2nodes = client.chunkedgraph.get_leaves(root_id,stop_layer=2)\nl2stats = client.l2cache.get_l2data(lvl2nodes, attributes=['size_nm3','area_nm2'])\nl2df = pd.DataFrame(l2stats).T\ntotal_area_um2=l2df.area_nm2.sum()/(1000*1000)\ntotal_volume_um3 = l2df.size_nm3.sum()/(1000*1000*1000)\nBy utilizing the bounds argument of get_leaves, you can also do simple spatially restricted analysis of objects. In fact, because you have data on each level2 node individually, you can segregate the neuron using any labelling of its topology.
"},{"location":"tutorials/l2cache/#skeletonization","title":"Skeletonization","text":"Level 2 nodes have \"cross chunk\" edges within the chunkedgraph which represent what level 2 nodes that object is locally connected to. This forms a graph between the level 2 nodes of the object that can be retrieved using the chunkedgraph function client.chunkedgraph. This graph represents a topological representation of the neuron at the resolution of individual chunks, and is guaranteed to be fully connected, unlike a voxel or mesh representation of the neuron which can have gaps where there are defects in the segmentation volume or incorrectly inferred edges at self contact locations.
The level 2 graph can be turned into a skeleton representation of a neuron using a graph based TEASAR like algorithm as described for skeletonizing meshes in this MeshParty Documentation. There is an implementation of this approach that utilizes the chunkedgraph and the L2cache if available here and on pypi as pcg-skel. In this implementation the l2cache is used to more accurately place the level 2 nodes in space using the rep_coord_nm value.
Note that there is detailed documentation on the Skeleton Client interface at the skeletonization tutorial.
"},{"location":"tutorials/l2cache/#trajectory-distributions","title":"Trajectory Distributions","text":"If one is interested in the bulk direction of processes in a region of the brain, one can start with supervoxels in a region, find level 2 nodes that correspond to them, filter out components based on size, (or other criteria such as whether they are part of objects that have components in some other brain area) and look at the distribution of PCA components to understand the directions that those processes are moving within that region of space.
"},{"location":"tutorials/materialization/","title":"Materialization","text":"The Materialization client allows one to interact with the materialized annotation tables, that were posted to the annotation service (the annotations tutorial).
To see the entire class visit the API doc.
The service regularly looks up all annotations and the segids underneath all the boundspatialpoints. You can then query these tables to find out the IDs that underlie the annotations, or the annotations that now intersect with certain IDs.
For example, one common pattern is that you have identified a cell based on the location of its cell body, and you have an annotation there.
You want to know what are the inputs onto the cell, so you first query the annotation table with your soma annotation, asking for the current ID underneath that soma. Then you query a synapse table for all synapse annotations that have a post-synaptic ID equal to the ID from your soma annotation.
In this way your code stays the same, as the proofreading changes and you can track the connectivity of your cell over time.
"},{"location":"tutorials/materialization/#initializing-the-client","title":"Initializing the client","text":"By default when you initialize the overall client, it will choose the most recent materialization version available. This may or may not be desirable depending on your use case. If your code involves using specific IDs then you should be using a specific version that is tied to a timepoint where those IDs are valid.
To see what versions are available, use the client.materialize.get_versions() function.
client.materialize.get_versions()\nEach version has a timestamp it was run on as well as a date when it will expire. You can query all this metadata for a specific version using client.materialize.get_version_metadata() or all versions using client.materialize.get_versions_metadata().
To change the default version, alter the .version property of the client. This will change the version for all subsequent calls which expect one, unless you specify a different version in the method call. Note that this also sets the timestamp property of the client to the timestamp of the version for methods which expect a timestamp.
client.version = 9\nYou can also specify the version when you initialize the client, e.g.:
client = CAVEclient('minnie65_public', version=661)\nOr, you can specify the version when making a particular method call.
"},{"location":"tutorials/materialization/#browsing-versions","title":"Browsing versions","text":"To see what tables are available in a version you can use client.materialize.get_tables().
If you want to read about the description of what that table is, use the annotationengine client client.materialize.get_table_metadata().
If you want to read more about the schema for the annotation table use the schema service client.schema.schema_definition().
Note, the materialization service has a human readable webpage that links to the other services that might be more convenient for you to browse, to get a link there in ipython display client.materialize.homepage
for some important tables, the info service has a pointer to which table you should use in the metadata for the datastack. client.info.get_datastack_info()['synapse_table'] and client.info.get_datastack_info()['soma_table'].
To see how many annotations are in a particular table use
nannotations=client.materialize.get_annotation_count('my_table')\nTo query a small table, you can just download the whole thing using client.materialize.query_table() which will return a dataframe of the table.
Note however, some tables, such as the synapse table might be very large 200-300 million rows and the service will only return the first 200,000 results, and not in a deterministic manner. NOTE! This API is not designed to enable enmass downloading of the entire synapse table there are more efficient ways of doing this. Contact your dataset administrator for more information if this is what you are looking to do.
To just get a preview, use the limit argument (but note again that this won't be a reproducible set)
df=client.materialize.query_table('my_table', limit=10)\nFor most applications, you will want to filter the query in some way.
We offer seven kinds of filters you can apply: filter_in_dict filter_out_dict, filter_equal_dict, filter_greater_dict, filter_less_dict, filter_greater_equal_dict, and filter_less_equal_dict. For query_table each is specified as a dictionary where the keys are column names, and the values are a list of values (or single value in the case of filter_equal).
So for example to query a synapse table for all synapses onto a neuron in flywire you would use
synapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n filter_equal_dict = {'post_pt_root_id': MYID})\nThe speed of querying is affected by a number of factors, including the size of the data. To improve the performance of results, you can reduce the number of columns returned using select_columns.
So for example, if you are only interested in the root_ids and locations of pre_synaptic terminals you might limit the query with select_columns. Also, it is convenient to return the with positions as a column of np.array([x,y,z]) coordinates for many purposes. However, sometimes you might prefer to have them split out as separate x, y, z columns. To enable this option use split_columns=True. split_columns=True is faster, as combining them is an extra step.
synapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n filter_equal_dict = {'post_pt_root_id': MYID},\n select_columns=['id','pre_pt_root_id', 'pre_pt_position'],\n split_columns=True)\nOften you want to have position information in different units. For example, to consider synapse locations or soma locations, you might want to have positions in nanometers or microns.
To create neuroglancer views, you might want positions in integer voxels of a size that aligns with the resolution you are used to using Neuroglancer at.
Annotation tables can be created and uploaded in varying resolutions according to whatever the user of the table felt was natural. This information is available in the metadata for that table. In addition, you may pass desired_resolution as a keyword argument which will automatically convert all spatial positions into voxels of that size in nanometers.
So if you want positions in nanometers, you would pass desired_resolution=[1,1,1]. If you want positions in microns you would pass desired_resolution=[1000,1000,1000]. If you want positions in 4,4,40nm voxel coordinates to use with cloud-volume or neuroglancer you would pass desired_resolution=[4,4,40].
You can also filter columns that are associated with spatial locations based upon being within a 3d bounding box.
This is done by adding a filter_spatial_dict argument to query_table. The units of the bounding box should be in the units of the voxel_resolution of the table (which can be obtained from client.materialize.get_table_metadata()).
bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\nsynapse_table = client.info.get_datastack_info('synapse_table')\ndf=client.materialize.query_table(synapse_table,\n filter_equal_dict = {'post_pt_root_id': MYID},\n filter_spatial_dict = {'post_pt_position': bounding_box})\nFor synapses in particular, we have a simplified method for querying them with a reduced syntax. client.materialize.synapse_query() lets you specify pre and post synaptic partners as keyword arguments and bounding boxes. The defaults make reasonable assumptions about what you want to query, namely that the synapse_table is the table that the info service advertises, and that if you specify a bounding box, that you want the post_pt_position. These can be overridden of course, but the above bounding box query is simplified to.
NOTE! This API is not designed to enable enmass downloading of the entire synapse table there are more efficient ways of doing this. Contact your dataset administrator for more information if this is what you are looking to do.
bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\ndf=client.materialize.query_table(post_ids = MYID,\n bounding_box=bounding_box)\nIn order to query the materialized tables above you can only use IDs that were present at the timestamp of the materialization. If you query the tables with an ID that is not valid during the time of the materialization you will get empty results.
To check if root_ids are valid at your materialization's timestamp, you can use client.chunkedgraph.is_latest_roots()
import numpy as np\nmat_time = client.materialize.get_timestamp()\nis_latest = client.chunkedgraph.is_latest_roots([MYID], timestamp=mat_time)\nassert(np.all(is_latest))\nIf you need to lookup what happened to that ID, you can use the chunkedgraph lineage tree, to look into the future or the past, depending on your application you can use client.chunkedgraph.get_lineage_graph().
Again, the ideal situation is that you have an annotation in the database which refers to your objects of interest, and querying that table by the id column will return the object in the most recent materialization.
However, sometimes you might be browsing and proofreadding the data and get an ID that is more recent that the most recent version available. For convenience, you can use client.materialize.live_query().
to automatically update the results of your query to a time in the future, such as now. For example, to pass now, use datetime.datetime.now(datetime.timezone.utc). Note all timestamps are in UTC throughout the codebase.
import datetime\nsynapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.live_query(synapse_table,\n datetime.datetime.now(datetime.timezone.utc),\n filter_equal_dict = {'post_pt_root_id': MYID})\nThis will raise an ValueError exception if the IDs passed in your filters are not valid at the timestamp given
You can also pass a timestamp directly to query_table and it will call live_query automatically.
import datetime\nsynapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n timestamp=datetime.datetime.now(datetime.timezone.utc),\n filter_equal_dict = {'post_pt_root_id': MYID})\nAlso, keep in mind if you run multiple queries and at each time pass datetime.datetime.now(datetime.timezone.utc), there is no guarantee that the IDs will be consistent from query to query, as proofreading might be happening at any time. For larger scale analysis constraining oneself to a materialized version will ensure consistent results.
Versions have varying expiration times in order to support the tradeoff between recency and consistency, so before undertaking an analysis project consider what version you want to query and what your plan will be to update your analysis to future versions.
"},{"location":"tutorials/materialization/#content-aware-interface-experimental","title":"Content-aware Interface (Experimental)","text":"As of version 5.8.0, we have introduced a new interface to query tables and views. This interface might have small but breaking changes in the near future. :::
In order to make the querying interface more consistent across tables, we have introduced an additional alternative interface to filtering and querying data via the client.materialize.tables object. When you instantiate this object, this object finds all of the existing tables and the list of their columns and lets you filter the tables as arguments in the function with suggestions. Moreover, the filtering arguments and the querying arguments are separated into two.
Let's see how this works with a simplest example --- downloading a table called nucleus_detection_v0. First, we reference the table as a function and then we run the query --- this is exactly the same as client.materialize.query_table('nucleus_detection_v0').
client = CAVEclient('minnie65_public')\nnuc_df = client.materialize.tables.nucleus_detection_v0().query()\nWhere things differ is when we add filters. If we want to query based on a set of values for the field \"id\", for example, we add that as an argument:
my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(id=my_ids).query()\nWhere in this example the id= queries the column id based on the schema. These values can be either individual elements (i.e. an integer or a string) or a list/array of elements, and any field can be used. The tooling will automatically sort out how to format the filtering appropriately when running the query. Importantly, the filtering is identical between querying all types of tables and queries. To see the complete list of fields that can be queried, you can tab-autocomplete or in Jupyter or IPython glance at the docstring with client.materialize.tables.nucleus_detection_v0?.
In addition to filtering by one or many values, you can also do spatial queries. To find only annotations within a particular bounding box, you need to specify the spatial point (e.g. pt_position). For each such spatial point, there will be an argument {spatial_point_position}_bbox (e.g. pt_position_bbox) that accepts a 2x3 list of coordinates specifying the upper and lower bounds for the query. For example, to find all nucleus centroids within a particular bounding box, it would be:
bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n pt_position_bbox=bounding_box\n).query()\nIf you are not using any filters, you can omit the parenthesis and use the get_all or get_all_live functions directly, which act similarly to the query and live_query functions respectively. The first example could be rewritten as:
nuc_df = client.materialize.tables.nucleus_detection_v0.get_all()\nIf you want to list all available fields, you can use the .fields attribute. Similarly, you can get all numeric fields with the .numeric_fields attribute and all spatial fields (allowing bounding box queries) with .spatial_fields.
nuc_df = client.materialize.tables.nucleus_detection_v0.spatial_fields\nIf you need to specify the table programmatically, you can also use a dictionary-style approach to getting the table filtering function. For example, an equivalent version of the above line would be:
my_ids = [373879, 111162]\nmy_table = 'nucleus_detection_v0'\nnuc_df = client.materialize.tables[my_table](id=my_ids).query()\nThe query function can also take arguments relating to timestamps or formatting where they act just like in the other query method. In particular, the arguments that apply to query are: select_columns, offset, limit, split_positions, materialization_version, timestamp, metadata, desired_resolution, and get_counts. For example, to add a desired resolution and split positions in the above query, it would look like:
my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n id=my_ids\n).query(\n split_positions=True,\n desired_resolution=[1,1,1],\n)\nInequalities can also be used in filtering numeric columns. Here, you can pass a dictionary instead of a list of values, with the keys being inequality operators (\">\", \">=\", \"<\", and \"<=\") and the values being the comparison. For example, to query for all nuclei with a volume greater than 1000:
client.materialize.tables.nucleus_detection_v0(\n volume={\">\": 1000}\n).query()\nYou can also use multiple inequalities in the same dictionary to filter within a range. For example, to query for all nuclei with a volume between 500 and 750:
client.materialize.tables.nucleus_detection_v0(\n volume={\">\": 500, \"<\": 750}\n).query()\nIf you want to do a live query instead of a materialized query, the filtering remains identical but we use the live_query function instead. The one required argument for live_query is the timestamp.
my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n id=my_ids\n).live_query(\n timestamp=datetime.datetime.now(datetime.timezone.utc),\n)\nThe live query functions have similar but slightly different arguments: timestamp (required), offset, limit, split_positions, metadata, desired_resolution, and allow_missing_lookups.
Note that way that IPython handles docstrings means that while you can use ? to get the docstring of the filtering part of the function, you can't simply do something like client.materialize.tables.nucleus_detection_v0().query?. It will tell you the function can't be found, because technically the query function does not yet exist until the table filtering function is called.
Instead, if you want to glimpse the docstring of the query or live_query functions, you need to split it into two lines:
qry_func = client.materialize.tables.nucleus_detection_v0().query\nqry_func?\nFinally, if the project you are working with has views, a similar interface is available to them via client.materialize.views. Currently views are not compatible with live query, and so only the .query function is available.
The EMAnnotationSchemas client lets one look up the available schemas and how they are defined. This is mostly used for programmatic interactions between services, but can be useful when looking up schema definitions for new tables.
"},{"location":"tutorials/schemas/#get-the-list-of-schema","title":"Get the list of schema","text":"One can get the list of all available schema with the schema method. Currently, new schema have to be generated on the server side, although we aim to have a generic set available to use.
client.schema.get_schemas()\nThe details of each schema can be viewed with the schema_definition method, formatted as per JSONSchema.
example_schema = client.schema.schema_definition('microns_func_coreg')\nexample_schema\nThis is mostly useful for programmatic interaction between services at the moment, but can also be used to inspect the expected form of an annotation by digging into the format.
example_schema['definitions']['FunctionalCoregistration']\nThe skeleton service enables you to generate and retrieve skeletons from the server. This saves you the trouble of performing skeletonization routines on your local machine, while also saving you the trouble of understanding many of the details inside of skeletonization. Furthermore, the server offers the ability to generate multiple skeletons simultaneously, in parallel. And lastly, the server caches each skeleton that is generated so that it only ever needs to be generated once. All subsequent requests for the same skeleton will simply return the cached skeleton with notable turn-around-time improvements.
"},{"location":"tutorials/skeletonization/#initializing-the-client","title":"Initializing the client","text":"The simplest way to initialize the CAVEclient is by merely providing the datastack of interest:
import caveclient as cc\n\nclient = cc.CAVEclient(<datastack_name>)\nWith a CAVEclient built, you can now investigate the current build version of the SkeletonClient:
client.skeleton.get_version()\nAnd you can see a list of available skeleton versions. In most cases you will want to use the highest (most recent) version provided.
client.skeleton.get_versions()\nRetrieve a skeleton using get_skeleton(). The simplest usage is:
sk = client.skeleton.get_skeleton(\n <root_id>,\n output_format=<output_format>,\n)\nwhere the availale output_formats (described below) are:
'dict' (default if unspecified)'swc' (a Pandas Dataframe)If the skeleton doesn't exist in the server cache, it may take 20-60 seconds to generate the skeleton before it is returned. This function will block during that time. Any subsequent retrieval of the same skeleton should go very quickly however.
To specify a nondefault skeleton version:
sk = client.skeleton.get_skeleton(\n <root_id>,\n skeleton_version=<sk_version>,\n output_format=<output_format>,\n)\nTo specify a specific datastack:
sk = client.skeleton.get_skeleton(\n <root_id>,\n datastack_name=<datastack_name>,\n skeleton_version=<sk_version>,\n output_format=<output_format>,\n)\nMost end-users shouldn't need to use the following function very much, but to see the contents of the cache for a given root id, set of root ids, root id prefix, or set of prefixes:
get_cache_contents(\n root_id_prefixes=<root_id>,\n)\nYou can also add additional parameters as needed:
get_cache_contents(\n datastack_name=<datastack_name>,\n skeleton_version=<sk_version>,\n root_id_prefixes=<root_id>,\n)\nThe primary parameter, root_id_prefixes, can be a list of root ids:
get_cache_contents(\n root_id_prefixes=[<root_id>, <root_id>, ...],\n)\nThe primary parameter can also be a root id prefix, which will match any associated root ids. Since this could potentially return a large list of results, there is an optional limit parameter so you don't overwhelm the memory of your processing environment, e.g., a Jupyter notebook or some other Python script running on your local machine:
get_cache_contents(\n root_id_prefixes=<root_id_prefix>,\n limit=<limit>,\n)\nNote that the limit only constraints the size of the return value. The internal operation of the function will still receive the full list when it passes the prefix on to CloudFiles. Consequently, calling this function for a short prefix may block for a long time.
And of course you can also pass in a list of prefixes (or a mixture of full ids and partial prefixes):
get_cache_contents(\n root_id_prefixes=[<root_id_prefix>, <root_id_prefix>, ...],\n limit=<limit>,\n)\nThe function shown above isn't necessarily the most direct way to simply inquire whether a skeleton exists in the cache for a given root id. For that purpose, the following function is better suited:
skeletons_exist(\n root_ids=<root_id>,\n)\nOr:
skeletons_exist(\n root_ids=[<root_id>, <root_id>, ...],\n)\nNote that this function doesn't accept prefixes, as supported by cache_query_contents(). Only full root ides are supported. When querying with as single root id, the return value will be a boolean. When querying with a list of ids, the return value will be a Python dictionary mapping from each id to a boolean.
This function also takes the same optional parameters described above:
skeletons_exist(\n datastack_name=<datastack_name>,\n skeleton_version=<sk_version>,\n root_ids=<root_id>, # Or [<root_id>, <root_id>, ...],\n)\nYou can retrieve a large set of skeletons in a single function call:
get_bulk_skeletons(\n root_ids=[<root_id>, <root_id>, ...],\n)\nIf any skeletons are not generated yet, the default behavior is to skip those root ids and only return skeletons that are already available. But you can override this default behavior:
get_bulk_skeletons(\n root_ids=[<root_id>, <root_id>, ...],\n generate_missing_skeletons=[False|True],\n)\nAny root ids for which skeletonization is required will be generated one at a time, at a cost of 20-60 seconds each. Consequently, there is a hard-coded limit of 10, after which all subsequent missing skeletons will not be returned.
By default, skeletons are returned in JSON format. However SWC is also supported, thusly:
get_bulk_skeletons(\n root_ids=[<root_id>, <root_id>, ...],\n output_format=<\"json\"|\"swc\">\n)\nAnd the usual defaults can be overridden again:
get_bulk_skeletons(\n root_ids=[<root_id>, <root_id>, ...],\n datastack_name=<datastack_name>,\n skeleton_version=<sk_version>,\n)\nget_bulk_skeletons() is not an effective way to produce a large number of skeletons since it operates synchronously, generating one skeleton at a time. In order to generate a large number of skeletons it is better to do so in parallel. The following function dispatches numerous root ids for skeletonization without returning anything immediately. The root ids are then distributed on the server for parallel skeletonization and eventual caching. Once they are in the cache, you can retrieve them. Of course, it can be tricky to know when they are available. That is addressed further below. Here's how to dispatch asynchronous bulk skeletonization:
generate_bulk_skeletons_async(\n root_ids=[<root_id>, <root_id>, ...],\n)\nAnd with the usual overrides:
get_bulk_skeletons(\n root_ids=[<root_id>, <root_id>, ...],\n datastack_name=<datastack_name>,\n skeleton_version=<sk_version>,\n)\nIn order to retrieve asynchronously generated skeletons, it is necessary to poll the cache for the availability of the skeletons and then eventually retrieve them. Here's an example of such a workflow:
# Dispatch multiple asynchronous, parallel skeletonization and caching processes\ngenerate_bulk_skeletons_async(root_ids)\n\n# Repeatedly query the cache for the existence of the skeletons until they are all available\nwhile True:\n skeletons_that_exist = client.skeleton.skeletons_exist(root_ids=rids)\n num_skeletons_found = sum(skeletons_that_exist.values())\n if num_skeletons_found == len(rids):\n break\n sleep(10) # Pause for ten seconds and check again\n\n# Retrieve the skeletons (remember, SWC is also offered)\nskeletons_json = get_bulk_skeletons(root_ids)\nWe store the JSON description of a Neuroglancer state in a simple database at the JSON Service. This is a convenient way to build states to distribute to people, or pull states to parse work by individuals. The JSON Client is at client.state
client.state\nJSON states are found simply by their ID, which you get when uploading a state. You can download a state with get_state_json.
example_id = 4845531975188480\nexample_state = client.state.get_state_json(test_id)\nexample_state['layers'][0]\nYou can also upload states with upload_state_json. If you do this, the state id is returned by the function. Note that there is no easy way to query what you uploaded later, so be VERY CAREFUL with this state id if you wish to see it again.
Note: If you are working with a Neuroglancer Viewer object or similar, in order to upload, use viewer.state.to_json() to generate this representation.
example_state['layers'][0]['name'] = 'example_name'\nnew_id = client.state.upload_state_json(example_state)\ntest_state = client.state.get_state_json(new_id)\ntest_state['layers'][0]['name']\nOnce you have a state ID, you want to turn it into a well-formatted link. So you don\\'t have to remember all the endpoints, we can do this from the state client.
ngl_base = 'neuromancer-seung-import.appspot.com'\nclient.state.build_neuroglancer_url(new_id, ngl_base)\nNote that the neuroglancer base can be found in the info service under client.info.viewer_site().
CAVE is short for Connectome Annotation Versioning Engine. CAVE is a set of microservices that provide a framework for storing and versioning connectomics data and large sets of dynamic annotations, metadata, and segmentations. This repository supplies client-side code to easily interact with the microservices in CAVE.
"},{"location":"#documentation-layout","title":"Documentation Layout","text":"To learn how to install caveclient, visit Installation.
To see hands-on examples of using caveclient in a notebook, visit the Tutorials.
To see API reference documentation for interacting with a caveclient.CAVEclient object, visit the Client API.
To see a glossary of terms used in the documentation, visit the Glossary (work in progress).
For information about how to contribute to the documentation or the package, visit the Contributing page. Feedback on the documentation is welcome! Please open an issue or use the \"Edit this page\" button at the top right of any page to suggest changes.
To read a full description of the Connectome Annotation Versioning Engine, including a description of the various services please read this paper.
"},{"location":"changelog/","title":"Changelog","text":""},{"location":"changelog/#743-december-20-2024","title":"7.4.3 (December 20, 2024)","text":"annotation module: Added support for \"None\", \"none\", etc. to remove notice_text and improved documentation on method
"},{"location":"changelog/#732-december-10-2024","title":"7.3.2 (December 10, 2024)","text":"Dropped oldest supported numpy requirements and declared python 3.8 support
"},{"location":"changelog/#731-december-5-2024","title":"7.3.1 (December 5, 2024)","text":"Improved documentation to annotation.update_table_metadata
"},{"location":"changelog/#730-december-5-2024","title":"7.3.0 (December 5, 2024)","text":"Added get_minimal_covering_nodes to chunkedgraph (#274)
"},{"location":"changelog/#721-december-3-2024","title":"7.2.1 (December 3, 2024)","text":"Added raw strings to avoid deprecation warning (#275)
"},{"location":"changelog/#720-december-3-2024","title":"7.2.0 (December 3, 2024)","text":"Allow get_state_json to accept direct URLs (#271)
"},{"location":"changelog/#710-december-3-2024","title":"7.1.0 (December 3, 2024)","text":"Added get_leaves_multiple to chunkedgraph module (#273)
"},{"location":"changelog/#700-november-20-2024","title":"7.0.0 (November 20, 2024)","text":"log_warnings=FalseCache is saved on a local file ~/.cloudvolume/secrets/cave_datastack_to_server_map.json
Cache will populate the first time caveclient.CAVEclient('my_datastack', server_address=\"https://my_server.com\") is called. Subsequent calls can then just be caveclient.CAVEclient('my_datastack').
"},{"location":"changelog/#510","title":"5.1.0","text":"Change all select_column calls to pass dictionaries rather than lists. Change all suffix calls to pass dictionaries rather than lists. Advocate for your server administrator to upgrade to MaterializationEngine 4.7.0 or later, so you can use the new MaterializationEngine 3.0 API and client.
"},{"location":"contributing/","title":"Contributing","text":"Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.
You can contribute in many ways:
"},{"location":"contributing/#types-of-contributions","title":"Types of Contributions","text":""},{"location":"contributing/#report-bugs","title":"Report Bugs","text":"Report bugs to our issues page.
If you are reporting a bug, please include:
Look through the GitHub issues for bugs. Anything tagged with \"bug\" and \"help wanted\" is open to whoever wants to implement it.
"},{"location":"contributing/#implement-features","title":"Implement Features","text":"Look through the GitHub issues for features. Anything tagged with \"enhancement\" and \"help wanted\" is open to whoever wants to implement it.
"},{"location":"contributing/#write-documentation","title":"Write Documentation","text":"caveclient could always use more documentation, whether as part of the official caveclient docs, in docstrings, or even on the web in blog posts, articles, and such.
The best way to send feedback is to create an issue on GitHub.
If you are proposing a feature:
Ready to contribute? Here's how to set up caveclient for local development.
git clone git@github.com:your_name_here/CAVEclient.git\nuv for various developer tasks. Ensure you have uv installed according to the installation instructions.Note
While we recommend using uv as described here, these tasks could also be achieved using pip to install and run the various required tools. You can view the development requirements and build/check commands in our pyproject.toml, so we avoid duplicating them here.
cd CAVEclient\nuv sync --python 3.12\ngit checkout -b name-of-your-bugfix-or-feature\nMake your changes locally
If you have added code that should be tested, add tests.
If you have modified dependencies in any way, make sure to run
uv sync\nuvx --from poethepoet poe doc-build\nuvx --from poethepoet poe lint-fix\nuvx --from poethepoet poe checks\nYou may be interested in running some of these checks individually, such as:
uvx --from poethepoet poe test\nuvx --from poethepoet poe doc-build\nuvx --from poethepoet poe lint\ngit add .\ngit commit -m \"Your detailed description of your changes.\"\ngit push origin name-of-your-bugfix-or-feature\nBefore you submit a pull request, check that it meets these guidelines:
The docs should be updated with whatever changes you have made. Put your new functionality into a function with a docstring, and make sure the new functionality is documented after building the documentation (described above).
Once you submit a pull request, automated checks will run. You may require administrator approval before running these checks if this is your first time contributing to the repo.
Warning
This glossary is a work in progress; for now we are documenting the commonly used terms that we need to define. Please feel free to contribute definitions or additional terms.
To install caveclient, run this command in your terminal:
pip install caveclient\nThis is the preferred method to install caveclient, as it will always install the most recent stable release.
You can also specify a particular version, e.g.
pip install caveclient==5.0.0\nIf you don't have pip installed, this Python installation guide can guide you through the process.
"},{"location":"installation/#from-source","title":"From source","text":"The source for caveclient can be downloaded from the Github repo.
You can either clone the public repository:
git clone git://github.com/CAVEconnectome/CAVEclient\nOr download the tarball:
curl -OJL https://github.com/CAVEconnectome/CAVEclient/tarball/master\nOnce you have a copy of the source, you can install it with:
pip install .\nOr in editable mode, it can be installed with:
pip install -e .\nThe most common method of interacting with the CAVE Framework is by instantiating a client (caveclient.CAVEclient) and then using that client to interact with various services. Under the hood, the CAVEclient is a collection of individual clients, which can be accessed via properties. For example, to access the materialization client, you can use client.materialize, which (up to the exact version) will actually return a MaterializationClient object.
These pages describe the functionality of each of the individual clients, but note that some features may not be available if the server you are accessing is using older versions of particular services.
"},{"location":"api/annotation/","title":"client.annotation","text":"Client for interacting with the annotation engine.
Methods:
Name Descriptioncreate_table Creates a new data table based on an existing schema
delete_annotation Delete one or more annotations in a table. Annotations that are
delete_table Marks a table for deletion requires super admin privileges
get_annotation Retrieve an annotation or annotations by id(s) and table name.
get_annotation_count Get number of annotations in a table
get_table_metadata Get metadata about a table
get_tables Gets a list of table names for a aligned_volume_name
post_annotation Post one or more new annotations to a table in the AnnotationEngine.
post_annotation_df Post one or more new annotations to a table in the AnnotationEngine.
process_position_columns Process a dataframe into a list of dictionaries
raise_for_status Raises requests.HTTPError, if one occurred.
stage_annotations Get a StagedAnnotations object to help produce correctly formatted annotations for a given table or schema.
update_annotation Update one or more new annotations to a table in the AnnotationEngine.
update_annotation_df Update one or more annotations to a table in the AnnotationEngine using a
update_metadata Update the metadata on an existing table
upload_staged_annotations Upload annotations directly from an Annotation Guide object.
Attributes:
Name Type Descriptionserver_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.create_table","title":"create_table(table_name, schema_name, description, voxel_resolution, reference_table=None, track_target_id_updates=None, flat_segmentation_source=None, user_id=None, aligned_volume_name=None, write_permission='PRIVATE', read_permission='PUBLIC', notice_text=None)","text":"Creates a new data table based on an existing schema
Parameters:
Name Type Description Defaulttable_name str Name of the new table. Cannot be the same as an existing table
requiredschema_name str Name of the schema for the new table.
requireddescription str Human readable description for what is in the table. Should include information about who generated the table What data it covers, and how it should be interpreted. And who should you talk to if you want to use it. An Example: a manual synapse table to detect chandelier synapses on 81 PyC cells with complete AISs [created by Agnes - agnesb@alleninstitute.org, uploaded by Forrest]
requiredvoxel_resolution List[float] voxel resolution points will be uploaded in, typically nm, i.e [1,1,1] means nanometers [4,4,40] would be 4nm, 4nm, 40nm voxels
requiredreference_table str If the schema you are using is a reference schema Meaning it is an annotation of another annotation. Then you need to specify what the target table those annotations are in.
None track_target_id_updates bool Indicates whether to automatically update reference table's foreign key if target annotation table row is updated.
None flat_segmentation_source str the source to a flat segmentation that corresponds to this table i.e. precomputed:\\gs:\\mybucket his_tables_annotation
None user_id int If you are uploading this schema on someone else's behalf and you want to link this table with their ID, you can specify it here Otherwise, the table will be created with your userID in the user_id column.
None aligned_volume_name str Name of the aligned_volume. If None, uses the one specified in the client.
None write_permission str What permissions to give the table for writing. One of PRIVATE: only you can write to this table (DEFAULT) GROUP: only members that share a group with you can write (excluding some groups) PUBLIC: Anyone can write to this table. Note all data is logged, and deletes are done by marking rows as deleted, so all data is always recoverable
'PRIVATE' read_permission str What permissions to give the table for reading. One of PRIVATE: only you can read this table. Intended to be used for sorting out bugs. GROUP: only members that share a group with you can read (intended for within group vetting) PUBLIC: anyone with permissions to read this datastack can read this data (DEFAULT)
'PUBLIC' notice_text str Text the user will see when querying this table. Can be used to warn users of flaws, and uncertainty in the data, or to advertise citations that should be used with this table. Defaults to None, no text. If you want to remove text, send empty string.
None Returns:
Type Descriptionjson Response JSON
Examples:
Basic annotation table:
description = \"Some description about the table\"\nvoxel_res = [4,4,40]\nclient.create_table(\"some_synapse_table\", \"synapse\", description, voxel_res)\ndelete_annotation(table_name, annotation_ids, aligned_volume_name=None)","text":"Delete one or more annotations in a table. Annotations that are deleted are recorded as 'non-valid' but are not physically removed from the table.
Parameters:
Name Type Description Defaulttable_name str Name of the table where annotations will be added
requiredannotation_ids (dict or list) A list of (or a single) dict of schematized annotation data matching the target table. each dict must contain an \"id\" field which is the ID of the annotation to update
requiredaligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionjson Response JSON: a list of new annotation IDs.
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.delete_table","title":"delete_table(table_name, aligned_volume_name=None)","text":"Marks a table for deletion requires super admin privileges
Parameters:
Name Type Description Defaulttable_name str name of table to mark for deletion
requiredaligned_volume_name str Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionjson Response JSON
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_annotation","title":"get_annotation(table_name, annotation_ids, aligned_volume_name=None)","text":"Retrieve an annotation or annotations by id(s) and table name.
Parameters:
Name Type Description Defaulttable_name str Name of the table
requiredannotation_ids int or iterable ID or IDS of the annotation to retreive
requiredaligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionlist Annotation data
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_annotation_count","title":"get_annotation_count(table_name, aligned_volume_name=None)","text":"Get number of annotations in a table
Parameters:
Name Type Description Defaulttable_name str name of table to mark for deletion
requiredaligned_volume_name str Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionint number of annotations
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_table_metadata","title":"get_table_metadata(table_name, aligned_volume_name=None)","text":"Get metadata about a table
Parameters:
Name Type Description Defaulttable_name str name of table to mark for deletion
requiredaligned_volume_name str Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionjson metadata about table
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_tables","title":"get_tables(aligned_volume_name=None)","text":"Gets a list of table names for a aligned_volume_name
Parameters:
Name Type Description Defaultaligned_volume_name str or None Name of the aligned_volume, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client
None Returns:
Type Descriptionlist List of table names
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.post_annotation","title":"post_annotation(table_name, data, aligned_volume_name=None)","text":"Post one or more new annotations to a table in the AnnotationEngine. All inserted annotations will be marked as 'valid'. To invalidate annotations refer to 'update_annotation', 'update_annotation_df' and 'delete_annotation' methods.
Parameters:
Name Type Description Defaulttable_name str Name of the table where annotations will be added
requireddata (dict or list) A list of (or a single) dict of schematized annotation data matching the target table.
requiredaligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionjson Response JSON
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.post_annotation_df","title":"post_annotation_df(table_name, df, position_columns, aligned_volume_name=None)","text":"Post one or more new annotations to a table in the AnnotationEngine. All inserted annotations will be marked as 'valid'. To invalidate annotations see 'update_annotation', 'update_annotation_df' and 'delete_annotation' methods.
Parameters:
Name Type Description Defaulttable_name str Name of the table where annotations will be added
requireddf DataFrame A pandas dataframe containing the annotations. Columns should be fields in schema, position columns need to be called out in position_columns argument.
requiredposition_columns Optional[Union[Iterable[str], Mapping[str, str]]] if None, will look for all columns with 'X_position' in the name and assume they go in fields called \"X\". if Iterable assumes each column given ends in _position. (i.e. ['pt_position'] if 'pt' is the name of the position field in schema) if Mapping, keys are names of columns in dataframe, values are the names of the fields (i.e. {'pt_column': 'pt'} would be correct if you had one column named 'pt_column' which needed to go into a schema with a position column called 'pt')
requiredaligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionjson Response JSON
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.process_position_columns","title":"process_position_columns(df, position_columns) staticmethod","text":"Process a dataframe into a list of dictionaries
Parameters:
Name Type Description Defaultdf DataFrame Dataframe to process
requiredposition_columns Optional[Union[Iterable[str], Mapping[str, str]]] See .post_annotation_df
Returns:
Type Descriptiondict Annotations ready for posting
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.raise_for_status","title":"raise_for_status(r, log_warning=True) staticmethod","text":"Raises requests.HTTPError, if one occurred.
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.stage_annotations","title":"stage_annotations(table_name=None, schema_name=None, update=False, id_field=False, table_resolution=None, annotation_resolution=None)","text":"Get a StagedAnnotations object to help produce correctly formatted annotations for a given table or schema. StagedAnnotation objects can be uploaded directly with upload_staged_annotations.
Parameters:
Name Type Description Defaulttable_name str Table name to stage annotations for, by default None.
None schema_name str Schema name to use to make annotations. Only needed if the table_name is not set, by default None
None update bool Set to True if individual annotations are going to be updated, by default False.
False id_field bool Set to True if id fields are to be specified. Not needed if update is True, which always needs id fields. Optional, by default False
False table_resolution list - like or None Voxel resolution of spatial points in the table in nanometers. This is found automatically from the info service if a table name is provided, by default None. If annotation_resolution is also set, this allows points to be scaled correctly for the table.
None annotation_resolution list - like Voxel resolution of spatial points provided by the user when creating annotations. If the table resolution is also available (manually or from the info service), annotations are correctly rescaled for the volume. By default, None.
None"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_annotation","title":"update_annotation(table_name, data, aligned_volume_name=None)","text":"Update one or more new annotations to a table in the AnnotationEngine. Updating is implemented by invalidating the old annotation and inserting a new annotation row, which will receive a new primary key ID.
NotesIf annotations ids were user provided upon insertion the database will autoincrement from the current max id in the table.
Parameters:
Name Type Description Defaulttable_name str Name of the table where annotations will be added
requireddata (dict or list) A list of (or a single) dict of schematized annotation data matching the target table. each dict must contain an \"id\" field which is the ID of the annotation to update
requiredaligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionjson Response JSON: a list of new annotation IDs.
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_annotation_df","title":"update_annotation_df(table_name, df, position_columns, aligned_volume_name=None)","text":"Update one or more annotations to a table in the AnnotationEngine using a dataframe as format. Updating is implemented by invalidating the old annotation and inserting a new annotation row, which will receive a new primary key ID.
NotesIf annotations ids were user provided upon insertion the database will autoincrement from the current max id in the table.
Parameters:
Name Type Description Defaulttable_name str Name of the table where annotations will be added
requireddf DataFrame A pandas dataframe containing the annotations. Columns should be fields in schema, position columns need to be called out in position_columns argument.
requiredposition_columns Iterable[str] or Mapping[str, str] or None if None, will look for all columns with 'X_position' in the name and assume they go in fields called \"X\". if Iterable assumes each column given ends in _position. (i.e. ['pt_position'] if 'pt' is the name of the position field in schema) if Mapping, keys are names of columns in dataframe, values are the names of the fields (i.e. {'pt_column': 'pt'} would be correct if you had one column named 'pt_column' which needed to go into a schema with a position column called 'pt')
requiredaligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Descriptionjson Response JSON
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_metadata","title":"update_metadata(table_name, description=None, flat_segmentation_source=None, read_permission=None, write_permission=None, user_id=None, notice_text=None, aligned_volume_name=None)","text":"Update the metadata on an existing table
Parameters:
Name Type Description Defaulttable_name str required description str Defaults to the None type, in which case no change will be made to this metadata field.
None flat_segmentation_source str Defaults to the None type, in which case no change will be made to this metadata field.
None read_permission str What permissions to give the table for reading. One of PRIVATE: only you can read this table. Intended to be used for sorting out bugs. GROUP: only members that share a group with you can read (intended for within group vetting) PUBLIC: anyone with permissions to read this datastack can read this data Defaults to the None type, in which case no change will be made to this metadata field.
None write_permission str What permissions to give the table for writing. One of PRIVATE: only you can write to this table GROUP: only members that share a group with you can write (excluding some groups) PUBLIC: Anyone can write to this table. Note all data is logged, and deletes are done by marking rows as deleted, so all data is always recoverable Defaults to the None type, in which case no change will be made to this metadata field.
None user_id int Note, if you use this you will not be able to update the metadata on this table any longer and depending on permissions may not be able to read or write to it Defaults to the None type, in which case no change will be made to this metadata field.
None notice_text str Text the user will see when querying this table. Can be used to warn users of flaws, and uncertainty in the data, or to advertise citations that should be used with this table. If you wish to remove the notice_text pass an empty string, or \"none\", \"None\", \"NONE\", or any other capitaliztion of the word \"none\". Defaults to the None type, in which case no change will be made to this metadata field.
None aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.upload_staged_annotations","title":"upload_staged_annotations(staged_annos, aligned_volume_name=None)","text":"Upload annotations directly from an Annotation Guide object. This method uses the options specified in the object, including table name and if the annotation is an update or not.
Parameters:
Name Type Description Defaultstaged_annos AnnotationGuide AnnotationGuide object with a specified table name and a collection of annotations already filled in.
requiredaligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type DescriptionList or dict If new annotations are posted, a list of ids. If annotations are being updated, a dictionary with the mapping from old ids to new ids.
"},{"location":"api/auth/","title":"client.auth","text":"Client to find and use auth tokens to access the other services.
Methods:
Name Descriptionget_group_users Get users in a group
get_new_token Currently, returns instructions for getting a new token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.
get_token Load a token with a given key the specified token file
get_tokens Get the tokens setup for this users
get_user_information Get user data.
save_token Conveniently save a token in the correct format.
setup_token Currently, returns instructions for getting your auth token based on the current settings and saving it to the local environment.
Attributes:
Name Type Descriptionrequest_header Formatted request header with the specified token
token Secret token used to authenticate yourself to the Connectome Annotation Versioning Engine services.
"},{"location":"api/auth/#caveclient.auth.AuthClient.request_header","title":"request_header property","text":"Formatted request header with the specified token
"},{"location":"api/auth/#caveclient.auth.AuthClient.token","title":"token property writable","text":"Secret token used to authenticate yourself to the Connectome Annotation Versioning Engine services.
"},{"location":"api/auth/#caveclient.auth.AuthClient.get_group_users","title":"get_group_users(group_id)","text":"Get users in a group
Parameters:
Name Type Description Defaultgroup_id int ID value for a given group
requiredReturns:
Type Descriptionlist List of dicts of user ids. Returns empty list if group does not exist.
"},{"location":"api/auth/#caveclient.auth.AuthClient.get_new_token","title":"get_new_token(open=False, no_text=False)","text":"Currently, returns instructions for getting a new token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.
Parameters:
Name Type Description Defaultopen bool If True, opens a web browser to the web page where you can generate a new token.
False"},{"location":"api/auth/#caveclient.auth.AuthClient.get_token","title":"get_token(token_key=None)","text":"Load a token with a given key the specified token file
Parameters:
Name Type Description Defaulttoken_key str or None key in the token file JSON, by default None. If None, uses 'token'.
None"},{"location":"api/auth/#caveclient.auth.AuthClient.get_tokens","title":"get_tokens()","text":"Get the tokens setup for this users
Returns:
Type Descriptionlist[dict]: a list of dictionary of tokens, each with the keys \"id\": the id of this token \"token\": the token (str) \"user_id\": the users id (should be your ID)
"},{"location":"api/auth/#caveclient.auth.AuthClient.get_user_information","title":"get_user_information(user_ids)","text":"Get user data.
Parameters:
Name Type Description Defaultuser_ids list of int user_ids to look up
required"},{"location":"api/auth/#caveclient.auth.AuthClient.save_token","title":"save_token(token=None, token_key=default_token_key, overwrite=False, token_file=None, switch_token=True, write_to_server_file=True, ignore_readonly=True, local_server=True)","text":"Conveniently save a token in the correct format.
After getting a new token by following the instructions in authclient.get_new_token(), you can save it with a fully default configuration by running:
token = 'my_shiny_new_token'
authclient.save_token(token=token)
Now on next load, authclient=AuthClient() will make an authclient instance using this token. If you would like to specify more information about the json file where the token will be stored, see the parameters below.
Parameters:
Name Type Description Defaulttoken str New token to save, by default None
None token_key str Key for the token in the token_file json, by default \"token\"
default_token_key overwrite bool Allow an existing token to be changed, by default False
False token_file str Path to the token file, by default None. If None, uses the default file location specified above.
None switch_token bool If True, switch the auth client over into using the new token, by default True
True write_to_server_file bool If True, will write token to a server specific file to support this machine interacting with multiple auth servers.
True ignore_readonly bool If True, will only attempt to save a token if the directory is writeable.
True local_server bool If True, saves the token to the local server file as well.
True"},{"location":"api/auth/#caveclient.auth.AuthClient.setup_token","title":"setup_token(make_new=True, open=True)","text":"Currently, returns instructions for getting your auth token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.
Parameters:
Name Type Description Defaultmake_new bool If True, will make a new token, else prompt you to open a page to retrieve an existing token.
True open bool If True, opens a web browser to the web page where you can retrieve a token.
True"},{"location":"api/chunkedgraph/","title":"client.chunkedgraph","text":"Client for interacting with the chunkedgraph.
Methods:
Name Descriptionget_roots Get the root ID for a list of supervoxels.
get_root_id Get the root ID for a specified supervoxel.
get_minimal_covering_nodes Get the minimal covering nodes for a list of root IDs.
get_merge_log Get the merge log (splits and merges) for an object.
get_change_log Get the change log (splits and merges) for an object.
get_user_operations Get operation details for a user ID. Currently, this is only available to
get_tabular_change_log Get a detailed changelog for neurons.
get_leaves_many Get all supervoxels for a list of root IDs.
get_leaves Get all supervoxels for a root ID.
do_merge Perform a merge on the chunked graph.
undo_operation Undo an operation.
execute_split Execute a multicut split based on points or supervoxels.
preview_split Get supervoxel connected components from a preview multicut split.
get_children Get the children of a node in the chunked graph hierarchy.
get_contact_sites Get contacts for a root ID.
find_path Find a path between two locations on a root ID using the level 2 chunked
get_subgraph Get subgraph of root id within a bounding box.
level2_chunk_graph Get graph of level 2 chunks, the smallest agglomeration level above supervoxels.
remesh_level2_chunks Submit specific level 2 chunks to be remeshed in case of a problem.
get_operation_details Get the details of a list of operations.
get_lineage_graph Returns the lineage graph for a root ID, optionally cut off in the past or
get_latest_roots Returns root IDs that are related to the given root_id at a given
get_original_roots Returns root IDs that are the latest successors of a given root ID.
is_latest_roots Check whether these root IDs are still a root at this timestamp.
suggest_latest_roots Suggest latest roots for a given root id, based on overlap of component
is_valid_nodes Check whether nodes are valid for given timestamp range.
get_root_timestamps Retrieves timestamps when roots where created.
get_past_ids For a set of root IDs, get the list of IDs at a past or future time point
get_delta_roots Get the list of roots that have changed between timetamp_past and
get_oldest_timestamp Get the oldest timestamp in the database.
Attributes:
Name Type Descriptionserver_version Optional[Version] The version of the service running on the remote server. Note that this
timestamp Optional[datetime] The default timestamp for queries which expect a timestamp. If None, uses the
segmentation_info Complete segmentation metadata
base_resolution MIP 0 resolution for voxels assumed by the ChunkedGraph
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.timestamp","title":"timestamp: Optional[datetime.datetime] property writable","text":"The default timestamp for queries which expect a timestamp. If None, uses the current time.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.segmentation_info","title":"segmentation_info property","text":"Complete segmentation metadata
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.base_resolution","title":"base_resolution property","text":"MIP 0 resolution for voxels assumed by the ChunkedGraph
Returns:
Type Descriptionlist 3-long list of x/y/z voxel dimensions in nm
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_roots","title":"get_roots(supervoxel_ids, timestamp=None, stop_layer=None)","text":"Get the root ID for a list of supervoxels.
Parameters:
Name Type Description Defaultsupervoxel_ids list or np.array of int Supervoxel IDs to look up.
requiredtimestamp datetime UTC datetime to specify the state of the chunkedgraph at which to query, by default None. If None, uses the timestamp property for this client, which defaults to the current time.
None stop_layer int or None If True, looks up IDs only up to a given stop layer. Default is None.
None Returns:
Type Descriptionnp.array of np.uint64 Root IDs containing each supervoxel.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_root_id","title":"get_root_id(supervoxel_id, timestamp=None, level2=False)","text":"Get the root ID for a specified supervoxel.
Parameters:
Name Type Description Defaultsupervoxel_id int Supervoxel id value
requiredtimestamp datetime UTC datetime to specify the state of the chunkedgraph at which to query, by default None. If None, uses the timestamp property for this client, which defaults to the current time.
None Returns:
Type Descriptionint64 Root ID containing the supervoxel.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_minimal_covering_nodes","title":"get_minimal_covering_nodes(node_ids)","text":"Get the minimal covering nodes for a list of root IDs.
Parameters:
Name Type Description Defaultnodes_ids Iterable of int or np.int64 List of root IDs to query.
requiredReturns:
Type Descriptionnp.array of np.int64: List of PCG node_ids that minimally and exactly cover the input nodes
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_merge_log","title":"get_merge_log(root_id)","text":"Get the merge log (splits and merges) for an object.
Parameters:
Name Type Description Defaultroot_id int Object root ID to look up.
requiredReturns:
Type Descriptionlist List of merge events in the history of the object.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_change_log","title":"get_change_log(root_id, filtered=True)","text":"Get the change log (splits and merges) for an object.
Parameters:
Name Type Description Defaultroot_id int Object root ID to look up.
requiredfiltered bool Whether to filter the change log to only include splits and merges which affect the final state of the object (filtered=True), as opposed to including edit history for objects which as some point were split from the query object root_id (filtered=False). Defaults to True.
True Returns:
Type Descriptiondict Dictionary summarizing split and merge events in the object history, containing the following keys:
get_user_operations(user_id, timestamp_start, include_undo=True, timestamp_end=None)","text":"Get operation details for a user ID. Currently, this is only available to admins.
Parameters:
Name Type Description Defaultuser_id int User ID to query (use 0 for all users (admin only)).
requiredtimestamp_start datetime Timestamp to start filter (UTC).
requiredinclude_undo bool Whether to include undos. Defaults to True.
True timestamp_end datetime Timestamp to end filter (UTC). Defaults to now.
None Returns:
Type DescriptionDataFrame DataFrame including the following columns:
get_tabular_change_log(root_ids, filtered=True)","text":"Get a detailed changelog for neurons.
Parameters:
Name Type Description Defaultroot_ids list of int Object root IDs to look up.
requiredfiltered bool Whether to filter the change log to only include splits and merges which affect the final state of the object (filtered=True), as opposed to including edit history for objects which as some point were split from the query objects in root_ids (filtered=False). Defaults to True.
True Returns:
Type Descriptiondict of pd.DataFrame The keys are the root IDs, and the values are DataFrames with the following columns and datatypes:
datetime.datetime.utcfromtimestamp(timestamp/1000).get_leaves_many(root_ids, bounds=None, stop_layer=None)","text":"Get all supervoxels for a list of root IDs.
Parameters:
Name Type Description Defaultroot_ids Iterable Root IDs to query.
requiredbounds If specified, returns supervoxels within a 3x2 numpy array of bounds [[minx,maxx],[miny,maxy],[minz,maxz]]. If None, finds all supervoxels.
None stop_layer int If specified, returns chunkedgraph nodes at layer stop_layer default will be stop_layer=1 (supervoxels).
None Returns:
Type Descriptiondict Dict relating ids to contacts
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_leaves","title":"get_leaves(root_id, bounds=None, stop_layer=None)","text":"Get all supervoxels for a root ID.
Parameters:
Name Type Description Defaultroot_id int Root ID to query.
requiredbounds If specified, returns supervoxels within a 3x2 numpy array of bounds [[minx,maxx],[miny,maxy],[minz,maxz]]. If None, finds all supervoxels.
None stop_layer int If specified, returns chunkedgraph nodes at layer stop_layer default will be stop_layer=1 (supervoxels).
None Returns:
Type Descriptionnp.array of np.int64 Array of supervoxel IDs (or node ids if stop_layer>1).
do_merge(supervoxels, coords, resolution=(4, 4, 40))","text":"Perform a merge on the chunked graph.
Parameters:
Name Type Description Defaultsupervoxels iterable An N-long list of supervoxels to merge.
requiredcoords array An Nx3 array of coordinates of the supervoxels in units of resolution.
resolution tuple What to multiply coords by to get nanometers. Defaults to (4,4,40).
(4, 4, 40)"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.undo_operation","title":"undo_operation(operation_id)","text":"Undo an operation.
Parameters:
Name Type Description Defaultoperation_id int Operation ID to undo.
requiredReturns:
Type Descriptiondict"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.execute_split","title":"execute_split(source_points, sink_points, root_id, source_supervoxels=None, sink_supervoxels=None)","text":"Execute a multicut split based on points or supervoxels.
Parameters:
Name Type Description Defaultsource_points array or list Nx3 list or array of 3d points in nm coordinates for source points (red).
requiredsink_points array or list Mx3 list or array of 3d points in nm coordinates for sink points (blue).
requiredroot_id int Root ID of object to do split preview.
requiredsource_supervoxels (array, list or None) If providing source supervoxels, an N-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.
None sink_supervoxels (array, list or None) If providing sink supervoxels, an M-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.
None Returns:
Name Type Descriptionoperation_id int Unique ID of the split operation
new_root_ids list of int List of new root IDs resulting from the split operation.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.preview_split","title":"preview_split(source_points, sink_points, root_id, source_supervoxels=None, sink_supervoxels=None, return_additional_ccs=False)","text":"Get supervoxel connected components from a preview multicut split.
Parameters:
Name Type Description Defaultsource_points array or list Nx3 list or array of 3d points in nm coordinates for source points (red).
requiredsink_points array or list Mx3 list or array of 3d points in nm coordinates for sink points (blue).
requiredroot_id int Root ID of object to do split preview.
requiredsource_supervoxels (array, list or None) If providing source supervoxels, an N-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.
None sink_supervoxels (array, list or None) If providing sink supervoxels, an M-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.
None return_additional_ccs bool If True, returns any additional connected components beyond the ones with source and sink points. In most situations, this can be ignored. By default, False.
False Returns:
Name Type Descriptionsource_connected_component list Supervoxel IDs in the component with the most source points.
sink_connected_component list Supervoxel IDs in the component with the most sink points.
successful_split bool True if the split worked.
other_connected_components (optional) : list of lists of int List of lists of supervoxel IDs for any other resulting connected components. Only returned if return_additional_ccs is True.
get_children(node_id)","text":"Get the children of a node in the chunked graph hierarchy.
Parameters:
Name Type Description Defaultnode_id int Node ID to query.
requiredReturns:
Type Descriptionnp.array of np.int64 IDs of child nodes.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_contact_sites","title":"get_contact_sites(root_id, bounds, calc_partners=False)","text":"Get contacts for a root ID.
Parameters:
Name Type Description Defaultroot_id int Root ID to query.
requiredbounds Bounds within a 3x2 numpy array of bounds [[minx,maxx],[miny,maxy],[minz,maxz]] for which to find contacts. Running this query without bounds is too slow.
calc_partners bool If True, get partner root IDs. By default, False.
False Returns:
Type Descriptiondict Dict relating ids to contacts
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.find_path","title":"find_path(root_id, src_pt, dst_pt, precision_mode=False)","text":"Find a path between two locations on a root ID using the level 2 chunked graph.
Parameters:
Name Type Description Defaultroot_id int Root ID to query.
requiredsrc_pt array 3-element array of xyz coordinates in nm for the source point.
requireddst_pt array 3-element array of xyz coordinates in nm for the destination point.
requiredprecision_mode bool Whether to perform the search in precision mode. Defaults to False.
False Returns:
Name Type Descriptioncentroids_list array Array of centroids along the path.
l2_path np.array of int Array of level 2 chunk IDs along the path.
failed_l2_ids np.array of int Array of level 2 chunk IDs that failed to find a path.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_subgraph","title":"get_subgraph(root_id, bounds)","text":"Get subgraph of root id within a bounding box.
Parameters:
Name Type Description Defaultroot_id int Root (or any node ID) of chunked graph to query.
requiredbounds array 3x2 bounding box (x,y,z) x (min,max) in chunked graph coordinates.
requiredReturns:
Type Descriptionnp.array of np.int64 Node IDs in the subgraph.
np.array of np.double Affinities of edges in the subgraph.
np.array of np.int32 Areas of nodes in the subgraph.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.level2_chunk_graph","title":"level2_chunk_graph(root_id, bounds=None)","text":"Get graph of level 2 chunks, the smallest agglomeration level above supervoxels.
Parameters:
Name Type Description Defaultroot_id int Root id of object
requiredbounds array 3x2 bounding box (x,y,z) x (min,max) in chunked graph coordinates (use client.chunkedgraph.base_resolution to view this default resolution for your chunkedgraph client). Note that the result will include any level 2 nodes which have chunk boundaries within some part of this bounding box, meaning that the representative point for a given level 2 node could still be slightly outside of these bounds. If None, returns all level 2 chunks for the root ID.
None Returns:
Type Descriptionlist of list Edge list for level 2 chunked graph. Each element of the list is an edge, and each edge is a list of two node IDs (source and target).
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.remesh_level2_chunks","title":"remesh_level2_chunks(chunk_ids)","text":"Submit specific level 2 chunks to be remeshed in case of a problem.
Parameters:
Name Type Description Defaultchunk_ids list List of level 2 chunk IDs.
required"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_operation_details","title":"get_operation_details(operation_ids)","text":"Get the details of a list of operations.
Parameters:
Name Type Description Defaultoperation_ids Iterable[int] List/array of operation IDs.
requiredReturns:
Type Descriptiondict of str to dict A dict of dicts of operation info, keys are operation IDs (as strings), values are a dictionary of operation info for the operation. These dictionaries contain the following keys:
get_lineage_graph(root_id, timestamp_past=None, timestamp_future=None, as_nx_graph=False, exclude_links_to_future=False, exclude_links_to_past=False)","text":"Returns the lineage graph for a root ID, optionally cut off in the past or the future.
Each change in the chunked graph creates a new root ID for the object after that change. This function returns a graph of all root IDs for a given object, tracing the history of the object in terms of merges and splits.
Parameters:
Name Type Description Defaultroot_id int Object root ID.
requiredtimestamp_past datetime or None Cutoff for the lineage graph backwards in time. By default, None.
None timestamp_future datetime or None Cutoff for the lineage graph going forwards in time. By default, uses the timestamp property for this client, which defaults to the current time.
None as_nx_graph If True, a NetworkX graph is returned.
False exclude_links_to_future If True, links from nodes before timestamp_future to after timestamp_future are removed. If False, the link(s) which has one node before timestamp and one node after timestamp is kept.
False exclude_links_to_past If True, links from nodes before timestamp_past to after timestamp_past are removed. If False, the link(s) which has one node before timestamp and one node after timestamp is kept.
False Returns:
Type Descriptiondict Dictionary describing the lineage graph and operations for the root ID. Not returned if as_nx_graph is True. The dictionary contains the following keys:
DiGraph NetworkX directed graph of the lineage graph. Only returned if as_nx_graph is True.
get_latest_roots(root_id, timestamp=None, timestamp_future=None)","text":"Returns root IDs that are related to the given root_id at a given timestamp. Can be used to find the \"latest\" root IDs associated with an object.
Parameters:
Name Type Description Defaultroot_id int Object root ID.
requiredtimestamp datetime or None Timestamp of where to query IDs from. If None, uses the timestamp property for this client, which defaults to the current time.
None timestamp_future datetime or None DEPRECATED name, use timestamp instead. Timestamp to suggest IDs from (note can be in the past relative to the root). By default, None.
None Returns:
Type Descriptionndarray 1d array with all latest successors.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_original_roots","title":"get_original_roots(root_id, timestamp_past=None)","text":"Returns root IDs that are the latest successors of a given root ID.
Parameters:
Name Type Description Defaultroot_id int Object root ID.
requiredtimestamp_past datetime or None Cutoff for the search going backwards in time. By default, None.
None Returns:
Type Descriptionndarray 1d array with all latest successors.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.is_latest_roots","title":"is_latest_roots(root_ids, timestamp=None)","text":"Check whether these root IDs are still a root at this timestamp.
Parameters:
Name Type Description Defaultroot_ids array-like of int Root IDs to check.
requiredtimestamp datetime Timestamp to check whether these IDs are valid root IDs in the chunked graph. If None, uses the timestamp property for this client, which defaults to the current time.
None Returns:
Type Descriptionnp.array of bool Array of whether these are valid root IDs.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.suggest_latest_roots","title":"suggest_latest_roots(root_id, timestamp=None, stop_layer=None, return_all=False, return_fraction_overlap=False)","text":"Suggest latest roots for a given root id, based on overlap of component chunk IDs. Note that edits change chunk IDs, and so this effectively measures the fraction of unchanged chunks at a given chunk layer, which sets the size scale of chunks. Higher layers are coarser.
Parameters:
Name Type Description Defaultroot_id int Root ID of the potentially outdated object.
requiredtimestamp datetime Datetime at which \"latest\" roots are being computed, by default None. If None, uses the timestamp property for this client, which defaults to the current time. Note that this has to be a timestamp after the creation of the root_id.
None stop_layer int Chunk level at which to compute overlap, by default None. No value will take the 4th from the top layer, which emphasizes speed and works well for larger objects. Lower values are slower but more fine-grained. Values under 2 (i.e. supervoxels) are not recommended except in extremely fine grained scenarios.
None return_all bool If True, return all current IDs sorted from most overlap to least, by default False. If False, only the top is returned.
False return_fraction_overlap bool If True, return all fractions sorted by most overlap to least, by default False. If False, only the top value is returned.
False"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.is_valid_nodes","title":"is_valid_nodes(node_ids, start_timestamp=None, end_timestamp=None)","text":"Check whether nodes are valid for given timestamp range.
Valid is defined as existing in the chunked graph. This makes no statement about these IDs being roots, supervoxel or anything in-between. It also does not take into account whether a root ID has since been edited.
Parameters:
Name Type Description Defaultnode_ids array-like of int Node IDs to check.
requiredstart_timestamp datetime Timestamp to check whether these IDs were valid after this timestamp. Defaults to None (assumes now).
None end_timestamp datetime Timestamp to check whether these IDs were valid before this timestamp. If None, uses the timestamp property for this client, which defaults to the current time.
None Returns:
Type Descriptionnp.array of bool Array of whether these are valid IDs.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_root_timestamps","title":"get_root_timestamps(root_ids, latest=False, timestamp=None)","text":"Retrieves timestamps when roots where created.
Parameters:
Name Type Description Defaultroot_ids Iterable of root IDs to query.
requiredlatest bool If False, returns the first timestamp that the root_id was valid for each root ID. If True, returns the newest/latest timestamp for each root ID. Note, this will return the timestamp at which the query was run when the root is currently valid. This means that you will get a different answer if you make this same query at a later time if you don't specify a timestamp parameter.
False timestamp datetime Timestamp to query when using latest=True. Use this to provide consistent results for a particular timestamp. If an ID is still valid at a point in the future past this timestamp, the query will still return this timestamp as the latest moment in time. An error will occur if you provide a timestamp for which the root ID is not valid. If None, uses the timestamp property for this client, which defaults to the current time.
None Returns:
Type Descriptionnp.array of datetime.datetime Array of timestamps when root_ids were created.
get_past_ids(root_ids, timestamp_past=None, timestamp_future=None)","text":"For a set of root IDs, get the list of IDs at a past or future time point that could contain parts of the same object.
Parameters:
Name Type Description Defaultroot_ids Iterable of int Iterable of root IDs to query.
requiredtimestamp_past datetime or None Time of a point in the past for which to look up root ids. Default is None.
None timestamp_future datetime or None Time of a point in the future for which to look up root ids. Not implemented on the server currently. Default is None.
None Returns:
Type Descriptiondict Dict with keys \"future_id_map\" and \"past_id_map\". Each is a dict whose keys are the supplied root_ids and whose values are the list of related root IDs at timestamp_past/timestamp_future.
get_delta_roots(timestamp_past, timestamp_future=datetime.datetime.now(datetime.timezone.utc))","text":"Get the list of roots that have changed between timetamp_past and timestamp_future.
Parameters:
Name Type Description Defaulttimestamp_past datetime Past timepoint to query
requiredtimestamp_future datetime Future timepoint to query. Defaults to datetime.datetime.now(datetime.timezone.utc).
now(utc) Returns:
Name Type Descriptionold_roots np.ndarray of np.int64 Roots that have expired in that interval.
new_roots np.ndarray of np.int64 Roots that are new in that interval.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_oldest_timestamp","title":"get_oldest_timestamp()","text":"Get the oldest timestamp in the database.
Returns:
Type Descriptiondatetime Oldest timestamp in the database.
"},{"location":"api/client/","title":"caveclient.CAVEclient","text":"A manager for all clients sharing common datastack and authentication information.
This client wraps all the other clients and keeps track of the things that need to be consistent across them. To instantiate a client:
from caveclient import CAVEclient\n\nclient = CAVEclient(datastack_name='my_datastack',\n server_address='www.myserver.com',\n auth_token_file='~/.mysecrets/secrets.json')\nThen
client.annotation is an AnnotationClient (see client.annotation)client.auth is an AuthClient (see client.auth)client.chunkedgraph is a ChunkedGraphClient (see client.chunkedgraph)client.info is an InfoServiceClient (see client.info)client.l2cache is an L2CacheClient (see client.l2cache)client.materialize is a MaterializationClient (see client.materialize)client.skeleton is a SkeletonClient (see client.skeleton)client.schema is a SchemaClient (see client.schema)client.state is a neuroglancer JSONService (see client.state)All subclients are loaded lazily and share the same datastack name, server address, and auth tokens where used.
Parameters:
Name Type Description Defaultdatastack_name str Datastack name for the services. Almost all services need this and will not work if it is not passed.
None server_address str or None URL of the framework server. If None, chooses the default server global.daf-apis.com. Optional, defaults to None.
None auth_token_file str or None Path to a json file containing the auth token. If None, uses the default location. See Auth client documentation. Optional, defaults to None.
default_token_file auth_token_key str Dictionary key for the token in the the JSON file. Optional, default is 'token'.
'token' auth_token str or None Direct entry of an auth token. If None, uses the file arguments to find the token. Optional, default is None.
None max_retries int or None Sets the default number of retries on failed requests. Optional, by default 2.
None pool_maxsize int or None Sets the max number of threads in a requests pool, although this value will be exceeded if pool_block is set to False. Optional, uses requests defaults if None.
None pool_block If True, prevents the number of threads in a requests pool from exceeding the max size. Optional, uses requests defaults (False) if None.
None desired_resolution Iterable[float] or None If given, should be a list or array of the desired resolution you want queries returned in useful for materialization queries.
None info_cache Pre-computed info cache, bypassing the lookup of datastack info from the info service. Should only be used in cases where this information is cached and thus repetitive lookups can be avoided.
None version Optional[int] The default materialization version of the datastack to use. If None, the latest version is used. Optional, defaults to None.
None See Also set_session_defaults
get_session_defaults
Methods:
Name Descriptionchange_auth Change the authentication token and reset services.
Attributes:
Name Type Descriptionannotation AnnotationClient A client for the annotation service. See client.annotation
auth AuthClient A client for the auth service. See client.auth for more information.
chunkedgraph ChunkedGraphClient A client for the chunkedgraph service. See client.chunkedgraph
datastack_name str The name of the datastack for the client.
info InfoServiceClient A client for the info service. See client.info for more information.
l2cache L2CacheClient A client for the L2 cache service. See client.l2cache
materialize MaterializationClient A client for the materialization service. See client.materialize
schema SchemaClient A client for the EM Annotation Schemas service. See client.schema
server_address The server address for the client.
skeleton SkeletonClient A client for the skeleton service. See client.skeleton
state JSONService A client for the neuroglancer state service. See client.state
timestamp Optional[datetime] The default timestamp to use for queries which rely on a timestamp.
version Optional[int] The default materialization version of the datastack to use for queries which
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.annotation","title":"annotation: AnnotationClient property","text":"A client for the annotation service. See client.annotation for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.auth","title":"auth: AuthClient property","text":"A client for the auth service. See client.auth for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.chunkedgraph","title":"chunkedgraph: ChunkedGraphClient property","text":"A client for the chunkedgraph service. See client.chunkedgraph for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.datastack_name","title":"datastack_name: str property","text":"The name of the datastack for the client.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.info","title":"info: InfoServiceClient property","text":"A client for the info service. See client.info for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.l2cache","title":"l2cache: L2CacheClient property","text":"A client for the L2 cache service. See client.l2cache for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.materialize","title":"materialize: MaterializationClient property","text":"A client for the materialization service. See client.materialize for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.schema","title":"schema: SchemaClient property","text":"A client for the EM Annotation Schemas service. See client.schema for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.server_address","title":"server_address property","text":"The server address for the client.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.skeleton","title":"skeleton: SkeletonClient property","text":"A client for the skeleton service. See client.skeleton for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.state","title":"state: JSONService property","text":"A client for the neuroglancer state service. See client.state for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.timestamp","title":"timestamp: Optional[datetime] property","text":"The default timestamp to use for queries which rely on a timestamp.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.version","title":"version: Optional[int] property writable","text":"The default materialization version of the datastack to use for queries which expect a version. Also sets the timestamp to the corresponding timestamp of the version for queries which rely on a timestamp.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.change_auth","title":"change_auth(auth_token_file=None, auth_token_key=None, auth_token=None)","text":"Change the authentication token and reset services.
Parameters:
Name Type Description Defaultauth_token_file str New auth token json file path, by default None, which defaults to the existing state.
None auth_token_key str New dictionary key under which the token is stored in the json file, by default None, which defaults to the existing state.
None auth_token str Direct entry of a new token, by default None.
None"},{"location":"api/config/","title":"Configuration","text":""},{"location":"api/config/#caveclient.set_session_defaults","title":"caveclient.set_session_defaults(max_retries=3, pool_block=False, pool_maxsize=10, backoff_factor=0.1, backoff_max=120, status_forcelist=(502, 503, 504))","text":"Set global default values to configure how all clients will communicate with servers. Should be done prior to initializing a client.
Note that these values are only used when not set at the client level.
Parameters:
Name Type Description Defaultmax_retries int The maximum number of retries each connection should attempt. Set to 0 to fail on the first retry.
3 pool_block bool Whether the connection pool should block for connections.
False pool_maxsize int The maximum number of connections to save in the pool.
10 backoff_factor Union[float, int] A backoff factor to apply between attempts after the second try (most errors are resolved immediately by a second try without a delay). The query will sleep for: {backoff factor} * (2 ^ ({number of total retries} - 1)) seconds. For example, if the backoff_factor is 0.1, then will sleep for [0.0s, 0.2s, 0.4s, 0.8s, \u2026] between retries. No backoff will ever be longer than backoff_max.
0.1 backoff_max Union[float, int] The maximum backoff time.
120 status_forcelist Optional[Collection] A set of integer HTTP status codes that we should force a retry on.
(502, 503, 504) Usage from caveclient import set_session_defaults\n\nset_session_defaults(\n max_retries=5, # would increase the default number of retries\n backoff_factor=0.5, # would increase the default backoff factor between retries\n backoff_max=240, # would increase the default maximum backoff time\n status_forcelist=(502, 503, 504, 505), # would add 505 to the default list\n)\n\nset_session_defaults() # would revert all defaults to their original values\nCalling this function will set the default values for all clients created after the call.
Calling this function with any arguments missing will reset that value to the default value.
See Also:urllib3.util.Retry
requests.adapters.HTTPAdapter
"},{"location":"api/config/#caveclient.get_session_defaults","title":"caveclient.get_session_defaults()","text":"Get the current default values for session configuration.
Returns:
Type Descriptiondict Dictionary of current default values for session configuration.
"},{"location":"api/datastack_lookup/","title":"Datastack lookup","text":""},{"location":"api/datastack_lookup/#caveclient.datastack_lookup.reset_server_address_cache","title":"caveclient.datastack_lookup.reset_server_address_cache(datastack, filename=None)","text":"Remove one or more datastacks from the datastack-to-server cache.
Parameters:
Name Type Description Defaultdatastack str or list of str Datastack names to remove from the cache, by default None
requiredfilename str Name of the cache file, by default None
None"},{"location":"api/info/","title":"client.info","text":"Client for interacting with the info service.
Methods:
Name Descriptionannotation_endpoint AnnotationEngine endpoint for a dataset.
get_aligned_volume_info Gets the info record for a aligned_volume
get_datastack_info Gets the info record for a datastack
get_datastacks Query which datastacks are available at the info service
get_datastacks_by_aligned_volume Lookup what datastacks are associated with this aligned volume
image_cloudvolume Generate a cloudvolume instance based on the image source, using authentication if needed and
image_source Cloud path to the imagery for the dataset
refresh_stored_data Reload the stored info values from the server.
segmentation_cloudvolume Generate a cloudvolume instance based on the segmentation source, using authentication if needed and
segmentation_source Cloud path to the chunkgraph-backed Graphene segmentation for a dataset
synapse_segmentation_source Cloud path to the synapse segmentation for a dataset
viewer_resolution Get the viewer resolution metadata for this datastack
viewer_site Get the base Neuroglancer URL for the dataset
Attributes:
Name Type Descriptionserver_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.annotation_endpoint","title":"annotation_endpoint(datastack_name=None, use_stored=True)","text":"AnnotationEngine endpoint for a dataset.
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the datastack to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True Returns:
Type Descriptionstr Location of the AnnotationEngine
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_aligned_volume_info","title":"get_aligned_volume_info(datastack_name=None, use_stored=True)","text":"Gets the info record for a aligned_volume
Parameters:
Name Type Description Defaultdatastack_name str datastack_name to look up. If None, uses the one specified by the client. By default None
None use_stored bool If True and the information has already been queried for that dataset, then uses the cached version. If False, re-queries the infromation. By default True
True Returns:
Type Descriptiondict or None The complete info record for the aligned_volume
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastack_info","title":"get_datastack_info(datastack_name=None, use_stored=True)","text":"Gets the info record for a datastack
Parameters:
Name Type Description Defaultdatastack_name str datastack to look up. If None, uses the one specified by the client. By default None
None use_stored bool If True and the information has already been queried for that datastack, then uses the cached version. If False, re-queries the infromation. By default True
True Returns:
Type Descriptiondict or None The complete info record for the datastack
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastacks","title":"get_datastacks()","text":"Query which datastacks are available at the info service
Returns:
Type Descriptionlist List of datastack names
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastacks_by_aligned_volume","title":"get_datastacks_by_aligned_volume(aligned_volume=None)","text":"Lookup what datastacks are associated with this aligned volume
Args: aligned_volume (str, optional): aligned volume to lookup. Defaults to None.
Raises: ValueError: if no aligned volume is specified
Returns: list: a list of datastack string
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.image_cloudvolume","title":"image_cloudvolume(**kwargs)","text":"Generate a cloudvolume instance based on the image source, using authentication if needed and sensible default values for reading CAVE resources. By default, fill_missing is True and bounded is False. All keyword arguments are passed onto the CloudVolume initialization function, and defaults can be overridden.
Requires cloudvolume to be installed, which is not included by default.
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.image_source","title":"image_source(datastack_name=None, use_stored=True, format_for='raw')","text":"Cloud path to the imagery for the dataset
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the datastack to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True format_for 'raw', 'cloudvolume', or 'neuroglancer' Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"precomputed://gs://\" type path is converted to a full https URL. If 'neuroglancer', a full https URL is converted to a \"precomputed://gs://\" type path.
'raw' Returns:
Type Descriptionstr Formatted cloud path to the flat segmentation
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.refresh_stored_data","title":"refresh_stored_data()","text":"Reload the stored info values from the server.
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.segmentation_cloudvolume","title":"segmentation_cloudvolume(use_client_secret=True, **kwargs)","text":"Generate a cloudvolume instance based on the segmentation source, using authentication if needed and sensible default values for reading CAVE resources. By default, fill_missing is True and bounded is False. All keyword arguments are passed onto the CloudVolume initialization function, and defaults can be overridden.
Requires cloudvolume to be installed, which is not included by default.
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.segmentation_source","title":"segmentation_source(datastack_name=None, format_for='raw', use_stored=True)","text":"Cloud path to the chunkgraph-backed Graphene segmentation for a dataset
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the datastack to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True format_for 'raw', 'cloudvolume', or 'neuroglancer' Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"graphene://https://\" type path is used If 'neuroglancer', a \"graphene://https://\" type path is used, as needed by Neuroglancer.
'raw' Returns:
Type Descriptionstr Formatted cloud path to the Graphene segmentation
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.synapse_segmentation_source","title":"synapse_segmentation_source(datastack_name=None, use_stored=True, format_for='raw')","text":"Cloud path to the synapse segmentation for a dataset
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the dataset to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True format_for 'raw', 'cloudvolume', or 'neuroglancer' Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"precomputed://gs://\" type path is converted to a full https URL. If 'neuroglancer', a full https URL is converted to a \"precomputed://gs://\" type path.
'raw' Returns:
Type Descriptionstr Formatted cloud path to the synapse segmentation
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.viewer_resolution","title":"viewer_resolution(datastack_name=None, use_stored=True)","text":"Get the viewer resolution metadata for this datastack
Parameters:
Name Type Description Defaultdatastack_name If None use the default one configured in the client
None use_stored Use the cached value, if False go get a new value from server
True Returns:
Type Descriptionarray Voxel resolution as a len(3) np.array
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.viewer_site","title":"viewer_site(datastack_name=None, use_stored=True)","text":"Get the base Neuroglancer URL for the dataset
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the datastack to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True Returns:
Type Descriptionstr Base URL for the Neuroglancer viewer
"},{"location":"api/l2cache/","title":"client.l2cache","text":"Client for interacting with the level2 cache service.
Methods:
Name Descriptioncache_metadata Retrieves the meta data for the cache
get_l2data Gets the attributed statistics data for L2 ids.
get_l2data_table Gets the attributed statistics data for L2 ids, returned as a dataframe.
has_cache Checks if the l2 cache is available for the dataset
table_mapping Retrieves table mappings for l2 cache.
Attributes:
Name Type Descriptionserver_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.cache_metadata","title":"cache_metadata()","text":"Retrieves the meta data for the cache
Returns:
Type Descriptiondict keys are attribute names, values are datatypes
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.get_l2data","title":"get_l2data(l2_ids, attributes=None)","text":"Gets the attributed statistics data for L2 ids.
Parameters:
Name Type Description Defaultl2_ids list or ndarray A list of level 2 ids.
requiredattributes list A list of attributes to retrieve. Defaults to None, which will return all that are available. Available stats are:
area_nm2chunk_intersect_countmax_dt_nmmean_dt_nmpcapca_valrep_coord_nmsize_nm3See the tutorial for more description.
None Returns:
Type Descriptiondict Keys are l2 ids, values are data.
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.get_l2data_table","title":"get_l2data_table(l2_ids, attributes=None, split_columns=True)","text":"Gets the attributed statistics data for L2 ids, returned as a dataframe.
Parameters:
Name Type Description Defaultl2_ids list or ndarray A list of level 2 ids.
requiredattributes list A list of attributes to retrieve. Defaults to None, which will return all that are available. Available stats are:
area_nm2chunk_intersect_countmax_dt_nmmean_dt_nmpcapca_valrep_coord_nmsize_nm3See the tutorial for more description.
None split_columns bool Whether to split columns with multiple values into separate columns.
True Returns:
Type DescriptionDataFrame A pandas dataframe with the requested attributes as columns and indexed by l2_id.
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.has_cache","title":"has_cache(datastack_name=None)","text":"Checks if the l2 cache is available for the dataset
Parameters:
Name Type Description Defaultdatastack_name str The name of the datastack to check, by default None (if None, uses the client's datastack)
None Returns:
Type Descriptionbool True if the l2 cache is available, False otherwise
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.table_mapping","title":"table_mapping()","text":"Retrieves table mappings for l2 cache.
Returns:
Type Descriptiondict keys are pcg table names, values are dicts with fields l2cache_id and cv_path.
Client for interacting with the materialization engine.
Methods:
Name Descriptionget_table_metadata Get metadata about a table
get_tables Gets a list of table names for a datastack
get_tables_metadata Get metadata about tables
get_timestamp Get datetime.datetime timestamp for a materialization version.
get_unique_string_values Get unique string values for a table
get_version_metadata Get metadata about a version
get_versions Get the versions available
get_versions_metadata Get the metadata for all the versions that are presently available and valid
get_view_metadata Get metadata for a view
get_view_schema Get schema for a view
get_view_schemas Get schema for a view
get_views Get all available views for a version
ingest_annotation_table Trigger supervoxel lookup and root ID lookup of new annotations in a table.
join_query Generic query on materialization tables
live_live_query Beta method for querying cave annotation tables with root IDs and annotations
live_query Generic query on materialization tables
lookup_supervoxel_ids Trigger supervoxel lookups of new annotations in a table.
map_filters Translate a list of filter dictionaries from a point in the
most_recent_version Get the most recent version of materialization for this datastack name
query_table Generic query on materialization tables
query_view Generic query on a view
synapse_query Convenience method for querying synapses.
Attributes:
Name Type Descriptioncg_client The chunked graph client.
datastack_name The name of the datastack.
homepage HTML The homepage for the materialization engine.
server_version Optional[Version] The version of the service running on the remote server. Note that this
tables TableManager The table manager for the materialization engine.
version int The version of the materialization. Can be used to set up the
views ViewManager The view manager for the materialization engine.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.cg_client","title":"cg_client property","text":"The chunked graph client.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.datastack_name","title":"datastack_name property","text":"The name of the datastack.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.homepage","title":"homepage: HTML property","text":"The homepage for the materialization engine.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.tables","title":"tables: TableManager property","text":"The table manager for the materialization engine.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.version","title":"version: int property writable","text":"The version of the materialization. Can be used to set up the client to default to a specific version when timestamps or versions are not specified in queries. If not set, defaults to the most recent version.
Note that if this materialization client is attached to a CAVEclient, the version must be set at the CAVEclient level.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.views","title":"views: ViewManager property","text":"The view manager for the materialization engine.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_table_metadata","title":"get_table_metadata(table_name, datastack_name=None, version=None, log_warning=True)","text":"Get metadata about a table
Parameters:
Name Type Description Defaulttable_name str name of table to mark for deletion
requireddatastack_name str or None Name of the datastack_name. If None, uses the one specified in the client.
None version int The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None log_warning bool Whether to print out warnings to the logger. Defaults to True.
True Returns:
Type Descriptiondict Metadata dictionary for table
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_tables","title":"get_tables(datastack_name=None, version=None)","text":"Gets a list of table names for a datastack
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the datastack, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client
None version int or None The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None Returns:
Type Descriptionlist List of table names
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_tables_metadata","title":"get_tables_metadata(datastack_name=None, version=None, log_warning=True)","text":"Get metadata about tables
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the datastack_name. If None, uses the one specified in the client.
None version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None log_warning bool Whether to print out warnings to the logger. Defaults to True.
True Returns:
Type Descriptiondict Metadata dictionary for table
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_timestamp","title":"get_timestamp(version=None, datastack_name=None)","text":"Get datetime.datetime timestamp for a materialization version.
Parameters:
Name Type Description Defaultversion int or None The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None datastack_name str or None Datastack name, by default None. If None, defaults to the value set in the client.
None Returns:
Type Descriptiondatetime Datetime when the materialization version was frozen.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_unique_string_values","title":"get_unique_string_values(table, datastack_name=None)","text":"Get unique string values for a table
Parameters:
Name Type Description Defaulttable str Table to query
requireddatastack_name Optional[str] Datastack to query. If None, uses the one specified in the client.
None Returns:
Type Descriptiondict[str] A dictionary of column names and their unique values
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_version_metadata","title":"get_version_metadata(version=None, datastack_name=None)","text":"Get metadata about a version
Parameters:
Name Type Description Defaultversion int or None The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None datastack_name str or None Datastack name, by default None. If None, defaults to the value set in the client.
None Returns:
Type Descriptiondict Dictionary of metadata about the version
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_versions","title":"get_versions(datastack_name=None, expired=False)","text":"Get the versions available
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the datastack, by default None. If None, uses the one specified in the client.
None expired bool Whether to include expired versions, by default False.
False Returns:
Type Descriptiondict Dictionary of versions available
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_versions_metadata","title":"get_versions_metadata(datastack_name=None, expired=False)","text":"Get the metadata for all the versions that are presently available and valid
Parameters:
Name Type Description Defaultdatastack_name str or None Datastack name, by default None. If None, defaults to the value set in the client.
None expired bool Whether to include expired versions, by default False.
False Returns:
Type Descriptionlist[dict] List of metadata dictionaries
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_metadata","title":"get_view_metadata(view_name, materialization_version=None, datastack_name=None, log_warning=True)","text":"Get metadata for a view
Parameters:
Name Type Description Defaultview_name str Name of view to query.
requiredmaterialization_version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None log_warning bool Whether to log warnings.
True Returns:
Type Descriptiondict Metadata of view
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_schema","title":"get_view_schema(view_name, materialization_version=None, datastack_name=None, log_warning=True)","text":"Get schema for a view
Parameters:
Name Type Description Defaultview_name str Name of view to query.
requiredmaterialization_version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None log_warning bool Whether to log warnings.
True Returns:
Type Descriptiondict Schema of view.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_schemas","title":"get_view_schemas(materialization_version=None, datastack_name=None, log_warning=True)","text":"Get schema for a view
Parameters:
Name Type Description Defaultmaterialization_version Optional[int] Version to query. If None, will use version set by client.
None log_warning bool Whether to log warnings.
True Returns:
Type Descriptiondict Schema of view.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_views","title":"get_views(version=None, datastack_name=None)","text":"Get all available views for a version
Parameters:
Name Type Description Defaultversion Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None datastack_name str Datastack to query. If None, uses the one specified in the client.
None Returns:
Type Descriptionlist List of views
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.ingest_annotation_table","title":"ingest_annotation_table(table_name, datastack_name=None)","text":"Trigger supervoxel lookup and root ID lookup of new annotations in a table.
Parameters:
Name Type Description Defaulttable_name str Table to trigger
requireddatastack_name str Datastack to trigger it. Defaults to what is set in client.
None Returns:
Type Descriptiondict Status code of response from server
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.join_query","title":"join_query(tables, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, suffixes=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, metadata=True, desired_resolution=None, random_sample=None, log_warning=True)","text":"Generic query on materialization tables
Parameters:
Name Type Description Defaulttables list of lists with length 2 or 'str' list of two lists: first entries are table names, second entries are the columns used for the join.
requiredfilter_in_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are allowed entries, by default None
None filter_out_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are not allowed entries, by default None
None filter_equal_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are specified entry, by default None
None filter_greater_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are exclusive upper-bound, by default None
None filter_less_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are exclusive lower-bound, by default None
None filter_greater_equal_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are inclusive upper-bound, by default None
None filter_less_equal_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are inclusive lower-bound, by default None
None filter_spatial_dict dict of dicts outer layer: keys are table names, inner layer: keys are column names. Values are bounding boxes as [[min_x, min_y,min_z],[max_x, max_y, max_z]], expressed in units of the voxel_resolution of this dataset. Defaults to None.
None filter_regex_dict dict of dicts outer layer: keys are table names. inner layer: keys are column names, values are regex strings. Defaults to None
None select_columns dict of lists of str keys are table names,values are the list of columns from that table. Defaults to None, which will select all tables. Will be passed to server as select_column_maps. Passing a list will be passed as select_columns which is deprecated.
None offset int result offset to use. Defaults to None. Will only return top K results.
None limit int maximum results to return (server will set upper limit, see get_server_config)
None suffixes dict suffixes to use for duplicate columns, keys are table names, values are the suffix
None datastack_name str datastack to query. If None defaults to one specified in client.
None return_df bool whether to return as a dataframe default True, if False, data is returned as json (slower)
True split_positions bool whether to break position columns into x,y,z columns default False, if False data is returned as one column with [x,y,z] array (slower)
False materialization_version int The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None metadata bool toggle to return metadata If True (and return_df is also True), return table and query metadata in the df.attr dictionary.
True desired_resolution Iterable What resolution to convert position columns to. Defaults to None will use defaults.
None random_sample int if given, will do a tablesample of the table to return that many annotations
None log_warning bool Whether to log warnings, by default True
True Returns:
Type DescriptionDataFrame a pandas dataframe of results of query
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.live_live_query","title":"live_live_query(table, timestamp, joins=None, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, split_positions=False, metadata=True, suffixes=None, desired_resolution=None, allow_missing_lookups=False, allow_invalid_root_ids=False, random_sample=None, log_warning=True)","text":"Beta method for querying cave annotation tables with root IDs and annotations at a particular timestamp. Note: this method requires more explicit mapping of filters and selection to table as its designed to test a more general endpoint that should eventually support complex joins.
Parameters:
Name Type Description Defaulttable str Principle table to query
requiredtimestamp datetime Timestamp to query
requiredjoins List of joins, where each join is a list of [table1,column1, table2, column2]
None filter_in_dict A dictionary with tables as keys, values are dicts with column keys and list values to accept.
None filter_out_dict A dictionary with tables as keys, values are dicts with column keys and list values to reject.
None filter_equal_dict A dictionary with tables as keys, values are dicts with column keys and values to equate.
None filter_greater_dict A dictionary with tables as keys, values are dicts with column keys and values as exclusive upper-bound.
None filter_less_dict A dictionary with tables as keys, values are dicts with column keys and values as exclusive lower-bound.
None filter_greater_equal_dict A dictionary with tables as keys, values are dicts with column keys and values as inclusive upper-bound.
None filter_less_equal_dict A dictionary with tables as keys, values are dicts with column keys and values as inclusive lower-bound.
None filter_spatial_dict A dictionary with tables as keys, values are dicts with column keys and values of 2x3 list of bounds.
None filter_regex_dict A dictionary with tables as keys, values are dicts with column keys and values of regex strings.
None select_columns A dictionary with tables as keys, values are lists of columns to select.
None offset int Value to offset query by.
None limit int Limit of query.
None datastack_name str Datastack to query. Defaults to set by client.
None split_positions bool Whether to split positions into separate columns, True is faster.
False metadata bool Whether to attach metadata to dataframe.
True suffixes dict What suffixes to use on joins, keys are table_names, values are suffixes.
None desired_resolution Iterable What resolution to convert position columns to.
None allow_missing_lookups bool If there are annotations without supervoxels and root IDs yet, allow results.
False allow_invalid_root_ids bool If True, ignore root ids not valid at the given timestamp, otherwise raise an error.
False random_sample int If given, will do a table sample of the table to return that many annotations.
None log_warning bool Whether to log warnings.
True Returns:
Type DescriptionResults of query
Examples:
>>> from caveclient import CAVEclient\n>>> client = CAVEclient('minnie65_public_v117')\n>>> live_live_query(\"table_name\", datetime.datetime.now(datetime.timezone.utc),\n>>> joins=[[table_name, table_column, joined_table, joined_column],\n>>> [joined_table, joincol2, third_table, joincol_third]]\n>>> suffixes={\n>>> \"table_name\":\"suffix1\",\n>>> \"joined_table\":\"suffix2\",\n>>> \"third_table\":\"suffix3\"\n>>> },\n>>> select_columns= {\n>>> \"table_name\":[ \"column\",\"names\"],\n>>> \"joined_table\":[\"joined_colum\"]\n>>> },\n>>> filter_in_dict= {\n>>> \"table_name\":{\n>>> \"column_name\":[included,values]\n>>> }\n>>> },\n>>> filter_out_dict= {\n>>> \"table_name\":{\n>>> \"column_name\":[excluded,values]\n>>> }\n>>> },\n>>> filter_equal_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_greater_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_less_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_greater_equal_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_less_equal_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_spatial_dict\"= {\n>>> \"table_name\": {\n>>> \"column_name\": [[min_x, min_y, min_z], [max_x, max_y, max_z]]\n>>> }\n>>> filter_regex_dict\"= {\n>>> \"table_name\": {\n>>> \"column_name\": \"regex_string\"\n>>> }\nlive_query(table, timestamp, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, split_positions=False, post_filter=True, metadata=True, merge_reference=True, desired_resolution=None, random_sample=None, log_warning=True)","text":"Generic query on materialization tables
Parameters:
Name Type Description Defaulttable str Table to query
requiredtimestamp datetime Time to materialize (in utc). Pass datetime.datetime.now(datetime.timezone.utc) for present time.
requiredfilter_in_dict dict Keys are column names, values are allowed entries.
None filter_out_dict dict Keys are column names, values are not allowed entries.
None filter_equal_dict dict Keys are column names, values are specified entry.
None filter_greater_dict dict Keys are column names, values are exclusive upper-bounds.
None filter_less_dict dict Keys are column names, values are exclusive lower-bounds.
None filter_greater_equal_dict dict Keys are column names, values are inclusive upper-bounds.
None filter_less_equal_dict dict Keys are column names, values are inclusive lower-bounds.
None filter_spatial_dict dict Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]].
None filter_regex_dict dict Keys are column names, values are regex strings.
None select_columns list of str Columns to select.
None offset int Offset in query result.
None limit int Maximum results to return (server will set upper limit, see get_server_config).
None datastack_name str Datastack to query. If None, defaults to one specified in client.
None split_positions bool Whether to break position columns into x,y,z columns. If False data is returned as one column with [x,y,z] array (slower).
False post_filter bool Whether to filter down the result based upon the filters specified. If False, it will return the query with present root_ids in the root_id columns, but the rows will reflect the filters translated into their past IDs. So if, for example, a cell had a false merger split off since the last materialization, those annotations on that incorrect portion of the cell will be included if this is False, but will be filtered down if this is True.
True metadata bool Toggle to return metadata. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.
True merge_reference bool Toggle to automatically join reference table. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that table.
True desired_resolution Iterable Desired resolution you want all spatial points returned in. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata.
None random_sample int If given, will do a tablesample of the table to return that many annotations.
None log_warning bool Whether to log warnings.
True Returns:
Type DescriptionDataFrame A pandas dataframe of results of query
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.lookup_supervoxel_ids","title":"lookup_supervoxel_ids(table_name, annotation_ids=None, datastack_name=None)","text":"Trigger supervoxel lookups of new annotations in a table.
Parameters:
Name Type Description Defaulttable_name str Table to trigger
requiredannotation_ids list List of annotation ids to lookup. Default is None, which will trigger lookup of entire table.
None datastack_name str Datastack to trigger it. Defaults to what is set in client.
None Returns:
Type Descriptiondict Status code of response from server
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.map_filters","title":"map_filters(filters, timestamp, timestamp_past)","text":"Translate a list of filter dictionaries from a point in the future to a point in the past
Parameters:
Name Type Description Defaultfilters list[dict] filter dictionaries with root_ids
requiredtimestamp datetime timestamp to query
requiredtimestamp_past datetime timestamp to query from
requiredReturns:
Type Descriptionlist[dict] filter dictionaries with past root_ids
dict mapping of future root_ids to past root_ids
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.most_recent_version","title":"most_recent_version(datastack_name=None)","text":"Get the most recent version of materialization for this datastack name
Parameters:
Name Type Description Defaultdatastack_name str or None Name of the datastack, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client
None Returns:
Type Descriptionint Most recent version of materialization for this datastack name
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.query_table","title":"query_table(table, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, timestamp=None, metadata=True, merge_reference=True, desired_resolution=None, get_counts=False, random_sample=None, log_warning=True)","text":"Generic query on materialization tables
Parameters:
Name Type Description Defaulttable str Table to query
requiredfilter_in_dict dict Keys are column names, values are allowed entries, by default None
None filter_out_dict dict Keys are column names, values are not allowed entries, by default None
None filter_equal_dict dict Keys are column names, values are specified entry, by default None
None filter_greater_dict dict Keys are column names, values are exclusive upper-bound value, by default None
None filter_less_dict dict Keys are column names, values are exclusive lower-bound value, by default None
None filter_greater_equal_dict dict Keys are column names, values are inclusive upper-bound value, by default None
None filter_less_equal_dict dict Keys are column names, values are inclusive lower-bound value, by default None
None filter_spatial_dict dict Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]], by default None
None filter_regex_dict dict Keys are column names, values are regex strings, by default None
None select_columns list of str Columns to select, by default None
None offset int Result offset to use, by default None. Will only return top K results.
None limit int Maximum results to return (server will set upper limit, see get_server_config), by default None
None datastack_name str Datastack to query, by default None. If None, defaults to one specified in client.
None return_df bool Whether to return as a dataframe, by default True. If False, data is returned as json (slower).
True split_positions bool Whether to break position columns into x,y,z columns, by default False. If False data is returned as one column with [x,y,z] array (slower)
False materialization_version int The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None timestamp datetime Timestamp to query, by default None. If passsed will do a live query. Error if also passing a materialization version
None metadata bool Toggle to return metadata (default True), by default True. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.
True merge_reference bool Toggle to automatically join reference table, by default True. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that
True desired_resolution Iterable[float] Desired resolution you want all spatial points returned in, by default None. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata
None get_counts bool Whether to get counts of the query, by default False
False random_sample int If given, will do a tablesample of the of the table to return that many annotations
None log_warning bool Whether to log warnings, by default True
True Returns:
Type DescriptionDataFrame A pandas dataframe of results of query
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.query_view","title":"query_view(view_name, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, metadata=True, merge_reference=True, desired_resolution=None, get_counts=False, random_sample=None)","text":"Generic query on a view
Parameters:
Name Type Description Defaultview_name str View to query
requiredfilter_in_dict dict Keys are column names, values are allowed entries, by default None
None filter_out_dict dict Keys are column names, values are not allowed entries, by default None
None filter_equal_dict dict Keys are column names, values are specified entry, by default None
None filter_greater_dict dict Keys are column names, values are exclusive upper-bound, by default None
None filter_less_dict dict Keys are column names, values are exclusive lower-bound, by default None
None filter_greater_equal_dict dict Keys are column names, values are inclusive upper-bound, by default None
None filter_less_equal_dict dict Keys are column names, values are inclusive lower-bound, by default None
None filter_spatial_dict dict Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]], by default None
None filter_regex_dict dict Keys are column names, values are regex strings, by default None
None select_columns list of str Columns to select, by default None
None offset int Result offset to use, by default None. Will only return top K results.
None limit int Maximum results to return (server will set upper limit, see get_server_config), by default None
None datastack_name str Datastack to query, by default None. If None, defaults to one specified in client.
None return_df bool Whether to return as a dataframe, by default True. If False, data is returned as json (slower).
True split_positions bool Whether to break position columns into x,y,z columns, by default False. If False data is returned as one column with [x,y,z] array (slower)
False materialization_version int The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None metadata bool Toggle to return metadata (default True), by default True. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.
True merge_reference bool Toggle to automatically join reference table, by default True. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that
True desired_resolution Iterable[float] Desired resolution you want all spatial points returned in, by default None. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata
None get_counts bool Whether to get counts of the query, by default False
False random_sample int If given, will do a tablesample of the of the table to return that many annotations
None Returns:
Type DescriptionDataFrame A pandas dataframe of results of query
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.synapse_query","title":"synapse_query(pre_ids=None, post_ids=None, bounding_box=None, bounding_box_column='post_pt_position', timestamp=None, remove_autapses=True, include_zeros=True, limit=None, offset=None, split_positions=False, desired_resolution=None, materialization_version=None, synapse_table=None, datastack_name=None, metadata=True)","text":"Convenience method for querying synapses.
Will use the synapse table specified in the info service by default. It will also remove autapses by default. NOTE: This is not designed to allow querying of the entire synapse table. A query with no filters will return only a limited number of rows (configured by the server) and will do so in a non-deterministic fashion. Please contact your dataset administrator if you want access to the entire table.
Parameters:
Name Type Description Defaultpre_ids Union[int, Iterable, ndarray] Pre-synaptic cell(s) to query.
None post_ids Union[int, Iterable, ndarray] Post-synaptic cell(s) to query.
None bounding_box Optional[Union[list, ndarray]] [[min_x, min_y, min_z],[max_x, max_y, max_z]] bounding box to filter synapse locations. Expressed in units of the voxel_resolution of this dataset.
None bounding_box_column str Which synapse location column to filter by.
'post_pt_position' timestamp datetime Timestamp to query. If passed recalculate query at timestamp, do not pass with materialization_version.
None remove_autapses bool Whether to remove autapses from query results.
True include_zeros bool Whether to include synapses to/from id=0 (out of segmentation).
True limit int Number of synapses to limit. Server-side limit still applies.
None offset int Number of synapses to offset query.
None split_positions bool Whether to split positions into separate columns, True is faster.
False desired_resolution Iterable[float] List or array of the desired resolution you want queries returned in useful for materialization queries.
None materialization_version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None metadata bool Whether to attach metadata to dataframe in the df.attr dictionary.
True Returns:
Type DescriptionDataFrame Results of query.
"},{"location":"api/schema/","title":"client.schema","text":"Client for interacting with the schema service.
Methods:
Name Descriptionget_schemas Get the available schema types
schema_definition Get the definition of a specified schema_type
schema_definition_all Get the definition of all schema_types
schema_definition_multi Get the definition of multiple schema_types
Attributes:
Name Type Descriptionserver_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.get_schemas","title":"get_schemas()","text":"Get the available schema types
Returns:
Type Descriptionlist List of schema types available on the Schema service.
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition","title":"schema_definition(schema_type)","text":"Get the definition of a specified schema_type
Parameters:
Name Type Description Defaultschema_type str Name of a schema_type
requiredReturns:
Type Descriptionjson Schema definition
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition_all","title":"schema_definition_all()","text":"Get the definition of all schema_types
Returns:
Type Descriptiondict Dictionary of schema definitions. Keys are schema names, values are definitions.
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition_multi","title":"schema_definition_multi(schema_types)","text":"Get the definition of multiple schema_types
Parameters:
Name Type Description Defaultschema_types list List of schema names
requiredReturns:
Type Descriptiondict Dictionary of schema definitions. Keys are schema names, values are definitions.
"},{"location":"api/skeleton/","title":"client.skeleton","text":"Client for interacting with the skeleton service.
Methods:
Name Descriptionget_skeleton Gets basic skeleton information for a datastack
get_cache_contents Mirror CloudFiles.list() for skeletons as a pass-through interface to the underlying service and bucket.
skeletons_exist Confirm or deny that a set of root ids have H5 skeletons in the cache.
get_bulk_skeletons Generates skeletons for a list of root ids without retrieving them.
generate_bulk_skeletons_async Generates skeletons for a list of root ids without retrieving them.
Attributes:
Name Type Descriptionserver_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.get_skeleton","title":"get_skeleton(root_id, datastack_name=None, skeleton_version=3, output_format='dict', log_warning=True, verbose_level=0)","text":"Gets basic skeleton information for a datastack
Parameters:
Name Type Description Defaultroot_id int The root id of the skeleton to retrieve
requireddatastack_name str The name of the datastack to check
None skeleton_version int The skeleton version to generate and retrieve. Options are documented in SkeletonService. Use 0 for Neuroglancer-compatibility. Use -1 for latest.
3 output_format string The format to retrieve. Options are: - 'dict': A dictionary - 'swc': A pandas DataFrame
'dict' Returns:
Type DescriptionSkeleton of the requested type. See output_format for details.
get_cache_contents(datastack_name=None, skeleton_version=3, root_id_prefixes=0, limit=0, log_warning=True)","text":"Mirror CloudFiles.list() for skeletons as a pass-through interface to the underlying service and bucket.
"},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.skeletons_exist","title":"skeletons_exist(datastack_name=None, skeleton_version=3, root_ids=0, log_warning=True)","text":"Confirm or deny that a set of root ids have H5 skeletons in the cache.
"},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.get_bulk_skeletons","title":"get_bulk_skeletons(root_ids, datastack_name=None, skeleton_version=3, output_format='dict', generate_missing_skeletons=False, log_warning=True, verbose_level=0)","text":"Generates skeletons for a list of root ids without retrieving them.
Parameters:
Name Type Description Defaultroot_ids List A list of root ids of the skeletons to generate
requireddatastack_name str The name of the datastack to check
None skeleton_version int The skeleton version to generate. Use 0 for Neuroglancer-compatibility. Use -1 for latest.
3"},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.generate_bulk_skeletons_async","title":"generate_bulk_skeletons_async(root_ids, datastack_name=None, skeleton_version=None, log_warning=True, verbose_level=0)","text":"Generates skeletons for a list of root ids without retrieving them.
Parameters:
Name Type Description Defaultroot_ids List A list of root ids of the skeletons to generate
requireddatastack_name str The name of the datastack to check
None skeleton_version int The skeleton version to generate. Use 0 for Neuroglancer-compatibility. Use -1 for latest.
None Returns:
Type Descriptionfloat The estimated time in seconds to generate all skeletons (a comparable message will be output to the console prior to return).
"},{"location":"api/state/","title":"client.state","text":"Client to interface with the JSON state service.
Methods:
Name Descriptionbuild_neuroglancer_url Build a URL for a Neuroglancer deployment that will automatically retrieve specified state.
get_neuroglancer_info Get the info field from a Neuroglancer deployment
get_property_json Download a Neuroglancer JSON state
get_state_json Download a Neuroglancer JSON state
save_state_json_local Save a Neuroglancer JSON state to a JSON file locally.
upload_property_json Upload a Neuroglancer JSON state
upload_state_json Upload a Neuroglancer JSON state
Attributes:
Name Type Descriptionserver_version Optional[Version] The version of the service running on the remote server. Note that this
state_service_endpoint str Endpoint URL for posting JSON state
"},{"location":"api/state/#caveclient.jsonservice.JSONService.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/state/#caveclient.jsonservice.JSONService.state_service_endpoint","title":"state_service_endpoint: str property","text":"Endpoint URL for posting JSON state
"},{"location":"api/state/#caveclient.jsonservice.JSONService.build_neuroglancer_url","title":"build_neuroglancer_url(state_id, ngl_url=None, target_site=None, static_url=False, format_properties=False)","text":"Build a URL for a Neuroglancer deployment that will automatically retrieve specified state. If the datastack is specified, this is prepopulated from the info file field \"viewer_site\". If no ngl_url is specified in either the function or the client, a fallback neuroglancer deployment is used.
Parameters:
Name Type Description Defaultstate_id int State id to retrieve
requiredngl_url str Base url of a neuroglancer deployment. If None, defaults to the value for the datastack or the client. As a fallback, a default deployment is used.
None target_site seunglab or cave - explorer or mainline or None Set this to 'seunglab' for a seunglab deployment, or either 'cave-explorer'/'mainline' for a google main branch deployment. If None, checks the info field of the neuroglancer endpoint to determine which to use. Default is None.
None static_url bool If True, treats \"state_id\" as a static URL directly to the JSON and does not use the state service.
False format_properties bool If True, formats the url as a segment_properties info file
False Returns:
Type Descriptionstr The full URL requested
"},{"location":"api/state/#caveclient.jsonservice.JSONService.get_neuroglancer_info","title":"get_neuroglancer_info(ngl_url=None)","text":"Get the info field from a Neuroglancer deployment
Parameters:
Name Type Description Defaultngl_url str(optional) URL to a Neuroglancer deployment. If None, defaults to the value for the datastack or the client.
None Returns:
Type Descriptiondict JSON-formatted info field from the Neuroglancer deployment
"},{"location":"api/state/#caveclient.jsonservice.JSONService.get_property_json","title":"get_property_json(state_id)","text":"Download a Neuroglancer JSON state
Parameters:
Name Type Description Defaultstate_id int ID of a JSON state uploaded to the state service.
requiredReturns:
Type Descriptiondict JSON specifying a Neuroglancer state.
"},{"location":"api/state/#caveclient.jsonservice.JSONService.get_state_json","title":"get_state_json(state_id)","text":"Download a Neuroglancer JSON state
Parameters:
Name Type Description Defaultstate_id int or string ID of a JSON state uploaded to the state service. If a string, treats this as the url to the json state.
requiredReturns:
Type Descriptiondict JSON specifying a Neuroglancer state.
"},{"location":"api/state/#caveclient.jsonservice.JSONService.save_state_json_local","title":"save_state_json_local(json_state, filename, overwrite=False)","text":"Save a Neuroglancer JSON state to a JSON file locally.
Parameters:
Name Type Description Defaultjson_state dict Dict representation of a neuroglancer state
requiredfilename str Filename to save the state to
requiredoverwrite bool Whether to overwrite the file if it exists. Default False.
False Returns:
Type DescriptionNone"},{"location":"api/state/#caveclient.jsonservice.JSONService.upload_property_json","title":"upload_property_json(property_json, state_id=None, max_size=2500000)","text":"Upload a Neuroglancer JSON state
Parameters:
Name Type Description Defaultpropery_json dict Dict representation of a neuroglancer segment properties json
requiredstate_id int ID of a JSON state uploaded to the state service. Using a state_id is an admin feature.
None max_size Optional[int] Maximum size in bytes for the data to upload. Default is 2.5MB. Set to None for no limit.
2500000 Returns:
Type Descriptionint state_id of the uploaded JSON state
"},{"location":"api/state/#caveclient.jsonservice.JSONService.upload_state_json","title":"upload_state_json(json_state, state_id=None, timestamp=None)","text":"Upload a Neuroglancer JSON state
Parameters:
Name Type Description Defaultjson_state dict Dict representation of a neuroglancer state
requiredstate_id int ID of a JSON state uploaded to the state service. Using a state_id is an admin feature.
None timestamp Optional[time] Timestamp for json state date. Requires state_id.
None Returns:
Type Descriptionint state_id of the uploaded JSON state
"},{"location":"api/testing/","title":"caveclient.tools.testing","text":"Functions:
Name DescriptionCAVEclientMock Created a mocked CAVEclient function for testing using the responses library to mock
get_materialiation_info Get the materialization versions and version metadata for the materialization service.
get_server_versions Get the server versions for the services used in testing.
get_server_information Generate the datastack name and server locations used in testing.
get_api_versions Get the API versions for the services used in testing.
default_info Generate a info service info file for testing
"},{"location":"api/testing/#caveclient.tools.testing.CAVEclientMock","title":"CAVEclientMock(datastack_name=None, global_server=None, local_server=None, info_file=None, chunkedgraph=False, chunkedgraph_server_version=DEFAULT_CHUNKEDGRAPH_SERVER_VERSION, chunkedgraph_api_versions=None, materialization=False, materialization_server_version=DEFAULT_MATERIALIZATION_SERVER_VERSON, materialization_api_versions=None, available_materialization_versions=None, set_version=None, set_version_metadata=None, json_service=False, json_service_server_version=DEFAULT_JSON_SERVICE_SERVER_VERSION, skeleton_service=False, skeleton_service_server_version=DEFAULT_SKELETON_SERVICE_SERVER_VERSION, schema_api_versions=None, l2cache=False, l2cache_disabled=False, global_only=False)","text":"Created a mocked CAVEclient function for testing using the responses library to mock the server responses. This function returns a drop-in replacement for the CAVEclient function that will be able to initialize itself and selected individual service clients with the selected options.
Note that the test configuration is intended to be purely for pytest purposes and should not actually result in calls to active endpoints.
Parameters:
Name Type Description Defaultdatastack_name str Name of the test datastack, by default None
None global_server str Test global server address, by default None
None local_server str Test local server address, by default None
None info_file dictionary Info service dictionary for the datastack, by default None
None chunkedgraph bool If True, configures the client to initialize a chunkedgraph subclient, by default False
False chunkedgraph_server_version str Sets the value of the chunkedgraph server version as a three-element semanatic version (e.g \"2.3.4\"), by default the value in DEFAULT_CHUNKEDGRAPH_SERVER_VERSION.
DEFAULT_CHUNKEDGRAPH_SERVER_VERSION chunkedgraph_api_versions list List of chunkedgraph API versions that the chunkedgraph client thinks exists, by default None. If None, returns the value in CHUNKEDGRAPH_API_VERSIONS.
None materialization bool If True, configures the client to initalize a materialization subclient, by default False Note that materialization being set to True will also configure the chunkedgraph client.
False materialization_server_version str Sets the value of the materialization server version as a three-element semanatic version (e.g \"2.3.4\"), by default the value in DEFAULT_MATERIALIZATION_SERVER_VERSON.
DEFAULT_MATERIALIZATION_SERVER_VERSON available_materialization_versions list List of materialization database versions that the materialization client thinks exists, by default None. If None, returns the value in DEFAULT_MATERIALIZATION_VERSONS.
None materialization_api_versions list List of materialization API versions that the materialization client thinks exists, by default None. If None, returns the value in MATERIALIZATION_API_VERSIONS.
None set_version Optional[int] If set, will set the version of the materialization server to the value of set_version, by default None. To work, this version must be in the list of available materialization versions.
None set_version_metadata Optional[dict] If set, will set the version metadata of the materialization server to the value of set_version_metadata. Default value is in DEFAULT_MATERIALIZATION_VERSION_METADATA.
None json_service bool If True, configures the client to initalize a materialization subclient, by default False
False json_service_server_version _type_ Sets the value of the json state server version as a three-element semanatic version (e.g \"2.3.4\"), by default the value in DEFAULT_JSON_SERVICE_SERVER_VERSION.
DEFAULT_JSON_SERVICE_SERVER_VERSION skeleton_service bool If True, configures the client to initalize a skeleton service subclient, by default False
False skeleton_service_server_version _type_ Sets the value of the skeleton service version as a three-element semanatic version (e.g \"2.3.4\"), by default the value in DEFAULT_SKELETON_SERVICE_SERVER_VERSION.
DEFAULT_SKELETON_SERVICE_SERVER_VERSION l2cache bool If True, configures the client to initialize an l2cache subclient, by default False Note that l2cache being set to True will also configure the chunkedgraph client.
False l2cache_disabled bool If True, allows a subclient to be initialized, but emulates a situation without an L2 cache, by default False Only used if l2cache is True.
False global_only bool If True, only initializes the global services and does not use a datastack, by default False.
False Returns:
Type DescriptionCAVEclient A mocked and initialized CAVEclient object for testing
Examples:
To make a basic pytest fixture to test chunkedgraph features with an initialized CAVEclient object in your pytest conftest.py file:
import pytest\nfrom caveclient.tools.testing import CAVEclientMock\n\ntest_datastack = \"test_stack\"\ntest_global_server = \"https://test.cave.com\"\ntest_local_server = \"https://local.cave.com\"\n\n@pytest.fixture()\ndef test_client():\n return CAVEclientMock(\n datastack_name=test_datastack,\n global_server=test_global_server,\n local_server=test_local_server,\n chunkedgraph=True,\n )\n\nYou can also create more complex fixtures with multiple services initialized and specific server versions:\n\n```python\n@pytest.fixture()\ndef fancier_test_client():\n return CAVEclientMock(\n datastack_name=test_datastack,\n global_server=test_global_server,\n local_server=test_local_server,\n chunkedgraph=True,\n chunkedgraph_server_version=\"3.0.2\",\n materialization=True,\n materialization_server_version=\"4.21.4\",\n l2cache=True,\n )\nget_materialiation_info(materialization_versions=DEFAULT_MATERIALIZATION_VERSONS, version_metadata=DEFAULT_MATERIALIZATION_VERSION_METADATA)","text":"Get the materialization versions and version metadata for the materialization service.
Parameters:
Name Type Description Defaultmaterialization_versions list List of materialization database versions that the materialization client thinks exists, by default DEFAULT_MATERIALIZATION_VERSONS.
DEFAULT_MATERIALIZATION_VERSONS version_metadata dict Version metadata for the materialization service, by default DEFAULT_MATERIALIZATION_VERSION_METADATA.
DEFAULT_MATERIALIZATION_VERSION_METADATA Returns:
Type Descriptiondict Dictionary with keys: \"materialization_versions\", \"version_metadata\".
"},{"location":"api/testing/#caveclient.tools.testing.get_server_versions","title":"get_server_versions(chunkedgraph_version=DEFAULT_CHUNKEDGRAPH_SERVER_VERSION, materialization_version=DEFAULT_MATERIALIZATION_SERVER_VERSON, skeleton_service_version=DEFAULT_SKELETON_SERVICE_SERVER_VERSION, json_service_version=DEFAULT_JSON_SERVICE_SERVER_VERSION)","text":"Get the server versions for the services used in testing.
Parameters:
Name Type Description Defaultchunkedgraph_version str Version of the chunkedgraph server, by default DEFAULT_CHUNKEDGRAPH_SERVER_VERSION.
DEFAULT_CHUNKEDGRAPH_SERVER_VERSION materialization_version str Version of the materialization server, by default DEFAULT_MATERIALIZATION_SERVER_VERSON.
DEFAULT_MATERIALIZATION_SERVER_VERSON skeleton_service_version str Version of the skeleton service server, by default DEFAULT_SKELETON_SERVICE_SERVER_VERSION.
DEFAULT_SKELETON_SERVICE_SERVER_VERSION json_service_version str Version of the json service server, by default DEFAULT_JSON_SERVICE_SERVER_VERSION.
DEFAULT_JSON_SERVICE_SERVER_VERSION Returns:
Type Descriptiondict Dictionary with keys: \"chunkedgraph_version\", \"materialization_version\", \"skeleton_service_version\", \"json_service_version\". Values are Version objects from packaging.versions.
"},{"location":"api/testing/#caveclient.tools.testing.get_server_information","title":"get_server_information(datastack_name=TEST_DATASTACK, global_server=TEST_GLOBAL_SERVER, local_server=TEST_LOCAL_SERVER)","text":"Generate the datastack name and server locations used in testing.
Parameters:
Name Type Description Defaultdatastack_name str Datastack value, by default the value in TEST_DATASTACK.
TEST_DATASTACK global_server str Server for global services, by default TEST_GLOBAL_SERVER.
TEST_GLOBAL_SERVER local_server str Server for local services, by default TEST_LOCAL_SERVER.
TEST_LOCAL_SERVER Returns:
Type Descriptiondict Dictionary with keys: \"datastack_name\", \"local_server\", \"global_server\".
"},{"location":"api/testing/#caveclient.tools.testing.get_api_versions","title":"get_api_versions(chunkedgraph_api_versions=CHUNKEDGRAPH_API_VERSIONS, materialization_api_versions=MATERIALIZATION_API_VERSIONS, schema_api_versions=SCHEMA_API_VERSIONS)","text":"Get the API versions for the services used in testing.
Parameters:
Name Type Description Defaultchunkedgraph_api_versions list List of chunkedgraph API versions that the chunkedgraph client thinks exists, by default CHUNKEDGRAPH_API_VERSIONS.
CHUNKEDGRAPH_API_VERSIONS materialization_api_versions list List of materialization API versions that the materialization client thinks exists, by default MATERIALIZATION_API_VERSIONS.
MATERIALIZATION_API_VERSIONS schema_api_versions list List of schema API versions that the schema client thinks exists, by default SCHEMA_API_VERSIONS.
SCHEMA_API_VERSIONS Returns:
Type Descriptiondict Dictionary with keys: \"chunkedgraph_api_versions\", \"materialization_api_versions\", \"schema_api_versions\".
"},{"location":"api/testing/#caveclient.tools.testing.default_info","title":"default_info(local_server=TEST_LOCAL_SERVER)","text":"Generate a info service info file for testing
Parameters:
Name Type Description Defaultlocal_server str Name of the local service, by default the value in TEST_LOCAL_SERVER.
TEST_LOCAL_SERVER Returns:
Type Descriptiondict Info file for the datastack.
"},{"location":"tutorials/","title":"Getting Started","text":"AnnotationFramework client is a package for simplifying interactions with HTML services associated with the CAVE (Connectome Annotation Versioning Engine).
For a larger introduction to CAVE and its services, see the main GitHub organization site: https://github.com/CAVEconnectome
"},{"location":"tutorials/#installation","title":"Installation","text":"The CAVEclient can be installed with pip:
$ pip install caveclient\nThe code is setup to work flexibly with any deployment of these services, but you need to specify the server_address if that address is not https://globalv1.daf-apis.com/ for each client when initializing it. Similarly, the clients can query the info service for metadata to simplify the interaction with a datastack, but you have to specify a datastack name.
"},{"location":"tutorials/advanced/","title":"Advanced Usage","text":""},{"location":"tutorials/advanced/#changing-session-configurations","title":"Changing session configurations","text":"It is possible to change details of how a client talks to the various servers it needs to interface with. For instance, the CAVEclient will attempt to retry specific failed requests to the server, but will only try a specific number of times, and will only wait specific amounts of time between retries. These values can be changed via the set_session_defaults method. For instance, to change the number of retries to 5, and to increase the delay between subsequent retries, you could do:
from caveclient import set_session_defaults\n\nset_session_defaults(max_retries=5, backoff_factor=0.5)\nNote that this needs to happen before initializing the client for this to work properly. Some of these parameters are also adjustable at the client level.
To view the current session defaults, you can use the get_session_defaults method:
from caveclient import get_session_defaults\n\nclient.get_session_defaults()\nMore information on the available parameters can be found in the API documentation.
"},{"location":"tutorials/advanced/#writing-unit-tests-using-caveclient","title":"Writing unit tests using caveclient","text":"If you have a module that relies on caveclient and you want to write unit tests for it, you can use the caveclient.testing module to help you create an initialized, mocked client with user-specified options. This helps deal with the complex conversation that caveclient has with various services when initializing itself so that you get the appropriate functionality. The function CAVEclientMock offers a number of options for how to build a client, including server versions for various services, available API versions, and more. Importantly, the module provides sensible defaults but also allows you to override them as needed. Note that using testing requires the responses to be installed, which is not a dependency of caveclient.
The CAVEclientMock function requires you to specify which services you expect to call on in order to ensure that your code is working as expected. For example, to create a pytest fixture that would utilize the chunkedgraph module, but otherwise use default nonsense test values, you could include this in your conftest.py:
import pytest\nfrom caveclient.tools.testing import CAVEclientMock\n\n@pytest.fixture()\ndef test_client():\n return CAVEclientMock(\n chunkedgraph=True,\n )\nThen, in your test module, you can use the test_client fixture to get a client. Note that after mocked initialization, the client will attempt normal network requests and thus you should mock the responses. If you only care to get a specific value back from a given function, you can use pytest-mock to mock the response to a given function call. For example, if you have a function do_something_with_roots that takes a caveclient that uses the get_roots function, you could mock the get_roots function so that it return a specific value:
from pytest_mock import mocker\nfrom conftest import test_client\nfrom my_module import do_something_with_roots\n\ndef test_get_roots(mocker, test_client):\n mocker.patch.object(test_client.chunkedgraph, 'get_roots', return_value=[1, 2, 3])\n test_data = # Something appropriate\n test_output = do_something_with_roots(test_client, test_data)\n assert test_output == # The expected answer\nNote that if you your own datastack info_file that has a different local_server address than the default value (TEST_LOCAL_SERVER variable, which defaults to https://local.cave.com) you will also need to specify a local_server with the same value as in the local_server field of your info file.
While sensible defaults are provided, you can also specify things like server versions to make sure your code works with the versions of the services you expect. For example, let's make a richer mock client that specifies the server versions for the chunkedgraph, materailization, and l2cache services:
@pytest.fixture()\ndef version_specified_client():\n return CAVEclientMock(\n chunkedgraph=True,\n chunkedgraph_server_version='3.0.1',\n materialization=True,\n materialization_server_version='2.0.0',\n l2cache=True,\n )\nNote that some services like l2cache do not currently use a server version to offer different functionality, and this value is not exposed for them currently. See the API documentation for more information.
You can also override default values like server_address or datastack_name:
@pytest.fixture()\ndef server_specified_client():\n return CAVEclientMock(\n datastack_name='my_datastack',\n server_address='http://my.server.com',\n chunkedgraph=True,\n materialization=True,\n )\nIf you want to get access to the various default values of server version, datastack name, datastack info file, and api versions, you can use functions get_server_versions, get_server_information, default_info, get_api_versions respectively. Each of these functions will return a dictionary of the values that can be used as a kwargs input into CAVEclientMock. If you specify your own override values, it will take precedence over the default value and you can just use the dictionary in your tests. See the caveclient tests for an example of how to use these functions.
The AnnotationClient is used to interact with the AnnotationEngine service to create tables from existing schema, upload new data, and download existing annotations. Note that annotations in the AnnotationEngine are not linked to any particular segmentation, and thus do not include any root ids. An annotation client is accessed with client.annotation.
A list of the existing tables for the datastack can be found with get_tables().
all_tables = client.annotation.get_tables()\nall_tables[0]\nEach table has three main properties that can be useful to know:
table_name : The table name, used to refer to it when uploading or downloading annotations. This is also passed through to the table in the Materialized database.schema_name : The name of the table's schema from EMAnnotationSchemas (see below).max_annotation_id : An upper limit on the number of annotations already contained in the table.You can download the JSON representation of a data point through the get_annotation() method. This can be useful if you need to look up information on unmaterialized data, or to see what a properly templated annotation looks like.
table_name = all_tables[0]['table_name'] # 'ais_analysis_soma'\nannotation_id = 100\nclient.annotation.get_annotation(annotation_ids=annotation_id, table_name=table_name)\nOne can create a new table with a specified schema with the create_table() method:
client.annotation.create_table(table_name='test_table',\n schema_name='microns_func_coreg',\n voxel_resolution = [1,1,1],\n description=\"some text to describe your table\")\nThe voxel resolution is the units your position columns will be uploaded in [1,1,1] would imply a nm location, where as [4,4,40] would correspond to voxels of that size. If you are uploading points from a neuroglancer session, you want this to match the units of that neuroglancer view.
Note there are some optional metadata parameters to create_table()
notice_text : This is text that will show up to users who access this data as a warning. This could be used to warn users that the data is not complete or checked yet, or to advertise that a particular publication should be cited when using this table.read_permission : one of \"PRIVATE\" which means only you can read data in this table. \"PUBLIC\" (default) which means anyone can read this table that has read permissions to this dataset. So if and only if you can read the segmentation results of this data, you can read this table. \"GROUP\" which means that you must share a common group with this user for them to be able to read. We need to make a way to discover what groups you are in and who you share groups with.write_permission: one of \"PRIVATE\" (default), which means only you can write to this table. \"PUBLIC\" which means anyone can write to this table that has write permissions to this dataset. Note although this means anyone can add data, no annotations are ever truly overwritten. \"GROUP\" which means that you must share a common group with this user for them to be able to write. We need to make a way to discover what groups you are in and who you share groups with.If you change your mind about what you want for metadata, some but not all fields can be updated with update_metadata(). This includes the description, the notice_text, and the permissions, but not the name, schema or voxel resolution.
# to update description\nclient.annotation.update_metadata(table_name='test_table',\n description=\"a new description for my table\")\n\n# to make your table readable by anybody who can read this dataset\nclient.annotation.update_metadata(table_name='test_table',\n notice_text=\"This table isn't done yet, don't trust it. Contact me\")\n\n# to make your table readable by anybody who can read this dataset\nclient.annotation.update_metadata(table_name='test_table',\n read_permisison=\"PUBLIC\")\nNew data can be generated as a dict or list of dicts following the schema and uploaded with post_annotation. For example, a microns_func_coreg point needs to have:
type set to microns_func_coregpt set to a dict with position as a key and the xyz location as a value.func_id set to an integer.The following could would create a new annotation and then upload it to the service. Note that you get back the annotation id(s) of what you uploaded.
new_data = {'type': 'microns_func_coreg',\n 'pt': {'position': [1,2,3]},\n 'func_id': 0}\nclient.annotation.post_annotation(table_name='test_table', data=[new_data])\nThere are methods to simplify annotation uploads if you have a pandas dataframe whose structure mirrors the struction of the annotation schema you want to upload
import pandas as pd\n\ndf = pd.DataFrame([{'id':0,\n 'type': 'microns_func_coreg',\n 'pt_position': [1,2,3]},\n 'func_id': 0},\n {'id':1,\n 'type': 'microns_func_coreg',\n 'pt_position': [3,2,1]},\n 'func_id': 2}])\nclient.annotation.post_annotation_df('test_table', df)\nNote that here I specified the IDs of my annotations, which you can do, but then its up to you to assure that the IDs don't collide with other IDs. If you leave them blank then the service will assign the IDs for you.
There is a similar method for updating update_annotation_df()
"},{"location":"tutorials/annotation/#staged-annotations","title":"Staged Annotations","text":"Staged annotations help ensure that the annotations you post follow the appropriate schema, both by providing guides to the field names and locally validating against a schema before uploading. The most common use case for staged annotations is to create a StagedAnnotation object for a given table, then add annotations to it individually or as a group, and finally upload to the annotation table.
To get a StagedAnnotation object, you can start with either a table name or a schema name. Here, we'll assume that there's already a table called \"my_table\" that is running a \"cell_type_local\" schema. If we want to add new annotations to the table, we simply use the table name with stage_annotations().
stage = client.annotation.stage_annotations(\"my_table\")\nThis stage object retrieves the schema for the table and hosts a local collection of annotations. Every time you add an annotation, it is immediately validated against the schema. To add an annotation, use the add method:
stage.add(\n cell_type = \"pyramidal_cell\",\n classification_system=\"excitatory\",\n pt_position=[100,100,10],\n)\nThe argument names derive from fields in the schema and you must provide all required fields. Any number of annotations can be added to the stage. A dataframe of annotations can also be added with stage.add_dataframe, and requires an exact match between column names and schema fields. The key difference between this and posting a dataframe directly is that annotations added to a StagedAnnotations are validated locally, allowing any issues to be caught before uploading.
You can see the annotations as a list of dictionary records with stage.annotation_list or as a Pandas dataframe with stage.annotation_dataframe. Finally, if you initialized the stage with a table name, this information is stored in the stage and you can simply upload it from the client.
client.annotation.upload_staged_annotations(stage)\nUpdating annotations requires knowing the annotation id of the annotation you are updating, which is not required in the schema otherwise. In order to stage updated annotations, set the update parameter to True when creating the stage.
update_stage = client.annotation.stage_annotations(\"my_table\", update=True)\nupdate_stage.add(\n id=1,\n cell_type = \"stellate_cell\",\n classification_system=\"excitatory\",\n pt_position=[100,100,10],\n)\nThe update also informs the framework client to treat the annotations as an update and it will use the appropriate methods automatically when uploading client.annotation.upload_staged_annotations.
If you want to specify ids when posting new annotations, id_field can be set to True when creating the StagedAnnotation object. This will enforce an id column but still post the data as new annotations.
If you might be adding spatial data in coordinates that might be different than the resolution for the table, you can also set the annotation_resolution when creating the stage. The stage will convert between the resolution you specify for your own annotations and the resolution that the table expects.
stage = client.annotation.stage_annotations(\"my_table\", annotation_resolution=[8,8,40])\nstage.add(\n cell_type='pyramidal_cell',\n classification_system=\"excitatory\",\n pt_position=[50,50,10],\n)\nAuthentication tokens are generally needed for programmatic access to our services. The AuthClient handles storing and loading your token or tokens and inserting it into requests in other clients.
We can access the auth client from client.auth. Once you have saved a token, you probably won't interact with this client very often, however it has some convenient features for saving new tokens the first time. Let's see if you have a token already. Probably not.
client = CAVEclient()\nauth = client.auth\nprint(f\"My current token is: {auth.token}\")\nTo get a new token, you will need to manually acquire it. For convenience, the function client.auth.get_new_token() provides instructions for how to get and save the token.
By default, the token is saved to ~/.cloudvolume/secrets/cave-secret.json as a string under the key token. This makes it compatible by default with Cloudvolume projects, which can come in handy. The following steps will save a token to the default location.
auth.get_new_token()\nnew_token = 'abcdef1234567890' #This is the text you see after you visit the website.\nauth.save_token(token=new_token)\nprint(f\"My token is now: {auth.token}\")\nNote that requesting a new token will invalidate your previous token on the same project. If you want to use the same token across different computers, you will need to share the same token information.
"},{"location":"tutorials/authentication/#loading-saved-tokens","title":"Loading saved tokens","text":"Try opening ~/.cloudvolume/secrets/cave-secret.json to see what we just created.
If we had wanted to use a different file or a different json key, we could have specified that in auth.save_token.
Because we used the default values, this token is used automatically when we initialize a new CAVEclient. If we wanted to use a different token file, token key, or even directly specify a token we could do so here.
client = CAVEclient(datastack_name)\nprint(f\"Now my basic token is: {client.auth.token}\")\n\nclient_direct = CAVEclient(datastack_name, auth_token='another_fake_token_678')\nprint(f\"A directly specified token is: {client_direct.auth.token}\")\nIf you use a CAVEclient, the AuthClient and its token will be automatically applied to any other services without further use.
"},{"location":"tutorials/chunkedgraph/","title":"Chunked Graph","text":"The chunkedgraph is a dynamic oct-tree connected components supervoxel graph.
A visual representation of an oct-tree (Wikipedia (WhiteTimberwolf) CC BY-SA 3.0)
As with any oct-tree, it is organized in hierarchical levels, with the bottom level 1 corresponding to the supervoxels of the segmentations, and the top level being the unique connected components of the supervoxel graph.
A figure illustrating the spatial chunking and editing of a the chunkedgraph. From Dorkenwald et. al 2021
The ChunkedGraph client allows one to interact with the ChunkedGraph service, which stores and updates the supervoxel agglomeration graph. This is most often useful for looking up an object root id of a supervoxel or looking up supervoxels belonging to a root id. The ChunkedGraph client is at client.chunkedgraph.
Usually in Neuroglancer, one never notices supervoxel ids, but they are important for programmatic work. In order to look up the root id for a location in space, one needs to use the supervoxel segmentation to get the associated supervoxel id. The ChunkedGraph client makes this easy using the client.chunkedgraph.get_root_id() method.
sv_id = 104200755619042523\nclient.chunkedgraph.get_root_id(supervoxel_id=sv_id)\nHowever, as proofreading occurs, the root id that a supervoxel belongs to can change. By default, this function returns the current state, however one can also provide a UTC timestamp to get the root id at a particular moment in history. This can be useful for reproducible analysis. Note below that the root id for the same supervoxel is different than it is now.
import datetime\n\n# I looked up the UTC POSIX timestamp from a day in early 2019.\ntimestamp = datetime.datetime.utcfromtimestamp(1546595253)\n\nsv_id = 104200755619042523\nclient.chunkedgraph.get_root_id(supervoxel_id=sv_id, timestamp=timestamp)\nIf you are doing this across lots of supervoxels (or any nodes) then you can do it more efficiently in one request with client.chunkedgraph.get_roots()
node_ids = [104200755619042523, 104200755619042524,104200755619042525]\nroot_ids = client.chunkedgraph.get_roots(node_ids)\nA root id is associated with a particular agglomeration of supervoxels, which can be found with the client.chunkedgraph.get_leaves() method. A new root id is generated for every new change in the chunkedgraph, so time stamps do not apply.
root_id = 648518346349541252\nclient.chunkedgraph.get_leaves(root_id)\nYou can also query the chunkedgraph not all the way to the bottom, using the stop_layer option
root_id = 648518346349541252\nclient.chunkedgraph.get_leaves(root_id,stop_layer=2)\nThis will get all the level 2 IDs for this root, which correspond to the lowest chunk of the hierarchy. An analogous option exists for client.chunkedgraph.get_roots(). This is useful to help find nodes to query within the l2cache, amongst other things.
There are a variety of other interesting functions to explore in client.chunkedgraph.
"},{"location":"tutorials/framework/","title":"Introduction","text":""},{"location":"tutorials/framework/#caveclient-one-client-for-all-services","title":"CAVEclient: one client for all services","text":"The CAVE Framework consists of a number of different services, each with a specific set of tasks that it can perform through REST endpoints. The CAVEclient is designed to ease programmatic interaction with all of the various endpoints. In addition, most programmatic access requires the use of authentication tokens. In order to collect a given server, datastack name, and user token together into a coherent package that can be used on multiple endpoints, the CAVEclient builds appropriately configured clients for each of the specific services. Each of the individual services has their own specific documentation as well.
"},{"location":"tutorials/framework/#global-and-local-services","title":"Global and Local Services","text":"There are two categories of data in CAVE: Global and local. Local services are associated with a single so-called datastack, which refers to a precise collection of imagery and segmentation data that function together. For example, EM imagery and a specific pychunkedgraph segmentation would be one datastack, while the same EM imagery but an initial static segmentation would be another. Datastacks are referred to by a short name, for instance pinky100_public_flat_v185.
Global services are those that are potentially shared across multiple different specific datastacks. These include the info service, which can describe the properties of all available datastacks, the authentication service, and the state service that hosts neuroglancer states. Global services are associated with a particular URL (by default http://globalv1.daf-apis.com), but not a single datastack.
Assuming that the services are on http://globalv1.daf-apis.com and authentication tokens are either not being used or set up with default values (see Authentication), a simple CAVEclient that can only access global services can be initialized:
from caveclient import CAVEclient\n\nclient = CAVEclient()\nJust to confirm that this works, let's see if we can get the EM image source from the InfoService. If you get a list of names of datastacks, all is good. If you have not yet set up an authentication token or you get an authentication error, look at Getting a new token for information about how to set up your auth token.
client.info.get_datastacks()\nIf you have a specific datastack you want to use, you can initialize your CAVEclient with it. This gives you access to the full range of client functions.
client = CAVEclient(datastack_name='my_datastack')\nIf your data is hosted by a different global server, you specify its address when initializing the client.
client = CAVEclient(datastack_name='my_datastack', server_address='http://global.myserver.com')\nBy default, if you pass both a server address and a datastack, the client will store the mapping from datastack to server address in the same location as the default for authentication tokens. Once stored, the client will automatically use the correct server address for the datastack if none is provided. You can override storing the server address by passing write_server_address=False. Datastacks can be removed from the cache using
caveclient.datastack_lookup.reset_server_address_cache(datastack_name).
"},{"location":"tutorials/framework/#accessing-specific-clients","title":"Accessing specific clients","text":"Each client can be accessed as a property of the main client. See the documentation at left for the capabilities of each. Assuming your client is named client, the subclients for each service are:
client.authclient.annotationclient.chunkedgraphclient.infoclient.schemasclient.stateclient.skeletonserviceA datastack has a number of complex paths to various data sources that together comprise a datastack. Rather than hardcode these paths, the InfoService allows one to query the location of each data source. This is also convenient in case data sources change.
An InfoClient is accessed at client.info.
client = CAVEclient(datastack_name)\nprint(f\"This is an info client for {client.info.datastack_name} on {client.info.server_address}\")\nAll of the information accessible for the datastack can be seen as a dict using get_datastack_info().
info.get_datastack_info()\nIndividual entries can be found as well. Use tab autocomplete to see the various possibilities.
info.graphene_source()\nBecause of the way neuroglancer looks up data versus cloudvolume, sometimes one needs to convert between gs:// style paths to https://storage.googleapis.com/ stype paths. All of the path sources in the info client accept a format_for argument that can handle this, and correctly adapts to graphene vs precomputed data sources.
neuroglancer_style_source = info.image_source(format_for='neuroglancer')\nprint(f\"With gs-style: { neuroglancer_style_source }\")\n\ncloudvolume_style_source = info.image_source(format_for='cloudvolume')\nprint(f\"With https-style: { cloudvolume_style_source }\")\nTo understand the level 2 cache, you must understand the structure of the chunkedgraph so see the chunkedgraph tutorial.
Nodes on the second level or layer of the graph, corresponds to all the supervoxels that are locally connected to one another within a single level 2 spatial \"chunk\" of the data. The Level 2 Cache, is a service whose job it is to track and update relevant statistics about every level 2 node within the a chunkedgraph. The source code of this service can be found here.
"},{"location":"tutorials/l2cache/#finding-level-2-nodes","title":"Finding Level 2 Nodes","text":"The chunkedgraph can be used to find the level2 nodes of a rootID using a stop_layer=2 keyword argument on the client.chunkedgraph.get_leaves(). Conversely the level 2 node of a supervoxel can be found using the same keyword argument of client.chunkedgraph.get_roots(). Note if you don't specify a timestamp it will give you the level2 node that is presently associated with the object.
The statistics that are available are:
Level 2 stats about nodes can be retreived using the client.l2cache.get_l2data() method. It simply takes a list of level 2 nodes you want to retrieve. Optionally you can specify only the attributes that you are interested in retrieving which will speed up the request.
"},{"location":"tutorials/l2cache/#missing-data","title":"Missing Data","text":"The service is constantly watching for changes made to objects and recalculating stats on new level2 nodes that are created, in order to keep its database of statistics current. This however takes some time, and is subject to sporadic rare failures. If you request stats on a level 2 node which are not in the database, you will receive an empty dictionary for that node. This will immediately trigger the system to recalculate the statistics of that missing data, and so it should be available shortly (on the order of seconds) if systems are operational. Please note that PCA is not calculated for very small objects because it is not meaningful. So if you are interested in differentiating whether PCA is not available because it hasn't been calculated, vs when its not available because it is not possible to calculate, you should ask for at least one other non PCA statistic as well. You will see that its availability correlates strongly with size_nm3.
Say you want to calculate the total surface area and volume of a object in the dataset. The areas and volume of each component can simply be added together to do this.
import pandas as pd\nroot_id = 648518346349541252\nlvl2nodes = client.chunkedgraph.get_leaves(root_id,stop_layer=2)\nl2stats = client.l2cache.get_l2data(lvl2nodes, attributes=['size_nm3','area_nm2'])\nl2df = pd.DataFrame(l2stats).T\ntotal_area_um2=l2df.area_nm2.sum()/(1000*1000)\ntotal_volume_um3 = l2df.size_nm3.sum()/(1000*1000*1000)\nBy utilizing the bounds argument of get_leaves, you can also do simple spatially restricted analysis of objects. In fact, because you have data on each level2 node individually, you can segregate the neuron using any labelling of its topology.
"},{"location":"tutorials/l2cache/#skeletonization","title":"Skeletonization","text":"Level 2 nodes have \"cross chunk\" edges within the chunkedgraph which represent what level 2 nodes that object is locally connected to. This forms a graph between the level 2 nodes of the object that can be retrieved using the chunkedgraph function client.chunkedgraph. This graph represents a topological representation of the neuron at the resolution of individual chunks, and is guaranteed to be fully connected, unlike a voxel or mesh representation of the neuron which can have gaps where there are defects in the segmentation volume or incorrectly inferred edges at self contact locations.
The level 2 graph can be turned into a skeleton representation of a neuron using a graph based TEASAR like algorithm as described for skeletonizing meshes in this MeshParty Documentation. There is an implementation of this approach that utilizes the chunkedgraph and the L2cache if available here and on pypi as pcg-skel. In this implementation the l2cache is used to more accurately place the level 2 nodes in space using the rep_coord_nm value.
Note that there is detailed documentation on the Skeleton Client interface at the skeletonization tutorial.
"},{"location":"tutorials/l2cache/#trajectory-distributions","title":"Trajectory Distributions","text":"If one is interested in the bulk direction of processes in a region of the brain, one can start with supervoxels in a region, find level 2 nodes that correspond to them, filter out components based on size, (or other criteria such as whether they are part of objects that have components in some other brain area) and look at the distribution of PCA components to understand the directions that those processes are moving within that region of space.
"},{"location":"tutorials/materialization/","title":"Materialization","text":"The Materialization client allows one to interact with the materialized annotation tables, that were posted to the annotation service (the annotations tutorial).
To see the entire class visit the API doc.
The service regularly looks up all annotations and the segids underneath all the boundspatialpoints. You can then query these tables to find out the IDs that underlie the annotations, or the annotations that now intersect with certain IDs.
For example, one common pattern is that you have identified a cell based on the location of its cell body, and you have an annotation there.
You want to know what are the inputs onto the cell, so you first query the annotation table with your soma annotation, asking for the current ID underneath that soma. Then you query a synapse table for all synapse annotations that have a post-synaptic ID equal to the ID from your soma annotation.
In this way your code stays the same, as the proofreading changes and you can track the connectivity of your cell over time.
"},{"location":"tutorials/materialization/#initializing-the-client","title":"Initializing the client","text":"By default when you initialize the overall client, it will choose the most recent materialization version available. This may or may not be desirable depending on your use case. If your code involves using specific IDs then you should be using a specific version that is tied to a timepoint where those IDs are valid.
To see what versions are available, use the client.materialize.get_versions() function.
client.materialize.get_versions()\nEach version has a timestamp it was run on as well as a date when it will expire. You can query all this metadata for a specific version using client.materialize.get_version_metadata() or all versions using client.materialize.get_versions_metadata().
To change the default version, alter the .version property of the client. This will change the version for all subsequent calls which expect one, unless you specify a different version in the method call. Note that this also sets the timestamp property of the client to the timestamp of the version for methods which expect a timestamp.
client.version = 9\nYou can also specify the version when you initialize the client, e.g.:
client = CAVEclient('minnie65_public', version=661)\nOr, you can specify the version when making a particular method call.
"},{"location":"tutorials/materialization/#browsing-versions","title":"Browsing versions","text":"To see what tables are available in a version you can use client.materialize.get_tables().
If you want to read about the description of what that table is, use the annotationengine client client.materialize.get_table_metadata().
If you want to read more about the schema for the annotation table use the schema service client.schema.schema_definition().
Note, the materialization service has a human readable webpage that links to the other services that might be more convenient for you to browse, to get a link there in ipython display client.materialize.homepage
for some important tables, the info service has a pointer to which table you should use in the metadata for the datastack. client.info.get_datastack_info()['synapse_table'] and client.info.get_datastack_info()['soma_table'].
To see how many annotations are in a particular table use
nannotations=client.materialize.get_annotation_count('my_table')\nTo query a small table, you can just download the whole thing using client.materialize.query_table() which will return a dataframe of the table.
Note however, some tables, such as the synapse table might be very large 200-300 million rows and the service will only return the first 200,000 results, and not in a deterministic manner. NOTE! This API is not designed to enable enmass downloading of the entire synapse table there are more efficient ways of doing this. Contact your dataset administrator for more information if this is what you are looking to do.
To just get a preview, use the limit argument (but note again that this won't be a reproducible set)
df=client.materialize.query_table('my_table', limit=10)\nFor most applications, you will want to filter the query in some way.
We offer seven kinds of filters you can apply: filter_in_dict filter_out_dict, filter_equal_dict, filter_greater_dict, filter_less_dict, filter_greater_equal_dict, and filter_less_equal_dict. For query_table each is specified as a dictionary where the keys are column names, and the values are a list of values (or single value in the case of filter_equal).
So for example to query a synapse table for all synapses onto a neuron in flywire you would use
synapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n filter_equal_dict = {'post_pt_root_id': MYID})\nThe speed of querying is affected by a number of factors, including the size of the data. To improve the performance of results, you can reduce the number of columns returned using select_columns.
So for example, if you are only interested in the root_ids and locations of pre_synaptic terminals you might limit the query with select_columns. Also, it is convenient to return the with positions as a column of np.array([x,y,z]) coordinates for many purposes. However, sometimes you might prefer to have them split out as separate x, y, z columns. To enable this option use split_columns=True. split_columns=True is faster, as combining them is an extra step.
synapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n filter_equal_dict = {'post_pt_root_id': MYID},\n select_columns=['id','pre_pt_root_id', 'pre_pt_position'],\n split_columns=True)\nOften you want to have position information in different units. For example, to consider synapse locations or soma locations, you might want to have positions in nanometers or microns.
To create neuroglancer views, you might want positions in integer voxels of a size that aligns with the resolution you are used to using Neuroglancer at.
Annotation tables can be created and uploaded in varying resolutions according to whatever the user of the table felt was natural. This information is available in the metadata for that table. In addition, you may pass desired_resolution as a keyword argument which will automatically convert all spatial positions into voxels of that size in nanometers.
So if you want positions in nanometers, you would pass desired_resolution=[1,1,1]. If you want positions in microns you would pass desired_resolution=[1000,1000,1000]. If you want positions in 4,4,40nm voxel coordinates to use with cloud-volume or neuroglancer you would pass desired_resolution=[4,4,40].
You can also filter columns that are associated with spatial locations based upon being within a 3d bounding box.
This is done by adding a filter_spatial_dict argument to query_table. The units of the bounding box should be in the units of the voxel_resolution of the table (which can be obtained from client.materialize.get_table_metadata()).
bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\nsynapse_table = client.info.get_datastack_info('synapse_table')\ndf=client.materialize.query_table(synapse_table,\n filter_equal_dict = {'post_pt_root_id': MYID},\n filter_spatial_dict = {'post_pt_position': bounding_box})\nFor synapses in particular, we have a simplified method for querying them with a reduced syntax. client.materialize.synapse_query() lets you specify pre and post synaptic partners as keyword arguments and bounding boxes. The defaults make reasonable assumptions about what you want to query, namely that the synapse_table is the table that the info service advertises, and that if you specify a bounding box, that you want the post_pt_position. These can be overridden of course, but the above bounding box query is simplified to.
NOTE! This API is not designed to enable enmass downloading of the entire synapse table there are more efficient ways of doing this. Contact your dataset administrator for more information if this is what you are looking to do.
bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\ndf=client.materialize.query_table(post_ids = MYID,\n bounding_box=bounding_box)\nIn order to query the materialized tables above you can only use IDs that were present at the timestamp of the materialization. If you query the tables with an ID that is not valid during the time of the materialization you will get empty results.
To check if root_ids are valid at your materialization's timestamp, you can use client.chunkedgraph.is_latest_roots()
import numpy as np\nmat_time = client.materialize.get_timestamp()\nis_latest = client.chunkedgraph.is_latest_roots([MYID], timestamp=mat_time)\nassert(np.all(is_latest))\nIf you need to lookup what happened to that ID, you can use the chunkedgraph lineage tree, to look into the future or the past, depending on your application you can use client.chunkedgraph.get_lineage_graph().
Again, the ideal situation is that you have an annotation in the database which refers to your objects of interest, and querying that table by the id column will return the object in the most recent materialization.
However, sometimes you might be browsing and proofreadding the data and get an ID that is more recent that the most recent version available. For convenience, you can use client.materialize.live_query().
to automatically update the results of your query to a time in the future, such as now. For example, to pass now, use datetime.datetime.now(datetime.timezone.utc). Note all timestamps are in UTC throughout the codebase.
import datetime\nsynapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.live_query(synapse_table,\n datetime.datetime.now(datetime.timezone.utc),\n filter_equal_dict = {'post_pt_root_id': MYID})\nThis will raise an ValueError exception if the IDs passed in your filters are not valid at the timestamp given
You can also pass a timestamp directly to query_table and it will call live_query automatically.
import datetime\nsynapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n timestamp=datetime.datetime.now(datetime.timezone.utc),\n filter_equal_dict = {'post_pt_root_id': MYID})\nAlso, keep in mind if you run multiple queries and at each time pass datetime.datetime.now(datetime.timezone.utc), there is no guarantee that the IDs will be consistent from query to query, as proofreading might be happening at any time. For larger scale analysis constraining oneself to a materialized version will ensure consistent results.
Versions have varying expiration times in order to support the tradeoff between recency and consistency, so before undertaking an analysis project consider what version you want to query and what your plan will be to update your analysis to future versions.
"},{"location":"tutorials/materialization/#content-aware-interface-experimental","title":"Content-aware Interface (Experimental)","text":"As of version 5.8.0, we have introduced a new interface to query tables and views. This interface might have small but breaking changes in the near future. :::
In order to make the querying interface more consistent across tables, we have introduced an additional alternative interface to filtering and querying data via the client.materialize.tables object. When you instantiate this object, this object finds all of the existing tables and the list of their columns and lets you filter the tables as arguments in the function with suggestions. Moreover, the filtering arguments and the querying arguments are separated into two.
Let's see how this works with a simplest example --- downloading a table called nucleus_detection_v0. First, we reference the table as a function and then we run the query --- this is exactly the same as client.materialize.query_table('nucleus_detection_v0').
client = CAVEclient('minnie65_public')\nnuc_df = client.materialize.tables.nucleus_detection_v0().query()\nWhere things differ is when we add filters. If we want to query based on a set of values for the field \"id\", for example, we add that as an argument:
my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(id=my_ids).query()\nWhere in this example the id= queries the column id based on the schema. These values can be either individual elements (i.e. an integer or a string) or a list/array of elements, and any field can be used. The tooling will automatically sort out how to format the filtering appropriately when running the query. Importantly, the filtering is identical between querying all types of tables and queries. To see the complete list of fields that can be queried, you can tab-autocomplete or in Jupyter or IPython glance at the docstring with client.materialize.tables.nucleus_detection_v0?.
In addition to filtering by one or many values, you can also do spatial queries. To find only annotations within a particular bounding box, you need to specify the spatial point (e.g. pt_position). For each such spatial point, there will be an argument {spatial_point_position}_bbox (e.g. pt_position_bbox) that accepts a 2x3 list of coordinates specifying the upper and lower bounds for the query. For example, to find all nucleus centroids within a particular bounding box, it would be:
bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n pt_position_bbox=bounding_box\n).query()\nIf you are not using any filters, you can omit the parenthesis and use the get_all or get_all_live functions directly, which act similarly to the query and live_query functions respectively. The first example could be rewritten as:
nuc_df = client.materialize.tables.nucleus_detection_v0.get_all()\nIf you want to list all available fields, you can use the .fields attribute. Similarly, you can get all numeric fields with the .numeric_fields attribute and all spatial fields (allowing bounding box queries) with .spatial_fields.
nuc_df = client.materialize.tables.nucleus_detection_v0.spatial_fields\nIf you need to specify the table programmatically, you can also use a dictionary-style approach to getting the table filtering function. For example, an equivalent version of the above line would be:
my_ids = [373879, 111162]\nmy_table = 'nucleus_detection_v0'\nnuc_df = client.materialize.tables[my_table](id=my_ids).query()\nThe query function can also take arguments relating to timestamps or formatting where they act just like in the other query method. In particular, the arguments that apply to query are: select_columns, offset, limit, split_positions, materialization_version, timestamp, metadata, desired_resolution, and get_counts. For example, to add a desired resolution and split positions in the above query, it would look like:
my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n id=my_ids\n).query(\n split_positions=True,\n desired_resolution=[1,1,1],\n)\nInequalities can also be used in filtering numeric columns. Here, you can pass a dictionary instead of a list of values, with the keys being inequality operators (\">\", \">=\", \"<\", and \"<=\") and the values being the comparison. For example, to query for all nuclei with a volume greater than 1000:
client.materialize.tables.nucleus_detection_v0(\n volume={\">\": 1000}\n).query()\nYou can also use multiple inequalities in the same dictionary to filter within a range. For example, to query for all nuclei with a volume between 500 and 750:
client.materialize.tables.nucleus_detection_v0(\n volume={\">\": 500, \"<\": 750}\n).query()\nIf you want to do a live query instead of a materialized query, the filtering remains identical but we use the live_query function instead. The one required argument for live_query is the timestamp.
my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n id=my_ids\n).live_query(\n timestamp=datetime.datetime.now(datetime.timezone.utc),\n)\nThe live query functions have similar but slightly different arguments: timestamp (required), offset, limit, split_positions, metadata, desired_resolution, and allow_missing_lookups.
Note that way that IPython handles docstrings means that while you can use ? to get the docstring of the filtering part of the function, you can't simply do something like client.materialize.tables.nucleus_detection_v0().query?. It will tell you the function can't be found, because technically the query function does not yet exist until the table filtering function is called.
Instead, if you want to glimpse the docstring of the query or live_query functions, you need to split it into two lines:
qry_func = client.materialize.tables.nucleus_detection_v0().query\nqry_func?\nFinally, if the project you are working with has views, a similar interface is available to them via client.materialize.views. Currently views are not compatible with live query, and so only the .query function is available.
The EMAnnotationSchemas client lets one look up the available schemas and how they are defined. This is mostly used for programmatic interactions between services, but can be useful when looking up schema definitions for new tables.
"},{"location":"tutorials/schemas/#get-the-list-of-schema","title":"Get the list of schema","text":"One can get the list of all available schema with the schema method. Currently, new schema have to be generated on the server side, although we aim to have a generic set available to use.
client.schema.get_schemas()\nThe details of each schema can be viewed with the schema_definition method, formatted as per JSONSchema.
example_schema = client.schema.schema_definition('microns_func_coreg')\nexample_schema\nThis is mostly useful for programmatic interaction between services at the moment, but can also be used to inspect the expected form of an annotation by digging into the format.
example_schema['definitions']['FunctionalCoregistration']\nThe skeleton service enables you to generate and retrieve skeletons from the server. This saves you the trouble of performing skeletonization routines on your local machine, while also saving you the trouble of understanding many of the details inside of skeletonization. Furthermore, the server offers the ability to generate multiple skeletons simultaneously, in parallel. And lastly, the server caches each skeleton that is generated so that it only ever needs to be generated once. All subsequent requests for the same skeleton will simply return the cached skeleton with notable turn-around-time improvements.
"},{"location":"tutorials/skeletonization/#initializing-the-client","title":"Initializing the client","text":"The simplest way to initialize the CAVEclient is by merely providing the datastack of interest:
import caveclient as cc\n\nclient = cc.CAVEclient(<datastack_name>)\nWith a CAVEclient built, you can now investigate the current build version of the SkeletonClient:
client.skeleton.get_version()\nAnd you can see a list of available skeleton versions. In most cases you will want to use the highest (most recent) version provided.
client.skeleton.get_versions()\nRetrieve a skeleton using get_skeleton(). The simplest usage is:
sk = client.skeleton.get_skeleton(\n <root_id>,\n output_format=<output_format>,\n)\nwhere the availale output_formats (described below) are:
'dict' (default if unspecified)'swc' (a Pandas Dataframe)If the skeleton doesn't exist in the server cache, it may take 20-60 seconds to generate the skeleton before it is returned. This function will block during that time. Any subsequent retrieval of the same skeleton should go very quickly however.
To specify a nondefault skeleton version:
sk = client.skeleton.get_skeleton(\n <root_id>,\n skeleton_version=<sk_version>,\n output_format=<output_format>,\n)\nTo specify a specific datastack:
sk = client.skeleton.get_skeleton(\n <root_id>,\n datastack_name=<datastack_name>,\n skeleton_version=<sk_version>,\n output_format=<output_format>,\n)\nMost end-users shouldn't need to use the following function very much, but to see the contents of the cache for a given root id, set of root ids, root id prefix, or set of prefixes:
get_cache_contents(\n root_id_prefixes=<root_id>,\n)\nYou can also add additional parameters as needed:
get_cache_contents(\n datastack_name=<datastack_name>,\n skeleton_version=<sk_version>,\n root_id_prefixes=<root_id>,\n)\nThe primary parameter, root_id_prefixes, can be a list of root ids:
get_cache_contents(\n root_id_prefixes=[<root_id>, <root_id>, ...],\n)\nThe primary parameter can also be a root id prefix, which will match any associated root ids. Since this could potentially return a large list of results, there is an optional limit parameter so you don't overwhelm the memory of your processing environment, e.g., a Jupyter notebook or some other Python script running on your local machine:
get_cache_contents(\n root_id_prefixes=<root_id_prefix>,\n limit=<limit>,\n)\nNote that the limit only constraints the size of the return value. The internal operation of the function will still receive the full list when it passes the prefix on to CloudFiles. Consequently, calling this function for a short prefix may block for a long time.
And of course you can also pass in a list of prefixes (or a mixture of full ids and partial prefixes):
get_cache_contents(\n root_id_prefixes=[<root_id_prefix>, <root_id_prefix>, ...],\n limit=<limit>,\n)\nThe function shown above isn't necessarily the most direct way to simply inquire whether a skeleton exists in the cache for a given root id. For that purpose, the following function is better suited:
skeletons_exist(\n root_ids=<root_id>,\n)\nOr:
skeletons_exist(\n root_ids=[<root_id>, <root_id>, ...],\n)\nNote that this function doesn't accept prefixes, as supported by cache_query_contents(). Only full root ides are supported. When querying with as single root id, the return value will be a boolean. When querying with a list of ids, the return value will be a Python dictionary mapping from each id to a boolean.
This function also takes the same optional parameters described above:
skeletons_exist(\n datastack_name=<datastack_name>,\n skeleton_version=<sk_version>,\n root_ids=<root_id>, # Or [<root_id>, <root_id>, ...],\n)\nYou can retrieve a large set of skeletons in a single function call:
get_bulk_skeletons(\n root_ids=[<root_id>, <root_id>, ...],\n)\nIf any skeletons are not generated yet, the default behavior is to skip those root ids and only return skeletons that are already available. But you can override this default behavior:
get_bulk_skeletons(\n root_ids=[<root_id>, <root_id>, ...],\n generate_missing_skeletons=[False|True],\n)\nAny root ids for which skeletonization is required will be generated one at a time, at a cost of 20-60 seconds each. Consequently, there is a hard-coded limit of 10, after which all subsequent missing skeletons will not be returned.
By default, skeletons are returned in JSON format. However SWC is also supported, thusly:
get_bulk_skeletons(\n root_ids=[<root_id>, <root_id>, ...],\n output_format=<\"json\"|\"swc\">\n)\nAnd the usual defaults can be overridden again:
get_bulk_skeletons(\n root_ids=[<root_id>, <root_id>, ...],\n datastack_name=<datastack_name>,\n skeleton_version=<sk_version>,\n)\nget_bulk_skeletons() is not an effective way to produce a large number of skeletons since it operates synchronously, generating one skeleton at a time. In order to generate a large number of skeletons it is better to do so in parallel. The following function dispatches numerous root ids for skeletonization without returning anything immediately. The root ids are then distributed on the server for parallel skeletonization and eventual caching. Once they are in the cache, you can retrieve them. Of course, it can be tricky to know when they are available. That is addressed further below. Here's how to dispatch asynchronous bulk skeletonization:
generate_bulk_skeletons_async(\n root_ids=[<root_id>, <root_id>, ...],\n)\nAnd with the usual overrides:
get_bulk_skeletons(\n root_ids=[<root_id>, <root_id>, ...],\n datastack_name=<datastack_name>,\n skeleton_version=<sk_version>,\n)\nIn order to retrieve asynchronously generated skeletons, it is necessary to poll the cache for the availability of the skeletons and then eventually retrieve them. Here's an example of such a workflow:
# Dispatch multiple asynchronous, parallel skeletonization and caching processes\ngenerate_bulk_skeletons_async(root_ids)\n\n# Repeatedly query the cache for the existence of the skeletons until they are all available\nwhile True:\n skeletons_that_exist = client.skeleton.skeletons_exist(root_ids=rids)\n num_skeletons_found = sum(skeletons_that_exist.values())\n if num_skeletons_found == len(rids):\n break\n sleep(10) # Pause for ten seconds and check again\n\n# Retrieve the skeletons (remember, SWC is also offered)\nskeletons_json = get_bulk_skeletons(root_ids)\nWe store the JSON description of a Neuroglancer state in a simple database at the JSON Service. This is a convenient way to build states to distribute to people, or pull states to parse work by individuals. The JSON Client is at client.state
client.state\nJSON states are found simply by their ID, which you get when uploading a state. You can download a state with get_state_json.
example_id = 4845531975188480\nexample_state = client.state.get_state_json(test_id)\nexample_state['layers'][0]\nYou can also upload states with upload_state_json. If you do this, the state id is returned by the function. Note that there is no easy way to query what you uploaded later, so be VERY CAREFUL with this state id if you wish to see it again.
Note: If you are working with a Neuroglancer Viewer object or similar, in order to upload, use viewer.state.to_json() to generate this representation.
example_state['layers'][0]['name'] = 'example_name'\nnew_id = client.state.upload_state_json(example_state)\ntest_state = client.state.get_state_json(new_id)\ntest_state['layers'][0]['name']\nOnce you have a state ID, you want to turn it into a well-formatted link. So you don\\'t have to remember all the endpoints, we can do this from the state client.
ngl_base = 'neuromancer-seung-import.appspot.com'\nclient.state.build_neuroglancer_url(new_id, ngl_base)\nNote that the neuroglancer base can be found in the info service under client.info.viewer_site().