feat: Add information_schema #115
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes #98 Cloud will be making requests directly to the database to get info about the contents of the database, including schemas, tables, and columns.
RustomMS
reviewed
Sep 20, 2022
Comment on lines
+396
to
+399
| let char_len: Option<u64> = match data_type { | ||
| Utf8 | Binary => Some(i32::MAX as u64), | ||
| LargeBinary | LargeUtf8 => Some(i64::MAX as u64), | ||
| _ => None, |
There was a problem hiding this comment.
Curious where I could find what the lengths for the var data is signed ints? I couldn't quite find it searching arrow crate or the arrow spec
There was a problem hiding this comment.
The arrow spec indexes by i32 or i64 depending on if the logical types are the "Large" variants. They seemed to have picked signed over unsigned for indexing to be compatible with the widest number of languages.
justinrubek
added a commit
that referenced
this pull request
Sep 27, 2022
* chore: Update to datafusion 12 (#114) * First draft of key layout using RocksDB (#116) * First draft of key layout using RocksDB * Additional future considerations from Sean * feat: Add information_schema (#115) * feat: Add information_schema Fixes #98 Cloud will be making requests directly to the database to get info about the contents of the database, including schemas, tables, and columns. * fix: Remove datafusion-proto crate (#119) We're not using if for anything yet, and this release seems to break building container images. ``` error: builder for '/nix/store/75lddm4kg8mzn2x5nz8lg36gdj16p7ka-glaredb-cli-0.1.0.drv' failed with exit code 101; last 10 log lines: > Caused by: > process didn't exit successfully: `/build/source/target/release/build/datafusion-proto-497b9ae0fe438eda/build-script-build` (exit status: 1) > --- stdout > cargo:rerun-if-env-changed=FORCE_REBUILD > cargo:rerun-if-changed=proto/datafusion.proto > Running: "/nix/store/2qg94y58v1jr4dw360bmpxlrs30m31ca-protobuf-3.19.4/bin/protoc" "--include_imports" "--include_source_info" "-o" "/build/prost-buildFXFfZG/prost-descriptor-set" "-I" "proto" "-I" "/nix/store/2qg94y58v1jr4dw360bmpxlrs30m31ca-protobuf-3.19.4/include" "proto/datafusion.proto" > > --- stderr > Error: "protobuf compilation failed: Permission denied (os error 13)" > warning: build failed, waiting for other jobs to finish... For full logs, run 'nix log /nix/store/75lddm4kg8mzn2x5nz8lg36gdj16p7ka-glaredb-cli-0.1.0.drv'. ``` Possibly related: apache/datafusion#3538 * feat: Implement raft via gRPC (#63) * Replace toy-rpc with tonic gRPC * implement glaredb cli for raft nodes and client * current progress * implement begin, allocate_table, and get_schema * implement scan * implement insert * cleanup * comment out old tests * clean up ConsensusClient * Implement change membership command * rewrite cluster tests to use RaftClientSource * add protoc to CI * switch raft to in-memory implementation * Remove application logic from raft cluster tests * cargo fmt * add tracing to RPC impls * Remove lemur from raft crate * remove raft_client example * Apply suggestions from code review Co-authored-by: Sean Smith <scsmithr@gmail.com> * remove protoc from ci * Remove lemur_impl from raft crate * Store tonic clients instead of endpoint in ConsensusClient * use shared n_retries * Add default num_retries * Apply suggestions from code review Co-authored-by: Rustom Shareef <2547411+RustomMS@users.noreply.github.com> * moved some mod.rs modules into their parent directories * implement ConsensusClient retry to find leader using macro * Fix missing delimiter * fix clippy issues * rewrite retry_rpc_on_leader macro to evaluate to an expression * remove panics in rpc server impls Co-authored-by: Sean Smith <scsmithr@gmail.com> Co-authored-by: Rustom Shareef <2547411+RustomMS@users.noreply.github.com> * build(nix): Use crane to cache cargo dependencies (#121) * Add crane * switch rust toolchain to come from fenix * touch buildscript before executing cargo build * add clippy and build checks * rename arrowstore build script * rename raft build script * Send back BindComplete intead of ParseComplete Also moved sending results into its own function since we need to send results back after Execute commands complete. * Add logical plan stub for SETting runtime vars Also fixes logic for checking pg message length. Co-authored-by: Rustom Shareef <2547411+RustomMS@users.noreply.github.com> Co-authored-by: Justin Rubek <25621857+justinrubek@users.noreply.github.com>
justinrubek
added a commit
that referenced
this pull request
Sep 27, 2022
* add extended query parse message * parse step * bind step * Some Describe + Some Drop table * close server on Terminate message * Get further with extended query protocol (#127) * chore: Update to datafusion 12 (#114) * First draft of key layout using RocksDB (#116) * First draft of key layout using RocksDB * Additional future considerations from Sean * feat: Add information_schema (#115) * feat: Add information_schema Fixes #98 Cloud will be making requests directly to the database to get info about the contents of the database, including schemas, tables, and columns. * fix: Remove datafusion-proto crate (#119) We're not using if for anything yet, and this release seems to break building container images. ``` error: builder for '/nix/store/75lddm4kg8mzn2x5nz8lg36gdj16p7ka-glaredb-cli-0.1.0.drv' failed with exit code 101; last 10 log lines: > Caused by: > process didn't exit successfully: `/build/source/target/release/build/datafusion-proto-497b9ae0fe438eda/build-script-build` (exit status: 1) > --- stdout > cargo:rerun-if-env-changed=FORCE_REBUILD > cargo:rerun-if-changed=proto/datafusion.proto > Running: "/nix/store/2qg94y58v1jr4dw360bmpxlrs30m31ca-protobuf-3.19.4/bin/protoc" "--include_imports" "--include_source_info" "-o" "/build/prost-buildFXFfZG/prost-descriptor-set" "-I" "proto" "-I" "/nix/store/2qg94y58v1jr4dw360bmpxlrs30m31ca-protobuf-3.19.4/include" "proto/datafusion.proto" > > --- stderr > Error: "protobuf compilation failed: Permission denied (os error 13)" > warning: build failed, waiting for other jobs to finish... For full logs, run 'nix log /nix/store/75lddm4kg8mzn2x5nz8lg36gdj16p7ka-glaredb-cli-0.1.0.drv'. ``` Possibly related: apache/datafusion#3538 * feat: Implement raft via gRPC (#63) * Replace toy-rpc with tonic gRPC * implement glaredb cli for raft nodes and client * current progress * implement begin, allocate_table, and get_schema * implement scan * implement insert * cleanup * comment out old tests * clean up ConsensusClient * Implement change membership command * rewrite cluster tests to use RaftClientSource * add protoc to CI * switch raft to in-memory implementation * Remove application logic from raft cluster tests * cargo fmt * add tracing to RPC impls * Remove lemur from raft crate * remove raft_client example * Apply suggestions from code review Co-authored-by: Sean Smith <scsmithr@gmail.com> * remove protoc from ci * Remove lemur_impl from raft crate * Store tonic clients instead of endpoint in ConsensusClient * use shared n_retries * Add default num_retries * Apply suggestions from code review Co-authored-by: Rustom Shareef <2547411+RustomMS@users.noreply.github.com> * moved some mod.rs modules into their parent directories * implement ConsensusClient retry to find leader using macro * Fix missing delimiter * fix clippy issues * rewrite retry_rpc_on_leader macro to evaluate to an expression * remove panics in rpc server impls Co-authored-by: Sean Smith <scsmithr@gmail.com> Co-authored-by: Rustom Shareef <2547411+RustomMS@users.noreply.github.com> * build(nix): Use crane to cache cargo dependencies (#121) * Add crane * switch rust toolchain to come from fenix * touch buildscript before executing cargo build * add clippy and build checks * rename arrowstore build script * rename raft build script * Send back BindComplete intead of ParseComplete Also moved sending results into its own function since we need to send results back after Execute commands complete. * Add logical plan stub for SETting runtime vars Also fixes logic for checking pg message length. Co-authored-by: Rustom Shareef <2547411+RustomMS@users.noreply.github.com> Co-authored-by: Justin Rubek <25621857+justinrubek@users.noreply.github.com> * fix clippy issues * replaced dbg usage with tracing * updated some comments * comment terminate message Co-authored-by: Sean Smith <scsmithr@gmail.com> Co-authored-by: Rustom Shareef <2547411+RustomMS@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #98
Cloud will be making requests directly to the database to get info about the contents of the database, including schemas, tables, and columns.