Writer conversion #80
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mehtaishita
commented
Apr 20, 2022
mehtaishita
commented
- Added Int64 or int32 for relevant number types
- Created StreamOptions and other config types to replace anonymous objects
- Several Buffer readers
- Parquet Schema file type to serialize and write metadata
feat(bloom-filter): add support Add ability to create bloom filters on columns according to the apache [specs](https://github.com/apache/parquet-format/blob/master/BloomFilter.md). Co-authored-by: shannonwells <shannonwells@users.noreply.github.com> Co-authored-by: Wil Wade <wil.wade@unfinished.com>
* Remove travis * Fix tiny syntax bug * Add org to name in package.json * Add templates for PRs and issues * Add GitHub Actions for testing and publishing * Linting not yet implemented * Commit the package-lock file for consistent builds
…uding directly (#6)
chore(README): update to include bloom filter Update README file to include bloom filter feature. Co-authored-by: Shannon Wells <shannonwells@users.noreply.github.com>
* use xxhash-wasm * updates after rebase * update tests due to new xxhash-wasm package * remove reference to old xxhash
* Cleanup package.json packages * Fix readme up * Thrift types * Update parquet thrift * Class requires new * Export Parquetjs Types
Add browser/node support for fetching http request.
* Configuration for bundling parquetjs that works in browser, via esbuild * Add to build targets * Fix all the things that got broken * Example server showing how it can be used * Updates to README * Disable LZO and Brotli in browser * Disable LZO in Node as a consequence of this attempt (see notes) * Use wasm-brotli: * Use async import to load wasm before loading compression.js * requires async loading to get the wasm instanc * bubble up all the asyncs * make the tests pass again * Webpack config file for possibly later * check in example parquet files Co-authored-by: enddy <enddy.code@gmail.com>
…ck (#19) * Add parquet.js explicitly to the "include" list in tsconfig. * Also regenerated package-lock.json to bump some patch versions.
Update bson package from 2.0.8 to 4.4.0 due to a vulnerability in v2. The interface changed, so update the files that use the package. Remove an unused TypeArray import. Remove a stray comment.
Co-authored-by: acruikshank <acruikshank@example.com>
* Added test cases for types and fixed bigint precision * missed one check on if statement * Added more test cases for invalid values to throw * Added test cases for real this time * Reorganized test cases and added helper function for readability and try catch for better error message * Fixed spacing on function * Added final tests to int64 and 96 * changed throw error code and fixed type on tests Co-authored-by: Wayland Li <wayland@Waylands-MacBook-Pro.local>
* Converted Independent files into typescript Co-authored-by: Wayland Li <wayland@Waylands-MacBook-Pro-2.local> Co-authored-by: Sidney Keese <me@sidke.com>
Co-authored-by: Wayland Li <wayland@Waylands-MacBook-Pro-2.local>
* Converted schema.js into typescript without using type any Co-authored-by: Wayland Li <wayland@Waylands-MacBook-Pro-2.local> Co-authored-by: Wayland Li <wayland@Waylands-MacBook-Pro.local>
* Converted shred.js into typescript without using type any Co-authored-by: Wayland Li <wayland@Waylands-MacBook-Pro-2.local> Co-authored-by: Wayland Li <wayland@Waylands-MacBook-Pro.local>
Converted util.js into typescript Co-authored-by: Wayland Li <wayland@Waylands-MacBook-Pro-2.local> Co-authored-by: Wayland Li <wayland@Waylands-MacBook-Pro.local>
* Converted reader file to typescript and made fixes dependent on it Co-authored-by: Wayland Li <wayland@Waylands-MacBook-Pro.local> Wip wip dump 4 errors left Wip parquet envelope writer and bloom filter placeholder type number conversion wip dump user meta data is a keyvalue array across the board so cleaning up how metadata values are pushed/concat simplified encoding enum calls next: clean up some function signatures and calls next: add options for column chunk data re commit? draft commit
more consistent parameters, number types, buffer types
* Converted reader file to typescript and made fixes dependent on it Co-authored-by: Wayland Li <wayland@Waylands-MacBook-Pro.local>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.