A Bulk Data Pipeline out of Cassandra. Aegisthus implements a reader for the SSTable format and provides a map/reduce program to create a compacted snapshot of the data contained in a column family.
Aegisthus is built via Gradle (http://www.gradle.org). To build from the command line: ./gradlew build
Please see the wiki or checkout the scripts directory to use our sstable2json wrapper for individual sstables.
To run the included tests from the command line: ./gradlew build
- Reading
- Commit log readers
- Code to do this previously existed in Aegisthus but was removed in commit 35a05e3f.
- Split compressed input files
- Currently compressed input files are only handled by a single mapper. See the discussion in issue #9. The relevant section of code is in getSSTableSplitsForFile in AegisthusInputFormat.
- Add CQL support
- This way the user doesn't have to add the key and column types as job parameters. Perhaps we will do this by requiring the table schema like SSTableExporter does.
- Commit log readers
- Writing
- Add an option to snappy compress output.
- Add an output format for easier downstream processing.
- See discussion on issue #36.
- Add a pivot format
- Create an output format that contains a column per row. This can be used to support very large rows without having to have all of the columns in memory at one time.
- Packaging
- Publish Aegisthus to Maven Central
- Publish Shaded/Shadowed/FatJar version of Aegisthus as well
Copyright 2013 Netflix, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.