Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spark parquet improvements #195

Closed
wants to merge 1 commit into from
Closed

Spark parquet improvements #195

wants to merge 1 commit into from

Commits on Apr 3, 2014

  1. SPARK-1383 [SQL] Improvements to ParquetRelation 

    Improvements to Spark-SQL's ParquetRelation:
- Instead of files a ParquetRelation is now backed by a directory, which
  simplifies importing data from other sources (e.g. ParquetAvro)
- InsertIntoParquetTable operation now supports switching between
  overwriting or appending
- Parquet tests now use the new API
- Parquet logging is now forwarded to Log4J
- Compression is enabled by default (default compression for Parquet
  files: GZIP, as in parquet-mr)

Also fixes SPARK-1367: NPE when joining ParquetRelations.
    @AndreSchumacher
    AndreSchumacher committed Apr 3, 2014
    Configuration menu
    Copy the full SHA
    54df314 View commit details
    Browse the repository at this point in the history