Skip to content

Libraries to write spark dataframe to various persistence systems like S3Bucket, GCSBucket, LocalFileSystem, etc.

Notifications You must be signed in to change notification settings

teamclairvoyant/data-scalaxy-writer

Repository files navigation

data-scalaxy-writer

This library provides APIs to write a spark dataframe to various target source systems.

This library supports below target systems:

  • Local File System
  • AWS
  • GCP

local-file-system

User can use this library to write/persist spark dataframe to the local file system in various file formats. Supported file formats are:

  • CSV
  • JSON
  • XML
  • Parquet

Please see the detailed documentation here.

aws

User can use this library to write/persist spark dataframe to various storage options provided by AWS.

Currently, we are supporting below options:

  • S3 Buckets
  • Redshift Tables

Please see the detailed documentation here.

gcp

User can use this library to write/persist spark dataframe to various storage options provided by google cloud system.

Currently, we are supporting below options:

  • GCS Buckets
  • BigQuery Tables

Please see the detailed documentation here.

About

Libraries to write spark dataframe to various persistence systems like S3Bucket, GCSBucket, LocalFileSystem, etc.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Languages