eneel is a cli utility for fast extracting and loading data to and from relational databases. The E and L in ELT.
Mostly uses the source databases proprietary tools for extracting and loading data.
Features:
- Fastest way to extract and load large amounts of data between different databases
- Automatic table creation and conversion of datatypes
- Load a query from one database to another
- Loads are executed in parallel
- Incremental loads
- Pretty runtime logging of load progress
- Database logging
- Limit rows load in development
- Configuration with yaml files. Just list the tables you want to replicate
- Configuration files approach makes versioning with git etc easier
Make sure you have the cli-tools for your required databases then:
-
With pip:
pip install eneel
-
Or clone the repository, go to the directory and:
python setup.py install
- With Oracle you need Oracles CLI client SQLplus
- With SQL server you need CLI tool BCP
After installation an example connections.yml file will be in your home directory (~/.eneel). That's where you configure your connection info to your sources and targets.
Next, create a project configuration file in a directory for you EL projects.
Go to the directory with the project configuration file and run eneel with the project file name excluding yml. I.e for a project configuration file as load_from_postgres_prod_to_dw.yml run:
eneel load_from_postgres_prod_to_dw
The output will then be something like below for at successfull run:
Optional parameters:
--connections: add a path to the connections.yml you would like to use--target: specify which target in connections.yml you would like to use. This will be applied on both sources and targets
| Database | Source | Target |
|---|---|---|
| Postgres | YES | YES |
| Sql Server | YES | YES |
| Oracle | YES | NO |
| Snowflake | NO | YES |
- Support for BigQuery
- Incremental loads with updates
- Incremental loads with deletes
- Go ahead and report an issue
- Want to help? Pull requests are most welcome!