test Website · Slack Community · Documentation · Blog
valmi.io is the open-source Reverse-ETL (data activation) platform to load data from warehouses into SaaS platforms, Webhook Apis etc.
-
See it in Action
Quickly witness a One Million (1M) Rows Sync at Live Sync.
-
We'd love to discuss your use-case and the problem you are trying to solve, if you are up for a 15-20 min call, please block a time on Co-Founder Raj Varkala's calendar
valmi.io uses some of the best tools to create an Open Source reverse-ETL (data activation) Platform. It is built over the airbyte protocol. dbt is the centerpiece of our source connectors, and duckdb for metrics. We engineered our orchestrator over dagster, and dagster dovetails perfectly with our vision of being a multi-persona tool.
-
Checkout docs at - https://www.valmi.io/docs/
-
Read the variety of usecases valmi.io can enable for your organization - https://www.valmi.io/blog/
-
- You can immediately create a sync at https://demo.valmi.io.
- Watch a demo video to create a sync.
-
Prerequisites: valmi.io relies heavily on both Docker and Docker-compose. Install Docker Desktop to get both docker and docker-compose.
-
Clone this repo and recursively clone submodules.
git clone git@github.com:valmi-io/valmi-activation.git cd valmi-activation git submodule update --init --recursive
-
Setup the environment.
cp .env-example .env cd valmi-app-backend cp .env-example .env cd ../valmi-app `For macos` cp .env-example.macos .env `For linux` cp .env-example.linux .env
-
Intermediate storage, We are adding support for object stores like S3, GCS. Until then, Local storage is used.
sudo mkdir -p /tmp/shared_dir/intermediate_store sudo chmod -R 777 /tmp/shared_dir/intermediate_store
-
Launch the reverse-etl service.
./valmi prod
-
To stop the service, run the following.
./valmi prod down
-
Please wait for about 2 minutes before you access the service, since valmi-app builds an optimized compiled version of the app UI. To access the service, please check the 'Accessing the service' section for local deployments.
3. Develop a connector locally to customize valmi.io as per your needs. You can just contact us too.
-
Clone, setup environment variables and create intermediate storage (see above section).
-
Create a new connector (Optional).
# Copy code base from any existing connectors from valmi-integrations folder (ex. destination-webhook) cd valmi-integrations/connectors cp -r destination-webhook destination-awesome_connector # Make necessary changes and build the connector cd destination-awesome_connector make build_docker version=latest # Add the new connector information to "valmi-app-backend/init_db/connector_def.json"
-
Run the service.
./valmi dev
-
To access the service, please check the 'Accessing the service' section for local deployments.
-
To Stop the service, run the following.
./valmi dev down
-
Syncs
http://localhost:3000Sync Runs
http://localhost:3000UI Backend Server API
http://localhost:4000/api/docsActivation Server API
http://localhost:8000/docs