Want to get all the juicy metadata out of your Aiven services? Check out the Aiven Metadata Parser!
Based on the aiven-client
python library, the Aiven Metadata Parser exports the metadata from your services in order to build a connected graph showing all interconnections.
Currently the Aiven Metadata Parser is a work in progress and extracts only a limited amount of information and connections.
You can use the create_services.sh
and delete_services.sh
to create and destroy a pre-set of services.
The create_services.sh
uses:
- jq to parse JSON output
- kcat for Apache Kafka®
- psql for PostgreSQL®
- The Aiven CLI to connect and create services
- The Python packages in the
requirements.txt
file to push data to Grafana
Before starting it, please install the required dependences, login with the Aiven CLI and execute:
./scripts/create_services.sh <PROJECT_NAME>
Where <PROJECT_NAME>
is the name of your Aiven project.
To delete the services you can call:
./scripts/delete_services.sh <PROJECT_NAME>
If <PROJECT_NAME>
is not passed, the default project will be used.
You need to be on Python 3.7, install the required libraries with:
pip install -r requirements.txt
Copy the conf.env.sample
file to conf.env
in the conf
folder and edit the token parameter and the project name from which you want to extract parameters.
If you don't have a project with services already running you can create a sample set of services with the create_services.sh
file, which requires the aiven-client
to be installed and the user to be logged in.
Once conf.env
is set, you can start the metadata extraction with:
python main.py
This will generate:
- A file
graph_data.dot
containing the information in DOT format - A file
graph_data.gml
containing the information in GML format - A file
nx.html
containing the complete interactive graph - (Disabled for the moment) A file
filtered.html
containing the complete interactive graph filtered on the node with idpg~demo-pg~schema~public~table~pasta
(this might error out if you don't have such node)
Furthermore if, after executing the main.py
you also execute the following:
python app.py
The app.py
reads the graph_data.gml
file generated at step 1 and creates a Reactive Web Applications with Plotly and Dash (code taken from here).
Lastly the following allows you to push the content to a PG database passing the PG URI:
python write_pg.py PG_URI
Warning The code is a bare minimum product, doesn't cover all services and options and is quite chaotic! It demonstrates the idea and how we could start implementing it.
- If you run
python app.py
and see an error sayingNo such file or directory: 'neato'
, you will have to installgraphviz
on your machine (the package version from pip doesn't seem to work) - find out how to install it here.
Wanna contribute? Check the CONTRIBUTING file
Aiven Metadata Extractor is licensed under the Apache license, version 2.0. Full license text is available in the LICENSE file.
Please note that the project explicitly does not require a CLA (Contributor License Agreement) from its contributors.
Bug reports and patches are very welcome, please post them as GitHub issues and pull requests at https://github.com/aiven/metadata-parser . To report any possible vulnerabilities or other serious issues please see our security policy.
All images under the src
folder and shown in nx.html
file are taken from https://www.flaticon.com/
All images under the src/services
folder are a property of Aiven
Apache, Apache Kafka, Kafka, Apache Flink, Flink, Apache Cassandra, and Cassandra are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. ClickHouse is a registered trademark of ClickHouse, Inc. https://clickhouse.com. M3, M3 Aggregator, M3 Coordinator, OpenSearch, PostgreSQL, MySQL, InfluxDB, Grafana, Terraform, and Kubernetes are trademarks and property of their respective owners. *Redis is a trademark of Redis Ltd. and the Redis box logo is a mark of Redis Ltd. Any rights therein are reserved to Redis Ltd. Any use by Aiven is for referential purposes only and does not indicate any sponsorship, endorsement or affiliation between Redis and Aiven. All product and service names used in this website are for identification purposes only and do not imply endorsement.