From 5df5668097e3f0480b848260f5b7c6b7b4fd50ad Mon Sep 17 00:00:00 2001 From: Sam Firke Date: Mon, 20 Nov 2023 18:35:50 +0000 Subject: [PATCH 1/4] refresh adding a db driver in docker --- docs/docs/databases/docker-add-drivers.mdx | 87 +++++++++------------- 1 file changed, 37 insertions(+), 50 deletions(-) diff --git a/docs/docs/databases/docker-add-drivers.mdx b/docs/docs/databases/docker-add-drivers.mdx index 03a971a979efc..1b9301a62b9a0 100644 --- a/docs/docs/databases/docker-add-drivers.mdx +++ b/docs/docs/databases/docker-add-drivers.mdx @@ -8,86 +8,73 @@ version: 1 ## Adding New Database Drivers in Docker Superset requires a Python database driver to be installed for each additional type of database you -want to connect to. When setting up Superset locally via `docker compose`, the drivers and packages -contained in -[requirements.txt](https://github.com/apache/superset/blob/master/requirements.txt) and -[requirements-dev.txt](https://github.com/apache/superset/blob/master/requirements-dev.txt) -will be installed automatically. +want to connect to. -In this section, we'll walk through how to install the MySQL connector library. The connector -library installation process is the same for all additional libraries and we'll end this section -with the recommended connector library for each database. +In this example, we'll walk through how to install the MySQL connector library. The connector +library installation process is the same for all additional libraries. ### 1. Determine the driver you need -To figure out how to install the [database driver](/docs/databases/installing-database-drivers) of your choice. +Consult the [list of database drivers](/docs/databases/installing-database-drivers) and find the PyPI +package needed to connect to your database. In this example, we're connecting to a MySQL database, +so we'll need the `mysqlclient` connector library. -In the example, we'll walk through the process of installing a MySQL driver in Superset. +### 2. Install the driver in the container -### 2. Install MySQL Driver +We need to get the `mysqlclient` library installed into the Superset docker container (it doesn't matter if it's +installed on the host machine). We could enter the running container with `docker exec -it bash` +and run `pip install mysqlclient` there, but that wouldn't persist permanently. -As we are currently running inside of a Docker container via `docker compose`, we cannot simply run -`pip install mysqlclient` on our local shell and expect the drivers to be installed within the -Docker containers for superset. +To address this, the Superset `docker compose` uses the convention of a `requirements-local.txt` file. All packages +listed in this file will be installed into the container from PyPI at runtime. This file will be ignored by Git for +the purposes of local development. -In order to address this, the Superset `docker compose` setup comes with a mechanism for you to -install packages locally, which will be ignored by Git for the purposes of local development. Please -follow these steps: - -Create `requirements-local.txt` +Create the file `requirements-local.txt` in a subdirectory called `docker` that exists in the directory with your +`docker-compose.yml` or `docker-compose-non-dev.yml` file. ``` -# From the repo root... +# Run from the repo root: touch ./docker/requirements-local.txt ``` -Add the driver selected in step above: +Add the driver identified in step above. You can use a text editor or do it from the command line like: ``` echo "mysqlclient" >> ./docker/requirements-local.txt ``` -Rebuild your local image with the new driver baked in: +**If you are running a stock (non-customized) Superset image**, you are done. Launch Superset with +`docker compose -f docker-compose-non-dev.yml up` and the driver should be present. -``` -docker compose build --force-rm -``` +You can check its presence by entering the running container with `docker exec -it +bash` and running `pip freeze`. The PyPI package should be present in the printed list. -After the rebuild of the Docker images is complete (which may take a few minutes) you can relaunch using the following command: +**If you're running a customized docker image**, rebuild your local image with the new driver baked in: ``` -docker compose up +docker compose build --force-rm ``` -The other option is to start Superset via Docker Compose is using the recipe in `docker-compose-non-dev.yml`, which will use pre-built frontend assets and skip the building of front-end assets: +After the rebuild of the Docker images is complete, relaunch Superset by running `docker compose up`. -``` -docker compose -f docker-compose-non-dev.yml pull -docker compose -f docker-compose-non-dev.yml up -``` ### 3. Connect to MySQL -Now that you've got a MySQL driver installed locally, you should be able to test it out. +Now that you've got a MySQL driver installed in your container, you should be able to connect to your +database via the Superset web UI. -We can now create a Datasource in Superset that can be used to connect to a MySQL instance. Assuming -your MySQL instance is running locally and can be accessed via localhost, use the following -connection string in “SQL Alchemy URI”, by going to Sources > Databases > + icon (to add a new -datasource) in Superset. +As an admin user, go to Settings -> Data: Database Connections and click the +DATABASE button. From +there, follow the steps on the [Using Database Connection UI page](https://superset.apache.org/docs/databases/db-connection-ui). -For Docker running in Linux: +Consult the page for your specific database type in the Superset documentation to determine the connection string and any other +parameters you need to input. For instance, on the [MySQL page](https://superset.apache.org/docs/databases/mysql), we see +that the connection string to a local MySQL database differs depending on whether the setup is running on Linux or OSX. +Click the “Test Connection” button, which should result in a popup message saying, "Connection looks good!". -``` -mysql://mysqluser:mysqluserpassword@localhost/example?charset=utf8 -``` - -For Docker running in OSX: - -``` -mysql://mysqluser:mysqluserpassword@docker.for.mac.host.internal/example?charset=utf8 -``` +### 4. Troubleshooting -Then click “Test Connection”, which should give you an “OK” message. If not, please look at your -terminal for error messages, and reach out for help. +If the test fails, review your docker logs for error messages. Superset uses SQLAlchemy to connect to databases; to find +the correct connection string for your database, you might start Python in the Superset application container and try to +connect directly to the desired database and fetch data. This eliminates Superset for the purposes of isolating the problem. -You can repeat this process for every database you want superset to be able to connect to. +Repeat this process for every different type of database you want Superset to be able to connect to. \ No newline at end of file From 23e13b7920ea5b895c4ca45fc7d819d9c7d67025 Mon Sep 17 00:00:00 2001 From: Sam Firke Date: Mon, 20 Nov 2023 18:58:37 +0000 Subject: [PATCH 2/4] tiny wording tweaks --- docs/docs/databases/docker-add-drivers.mdx | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/docs/databases/docker-add-drivers.mdx b/docs/docs/databases/docker-add-drivers.mdx index 1b9301a62b9a0..c63147e14f6b9 100644 --- a/docs/docs/databases/docker-add-drivers.mdx +++ b/docs/docs/databases/docker-add-drivers.mdx @@ -25,7 +25,7 @@ We need to get the `mysqlclient` library installed into the Superset docker cont installed on the host machine). We could enter the running container with `docker exec -it bash` and run `pip install mysqlclient` there, but that wouldn't persist permanently. -To address this, the Superset `docker compose` uses the convention of a `requirements-local.txt` file. All packages +To address this, the Superset `docker compose` deployment uses the convention of a `requirements-local.txt` file. All packages listed in this file will be installed into the container from PyPI at runtime. This file will be ignored by Git for the purposes of local development. @@ -46,7 +46,7 @@ echo "mysqlclient" >> ./docker/requirements-local.txt **If you are running a stock (non-customized) Superset image**, you are done. Launch Superset with `docker compose -f docker-compose-non-dev.yml up` and the driver should be present. -You can check its presence by entering the running container with `docker exec -it +You can check its presence by entering the running container with `docker exec -it bash` and running `pip freeze`. The PyPI package should be present in the printed list. **If you're running a customized docker image**, rebuild your local image with the new driver baked in: @@ -68,13 +68,15 @@ there, follow the steps on the [Using Database Connection UI page](https://super Consult the page for your specific database type in the Superset documentation to determine the connection string and any other parameters you need to input. For instance, on the [MySQL page](https://superset.apache.org/docs/databases/mysql), we see -that the connection string to a local MySQL database differs depending on whether the setup is running on Linux or OSX. +that the connection string to a local MySQL database differs depending on whether the setup is running on Linux or Mac. + Click the “Test Connection” button, which should result in a popup message saying, "Connection looks good!". ### 4. Troubleshooting -If the test fails, review your docker logs for error messages. Superset uses SQLAlchemy to connect to databases; to find -the correct connection string for your database, you might start Python in the Superset application container and try to -connect directly to the desired database and fetch data. This eliminates Superset for the purposes of isolating the problem. +If the test fails, review your docker logs for error messages. Superset uses SQLAlchemy to connect to databases; to +troubleshoot the connection string for your database, you might start Python in the Superset application container or host +environment and try to connect directly to the desired database and fetch data. This eliminates Superset for the +purposes of isolating the problem. -Repeat this process for every different type of database you want Superset to be able to connect to. \ No newline at end of file +Repeat this process for each different type of database you want Superset to be able to connect to. \ No newline at end of file From 24808b4f77c1a38533912861d9cfa588b55708be Mon Sep 17 00:00:00 2001 From: Sam Firke Date: Mon, 20 Nov 2023 19:18:18 +0000 Subject: [PATCH 3/4] fix linting --- docs/docs/databases/docker-add-drivers.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/docs/databases/docker-add-drivers.mdx b/docs/docs/databases/docker-add-drivers.mdx index c63147e14f6b9..1e3d806aa4ca0 100644 --- a/docs/docs/databases/docker-add-drivers.mdx +++ b/docs/docs/databases/docker-add-drivers.mdx @@ -22,7 +22,7 @@ so we'll need the `mysqlclient` connector library. ### 2. Install the driver in the container We need to get the `mysqlclient` library installed into the Superset docker container (it doesn't matter if it's -installed on the host machine). We could enter the running container with `docker exec -it bash` +installed on the host machine). We could enter the running container with `docker exec -it bash` and run `pip install mysqlclient` there, but that wouldn't persist permanently. To address this, the Superset `docker compose` deployment uses the convention of a `requirements-local.txt` file. All packages @@ -30,7 +30,7 @@ listed in this file will be installed into the container from PyPI at runtime. T the purposes of local development. Create the file `requirements-local.txt` in a subdirectory called `docker` that exists in the directory with your -`docker-compose.yml` or `docker-compose-non-dev.yml` file. +`docker-compose.yml` or `docker-compose-non-dev.yml` file. ``` # Run from the repo root: @@ -79,4 +79,4 @@ troubleshoot the connection string for your database, you might start Python in environment and try to connect directly to the desired database and fetch data. This eliminates Superset for the purposes of isolating the problem. -Repeat this process for each different type of database you want Superset to be able to connect to. \ No newline at end of file +Repeat this process for each different type of database you want Superset to be able to connect to. From e68dd078fb6985981135988476d6094b8d7572ba Mon Sep 17 00:00:00 2001 From: Sam Firke Date: Wed, 22 Nov 2023 21:50:51 +0000 Subject: [PATCH 4/4] undo manual line breaks --- docs/docs/databases/docker-add-drivers.mdx | 43 ++++++---------------- 1 file changed, 12 insertions(+), 31 deletions(-) diff --git a/docs/docs/databases/docker-add-drivers.mdx b/docs/docs/databases/docker-add-drivers.mdx index 1e3d806aa4ca0..cb7c550c072cd 100644 --- a/docs/docs/databases/docker-add-drivers.mdx +++ b/docs/docs/databases/docker-add-drivers.mdx @@ -7,30 +7,21 @@ version: 1 ## Adding New Database Drivers in Docker -Superset requires a Python database driver to be installed for each additional type of database you -want to connect to. +Superset requires a Python database driver to be installed for each additional type of database you want to connect to. -In this example, we'll walk through how to install the MySQL connector library. The connector -library installation process is the same for all additional libraries. +In this example, we'll walk through how to install the MySQL connector library. The connector library installation process is the same for all additional libraries. ### 1. Determine the driver you need -Consult the [list of database drivers](/docs/databases/installing-database-drivers) and find the PyPI -package needed to connect to your database. In this example, we're connecting to a MySQL database, -so we'll need the `mysqlclient` connector library. +Consult the [list of database drivers](/docs/databases/installing-database-drivers) and find the PyPI package needed to connect to your database. In this example, we're connecting to a MySQL database, so we'll need the `mysqlclient` connector library. ### 2. Install the driver in the container -We need to get the `mysqlclient` library installed into the Superset docker container (it doesn't matter if it's -installed on the host machine). We could enter the running container with `docker exec -it bash` -and run `pip install mysqlclient` there, but that wouldn't persist permanently. +We need to get the `mysqlclient` library installed into the Superset docker container (it doesn't matter if it's installed on the host machine). We could enter the running container with `docker exec -it bash` and run `pip install mysqlclient` there, but that wouldn't persist permanently. -To address this, the Superset `docker compose` deployment uses the convention of a `requirements-local.txt` file. All packages -listed in this file will be installed into the container from PyPI at runtime. This file will be ignored by Git for -the purposes of local development. +To address this, the Superset `docker compose` deployment uses the convention of a `requirements-local.txt` file. All packages listed in this file will be installed into the container from PyPI at runtime. This file will be ignored by Git for the purposes of local development. -Create the file `requirements-local.txt` in a subdirectory called `docker` that exists in the directory with your -`docker-compose.yml` or `docker-compose-non-dev.yml` file. +Create the file `requirements-local.txt` in a subdirectory called `docker` that exists in the directory with your `docker-compose.yml` or `docker-compose-non-dev.yml` file. ``` # Run from the repo root: @@ -43,11 +34,9 @@ Add the driver identified in step above. You can use a text editor or do it fro echo "mysqlclient" >> ./docker/requirements-local.txt ``` -**If you are running a stock (non-customized) Superset image**, you are done. Launch Superset with -`docker compose -f docker-compose-non-dev.yml up` and the driver should be present. +**If you are running a stock (non-customized) Superset image**, you are done. Launch Superset with `docker compose -f docker-compose-non-dev.yml up` and the driver should be present. -You can check its presence by entering the running container with `docker exec -it -bash` and running `pip freeze`. The PyPI package should be present in the printed list. +You can check its presence by entering the running container with `docker exec -it bash` and running `pip freeze`. The PyPI package should be present in the printed list. **If you're running a customized docker image**, rebuild your local image with the new driver baked in: @@ -57,26 +46,18 @@ docker compose build --force-rm After the rebuild of the Docker images is complete, relaunch Superset by running `docker compose up`. - ### 3. Connect to MySQL -Now that you've got a MySQL driver installed in your container, you should be able to connect to your -database via the Superset web UI. +Now that you've got a MySQL driver installed in your container, you should be able to connect to your database via the Superset web UI. -As an admin user, go to Settings -> Data: Database Connections and click the +DATABASE button. From -there, follow the steps on the [Using Database Connection UI page](https://superset.apache.org/docs/databases/db-connection-ui). +As an admin user, go to Settings -> Data: Database Connections and click the +DATABASE button. From there, follow the steps on the [Using Database Connection UI page](https://superset.apache.org/docs/databases/db-connection-ui). -Consult the page for your specific database type in the Superset documentation to determine the connection string and any other -parameters you need to input. For instance, on the [MySQL page](https://superset.apache.org/docs/databases/mysql), we see -that the connection string to a local MySQL database differs depending on whether the setup is running on Linux or Mac. +Consult the page for your specific database type in the Superset documentation to determine the connection string and any other parameters you need to input. For instance, on the [MySQL page](https://superset.apache.org/docs/databases/mysql), we see that the connection string to a local MySQL database differs depending on whether the setup is running on Linux or Mac. Click the “Test Connection” button, which should result in a popup message saying, "Connection looks good!". ### 4. Troubleshooting -If the test fails, review your docker logs for error messages. Superset uses SQLAlchemy to connect to databases; to -troubleshoot the connection string for your database, you might start Python in the Superset application container or host -environment and try to connect directly to the desired database and fetch data. This eliminates Superset for the -purposes of isolating the problem. +If the test fails, review your docker logs for error messages. Superset uses SQLAlchemy to connect to databases; to troubleshoot the connection string for your database, you might start Python in the Superset application container or host environment and try to connect directly to the desired database and fetch data. This eliminates Superset for the purposes of isolating the problem. Repeat this process for each different type of database you want Superset to be able to connect to.