End to End tests: Publish using functions core tools

Overview

The end-to-end tests are located inside the directory .ci/e2e/. The setup and tests are inside the sub-directory publish_tests. Additionally, there's a Dockerfile running_environments/Docker/test_runner.Dockerfile for building a test container and running the tests locally on an environment.

These tests are also picked up by a nightly build specified in the azure-pipelines-nightly.yml file in the project root directory.

Coverage

These tests cover testing scenarios for publishing from azure-functions-core-tools (func) using the azure-functions-python-worker and azure-functions-docker (docker). func uses a docker image to build a publishing package. The end-to-end test cases test the scenarios of using the production and development versions of func with those of the docker images by publishing different Python function projects to production Azure function apps.

Tests Structure

Inside .ci/e2e/publish_tests, there are two setup directories each for setting up a dev build of the azure-functions-core-tools and a dev image for the Python image from azure-functions-docker. The func_tests_core is the sub-directory that holds all the actual tests. The test_runners holds all the tests that use different combinations of func and appropriate docker image to run a test. Additionally, the file publish_config.json specifies all the configurable options for running these apps.

Setup Scripts

Environment Setup

The test_runners\setup_container_environment.sh hold the commands to setup any testing environment by installing dependencies like docker, azure-cli, production func, jq, etc.

Artifacts Setup

The test_runners\setup_test_environment.sh use the scripts in dev_docker_setup and the dev_func_setup to set the development version of the artifacts to be tested.

• dev_docker_setup: It downloads the azure-functions-docker repository and build the base image and the Python image locally from the Dockerfile in that repository. It then builds another Dockerfile dev_docker_setup/dev.Dockerfile that builds on top of the local Python image (built in the last step), and adds the newest azure-functions-python-worker code to it. The script then tags that image and pushes it to an Azure Container Registry (specified in publish_config.json passed as params by setup_test_environment) to be used by the func CLI in the tests.

• dev_func_setup: This script downloads the func CLI from the link received in params (passed by setup_test_environment specified in publish_config.json)

Core Tests Scripts

These tests are located in func_tests_core sub-directory. There are two sub-directories or suites of tests. One for an existing function project (customer_churn_app) and the other one (new_functionapp) that creates a new function project. Each of these have two or three tests depending on the publishing flow availables-

• packapp_test.sh tests func azure functionapp publish <appname>
• build_native_deps_test.sh tests func azure functionapp publish <appname> --build-native-deps
• no_bundler_test.sh tests func azure functionapp publish <appname> --build-native-deps --no-bundler

Each of the tests need 4 command line arguments- name of working dir, name of function app to publish, path to the func executable and the name of the docker image to use. These core-tests are exploited by the test runners with the appropriate parameters.

Test Runner Scripts

The test_runners directory has four test-suites (dev_func_dev_docker, dev_func_prod_docker, prod_func_dev_docker, dev_func_prod_docker) each is a different combination of func CLI and docker image. Each of those have 5 test runners. Each test runner runs a test from func_tests_core with the appropriate parameters. Example- each of the test in dev_func_dev_docker with call the adjacent func_tests_core script with the path to the dev func artifact and the name of the dev docker image. Each script also reads other configuration from publish_settings.json for variables like Azure function app name to publish.

The test_runners directory also includes two test starter scripts- run_all_parallel.sh and run_all_serial.sh. Each of these script will run all the tests described above and emit a table with the results. These starter scripts will also store the logs of the tests and can be configured to be verbose or silent.

run_all_parallel.sh is not recommended unless some of the tests are commented out or skipped. If using this, please make sure that the function apps specified in publish_config.json are different for each tests as no locking mechanism is implemented yet.

The setup_container_environment.sh and setup_test_environment.sh are used to setup the runners (explained above).

Run the tests

You would need a couple of Azure resources for a one time setup-

• An Azure Service Principal with access to all resources below
• An Azure Container Registry
• An Azure Function App (or multiple if you want to run in parallel)

Once you have the above resources, fill out publish_config.json with the appropriate information (if running locally).

After that, you can quickly start a test execution in one of the two ways-

1. Using Azure DevOps Pipeline: You can use the azure-pipelines-nightly.yml to create a DevOps pipeline. Make sure to either modify publish_config.json or set the pipeline variable appropriately.

2. Using a Docker Container locally: After you have populated publish_config.json, you just need to run these two commands:

> docker build -f .ci\e2e\running-environments\Docker\test_runner.Dockerfile -t my-test-image <path-to-python-worker>
> docker run -v /var/run/docker.sock:/var/run/docker.sock my-test-image

The <path-to-python-worker> should the path to your local repository for azure-functions-python-worker. You need to mount the docker socket -v /var/run/docker.sock:/var/run/docker.sock so that your container has permissions to spawn sister containers. This is needed because the tests involve running docker containers. The Docker container will hang infinitely after the execution for you to allow to exec bash in your container and look at logs if you desire.

Once execution is complete, you should see an output like this:

In addition to the above steps, you can also run any test individually through the runner as long as you have the settings populated.

Add a test

Let's say we have a new publish flag in func where you can publish like- func azure functionapp publish <app-name> --do-magic.

1. First, we need to add a testing scenario each for new_functionapp and customer_churn_predict because we would want to test for both cases- creating a new function project and publishing an existing with the --do-magic flag. We will create a file do_magic_test.sh in both new_functionapp and customer_churn_predict. The test will be of a similar structure as the other tests, except it should do publishing like this FUNCTIONS_PYTHON_DOCKER_IMAGE=$4 "$3" azure functionapp publish "$2" --do-magic.

2. Then, inside test-runners, we will add two files to each test-suite (dev_func_dev_docker, dev_func_prod_docker, prod_func_dev_docker, dev_func_prod_docker). Let's call the two files- customer_do_magic.sh and new_do_magic.sh. We would have added 8 files in test-runners. Each of those test runners would be similar to the others, except they would call the new core tests we just added. It would call the core tests with the parameters appropriate to the artifacts being tested (dev or prod) just like other runners.

3. Now that we have added the tests and the runners, we just need to update our starter scripts run_all_parallel.sh and run_all_serial.sh to include the new 8 test-runners we just added.

4. That's it. You should be able to run all the tests now (including the new ones).