diff --git a/README.md b/README.md index 2b85f2a90..0e76ffc63 100644 --- a/README.md +++ b/README.md @@ -111,7 +111,7 @@ dotnet ef migrations add TestMigration Besides ordinary unit and integration tests, there are test suites for both functional and non-functional end-to-end tests implemented with [K6](https://k6.io/). -See `tests/k6/README.md` for more information. +See [tests/k6/README.md](tests/k6/README.md) for more information. ## Health Checks diff --git a/tests/k6/README.md b/tests/k6/README.md index b47da07df..c2d043be8 100644 --- a/tests/k6/README.md +++ b/tests/k6/README.md @@ -23,6 +23,14 @@ Run `Get-Help .\run.ps1` or `./run.sh --help` for usage information. The scripts will use locally installed `k6` if available. Failing that, it will attempt to use Docker with [grafana/k6](https://hub.docker.com/r/grafana/k6) +## Performance tests + +For information about the performance tests, see the following README files: + +- [ServiceOwner](tests/serviceowner/performance/README.md) +- [GraphQL](tests/graphql/performance/README.md) +- [EndUser](tests/enduser/performance/README.md) + ## Test suites Various test suites are defined withing the `suites` directory. A suite consists of diff --git a/tests/k6/tests/enduser/performance/README.md b/tests/k6/tests/enduser/performance/README.md index 0a88e425a..9a9810e36 100644 --- a/tests/k6/tests/enduser/performance/README.md +++ b/tests/k6/tests/enduser/performance/README.md @@ -10,7 +10,7 @@ The test file associated with this performance test is - `enduser-search.js` ## Test description -The test has a list of enduser (ssn) with pre-generated tokens, and the following endpoints are visited in +The test has a list of enduser (ssn), and the following endpoints are visited in sequence for each enduser: - api/v1/enduser/dialogs?Party=urn:altinn:person:identifier-no:`` - api/v1/enduser/dialogs/`` @@ -28,19 +28,14 @@ sequence for each enduser: ```shell cd tests/k6/tests/enduser/performance ``` -2. Generate tokens using the script below. Make sure to replace ``, `` and `` with your desired values: -```shell -TOKEN_GENERATOR_USERNAME= \ -TOKEN_GENERATOR_PASSWORD= API_ENVIRONMENT= \ -../../scripts/generate_tokens.sh ../../performancetest_data personal -``` -3. Run the test using the following command. Replace ``, ``, and `` with the desired values: +2. Run the test using the following command. Replace ``, ``, and `` with the desired values: ```shell +TOKEN_GENERATOR_USERNAME= TOKEN_GENERATOR_PASSWORD= \ k6 run enduser-search.js -e API_VERSION=v1 \ -e API_ENVIRONMENT= \ --vus= --duration= ``` -4. Refer to the k6 documentation for more information on usage. +3. Refer to the k6 documentation for more information on usage. ### From GitHub Actions To run the performance test using GitHub Actions, follow these steps: @@ -56,26 +51,16 @@ To run the performance test locally using GitHub Actions and act, perform the fo ```file TOKEN_GENERATOR_USERNAME: TOKEN_GENERATOR_PASSWORD: -K6_CLOUD_PROJECT_ID=** -K6_CLOUD_TOKEN=** -K6_PROMETHEUS_RW_USERNAME=** -K6_PROMETHEUS_RW_PASSWORD=** -K6_PROMETHEUS_RW_SERVER_URL=** ``` - Replace `` and ``, same as for generating tokens above. Fill in the K6_* values if available, - used for reporting to Grafana cloud + Replace `` and ``, same as for generating tokens above. ##### IMPORTANT: Ensure this file is added to .gitignore to prevent accidental commits of sensitive information. Never commit actual credentials to version control. 4. Run `act` using the command below. Replace `` and `` with the desired values: ```shell act workflow_dispatch -j k6-performance -s GITHUB_TOKEN=`gh auth token` \ --container-architecture linux/amd64 --artifact-server-path $HOME/.act \ --input vus= --input duration= \ ---input testSuitePath=tests/k6/tests/enduser/performance/enduser-search.js \ ---input tokens=personal +--input testSuitePath=tests/k6/tests/enduser/performance/enduser-search.js ``` ## Test Results -Test results can be found in GitHub action run log and in App Insights. We are prepared for exporting results to grafana, but so far results are exported to a private grafana instance only, as can be seen from the `.secrets`listed earlier - -## TODO -Fix reporting +Test results can be found in GitHub action run log, grafana and in App Insights. diff --git a/tests/k6/tests/graphql/performance/README.md b/tests/k6/tests/graphql/performance/README.md index 0faa92506..011f45af3 100644 --- a/tests/k6/tests/graphql/performance/README.md +++ b/tests/k6/tests/graphql/performance/README.md @@ -15,19 +15,14 @@ The test file associated with this performance test is ```shell cd tests/k6/tests/graphql/performance ``` -2. Generate tokens using the script below. Make sure to replace ``, `` and `()` with your desired values: -```shell -TOKEN_GENERATOR_USERNAME= \ -TOKEN_GENERATOR_PASSWORD= API_ENVIRONMENT=<(test|staging|yt01)> \ -../../scripts/generate_tokens.sh ../../performancetest_data personal -``` -3. Run the test using the following command. Replace `<(test|staging|yt01)>`, ``, and `` with the desired values: +2. Run the test using the following command. Replace `<(test|staging|yt01)>`, ``, and `` with the desired values: ```shell +TOKEN_GENERATOR_USERNAME= TOKEN_GENERATOR_PASSWORD= \ k6 run graphql-search.js -e API_VERSION=v1 \ -e API_ENVIRONMENT=<(test|staging|yt01)> \ --vus= --duration= ``` -4. Refer to the k6 documentation for more information on usage. +3. Refer to the k6 documentation for more information on usage. ### From GitHub Actions To run the performance test using GitHub Actions, follow these steps: 1. Go to the [GitHub Actions](https://github.com/digdir/dialogporten/actions/workflows/dispatch-k6-performance.yml) page. @@ -42,26 +37,16 @@ To run the performance test locally using GitHub Actions and act, perform the fo ```file TOKEN_GENERATOR_USERNAME: TOKEN_GENERATOR_PASSWORD: -K6_CLOUD_PROJECT_ID=** -K6_CLOUD_TOKEN=** -K6_PROMETHEUS_RW_USERNAME=** -K6_PROMETHEUS_RW_PASSWORD=** -K6_PROMETHEUS_RW_SERVER_URL=** ``` - Replace `` and ``, same as for generating tokens above. Fill in the K6_* values if available, - used for reporting to Grafana cloud + Replace `` and ``, same as for generating tokens above. ##### IMPORTANT: Ensure this file is added to .gitignore to prevent accidental commits of sensitive information. Never commit actual credentials to version control. 4. Run `act` using the command below. Replace `` and `` with the desired values: ```shell act workflow_dispatch -j k6-performance -s GITHUB_TOKEN=`gh auth token` \ --container-architecture linux/amd64 --artifact-server-path $HOME/.act \ --input vus= --input duration= \ ---input testSuitePath=tests/k6/tests/graphql/performance/graphql-search.js \ ---input tokens=personal +--input testSuitePath=tests/k6/tests/graphql/performance/graphql-search.js ``` ## Test Results -Test results can be found in GitHub action run log and in App Insights. We are prepared for exporting results to grafana, but so far results are exported to a private grafana instance only, as can be seen from the `.secrets`listed earlier - -## TODO -Fix reporting +Test results can be found in GitHub action run log, grafana and in App Insights. \ No newline at end of file diff --git a/tests/k6/tests/scripts/generate_tokens.sh b/tests/k6/tests/scripts/generate_tokens.sh deleted file mode 100755 index f9d39ab9a..000000000 --- a/tests/k6/tests/scripts/generate_tokens.sh +++ /dev/null @@ -1,111 +0,0 @@ -#!/bin/bash - -# Check if required environment variables are set -if [ -z "$TOKEN_GENERATOR_USERNAME" ] || [ -z "$TOKEN_GENERATOR_PASSWORD" ] || [ -z "$API_ENVIRONMENT" ]; then - echo "Error: TOKEN_GENERATOR_USERNAME, TOKEN_GENERATOR_PASSWORD, and API_ENVIRONMENT must be set" - exit 1 -fi - -# Function to display usage information -usage() { - echo "Usage: $0 " - echo " : Path to the test data files" - echo " : Type of tokens to generate (both, enterprise, or personal)" - echo " : limit number of tokens to generate. 0 means generate all" - echo " : Time to live in seconds for the generated tokens" - echo "Example: $0 /path/to/testdata both 10 3600" - exit 1 -} - -# Validate arguments -if [ $# -ne 4 ]; then - usage -fi - -tokengenuser=${TOKEN_GENERATOR_USERNAME} -tokengenpasswd=${TOKEN_GENERATOR_PASSWORD} - -env="" -case $API_ENVIRONMENT in - "test") - env="at21" ;; - "staging") - env="tt02" ;; - "yt01") - env="yt01" ;; - *) - echo "Error: Unknown api environment $API_ENVIRONMENT" - exit 1 ;; -esac - -testdatafilepath=$1 -tokens=$2 -limit=$3 -ttl=$4 - -# Validate tokens argument -if [[ ! "$tokens" =~ ^(both|enterprise|personal)$ ]]; then - echo "Error: Invalid token type. Must be 'both', 'enterprise', or 'personal'." - usage -fi - -serviceowner_datafile="$testdatafilepath/serviceowners-$API_ENVIRONMENT.csv" -serviceowner_tokenfile="$testdatafilepath/.serviceowners-with-tokens.csv" -enduser_datafile="$testdatafilepath/endusers-$API_ENVIRONMENT.csv" -enduser_tokenfile="$testdatafilepath/.endusers-with-tokens.csv" - -if [ "$tokens" = "both" ] || [ "$tokens" = "enterprise" ]; then - if [ ! -f "$serviceowner_datafile" ]; then - echo "Error: Input file not found: $serviceowner_datafile" - exit 1 - fi - echo "org,orgno,scopes,resource,token" > $serviceowner_tokenfile - generated=0 - while IFS=, read -r org orgno scopes resource - do - if [ $limit -gt 0 ] && [ $generated -ge $limit ]; then - break - fi - url="https://altinn-testtools-token-generator.azurewebsites.net/api/GetEnterpriseToken?org=$org&env=$env&orgno=$orgno&ttl=$ttl" - token=$(curl -s -f --get --data-urlencode "scopes=$scopes" $url -u "$tokengenuser:$tokengenpasswd" ) - if [ $? -ne 0 ]; then - echo "Error: Failed to generate enterprise token for: $env, $org, $orgno, $scopes " - continue - fi - echo "$org,$orgno,$scopes,$resource,$token" >> $serviceowner_tokenfile - status=$? - if [ $status -ne 0 ]; then - echo "Error: Failed to write enterprise token to file for: $env, $org, $orgno, $scopes" - else - ((generated++)) - fi - done < <(tail -n +2 $serviceowner_datafile) -fi - -if [ "$tokens" = "both" ] || [ "$tokens" = "personal" ]; then - if [ ! -f "$enduser_datafile" ]; then - echo "Error: Input file not found: $enduser_datafile" - exit 1 - fi - echo "ssn,resource,scopes,token" > $enduser_tokenfile - generated=0 - while IFS=, read -r ssn resource scopes - do - if [ $limit -gt 0 ] && [ $generated -ge $limit ]; then - break - fi - url="https://altinn-testtools-token-generator.azurewebsites.net/api/GetPersonalToken?env=$env&scopes=$scopes&pid=$ssn&ttl=$ttl" - token=$(curl -s -f $url -u "$tokengenuser:$tokengenpasswd" ) - if [ $? -ne 0 ]; then - echo "Error: Failed to generate personal token for: $ssn, $scopes " - continue - fi - echo "$ssn,$resource,$scopes,$token" >> $enduser_tokenfile - status=$? - if [ $status -ne 0 ]; then - echo "Error: Failed to write personal token to file for: $ssn, $scopes" - else - ((generated++)) - fi - done < <(tail -n +2 $enduser_datafile) -fi diff --git a/tests/k6/tests/serviceowner/performance/README.md b/tests/k6/tests/serviceowner/performance/README.md index 4155dfb6f..240592131 100644 --- a/tests/k6/tests/serviceowner/performance/README.md +++ b/tests/k6/tests/serviceowner/performance/README.md @@ -10,6 +10,7 @@ Before running the performance test, make sure you have met the following prereq The test files associated with this performance test are - `create-dialog.js` - `create-remove-dialog.js` +- `create-transmissions.js` - `serviceowner-search.js` - `purge-dialogs.js` (used for cleanup after test) @@ -21,19 +22,14 @@ To run the performance test, follow the instructions below: ```shell cd tests/k6/tests/serviceowner/performance ``` -2. Generate tokens using the script below. Make sure to replace ``, `` and `<(test|staging|yt01)>` with your actual desired values: -```shell -TOKEN_GENERATOR_USERNAME= \ -TOKEN_GENERATOR_PASSWORD= API_ENVIRONMENT=<(test|staging|yt01)> \ -../../scripts/generate_tokens.sh ../../performancetest_data both -``` -3. Run the test using the following command. Replace ``, `<(test|staging|yt01)>`, ``, and `` with the desired values: +2. Run the test using the following command. Replace ``, `<(test|staging|yt01)>`, ``, and `` with the desired values: ```shell +TOKEN_GENERATOR_USERNAME= TOKEN_GENERATOR_PASSWORD= \ k6 run -e API_VERSION=v1 \ -e API_ENVIRONMENT=<(test|staging|yt01)> \ --vus= --duration= ``` -4. Refer to the k6 documentation for more information on usage. +3. Refer to the k6 documentation for more information on usage. #### From GitHub Actions To run the performance test using GitHub Actions, follow these steps: @@ -49,22 +45,15 @@ To run the performance test locally using GitHub Actions and act, perform the fo ```file TOKEN_GENERATOR_USERNAME: TOKEN_GENERATOR_PASSWORD: -K6_CLOUD_PROJECT_ID=** -K6_CLOUD_TOKEN=** -K6_PROMETHEUS_RW_USERNAME=** -K6_PROMETHEUS_RW_PASSWORD=** -K6_PROMETHEUS_RW_SERVER_URL=** ``` - Replace `` and ``, same as for generating tokens above. Fill in the K6_* values if available, - used for reporting to Grafana cloud + Replace `` and ``, same as for generating tokens above. ##### IMPORTANT: Ensure this file is added to .gitignore to prevent accidental commits of sensitive information. Never commit actual credentials to version control. 4. Run `act` using the command below. Replace ``, ``, `` and `<(personal|enterprise|both)>` with the desired values: ```shell act workflow_dispatch -j k6-performance -s GITHUB_TOKEN=`gh auth token` \ --container-architecture linux/amd64 --artifact-server-path $HOME/.act \ --input vus= --input duration= \ ---input testSuitePath= \ ---input tokens=<(personal|enterprise|both)> +--input testSuitePath= ``` Example of command: @@ -72,8 +61,7 @@ Example of command: act workflow_dispatch -j k6-performance -s GITHUB_TOKEN=`gh auth token` \ --container-architecture linux/amd64 --artifact-server-path $HOME/.act \ --input vus=10 --input duration=5m \ ---input testSuitePath=tests/k6/tests/serviceowner/performance/create-dialog.js \ ---input tokens=enterprise +--input testSuitePath=tests/k6/tests/serviceowner/performance/create-dialog.js ``` #### Clean up @@ -97,7 +85,4 @@ Replace `<(test|staging|yt01)>` with the appropriate environment where the test This script will remove any dialogs created during the performance test, ensuring a clean state for future tests. ### Test Results -The test results can be found in the GitHub Actions run log and in App Insights. Currently, the results are exported to a private Grafana instance. Refer to the `.secrets` file mentioned earlier for more details. - -### TODO -- Fix reporting +The test results can be found in the GitHub Actions run log, grafana and in App Insights.