lagoon-core v2.10.0

Lagoon 2.10 - so many changes!!!

lagoon-charts

Lagoon operators should check the lagoon-charts release notes carefully, as there may be action needed - especially if you use custom image references for the docker-host, drush-alias, overrideBuildDeployImage or UI images

Service consolidation

A large number of services have either been consolidated, removed or transferred to new repositories - this is aiming towards Lagoon having a leaner, meaner core, and having more predictable release cycles for the various components.

• kubectl-build-deploy-dind has now been consolidated and transferred to https://www.github.com/uselagoon/build-deploy-tool, and the deprecated kubectl, oc and oc-build-deploy-dind images now no longer built
• lagoon-ui has now been transferred to https://www.github.com/uselagoon/lagoon-ui and adapted for more rapid build cycles (and easier contribution)
• a host of service images (docker-host, drush-alias, athenapdf, logs-concentrator, logs-dispatcher) have now been transferred to https://www.github.com/uselagoon/lagoon-service-images as they no longer depend on lagoon-core, and can be independently updated
• the individual logs2x services (slack, teams, rocketchat, email, webhooks, s3) have all been consolidated into a single logs2notifications service, that operates identically to the previous services.
• Lagoon can now support running the storage-calculator in lagoon-remote as well as lagoon-core. This will be made available shortly at https://github.com/uselagoon/storage-calculator and added as an option to lagoon-remote in a future release.

Database updates

A large focus of this release is to prepare for a larger database upgrade in 2.11 - namely migrating from the MariaDB 10.4 pod, to a newer version of mariadb, with the added option of using a managed database service, such as RDS etc

• All foreign keys have been removed from the api-db - they were non-functional anyway (and a couple were misconfigured). We will reassess whether we should use any foreign keys in future. This will be the final migration written to operate in api-db via raw SQL. As of 2.11, we will utilise an init process to trigger knex-based migrations.

Service updates

• The golang based services have all been updated to Golang 1.18 and rejigged to share a common structure
• The storage-calculator has been updated to a more recent version of kubectl and given improved error handling abilities

API updates

• The ability to disable harbor-core integration (to be depracated in favor of harbor-remote) and opensearch integration (for installs not using lagoon-logs) has been added to the API as feature flags.
• Specifying the lagoon-build-image to be used for a lagoon-remote has now been added to the API, along with sensible defaults to match the current release - this will avoid mismatches between lagoon-core and lagoon-remote releases, and place more control in the hands of Lagoon administrators. We will also shortly start work on compatibility checking between core and remote.
• Environment and Project cleanup has been improved, with the API not allowing a project to be deleted if environments exist, and the removal of a project or environment triggering the removal of any associated variables from the API.
• GraphQL support for a couple of new features in the UI - being able to filter deployments, and being able to request Lagoon Insights files from S3.
• Additional fixes to Active/Standby, build variable merging, DeployTarget config and reading S3 buckets on S3-compatible hosts.
• Audit logging now goes to a single consolidated index.

Local development and Test changes

• The local development docker-compose has been updated to remove or repoint the consolidated/removed/transferred services
• Some test routines have been reorganised to closer align durations to avoid potential issues in github actions

What's Changed

• chore: flag to disable harbor core integration on add project by @shreddedbacon in #3267
• chore: use bridge for s3_files_host for local by @shreddedbacon in #3254
• Feature: add lagoon-core version to build payload by @shreddedbacon in #3260
• Feature: Configure remote target build image in API by @shreddedbacon in #3244
• Chore: remove logs2logs-db by @shreddedbacon in #3226
• Go svc updates by @CGoodwin90 in #3245
• Sets stricter mandatory fields for addDeployTargetConfig by @bomoko in #3239
• Update install-harbor.md by @noskovao in #3269
• Feature: return error if environments exist when deleting project by @shreddedbacon in #3261
• chore: update storage-calculator for kubernetes by @shreddedbacon in #3201
• Makes the opensearch api integration feature flaggable by @bomoko in #3268
• Feature: remove associated variables from api when environments and projects are deleted by @shreddedbacon in #2946
• Fixing CLI docs link. by @AlannaBurke in #3278
• remove kubectl, kbdd, oc and references by @tobybellwood in #3271
• Feature: logs2notifications by @shreddedbacon in #3077
• consolidated docs update by @tobybellwood in #3279
• add test step to check/remove existing kind clusters by @tobybellwood in #3265
• remove all logs2x remnants by @tobybellwood in #3280
• Fix: active/standby support for environments with slash in the name by @shreddedbacon in #3214
• Adds self excluding task source environment argument by @bomoko in #3276
• Fix anchor to SSL certificates help doc by @christopher-hopper in #3282
• use longer varchar for self exclude migration by @tobybellwood in #3285
• Fix documentation on how to generate JWT token by @achton in #3295
• Fixes logic error in build var merging by @bomoko in #3296
• Increases width of varchar(74) temporary tables in migrations by @bomoko in #3298
• remove UI from lagoon-core repo by @tobybellwood in #3290
• Make the instructions clearer for custom backup and restore buckets. by @seanhamlin in #3287
• docs: fix up hsts documentation by @shreddedbacon in #3274
• Adds new deploymentByFilter query to api by @bomoko in #3293
• feat: add storage-calculator support to actions-handler by @shreddedbacon in #3291
• build lagoon-service-images in uselagoon/lagoon-service-images by @tobybellwood in #3288
• move cronjobs and var tests out of features-kubernetes by @tobybellwood in #3301
• Filters out deleted environments in getDeploymentsByFilter by @bomoko in #3300
• Introducing s3 insights file handling to the API by @timclifford in #3056
• minor localdev fixup by @tobybellwood in #3302
• Remove (unused) foreign keys from api-db by @CGoodwin90 in #3292
• tidyup allocation of lagoon-logs audit events to projects by @tobybellwood in #3303

New Contributors

• @noskovao made their first contribution in #3269
• @achton made their first contribution in #3295

Full Changelog: v2.9.2...v2.10.0

lagoon-core v2.9.2

Updated in this release

This is a patch release of lagoon-core, addressing two main issues:

• Adding a feature flag to non-default backup configurations (different prod and env retentions or schedules) - as this WILL have unintended consequences with the current compatible version of k8up
• Minor fixes to the creation of autogenerated routes and the length of ingress labels.

The version of the build-deploy-tool used in this release is v0.15.2

What's Changed

• chore: update build-deploy-tool to v0.15.0 by @shreddedbacon in #3242
• chore: add support for basic templates to get autogenerated routes by @shreddedbacon in #3247
• chore: bump build-deploy-tool to v0.15.2 by @shreddedbacon in #3249
• fix: use restore bucket protocol from the restore url by @shreddedbacon in #3252
• refactor: feature flag backup configurations by @shreddedbacon in #3251

Full Changelog: v2.9.1...v2.9.2

lagoon-core v2.9.1

This release updates the build-deploy-tool to v0.13.6 - bringing with it some improvements to routes and tasks.

What's Changed

• Update build deploy tool with route,task fixes by @shreddedbacon in #3218
• Fix: Set podsecuritycontext values for openshift to supplemental group id by @shreddedbacon in #3228
• Feature: add image size information for built images by @shreddedbacon in #3232
• Feature: basic and worker template types by @shreddedbacon in #3221
• fix: add custom dnsConfig to all cronjobs by @smlx in #3233

New Contributors

• @CGoodwin90 made their first contribution in #3222

Full Changelog: v2.9.0...v2.9.1

lagoon-core v2.9.0

Major changes this release:

• Keycloak 16 upgrade - The keycloak-db has now been upgraded to v16 of Keycloak - this brings some great performance improvements, as well as some extra security enhancements
• Implementation of build-deploy-tool for ingress and route tasks - initially released in v2.8.0, this has been refined to better handle edge cases. In builds where the docker-compose file is not valid or readable, the build will fail with the error message that requires fixing
• Implementation of build-deploy-tool for conditional pre/post rollout tasks - as well as migrating the code for pre & post rollout tasks to the new build-deploy-tool, these tasks now support conditional definitions
• Numerous documentation changes - now bigger and better than ever
• Lagoon now builds and tests Kubernetes 1.23 in ci, in preparation for the next set of upgrades for 1.24

What's Changed

• Adding project naming requirements to documentation by @cdchris12 in #3148
• fix: document a more robust SSH file copy method by @smlx in #3149
• Updating backup docs to add info on custom backup schedules by @cdchris12 in #3154
• Updating the docs on internal container registry variable usage by @cdchris12 in #3153
• Making the cdn IP and CNAME data more easy to read by @cdchris12 in #3151
• Upgrade keycloak to v16 by @rocketeerbkw in #3119
• fix: indentation on deploytarget docs by @shreddedbacon in #3167
• fix: check for task "complete" status, as succeeded is being deprecated by @shreddedbacon in #3168
• Remove duplicate annotations key in nginx-php-persistent helm chart by @yusufhm in #3157
• fix: add group-lagoon-project-ids to project-default groups by @smlx in #3166
• feat: add bulkdeploy related variables to builds by @shreddedbacon in #3147
• remove deprecated/non-functional github actions by @tobybellwood in #3169
• fix cronjob templates by @tobybellwood in #3156
• Improve project page performance by @rocketeerbkw in #3160
• update all package versions in makefile for Kubernetes 1.23 by @tobybellwood in #3170
• docs: update docs for updating lagoon by @shreddedbacon in #3172
• Replace pre and post-rollout tasks bash scripts with build-deploy-tool by @bomoko in #3146
• Updating the redis docs by @cdchris12 in #3158
• feat: hopefully improve the wording of pre-rollout task documentation by @smlx in #3124
• [DOCS] Added a link to setting up notification system. by @AlexSkrypnyk in #3108
• Assorted docs updates - requirements and gists by @tobybellwood in #3175
• disable dev env retention policy adjustments by @shreddedbacon in #3179
• Add update deploy target for environment mutation by @bomoko in #3176
• update build-deploy-tool and add docker-compose validation step by @shreddedbacon in #3181
• always create backend_role RoleMapping for roles in OpenSearch by @tobybellwood in #3186
• chore: bump bdt tool to latest v0.11.0 by @shreddedbacon in #3188
• Docs fixes by @AlannaBurke in #3190
• update docs to remove Openshift references by @tobybellwood in #3195
• Adds LAGOON_KUBERNETES to build-deploy-docker-compose.sh by @bomoko in #3192
• Removing references in docs to community RocketChat and replacing with Discord. by @AlannaBurke in #3194
• [DOCS] Updated $BRANCHNAME with actual env variable names $LAGOON_GIT_BRANCH and $LAGOON_GIT_SAFE_BRANCH used in the script. by @AlexSkrypnyk in #3196
• chore: minor improvments to redis documentation by @smlx in #3185
• fix: updateEnvironment environment type check by @shreddedbacon in #3199
• update build-deploy-tool to latest by @shreddedbacon in #3209
• update active standby image references and timeout by @shreddedbacon in #3205
• fail builds early for invalid docker-compose files by @shreddedbacon in #3210
• Minor fix to build-deploy-tool for dbaas-endpoint check by @shreddedbacon in #3211
• fix: build required task-activestandby image by @smlx in #3213

New Contributors

• @yusufhm made their first contribution in #3157

Full Changelog: v2.8.0...v2.9.0

lagoon-core v2.8.4

This patch release of Lagoon fully reverts the changes introduced in #3133 and #3141

The Lagoon team comprehensively tested these changes prior to release. However, on release, it became apparent that a very small minority of users were using docker-compose.yml files that were incorrectly formatted, and would therefore not be able to be effectively read using our automated tools.

These errors range from full syntax errors (that would prevent docker-compose from working) to minor issues that would cause it to work with some limitations, to other use cases not envisioned in developing our features.

In order to provide the best possible Lagoon experience, this release reverts the new functionality and instead introduces a docker-compose validate step that will raise an issue in the build logs if Lagoon discovers one of these errors. We still intend to release the updated build deploy code in Lagoon v2.9.0 but intend to monitor builds in the meantime to ensure that we've captured a wide range of configurations

tags v2.8.2 and v.2.8.3 have not been converted into releases and can be safely skipped.

Full Changelog: v2.8.1...v2.8.4

lagoon-core v2.8.1

This release is a hotfix release to remedy the issue raised in #3179, and a specific use case experienced when a user is a member of multiple projects. It has backported three PRs that will be released in Lagoon v2.9.0

If you are not utilising the LAGOON_BACKUP_DEV_RETENTION and LAGOON_BACKUP_PR_RETENTION variables to set alternate retention periods for development environments, you can skip this release and wait for v2.9.0 (released in the next day or so after this)

Changes in this release

• [BACKPORT] disable dev env retention policy adjustments @shreddedbacon (#3179)
• [BACKPORT] fix: check for task "complete" status, as succeeded is being deprecated @shreddedbacon (#3168)
• [BACKPORT] Improve project page performance @rocketeerbkw (#3160)

Full Changelog: v2.8.0...v2.8.1

lagoon-core v2.8.0

In this release

This release contains a security fix for an opendistro index creation permission issue (see GHSA-7jj3-wwp7-989p)

This release also introduces a new set of logic for the build-deploy image, processing routes and ingresses via the new https://github.com/uselagoon/build-deploy-tool - gradually more of the logic will transfer across - see uselagoon/build-deploy-tool#27 for details

Additionally, the permissions needed to add environment variables to projects have been revised to match the use case, and can now be actioned by a user with "maintainer" permission, instead of the previous "owner".

This release is built on the https://github.com/uselagoon/lagoon-images/releases/tag/22.4.1 images

The lagoon-charts update will follow shortly.

What's Changed

• Security fix - Improper creation of OpenDistro/OpenSearch index role permissions by the Lagoon API
• fixup ingress and persistant storage for node-based tests by @tobybellwood in #3137
• add quiet to syft command and clean up setx by @tobybellwood in #3131
• update RBAC for project add env var to maintainer by @tobybellwood in #3075
• feat: use build tool to generate ingress templates by @shreddedbacon in #3133
• remove autoidler by @shreddedbacon in #3140
• feat: upgrade lagoon-linter v0.5.0 -> v0.7.0 by @smlx in #3138
• feat: use build tool to generate autogen routes by @shreddedbacon in #3141
• update build-deploy tool for autogen route improvements by @shreddedbacon in #3142

Full Changelog: v2.7.1...v2.8.0

lagoon-core v2.7.1

This is a patch release to remedy a couple of issues found since releasing 2.7.0

What's Changed

• fix up counter check for dbaas provisioning by @shreddedbacon in #3127
• Set OpenDistro cluster permissions correcly and configure OpenSearch backend_roles by @tobybellwood in #3123
• more explicit controller Harbor robot actions by @tobybellwood in #3128

Full Changelog: v2.7.0...v2.7.1

lagoon-core v2.7.0

This release is built on the https://github.com/uselagoon/lagoon-images/releases/tag/22.4.1 release

There are two feature updates in this release:

• Making tasks more like deployments - with similar progress stages for reporting and tracking
• Configurable backup schedules for non-production environments, with the ability to differentiate between PR and branch deployments. More information on how to configure them is in #3087

This release also remediates two earlier issues with S3 restore checks from v2.6.0 and some incorrect database migrations logic that caused the migrations not to run correctly.

What's Changed

• Fix the s3 restore check by @shreddedbacon in #3115
• Fix add_ecdsa_ssh_key_types migration by @rocketeerbkw in #3117
• Make tasks more like deployments by @shreddedbacon in #3114
• feature: configurable backup schedules and retention for development environments by @shreddedbacon in #3087
• fix errors in migrations sql by @tobybellwood in #3120

Full Changelog: v2.6.0...v2.7.0

lagoon-core v2.6.0

Security FIx

This release of Lagoon patches a security vulnerability present in Lagoon v2.5.0 only. In this release, a service-api client was added to Keycloak without a secret being automatically generated. This client isn't in a release yet (it is pre-work for the SSH portal coming shortly). THis v2.6.0 releae resolves this by automatically creating or rotating a secret. The corresponding charts release also allows for the definition of a secret, although this isn't supported in our version of keycloak yet.
If you are unable to upgrade to v2.6.0 immediately, you can log in to Keycloak, go to the service-api client, and click "Regenerate secret".

What's Changed

• fix api-db fix-permissions permissions to actually fix api-db permissions by @tobybellwood in #3081
• feature: add a timeout on the rollout status watch by @shreddedbacon in #3089
• Typo: Correct version number for Solr 8 image by @kasperg in #3054
• Typo fix by @mxr576 in #3091
• Update README.md by @AlannaBurke in #3084
• feature: add a failure notice message to pod rollout failures by @shreddedbacon in #3088
• Minor fixes to Logging and Contributing documentation by @smlx in #3079
• add rootless rsync commands to drush rsync task by @Schnitzel in #3080
• Adds ack for environments with no workflows by @bomoko in #3078
• Add example for pinning Node.js version in php-cli images by @rocketeerbkw in #3076
• refactor: capture errors for deploytargets by @shreddedbacon in #3090
• fix autogenerated urls to correctly truncate if they are too long by @shreddedbacon in #3098
• Add support in kubectl-build-deploy-dind for running rootless by @smlx in #2572
• strip acl param from multipart task file uploads by @shreddedbacon in #3097
• ECDSA ssh key type support by @cdchris12 in #3099
• Dep. trivy integration from core by @bomoko in #3083
• feat: validate TLS for all k8s API interactions by @smlx in #3107
• initial k8s install docs update by @tobybellwood in #3085
• check s3 object exists before generating signed url by @shreddedbacon in #3105
• Feature/confirmation text for custom tasks by @bomoko in #3094
• Fixing links. by @AlannaBurke in #3109
• Update upstream images and deprecate oc-build-deploy-dind by @tobybellwood in #3110
• Adds custom task argument documentation to docs by @bomoko in #3111

New Contributors

• @kasperg made their first contribution in #3054
• @mxr576 made their first contribution in #3091

Full Changelog: v2.5.0...v2.6.0