Logs Logs Logs and more!

Today marks another big step for Lagoon, with the v0.15.0 release we have now a better logging system (#478, #481) which will allow us to keep logs of:

• Router Logs (each single Hit)
• Pod Logs (stderr and stdout of every single pod)
• Lagoon Logs (build fails, etc.)
• Syslog Logs (Logs emitted via Syslog from the Applications)
  As the system is quite complicated and depending on a lot of different services, this probably will only fully automatically work with v0.16 but it's a step in the right direction.

Beside of that we have many other improvements:

• Drupal 7 Support in Varnish #479 (thanks to @ron_s for working with us on this)
• Possibility to set another memory limit for Varnish than the default 100MB - #474 (thanks to @dasrecht)
• Better Default Character sets for MariaDB so we can support all the emojis 😱💚 #490 (thanks @twardnw)
• Possibility to overwrite the storage size for MariaDB PVCs, because who would have thought that there are Databases bigger than 5GB?! - #487 thanks to @twardnw
• Better Idling behavior: Don't idle Databases (they take quite long and generate all kind of timeouts), Idle Projects only after they have not been used for 4 hours (was 1 hour before) - #473 thanks @rtprio
• Some small maxscale improvements - #494
• More documentation: #471

And a special big shout out to @markxtji which is an active user of Lagoon and started to contribute back to Lagoon, with improving the elasticsearch PVC configurations: #463 (thanks also to @thom8 for reviewing). OpenSource Love all the way!

v0.9.1

Merge remote-tracking branch 'origin/289-elasticsearch-cluster-rebase'

v0.9.0: MariaDB Galera Cluster, Auto Idle, Postgres, Remote Shell, OpenShift 3.7, and moar!

We decided to skip v0.8 and release v0.9 directly. There are 728 commits into master since then, thanks everybody who worked on this <3

Changelog

• Much better Documentation (we know we lag here a bit behind, there will be much more coming), thanks @twardnw for taking the first stab at the documentation
• MariaDB is now also an Alpine Image
• We have now MariaDB Galera Cluster support in a Statefulset. Huge thanks to @rtprio for figuring out how to build galera and maxscale within Alpine.
• MariaDB has it's own cronjob that runs mysqldump once per day
• Better Local development environments via better Make minishift commands.
• by default we disable cookies in the openshift router
• better support for drush rsync with openshift
• logs2logs-db (our logstash) can now parse logs from the openshift router (haproxy)
• Full support for idleing CLIs and non-production Services (with possibility to disable idleing for specific projects)
• CLI containers have auto logout to enable better idleing
• Lagoon Services log into elasticsearch if they loose connection to rabbitmq
• Elasticsearch watchers that alert if services log that they lost rabbitmq connection (thanks @twardnw)
• Better Pull Request Build support: Filter by regex in PR title, additional Env variables
• Cronjob support in .lagoon.yml
• Support do disable monitoring of deploymentconfig - thanks @dasrecht
• better project and customer permissions in GraphlQL API, thanks @ryyppy
• Functionality to promote Docker Images from one environment to the other instead of fully building them from scratch
• Postgres and Postgres support for Drupal
• some Drupal Nginx Config improvements
• support for Drupal Varnish Purge Module in Varnish Config
• PHP 7.2 Images
• Node 9 Images
• Lagoon Tug Build support
• Remote Shell via SSH refactoring: Allow to define which service and container should be connected to. Allow to connect to another OpenShift Server defined in Lagoon API.
• Support for Gitlab Pull Requests
• Fix for some race conditions with oc rsh
• Save Master Images on S3 for easier local development (basically removes the need to rebuild all Lagoon images by hand, use via make s3-load)
• Better default FastCGI configs in Drupal Nginx Config
• Support for OpenShift 3.7: Don't use ImageTriggers anymore, inject fully pullable docker image string into deploymentconfigs (support for projects that still include an older custom deploymentconfig)

Update Instructions:

1. Run Deployment of Lagoon
2. Run /rerun_initdb.sh inside api-db container
3. Enjoy!

Breaking Changes

• All environments which are marked as type=development or all projects which have autoidle=true are now automatically idled every 30minutes! To prevent this, make sure the correct branch is marked as production_enviornment in the project table (and redeploy after change), or set auto_idle to 0.
• The nginx fastcgi config has now better defaults: https://github.com/amazeeio/lagoon/blob/b6a8b7d3f017105565c840500cd68017e3d619d4/images/nginx/fastcgi.conf#L46-L48 if your custom nginx config defines the same nginx configs it might fail as nginx does not allow to define the same config parameter twice (huh?). So you need to remove it and add it as an environment variable in the Dockerfile (this is anyway better for future compatibility)

Help?

Find us in #lagoon in https://slack.amazee.io

v0.7.0: Cronjobs, Custom Routes, SSH wrappers, first Production Sites!

Whew! What a last couple of weeks. After 316 commits later we are super proud to present Lagoon Version 0.7.0. Here a short overview of the new goodies:

1. The API learned more Objects. We have now a new Object called Environments which is automatically propagated by new deployments. It keeps track of the generated OpenShift Projects and the type of environment (development or production). They are also automatically removed when a Branch or PR is deleted.
2. The API is now authenticated by User. JSON Web Tokens for Users include now the SSH Key of the user and with that we can figure out to which projects and customers a User belongs. Based on that the API will only show API Objects that should actually be seen by the User. (Authenticated Create, Delete and Update is still in the works)
3. The varnish Image has been split up into varnish and varnish-drupal 🔀
4. Drupal specific things of the php-cli image (drush, console) have been removed and moved into php-cli-drupal
5. There is a new service called ssh which has magic powers 🎩! It can do two things:
   • Generate JSON Web Tokens for a user with the SSH Key of the connecting User
   • Provide remote shell access to a running container of an OpenShift Project. So it basically feels like you connect via ssh to the container even though you don't :) (this also provides possibilities for rsync and other tools that depend on rsync).
6. A new lagoon type called cli-persistent which allows to mount an already existing persistent storage name into the cli container
7. Refactored Drush Aliases that auto generate a JWT for the User and makes API calls with that token, plus it also uses the new ssh connection to run drush sql-sync, drush rsync and drush ssh.
8. The php-cli-drupal images are shipped with an drushrc.php that tells Drush where it finds the Drupal root and what routes it currently has. This removes the need to cd into the correct drupal folder before running Drush commands! 💚
9. Injecting the already existing ssh key into the cli containers so that the cli itself can make remote shell commands (needed to sync a remote environment with another remote environment)
10. Updates to the nginx image:
    • Auto adding x-robots-tag if we are on a development environment
    • an easy redirect-map file that allows for easy redirects without the need to write actual nginx configs
    • an easy way to create basic auth via Environment Variables (auto generated the .htpasswd file! 💪)
11. Solr 5 and 6 Images! Also with a Drupal Flavor.
12. MongoDB Images!
13. A fully function LAGOON_ENVIRONMENT_TYPE environment variable. Also production environments cannot be deleted automatically and need to beforced
14. Mail Sending Support for php with an external Mail Relay (either provided by the hosting Infra, or something like Mailgun, etc.)
15. Custom Routes per Environment definable in .lagoon.yml
16. Cronjobs definable in .lagoon.yml
17. Much much nicer shell and bash prompts
18. Updated drupal-example with all the above mentioned changes: https://github.com/amazeeio/drupal-example
19. Fought and conquered Spectre and Meltdown on the production servers. ⚔️

And probably the most exciting but not visible part of Lagoon:
We are running the first fully Dockerized Production environment sites (we always had sites that where hybrid of the new and old system, but now it's actually all in the new system).
If you are an amazee.io client we will contact you soon how we will migrate everybody over to Lagoon. Stay tuned!

v0.6.0: Bye bye Jenkins & Hiera, Welcome Alpine & MySQL

This is the first "official" Release of Lagoon and it's a big and important one. Here is what has happened:

1. We got completely rid of Jenkins. We still use a Docker container which itself does most of the heavy lifting (it's called oc-build-deploy-dind). But instead of inside a Jenkins Job it now runs within a OpenShift Build. This has many many advantages:
   • Less services and resource usage (Jenkins is a hungry animal)
   • No need to run an external Jenkins, we are now fully self contained within OpenShift
   • The Build Container is executed within the same OpenShift Project that will also contain all necessary Ressources to run the Application (Deploymentconfigs, Routes, Services, etc.). This increases security, performance and it's much easier to see which build actually failed, as it is at the same place.
2. There is a new Service Called OpenshiftBuildDeployMonitor. In the past the OpenshiftBuildDeploy Service was continuously connected to the Build Container and waited for it to be finished. This had two disadvantages. It used a possible slot for another Build to be started and so we could have ended up in big deploy queues. If the connection between the Service and Build Container broke (TCP issues, Internet hick-ups, whatever). The Build and Deployment probably still finished but the status was lost and we could not inform the developers about the status. Now the OpenshiftBuildDeploy Service creates the Build Container and after that creates a new RabbitMQ Message inside the lagoon-tasks-monitor exchange, which then will be used by OpenShiftBuildDeployMonitor to continuously (every 5 secs) to connect tot he OpenShift and check if the Build is done. This frees up resources, makes the system more robust and overall just better.
3. We completely de-amazeeioified Lagoon. Meaning everything is now called lagoon. This ranges from .lagoon.yml Files, over RabbitMQ queue names and up to comments. As amazee.io we are proud of our name, but we want to keep Lagoon as clean as possible. The Project is called 'lagoon' it's just maintained by amazee.io and friends.
4. We are replacing the CentOS Images with Alpine Images. With v0.6.0 all of them are gone except the mariadb ones, as there are no upstream alpine-mariadb images that we easily could connect to (see #61 for process on that). All other images are now based on Alpine and with that are not only smaller and faster but also more secure als they provide less attack vector.
5. Hiera is no more. Out of legacy reasons of the amazee.io Hosting Platform Version 3 (which is built on Puppet), the Lagoon internal API storage was based on YAML Files saved within a Git repo. Sounds weird and painful? Yes it was. So we replaced it with a proper Database, one that we already know: MariaDB.
6. Moar GraphQL. We are improving our Lagoon API more and more. With the v0.6.0 release there is the possibility to create new Projects, Customers, and all other API Objects directly via GraphQL. The deletion and update of the same is still in the works (#102) and we need to authenticate our users (#109) as today there is just a full admin access to everything. But we are finally getting where we want to be: Full API Access.
7. Better Install Documentation! (We are very sorry for all the people that tried their best to get Lagoon up and running and where failing because we didn't provide all the required documentation). We really wanted to get all changes in this release in first before anybody else tried to run it, unfortunately our planning didn't really work out so we had a couple of weeks without proper documentation. But this is now solved. Find it here.
8. We're testing with OpenShift 3.6.1 now
9. Updated the Lagoon Schema to reflect the removed Jenkins and new Services

We want to thank everybody who helped pushing Lagoon forwards, is it via actual code contributions, fixing documentation, bringing up ideas or just trying it out.

The amazee.io team.