GA type selection and registration abandonment, refactor deployment process, and enhance Docker image handling #437
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: jankun4 <michaljankun@gmail.com>
…ment efficiency This commit reorganizes the build and deployment scripts by introducing separate Makefiles for the backend and frontend components, along with common utilities and informational targets. The new structure aims to improve modularity, maintainability, and clarity of the deployment process. Component-specific Makefiles handle Docker image construction and deployment more effectively, while shared utilities and variables are centralized for ease of use. Additionally, the update simplifies Docker tag generation by incorporating it directly into the Makefiles, eliminating the need for separate versioning scripts. The README has been updated to guide users through the new deployment process.
…stem This commit eliminates the prepare-config.sh script, incorporating its configuration preparation logic into the Makefile workflow with a new config.mk file. This adjustment enhances the consistency of the deployment process, aligning it with the modular Makefile approach. It simplifies the management of configurations by dynamically generating necessary files from templates, thereby improving the security and maintainability of configuration secrets. Adjustments to related Makefiles files have been made to accommodate these changes, streamlining the entire configuration and deployment pipeline.
This commit streamlines the deployment process by consolidating the Docker Compose configurations into a single template file, which is then dynamically adjusted by the Makefile based on specific deployment environments. This enhancement reduces redundancy and facilitates easier management of environment-specific settings. The transition involves updating the `Makefile` and `config.mk` to support the generation of Docker Compose files from the template, as well as modifying the `README.md` to align with the new configuration approach. This change marks a significant improvement in the deployment workflow, making it more efficient and maintainable.
…ation This commit updates the CHANGELOG.md to reflect two significant improvements: - The adoption of a unified policy for tagging Docker images, addressing issue 320. This change ensures consistency in how Docker images are tagged across the project, facilitating easier management and deployment of images. - The reorganisation of deployment Makefiles as part of issue 385. This enhancement aims at better documenting the deployment process and simplifying the management of deployment scripts. The new structure of Makefiles contributes to a more streamlined and understandable deployment process. These updates are part of ongoing efforts to enhance the deployment process by refining project documentation and operational scripts.
…ling Optimized the build-and-deploy GitHub Actions workflow to streamline the deployment process. Removed redundant `make docker-login` and `make build-[backend|frontend]` steps, focusing directly on pushing the pre-built Docker images with `make push-backend` and `make push-frontend`. Added a crucial verification step to ensure the deployment environment is properly defined, enhancing the reliability of the deployment pipeline. This update ensures a more efficient and error-resilient deployment process by minimizing unnecessary steps and reinforcing environmental checks.
Consolidated and streamlined the GitHub Actions workflow for deploying the application by removing redundant build steps and focusing solely on deployment tasks. This simplification entails a direct approach to deploying the app, removing previous conditional steps for building Docker images and checking environment existence. The workflow now concentrates on setting up AWS credentials, SSH agent configuration, and executing a singular deployment command. This change leads to a more efficient, straightforward deployment process, enhancing the maintainability and readability of the CI/CD pipeline.
Eliminated unnecessary fake database synchronization settings from the GitHub Actions deployment workflow and the backend configuration template. This action streamlines the deployment and configuration process by removing redundant settings, focusing solely on the essential configurations required for real database synchronization. The simplification contributes to clearer, more maintainable deployment scripts and backend configuration, ensuring an efficient setup for actual operational needs.
Standardized the format of placeholders within template files across the deployment setup to improve the consistency and reliability of dynamic configuration generation. Modified `config.mk`, `backend-config.json.tpl`, and `grafana-provisioning/alerting/alerting.yml` to use a uniform placeholder notation, enclosed in angle brackets (e.g., `<PLACEHOLDER>`), for all configurable values. This refinement simplifies the template processing logic, ensuring a more streamlined and error-free substitution of environment-specific values during the deployment process, thereby enhancing the overall maintainability of the deployment infrastructure.
This commit enhances the efficiency of the Docker Compose execution process within the deployment script. By modifying the deploy-stack target in the Makefile, we've streamlined the command to not only update containers as necessary but also to clean up any orphaned containers in a more efficient manner. The key changes include: - Unified Pull and Deploy: Merging the pull and up commands into a single docker compose up command with the --detach, --pull always, and --remove-orphans flags. This optimization ensures that all required images are pulled fresh before starting containers, thereby reducing the risk of running outdated versions. - Simplification of Deployment Process: Removing separate pull commands simplifies the deployment process, making it faster and reducing the potential for errors during manual deployment steps. - Automatic Orphan Removal: The addition of --remove-orphans flag automatically cleans up any containers not defined in the compose file but previously running, keeping the environment clean and reducing manual cleanup efforts. These modifications are designed to facilitate a more streamlined, efficient, and error-resistant deployment process, directly supporting the goal of improving deployment traceability and operational efficiency within the project's continuous deployment pipeline.
This commit introduces a new utility target, reload-grafana, to the project's utility makefile (utils.mk). This target provides an automated way to trigger the reload of Grafana provisioning configurations, including alerting rules, dashboards, and notification channels. This enhancement directly supports the initiative for improved deployment traceability and operational efficiency by ensuring that monitoring and alerting configurations can be updated and applied with minimal downtime and manual effort.
This commit enhances the GitHub Actions CI/CD workflow to more efficiently and clearly set the deployment domain based on the target environment. The improvements made in the .github/workflows/build-and-deploy.yml file include: - Consolidated Domain Setting Step: Introduced a dedicated step to set the DOMAIN environment variable once, before the deployment and provisioning actions. This eliminates redundancy and potential errors by ensuring the DOMAIN is consistently set for all subsequent steps. - Environment-Specific Domain Logic: Implemented a clear, conditional logic block that sets the DOMAIN variable based on the inputs.environment value. This approach supports multiple environments (staging, beta, and others) by assigning them appropriate domain names in a single, centralized location. - Fallback Domain Naming Convention: For environments not explicitly defined (other than staging or beta), a default naming convention is applied, combining the environment name, the CARDANO_NETWORK variable, and a base domain. This ensures flexibility and scalability in deploying to new or temporary environments without needing workflow modifications. This optimization supports a more maintainable and error-proof deployment process by simplifying domain management across different deployment environments, directly contributing to the overall goal of improving deployment traceability and operational efficiency.
This commit reintroduces a previously removed crucial step in the .github/workflows/build-and-deploy.yml CI/CD pipeline that conditionally destroys and resets the Cardano node, DB sync, and Postgres components. The key aspects of this restoration include: - Conditional Step Execution: The step is guarded by a condition checking inputs.resync_cardano_node_and_db. This ensures that the infrastructure reset only occurs when explicitly required, minimizing disruption and unnecessary operations. - Infrastructure Clean-up: By executing make destroy-cardano-node-and-dbsync, this step ensures a clean state for these components when necessary, facilitating scenarios where a fresh start is required for accurate testing or deployment. - Support for Deployment Traceability: This action aligns with the overarching goal of deployment traceability by allowing for controlled environment resets, ensuring that deployments can be accurately traced and managed across different states of the infrastructure. This restoration corrects a critical oversight in the deployment process, reinstating the ability to manage the project's infrastructure effectively and ensuring that deployments are performed in a clean and predictable environment when necessary.
This commit introduces Loki and Promtail to the deployment stack, enhancing the project's logging infrastructure by enabling advanced log aggregation, visualization, and analysis capabilities. The integration involves modifications across multiple configuration files and the introduction of new templates for Loki and Promtail. Here's a breakdown of the changes: - config.mk Adjustments: Updated the prepare-config target to include steps for preparing Loki and Promtail configuration files, ensuring these services are configured as part of the deployment setup. - Docker Compose Template Update: Extended the docker-compose.yml.tpl to define loki and promtail services, including their respective Docker images, volumes, and dependencies. This update ensures that both Loki for log aggregation and Promtail for log shipping are part of the containerized environment. - Grafana Data Sources Configuration: Modified the grafana-provisioning/datasources/datasource.yml to include Loki as a data source, enabling Grafana to query log data collected by Loki. This allows for comprehensive log visualization alongside metrics within Grafana dashboards. - Loki Configuration Template (loki.yml): Introduced a new configuration template for Loki, specifying its operational parameters such as server ports, ingester configuration, schema, storage, and retention policies. This configuration is designed to optimize Loki's log aggregation capabilities within the deployment environment. - Promtail Configuration Template (promtail.yml): Added a new configuration template for Promtail, detailing how log files are located, scraped, and forwarded to Loki. The configuration includes the definition of job names, static configs, and labels for log categorization. The integration of Loki and Promtail represents a significant enhancement to the project's observability infrastructure. It enables efficient log collection, storage, and querying, providing developers and operators with powerful tools for troubleshooting, performance monitoring, and operational analysis.
This commit enhances the infrastructure setup process by incorporating an initialization step for a deployment log within the EC2 instance's user data script. This change is aimed at bolstering deployment traceability and logging mechanisms for the GovTool project. Key aspects of the update include: - Deployment Log File Creation: Added commands to create a new log file, /var/log/deployment.log, during the initial setup of the EC2 instance. This log file is intended to capture and store deployment-related events, making it easier to track and diagnose issues that may arise during or after the deployment process. - Log File Ownership Adjustment: The ownership of the newly created deployment log file is set to ubuntu:ubuntu, ensuring that the application, running under the ubuntu user, has the necessary permissions to write deployment events to the file. This modification directly supports the project's objective to enhance deployment traceability and operational monitoring by providing a dedicated mechanism for logging deployment activities, thereby facilitating better insight into deployment processes and outcomes.
…process - Modified 'all' target in Makefile to exclude 'info' and directly include 'notify', simplifying the deployment sequence. - Introduced 'log-deployment' in info.mk for logging deployment details to /var/log/deployment.log. This includes capturing essential deployment metadata such as the environment, branch, commit, and versions of the deployed components, formatted with a timestamp and marked as INFO for clarity. - Updated 'notify' target to depend on 'info' and 'log-deployment', ensuring comprehensive deployment information is shared and recorded. This includes deployment success notifications with detailed component versions and source information. - Adjusted the format for logging deployment information to improve readability and consistency in monitoring tools. These changes enhance the project's deployment traceability, allowing for better tracking of deployments and easier debugging and auditing of deployed versions.
- Added a new panel to the GovTool Grafana dashboard configuration for visualizing deployment logs. - This panel uses the Loki data source to display logs from `/var/log/deployment.log`, providing a trace of every deployment made. - The panel is configured to display logs in descending order, focusing on deployment events. This aims to facilitate quick identification of deployment activities and troubleshooting issues. - Updated the dashboard version to reflect these changes, ensuring that the dashboard remains up-to-date and functional with the latest configurations. This enhancement improves the observability of the deployment process, making it easier for operators to monitor deployment activities and investigate issues directly from the Grafana interface.
…ages tagging This commit modifies the build-and-deploy workflow file to include fetching the full history of the repository during deployment. This change is essential for accurately calculating the Docker image tags for both backend and frontend components in the CI/CD pipeline. By setting the `fetch-depth` to 0 in the step that checks out the code, the workflow now ensures that all historical commits are available, enabling precise version information retrieval for tagging the Docker images. Changes made: - Modified: .github/workflows/build-and-deploy.yml to add `fetch-depth: 0` parameter in the step that checks out the code, ensuring full repository history is fetched during deployment.
This commit adds logging of target indicators during the deployment process to improve visibility and tracking of deployment steps. Changed the deployment workflow in .github/workflows/build-and-deploy.yml to include target indicators logging when performing certain actions, such as destroying the Cardano Node and DB sync, deploying the application, and reprovisioning Grafana. The changes involve adding the '--debug=w' flag to the 'make' commands for destroying Cardano Node and DB sync, deploying the application, and reloading Grafana, allowing for more detailed logging during these actions.
This commit corrects the debug option in make commands in the CI/CD pipeline workflow, so that the deployment process in the CI can execute without issues related to unrecognized debug flags. The existing 'make' commands were utilizing the debug flag '--debug=w' which was causing compatibility issues in the CI environment where 'why' debug option was not recognized. This commit addresses the problem by replacing the 'w' with 'b' in the relevant 'make' commands in the '.github/workflows/build-and-deploy.yml' file, ensuring smooth execution of the deployment tasks.
Added a grafana panel to track all the deploys on the target machines. This change was made to address Issue 361.
In order to ensure accurate deployment traceability, it is necessary to use the highest existing version of the Grafana dashboard incremented by 1. This adjustment is crucial as the target instances have intermediate versions beyond version 35, and leveraging the most current version will maintain consistency in the deployment process. Changes: - Updated the version of the Grafana dashboard to 44 in the `govtool.json` file under `scripts/govtool/config/templates/grafana-provisioning/dashboards/`. By adjusting the version number, we align the dashboard with the latest iteration, thereby enhancing traceability and ensuring compatibility with the current deployment environment.
pmbinapps
requested review from
placek,
adgud,
kickloop,
MSzalowski,
jankun4,
Sworzen1 and
JanJaroszczak
as code owners
placek
approved these changes
Mar 8, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[#358] [#359] [#151] choose GA type/ abadoning registration and creat…
[#358] delete console.log
[#360] [#110] add textarea atom/ form for GAs
[#360] [#110] add to changelog
[#361] Implement deployment traceability and streamline notification …
[#361] Initialized deployment log in EC2 user data script
[#361] Integrate Loki panel for deployment log visualization in Grafana
[#361] Integrated Loki and Promtail into Deployment Stack
[#361] Use the highest dashboard version for deployment traceability
[#362] add review submission page
[#362] dont show links when unavailable
[#362] fix display data only for one type of GA
[#364] add information storage screens
[#379] create the propose a governance action card
[#382] add create governance action information screen
[#385] Centralize Docker Compose configurations using a dynamic template
[#385] Integrate prepare-config script functionality into Makefile sy…
[#385] Refactor Makefile structure for enhanced modularity and deploy…
[#385] Refine GitHub Actions workflow for optimized Docker image hand…
[#385] Simplify GitHub Actions deployment workflow
[#385] Unify placeholder usage in configuration templates
[#427] Fetch full repo history during deployment for proper Docker im…
add to changelog
Added Grafana provisioning reload target to utility makefile
change - func name/ number of links
change cancel to back/ fix card layout
change type on @type
Hotfix for 'why' debug option in make
Log a targets indicators in deployment process
Optimized Docker Compose execution in deployment script
Remove unused fake DB sync configurations from deployment setup
replace link with sancho dosc
Restored missing deployment step for conditional infrastructure reset
Streamlined domain configuration in CI/CD workflow
Update changelog to reflect recent changes
Update CHANGELOG.md with Docker tagging policy and Makefile reorganis…