Initial commit

.github/workflows/main.yml

@@ -0,0 +1,39 @@
+# This workflow will install Python dependencies, run tests and lint with a single version of Python
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
+
+name: ci
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+
+permissions:
+  contents: read
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python 3.10
+      uses: actions/setup-python@v3
+      with:
+        python-version: "3.10"
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install flake8 pytest
+        pip install .
+    - name: Lint with flake8
+      run: |
+        # stop the build if there are Python syntax errors or undefined names
+        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
+        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
+        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+    - name: Test with pytest
+      run: |
+        pytest

.gitignore

@@ -0,0 +1,14 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+.coverage
+# uv files
+.python-version
+# Virtual environments
+.venv
+# Development
+local_dev/

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Ali Masri
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,120 @@
+# Google Cloud Build YAML Validator
+
+A robust and extensible tool for validating Google Cloud Build YAML configuration files against [schema specifications](https://cloud.google.com/build/docs/build-config-file-schema) and custom rules.
+By providing a comprehensive set of validation checks and the ability to extend its functionality, this program helps ensure the correctness and consistency of your Cloud Build configuration files, potentially saving time and resources in your CI/CD pipeline
+
+## Features
+
+The Cloud Build YAML Validator performs comprehensive checks on your configuration files:
+
+- **YAML Syntax**: Ensures the file is a valid YAML document.
+- **Schema Compliance**: Validates the YAML structure against Cloud Build specifications.
+- **Duplicate Step IDs**: Identifies duplicate step IDs within the configuration file.
+- **Step Dependencies**: Verifies that all `waitFor` references point to valid step IDs.
+- **Substitution Variables**: Checks for unreferenced substitution variables and ensures they start with an underscore (`_`).
+- **Custom Validations**: Easily extendable with additional custom validation rules.
+
+## Installation
+
+You can install the Cloud Build YAML Validator using `pip`, `uv`, or your preferred Python package manager:
+
+```bash
+git clone https://github.com/alimasri/google-cloudbuild-yaml-validator
+cd google-cloudbuild-yaml-validator
+pip install -e .
+```
+
+## Usage
+
+### Command Line Interface
+
+The validator can be run from the command line with the following syntax:
+
+```bash
+cloudbuild-validator [-h] [-s SCHEMA] -f FILE
+```
+
+### Options:
+- `-h, --help`: Show the help message and exit
+- `-s SCHEMA, --schema SCHEMA`: Path to the schema file to validate against
+- `-f FILE, --file FILE`: Path to the content file to validate
+
+### Example
+
+```bash
+cloudbuild-validator -f /path/to/cloudbuild.yaml
+```
+
+### Programmatic Usage
+
+You can also use the validator as a Python library:
+
+```python
+from cloudbuild_validator.core import CloudbuildValidator
+
+validator = CloudbuildValidator(speficifactions_file="/path/to/specifications/file.yaml")
+validator.validate_file('/path/to/cloudbuild.yaml')
+```
+
+## Specifications
+
+The validator enforces schema specifications for Google Cloud Build YAML configuration files, based on the official Cloud Build documentation. Users can provide a custom schema file using the `-s` or `--schema` option. The default schema file is located at `src/cloudbuild_validator/data/cloudbuild-specifcations.yaml`, which can be used as a reference for creating custom schemas.
+
+By adhering to this schema, users ensure their Cloud Build configuration files are valid and correctly interpreted by the Cloud Build service. Example modifications could include adding organization-specific patterns for image names, environment variables, or other configuration options.
+
+## Extending the Validator
+
+### Adding New Validations
+
+#### Method 1: Extending the default validations
+
+The validator automatically discovers and executes all `Validator` subclasses in the `validators.py` file. To add a new validation rule:
+
+1. Create a new class that inherits from `cloudbuild_validator.validators.Validator`
+2. Implement the `validate` method
+
+The `validate` method should accept a dictionary representing the Cloud Build configuration file and raise a `cloudbuild_validator.exceptions.CloudBuildValidationError` if the validation fails.
+
+##### Example
+
+```python
+class StepIdPrefixValidator(Validator):
+    """Ensures that step IDs start with a specific prefix."""
+
+    def __init__(self, prefix: str):
+        super().__init__()
+        self.prefix = prefix
+
+    def validate(self, content: dict) -> None:
+        for step in content.get('steps', []):
+            step_id = step.get('id', '')
+            if not step_id.startswith(self.prefix):
+                raise CloudBuildValidationError(f"Step ID '{step_id}' does not start with the expected prefix '{self.prefix}'.")
+```
+
+#### Method 2: Using the `add_validator` method
+
+The `CloudbuildValidator` class provides an `add_validator` method that allows users to add custom validation rules. This method accepts a `Validator` subclass and adds it to the list of validators that will be executed during the validation process.
+
+##### Example
+
+```python
+from cloudbuild_validator import CloudbuildValidator
+from cloudbuild_validator.validators import Validator
+
+class CustomValidator(Validator):
+    def validate(self, content: dict) -> None:
+        # Custom validation logic here
+        pass
+
+validator = CloudbuildValidator()
+validator.add_validator(CustomValidator())
+```
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.
+
+## License
+
+This project is distributed under the MIT License. See the [LICENSE](LICENSE) file for more information.

pyproject.toml

@@ -0,0 +1,31 @@
+[project]
+name = "cloudbuild-validator"
+version = "0.1.0"
+description = "A robust and extensible tool for validating Google Cloud Build YAML configuration files against schema specifications and custom rules."
+readme = "README.md"
+authors = [
+    { name = "Ali Masri", email = "alimasri1991@gmail.com" }
+]
+requires-python = ">=3.10"
+dependencies = [
+    "loguru>=0.7.2",
+    "pydantic-settings>=2.6.1",
+    "pydantic>=2.9.2",
+    "yamale>=5.2.1",
+]
+
+[project.scripts]
+cloudbuild-validator = "cloudbuild_validator:main.run"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[dependency-groups]
+dev = [
+    "coverage>=7.6.4",
+    "pytest>=8.3.3",
+    "pytest-cov>=6.0.0",
+    "pytest-sugar>=1.0.0",
+    "ruff>=0.7.3",
+]

src/cloudbuild_validator/config.py

@@ -0,0 +1,31 @@
+from pydantic_settings import BaseSettings
+
+
+class Settings(BaseSettings):
+    DEFAULT_SUBSTITUTIONS: list[str] = [
+        "PROJECT_ID",
+        "BUILD_ID",
+        "PROJECT_NUMBER",
+        "LOCATION",
+        "TRIGGER_NAME",
+        "COMMIT_SHA",
+        "REVISION_ID",
+        "SHORT_SHA",
+        "REPO_NAME",
+        "REPO_FULL_NAME",
+        "BRANCH_NAME",
+        "TAG_NAME",
+        "REF_NAME",
+        "TRIGGER_BUILD_CONFIG_PATH",
+        "SERVICE_ACCOUNT_EMAIL",
+        "SERVICE_ACCOUNT",
+        "_HEAD_BRANCH",
+        "_BASE_BRANCH",
+        "_HEAD_REPO_URL",
+        "_PR_NUMBER",
+    ]
+
+    SUBSTITUTION_VARIABLE_PATTERN: str = r"\$\{(_\w+)\}"
+
+
+settings = Settings()

src/cloudbuild_validator/core.py

@@ -0,0 +1,46 @@
+from pathlib import Path
+from typing import List
+
+import yamale
+
+from cloudbuild_validator import validators
+
+
+class CloudBuildValidator:
+    def __init__(self, speficifactions_file: Path, add_default_validators: bool = True):
+        self.validators = []
+        self.schema = yamale.make_schema(speficifactions_file)
+        if add_default_validators:
+            for validator in dir(validators):
+                if (
+                    isinstance(getattr(validators, validator), type)
+                    and issubclass(getattr(validators, validator), validators.Validator)
+                    and getattr(validators, validator) != validators.Validator
+                ):
+                    self.add_validator(getattr(validators, validator)())
+
+    def add_validator(self, validator: validators.Validator):
+        self.validators.append(validator)
+
+    def remove_validator(self, validator: validators.Validator):
+        self.validators.remove(validator)
+
+    def validate(self, yaml_file_path: Path) -> List[str]:
+        content = yamale.make_data(yaml_file_path)
+        if len(content) > 1:
+            raise validators.CloudBuildValidationError(
+                "Multiple documents found in the file"
+            )
+        try:
+            yamale.validate(self.schema, content)
+        except yamale.YamaleError as e:
+            raise validators.CloudBuildValidationError(e) from e
+        content = content[0][0]
+
+        errors = []
+        for validator in self.validators:
+            try:
+                validator.validate(content)
+            except validators.CloudBuildValidationError as e:
+                errors.append(str(e))
+        return errors

src/cloudbuild_validator/data/cloudbuild-specifications.yaml

@@ -0,0 +1,50 @@
+steps: list(include('Step'), min=1)
+timeout: str(required=False)
+queueTtl: str(required=False)
+logsBucket: str(required=False)
+options:
+  env: list(str(), required=False)
+  secretEnv: str(required=False)
+  volumes: include('Volume', required=False)
+  sourceProvenanceHash: enum('MD5', 'SHA256', 'SHA1', required=False)
+  machineType: enum('UNSPECIFIED', 'N1_HIGHCPU_8', 'N1_HIGHCPU_32', 'E2_HIGHCPU_8', 'E2_HIGHCPU_32', required=False)
+  diskSizeGb: int(required=False)
+  substitutionOption: enum('MUST_MATCH', 'ALLOW_LOOSE', required=False)
+  dynamicSubstitutions: bool(required=False)
+  automapSubstitutions: bool(required=False)
+  logStreamingOption: enum('STREAM_DEFAULT', 'STREAM_ON', 'STREAM_OFF', required=False)
+  logging: enum('GCS_ONLY', 'CLOUD_LOGGING_ONLY', required=False)
+  defaultLogsBucketBehavior: str(required=False)
+  pool: map(required=False)
+  requestedVerifyOption: enum('NOT_VERIFIED', 'VERIFIED', required=False)
+  workerPool: str(required=True)
+substitutions: map(str(), str(), required=False)
+tags: list(str(), required=False)
+serviceAccount: str()
+secrets: map(required=False)
+availableSecrets: map(required=False)
+artifacts: include('Artifact', required=False)
+images: list(list(str()), required=False)
+---
+Artifact:
+  mavenArtifacts: list(map(), required=False)
+  pythonPackages: list(map(), required=False)
+  npmPackages: list(map(), required=False)
+Volume: list(map(name=str(), path=str()), required=False)
+TimeSpan:
+  startTime: str()
+  endTime: str()
+Step:
+  name: str()
+  args: list(str(), required=False)
+  env: list(str(), required=False)
+  allowFailure: bool(required=False)
+  dir: str(required=False)
+  id: str()
+  waitFor: list(str(), required=False)
+  entrypoint: str(required=False)
+  secretEnv: list(str(), required=False)
+  volumes: include('Volume', required=False)
+  timeout: str(required=False)
+  script: str(required=False)
+  automapSubstitutions: bool(required=False)