SAT: test strictness level migration tooling (create issues script) (#…

…19020)

airbyte-integrations/bases/source-acceptance-test/tools/strictness_level_migration/README.md

@@ -0,0 +1,36 @@
+# Tooling for `test_strictness_level` migration
+
+This directory contains scripts that can help us manage the migration of connectors's `acceptance-test-config.yml` to `high` test strictness level.
+Before running these scripts you need to set up a local virtual environment in the **current directory**:
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+```
+## Requirements
+* [GitHub CLI](https://cli.github.com/) (`brew install gh`)
+
+## Create migration issue for GA connectors (`create_issues.py`)
+This script will create one issue per GA connectors to migrate to `high` test strictness level.
+
+### What it does:
+1. Find all GA connectors in `../../../../../airbyte-config/init/src/main/resources/seed/source_definitions.yaml`
+2. Generate an issue content (title, body, labels, project), using `./templates/issue.md.j2`
+3. Find an already existing issue with the same title.
+4. Create the issue and return its url if it does not exist.
+
+Issues get created with the following labels:
+* `area/connectors`
+* `team/connectors-python` 
+* `type/enhancement` 
+* `test-strictness-level`
+
+Issues are added to the following project: `SAT-high-test-strictness-level`
+
+### How to run:
+**Dry run**:
+`python create_issues.py`
+
+**Real execution**:
+`python create_issues.py --dry False`
+

airbyte-integrations/bases/source-acceptance-test/tools/strictness_level_migration/__init__.py

@@ -0,0 +1,3 @@
+#
+# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
+#

airbyte-integrations/bases/source-acceptance-test/tools/strictness_level_migration/create_issues.py

@@ -0,0 +1,83 @@
+#
+# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
+#
+
+
+import argparse
+import json
+import logging
+import os
+import subprocess
+import tempfile
+
+from definitions import GA_DEFINITIONS
+from jinja2 import Environment, FileSystemLoader
+
+TEMPLATES_FOLDER = "./templates/"
+COMMON_ISSUE_LABELS = ["area/connectors", "team/connectors-python", "type/enhancement", "test-strictness-level"]
+GITHUB_PROJECT_NAME = "SAT-high-test-strictness-level"
+
+logging.basicConfig(level=logging.DEBUG)
+environment = Environment(loader=FileSystemLoader(TEMPLATES_FOLDER))
+
+parser = argparse.ArgumentParser(description="Create issues for migration of GA connectors to high test strictness level in SAT")
+parser.add_argument("-d", "--dry", default=True)
+
+
+def get_issue_content(source_definition):
+    issue_title = f"Source {source_definition['name']}: enable `high` test strictness level in SAT"
+
+    template = environment.get_template("issue.md.j2")
+    issue_body = template.render(connector_name=source_definition["name"], release_stage=source_definition["releaseStage"])
+    file_definition, issue_body_path = tempfile.mkstemp()
+
+    with os.fdopen(file_definition, "w") as tmp:
+        # do stuff with temp file
+        tmp.write(issue_body)
+
+    return {"title": issue_title, "body_file": issue_body_path, "labels": COMMON_ISSUE_LABELS}
+
+
+def create_issue(source_definition, dry_run=True):
+    issue_content = get_issue_content(source_definition)
+    list_command_arguments = ["gh", "issue", "list", "--state", "open", "--search", f"'{issue_content['title']}'", "--json", "url"]
+
+    create_command_arguments = [
+        "gh",
+        "issue",
+        "create",
+        "--title",
+        issue_content["title"],
+        "--body-file",
+        issue_content["body_file"],
+        "--project",
+        GITHUB_PROJECT_NAME,
+    ]
+    for label in issue_content["labels"]:
+        create_command_arguments += ["--label", label]
+
+    list_existing_issue_process = subprocess.Popen(list_command_arguments, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    stdout, stderr = list_existing_issue_process.communicate()
+    existing_issues = json.loads(stdout.decode())
+    already_created = len(existing_issues) > 0
+    if already_created:
+        logging.warning(f"An issue was already created for this definition: {existing_issues[0]}")
+    if not already_created:
+        if not dry_run:
+            process = subprocess.Popen(create_command_arguments, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+            stdout, stderr = process.communicate()
+            if stderr:
+                logging.error(stderr.decode())
+            else:
+                created_issue_url = stdout.decode()
+                logging.info(f"Created issue for {source_definition['name']}: {created_issue_url}")
+        else:
+            logging.info(f"[DRY RUN]: {' '.join(create_command_arguments)}")
+    os.remove(issue_content["body_file"])
+
+
+if __name__ == "__main__":
+    args = parser.parse_args()
+    dry_run = False if args.dry == "False" or args.dry == "false" else True
+    for definition in GA_DEFINITIONS:
+        create_issue(definition, dry_run=dry_run)

airbyte-integrations/bases/source-acceptance-test/tools/strictness_level_migration/definitions.py

@@ -0,0 +1,21 @@
+#
+# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
+#
+
+
+import yaml
+
+SOURCE_DEFINITIONS_FILE_PATH = "../../../../../airbyte-config/init/src/main/resources/seed/source_definitions.yaml"
+
+
+def read_source_definitions():
+    with open(SOURCE_DEFINITIONS_FILE_PATH, "r") as source_definitions_file:
+        return yaml.safe_load(source_definitions_file)
+
+
+def find_by_release_stage(source_definitions, release_stage):
+    return [definition for definition in source_definitions if definition.get("releaseStage") == release_stage]
+
+
+ALL_DEFINITIONS = read_source_definitions()
+GA_DEFINITIONS = find_by_release_stage(ALL_DEFINITIONS, "generally_available")

airbyte-integrations/bases/source-acceptance-test/tools/strictness_level_migration/requirements.txt

@@ -0,0 +1,4 @@
+Jinja2==3.1.2
+MarkupSafe==2.1.1
+pyaml==21.10.1
+PyYAML==6.0

airbyte-integrations/bases/source-acceptance-test/tools/strictness_level_migration/templates/issue.md.j2

@@ -0,0 +1,15 @@
+## What
+A `test_strictness_level` field was introduced to Source Acceptance Tests (SAT).
+{{ connector_name }} is a {{ release_stage }} connector, we want it to have a `high` test strictness level.
+
+**This will help**:
+- maximize the SAT coverage on this connector.
+- document its potential weaknesses in term of test coverage.
+
+## How
+1. Migrate the existing `acceptance-test-config.yml` file to the latest configuration format. (See instructions [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/bases/source-acceptance-test/README.md#L61))
+2. Enable `high` test strictness level in `acceptance-test-config.yml`. (See instructions [here](https://github.com/airbytehq/airbyte/blob/master/docs/connector-development/testing-connectors/source-acceptance-tests-reference.md#L240))
+3. Commit changes on `acceptance-test-config.yml` and open a PR.
+4. Run SAT with the `/test` command on the branch.
+5. If tests are failing please fix the failing test or use `bypass_reason` fields to explain why a specific test can't be run.
+