Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix running tests locally (outside of the CI) #227

Merged
merged 1 commit into from
Oct 8, 2024
Merged

Conversation

tatiana
Copy link
Collaborator

@tatiana tatiana commented Oct 8, 2024

Before this change, they raised the following error when running AIRFLOW_HOME=pwdmake test:

========================================================== FAILURES ==========================================================
________________________________________ test_get_dag_params_with_template_searchpath ________________________________________

    def test_get_dag_params_with_template_searchpath():
        from dagfactory import utils

        td = dagbuilder.DagBuilder(
            "test_dag", {"template_searchpath": ["./sql"]}, DEFAULT_CONFIG
        )
        error_message = "template_searchpath must be absolute paths"
        with pytest.raises(Exception, match=error_message):
            td.get_dag_params()

        td = dagbuilder.DagBuilder(
            "test_dag", {"template_searchpath": ["/sql"]}, DEFAULT_CONFIG
        )
        error_message = "template_searchpath must be existing paths"
        with pytest.raises(Exception, match=error_message):
            td.get_dag_params()

        td = dagbuilder.DagBuilder(
            "test_dag", {"template_searchpath": "./sql"}, DEFAULT_CONFIG
        )
        error_message = "template_searchpath must be absolute paths"
        with pytest.raises(Exception, match=error_message):
            td.get_dag_params()

        td = dagbuilder.DagBuilder(
            "test_dag", {"template_searchpath": "/sql"}, DEFAULT_CONFIG
        )
        error_message = "template_searchpath must be existing paths"
        with pytest.raises(Exception, match=error_message):
            td.get_dag_params()

        assert utils.check_template_searchpath(123) == False
>       assert utils.check_template_searchpath("/home/runner/work") == True

tests/test_dagbuilder.py:657:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

template_searchpath = '/home/runner/work'

    def check_template_searchpath(template_searchpath: Union[str, List[str]]) -> bool:
        """
        Check if template_searchpath is valid
        :param template_searchpath: a list or str to test
        :type template_searchpath: Union[str, List[str]]
        :return: result to check
        :type: bool
        """
        if isinstance(template_searchpath, str):
            if not os.path.isabs(template_searchpath):
                raise DagFactoryException("template_searchpath must be absolute paths")
            if not os.path.isdir(template_searchpath):
>               raise DagFactoryException("template_searchpath must be existing paths")
E               dagfactory.exceptions.DagFactoryException: template_searchpath must be existing paths

dagfactory/utils.py:204: DagFactoryException

--------- coverage: platform darwin, python 3.10.14-final-0 ----------
Coverage XML written to file coverage.xml

================================================== short test summary info ===================================================
FAILED tests/test_dagbuilder.py::test_get_dag_params_with_template_searchpath - dagfactory.exceptions.DagFactoryException: template_searchpath must be existing paths
================================================ 1 failed, 79 passed in 1.17s ================================================
py310-airflow2: exit 1 (1.57 seconds) /Users/tati/Code/dag-factory> pytest --cov=dagfactory tests -p no:warnings --verbose --color=yes --cov-report=xml pid=55426
.pkg: _exit> python /Users/tati/Code/dag-factory/venv/lib/python3.11/site-packages/pyproject_api/_backend.py True hatchling.build
  py38-airflow1108: OK (2.16 seconds)
  py38-airflow2: FAIL code 1 (5.23=setup[2.41]+cmd[1.23,1.59] seconds)
  py39-airflow1108: OK (1.85 seconds)
  py39-airflow2: FAIL code 1 (6.48=setup[2.18]+cmd[2.61,1.70] seconds)
  py310-airflow1108: OK (2.24 seconds)
  py310-airflow2: FAIL code 1 (6.71=setup[2.53]+cmd[2.60,1.57] seconds)
  evaluation failed :( (24.70 seconds)
make: *** [test] Error 255

Before this change, they raised the following error when running `AIRFLOW_HOME=`pwd` make test `:

```
========================================================== FAILURES ==========================================================
________________________________________ test_get_dag_params_with_template_searchpath ________________________________________

    def test_get_dag_params_with_template_searchpath():
        from dagfactory import utils

        td = dagbuilder.DagBuilder(
            "test_dag", {"template_searchpath": ["./sql"]}, DEFAULT_CONFIG
        )
        error_message = "template_searchpath must be absolute paths"
        with pytest.raises(Exception, match=error_message):
            td.get_dag_params()

        td = dagbuilder.DagBuilder(
            "test_dag", {"template_searchpath": ["/sql"]}, DEFAULT_CONFIG
        )
        error_message = "template_searchpath must be existing paths"
        with pytest.raises(Exception, match=error_message):
            td.get_dag_params()

        td = dagbuilder.DagBuilder(
            "test_dag", {"template_searchpath": "./sql"}, DEFAULT_CONFIG
        )
        error_message = "template_searchpath must be absolute paths"
        with pytest.raises(Exception, match=error_message):
            td.get_dag_params()

        td = dagbuilder.DagBuilder(
            "test_dag", {"template_searchpath": "/sql"}, DEFAULT_CONFIG
        )
        error_message = "template_searchpath must be existing paths"
        with pytest.raises(Exception, match=error_message):
            td.get_dag_params()

        assert utils.check_template_searchpath(123) == False
>       assert utils.check_template_searchpath("/home/runner/work") == True

tests/test_dagbuilder.py:657:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

template_searchpath = '/home/runner/work'

    def check_template_searchpath(template_searchpath: Union[str, List[str]]) -> bool:
        """
        Check if template_searchpath is valid
        :param template_searchpath: a list or str to test
        :type template_searchpath: Union[str, List[str]]
        :return: result to check
        :type: bool
        """
        if isinstance(template_searchpath, str):
            if not os.path.isabs(template_searchpath):
                raise DagFactoryException("template_searchpath must be absolute paths")
            if not os.path.isdir(template_searchpath):
>               raise DagFactoryException("template_searchpath must be existing paths")
E               dagfactory.exceptions.DagFactoryException: template_searchpath must be existing paths

dagfactory/utils.py:204: DagFactoryException

--------- coverage: platform darwin, python 3.10.14-final-0 ----------
Coverage XML written to file coverage.xml

================================================== short test summary info ===================================================
FAILED tests/test_dagbuilder.py::test_get_dag_params_with_template_searchpath - dagfactory.exceptions.DagFactoryException: template_searchpath must be existing paths
================================================ 1 failed, 79 passed in 1.17s ================================================
py310-airflow2: exit 1 (1.57 seconds) /Users/tati/Code/dag-factory> pytest --cov=dagfactory tests -p no:warnings --verbose --color=yes --cov-report=xml pid=55426
.pkg: _exit> python /Users/tati/Code/dag-factory/venv/lib/python3.11/site-packages/pyproject_api/_backend.py True hatchling.build
  py38-airflow1108: OK (2.16 seconds)
  py38-airflow2: FAIL code 1 (5.23=setup[2.41]+cmd[1.23,1.59] seconds)
  py39-airflow1108: OK (1.85 seconds)
  py39-airflow2: FAIL code 1 (6.48=setup[2.18]+cmd[2.61,1.70] seconds)
  py310-airflow1108: OK (2.24 seconds)
  py310-airflow2: FAIL code 1 (6.71=setup[2.53]+cmd[2.60,1.57] seconds)
  evaluation failed :( (24.70 seconds)
make: *** [test] Error 255
```
Copy link
Contributor

@pankajkoti pankajkoti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this!

@tatiana tatiana merged commit 4789077 into main Oct 8, 2024
6 checks passed
@tatiana tatiana added this to the DAG Factory 0.20.0 milestone Oct 10, 2024
@tatiana tatiana self-assigned this Oct 11, 2024
@tatiana tatiana mentioned this pull request Oct 17, 2024
tatiana added a commit that referenced this pull request Oct 22, 2024
### Added
- Support using envvar in config YAML by @tatiana in #236
- **Callback improvements**
- Support installed code via python callable string by @john-drews in
#221
- Add `callback_file` & `callback_name` to `default_args` DAG level by
@subbota19 in #218
- Cast callbacks to functions when set with `default_args` on TaskGroups
by @Baraldo and @pankajastro in #235

- **Telemetry**
- For more information, please, read the [Privacy
Notice](https://github.com/astronomer/dag-factory/blob/main/PRIVACY_NOTICE.md#collection-of-data).
  - Add scarf to readme for website analytics by @cmarteepants in #219
- Support telemetry during DAG parsing emitting data to Scarf by
@tatiana in #250.

### Fixed
- Build DAGs when tehre is an invalid YAML in the DAGs folder by @quydx
and @tatiana in #184

### Others
- Development tools
  - Fix make docker-run by @pankajkoti in #249
  - Add vim dot files to .gitignore by @tatiana in #228
  - Use Hatchling to modern package building by @kaxil in #208
- CI
  - Fix static check failures in PR #218 by @pankajkoti in #251
  - Fix pre-commit checks by @tatiana in #247
  - Remove tox and corresponding build jobs in CI by @pankajkoti in #248
- Install Airflow with different versions in the CI by @pankajkoti in
#237
  - Run pre-commit hooks on all existing files by @pankajkoti in #245
  - Add Python 3.11 and 3.12 to CI test pipeline by @pankajkoti in #229
- Tests
  - Fix duplicate test name by @pankajastro in #234
  - Add static check by @pankajastro in #231
  - Fix running tests locally (outside the CI) by @tatiana in #227
  - Add the task_2 back to dataset example by @cmarteepants in #204
  - Remove unnecessary config line by @jlaneve in #202
- Documentation
  - Update the license from MIT to Apache 2.0 by @pankajastro in #191
- Add registration icon and links to Airflow references by @cmarteepants
in #190
  - Update quickstart and add feature examples by @cmarteepants #189

### Breaking changes
- Removed support for Python 3.7
- The license was changed from MIT to Apache 2.0

Closes: #217
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants