Transition CI/CD to GitHub Workflows #2378

wbarnha · 2023-08-02T14:06:29Z

Placeholder text, will add more notes later.

This change is

github-advanced-security · 2023-08-02T14:09:07Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

This reverts commit 2b600c7.

This reverts commit 674ab78.

.github/CODE_OF_CONDUCT.md

build_integration.sh

dpkp · 2023-08-02T19:21:48Z

setup.py

    classifiers=[
        "Development Status :: 5 - Production/Stable",
        "Intended Audience :: Developers",
        "License :: OSI Approved :: Apache Software License",
        "Programming Language :: Python",
-        "Programming Language :: Python :: 2",


why drop the python2 support notice?

GitHub dropped support for testing Python 2.7 back in June as explained in actions/runner-images#7401. I figure it'd make sense to drop support for it if we can't test it on any runners in our CI/CD workflows.

If we're dropping support, we should probably add python_requires to the setup.py file.

I'm 👍 on dropping python 2 support (in fact I'm all for aligning with upstream Python EOL cadence, as that seems pretty conservative to have 5 years to support a version), but I'd do it separate from this PR... keep this one focused on the CI workflow migration.

Sounds good, I'll revert my tag drop for now.

jeffwidman · 2023-08-02T21:26:59Z

Thanks for working on this, really appreciated!

I'll try to give a more thorough review tomorrow, but if I don't get to it, please merge w/o waiting for me...

wbarnha · 2023-08-03T03:59:12Z

I should create a separate PR at some point for a workflow that will generate docs and push to readthedocs for each release. I've written similar workflows for GitHub Pages, I just need to know what our authentication situation is for readthedocs.

wbarnha · 2023-08-03T13:25:13Z

requirements-dev.txt

-coveralls==2.1.2
-crc32c==2.1
-docker-py==1.10.6
-flake8==3.8.3
-lz4==3.1.0
-mock==4.0.2
-py==1.9.0
-pylint==2.6.0
-pytest==6.0.2
-pytest-cov==2.10.1
-pytest-mock==3.3.1
-pytest-pylint==0.17.0
-python-snappy==0.5.4
-Sphinx==3.2.1
-sphinx-rtd-theme==0.5.0
-tox==3.20.0
-xxhash==2.0.0
+coveralls
+crc32c
+docker-py
+flake8
+lz4
+mock
+py
+pylint
+pytest
+pytest-cov
+pytest-mock
+pytest-pylint
+python-snappy
+Sphinx
+sphinx-rtd-theme
+tox
+xxhash


In my personal opinion, we don't necessarily need to pin versions until we get all of our workflows in order. We can revisit version pinning at a later point, many of these pinned versions are outdated.

wbarnha · 2023-08-03T13:26:01Z

tox.ini

-    py.test {posargs:--pylint --pylint-rcfile=pylint.rc --pylint-error-types=EF --cov=kafka --cov-config=.covrc}
+    pytest {posargs:--pylint --pylint-rcfile=pylint.rc --pylint-error-types=EF --cov=kafka --cov-config=.covrc}
 setenv =
    CRC32C_SW_MODE = auto
    PROJECT_ROOT = {toxinidir}
 passenv = KAFKA_VERSION

-[testenv:py26]
-# pylint doesn't support python2.6
-commands = py.test {posargs:--cov=kafka --cov-config=.covrc}

 [testenv:pypy]
 # pylint is super slow on pypy...
-commands = py.test {posargs:--cov=kafka --cov-config=.covrc}
+commands = pytest {posargs:--cov=kafka --cov-config=.covrc}


Calls to py.test are deprecated in favor of pytest, https://stackoverflow.com/a/41893170/9852671 provides a useful explanation.

wbarnha · 2023-08-03T13:37:24Z

test/test_assignors.py

-    all_topics = set(['t{}'.format(i) for i in range(1, n_topics + 1)])
+    all_topics = sorted(['t{}'.format(i) for i in range(1, n_topics + 1)])


sorted is a required call for Python 3.11+, otherwise the test fails.

wbarnha · 2023-08-03T13:38:02Z

.github/workflows/python-package.yml

+          - "0.8.2.2"
+          - "0.9.0.1"
+          - "0.10.2.2"
+          - "0.11.0.2"
+          - "0.11.0.3"
+          - "1.1.1"
+          - "2.4.0"
+          - "2.5.0"
+          - "2.6.0"


It'd be nice to keep all of these versions in a file somewhere for convenience, so we don't need to edit supported Kafka versions in multiple places.

wbarnha · 2023-08-03T13:39:19Z

.github/workflows/codeql-analysis.yml

+# the `language` matrix defined below to confirm you have the correct set of
+# supported CodeQL languages.
+#
+name: CodeQL


Not critical to include this, but it has served me well in other projects.

wbarnha · 2023-08-04T00:17:22Z

Just want to check with you both before I merge this. I'm keen on getting this into the main branch so I can test existing PRs against the pipeline and draft a workflow for docs generation.

The only PRs I plan on merging are documentation improvements. I was thinking we do a release of the library in its current state now, and announce deprecation of older Python versions. What do you think?

wbarnha · 2023-08-04T21:57:31Z

I'll go ahead and merge this since no core functionality is changed.

* Create GH workflows to test code * Update tests for future Python versions

wbarnha added 4 commits August 2, 2023 09:08

update readme and setup.py tags

98e427c

remove pinned versions from requirements-dev.txt for now

e2acd92

commit basic workflow for now

77c345b

disable setup-kafka requirement for now

bd1b0fe

wbarnha added 24 commits August 2, 2023 10:10

install libsnappy-dev

4a62b02

remove py3.7

4f03240

pull java and fix some tests

57c523c

cleanup commands

5ddacca

remove <distribution>

75a9ef7

specify oracle distribution

21a36f8

just use temurin 11 for now

9759af4

run ./build_integration.sh lol

fa4b11a

add more releases to build_integration.sh

d05e798

specify kafka version in env

60ad460

specify kafka versions

ddd2c0d

just pull major and minor versions

2b600c7

Revert "just pull major and minor versions"

4d1077c

This reverts commit 2b600c7.

split kafka and python tests

1f6ce72

remove kafka note

4ec8f04

remove env from name again

ea2e8a0

replace py.test with pytest

24ce096

fix pypy syntax

1a13bae

install pylint for everyone

873bc2d

disable travis_java_install

c895bba

try java 8

f186ae1

extend timeout from 35s to 60s since PyPy tests are slow

674ab78

Revert "extend timeout from 35s to 60s since PyPy tests are slow"

5a0a662

This reverts commit 674ab78.

change syntax to pypy-3.9

6baa9d7

dpkp reviewed Aug 2, 2023

View reviewed changes

.github/CODE_OF_CONDUCT.md Outdated Show resolved Hide resolved

wbarnha added 2 commits August 2, 2023 15:18

mark PyPy as experimental in CI/CD until we know why it keeps failing

4d05fd9

remove automatic publish

4ef5dbc

dpkp reviewed Aug 2, 2023

View reviewed changes

build_integration.sh Outdated Show resolved Hide resolved

dpkp reviewed Aug 2, 2023

View reviewed changes

wbarnha added 2 commits August 2, 2023 15:21

revert changes to ALL_RELEASES

f43e002

fix typo

b8e9e51

wbarnha marked this pull request as ready for review August 2, 2023 19:24

wbarnha added 4 commits August 2, 2023 15:31

Update dependabot.yml

30bfa8c

Update README.rst

7fc76d1

revert tags to setup.py

63837d6

Update setup.py to have " "Programming Language :: Python :: 2"," again

9ef7d62

wbarnha self-assigned this Aug 2, 2023

wbarnha mentioned this pull request Aug 3, 2023

Remove pylint workaround since upstream fixed #2252

Closed

wbarnha commented Aug 3, 2023

View reviewed changes

wbarnha merged commit f8a7e9b into master Aug 4, 2023

wbarnha deleted the update-cicd branch August 4, 2023 21:57

bradenneal1 pushed a commit to bradenneal1/kafka-python that referenced this pull request May 16, 2024

Transition CI/CD to GitHub Workflows (dpkp#2378)

f5cdb5c

* Create GH workflows to test code * Update tests for future Python versions

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transition CI/CD to GitHub Workflows #2378

Transition CI/CD to GitHub Workflows #2378

wbarnha commented Aug 2, 2023 •

edited by dpkp

Loading

github-advanced-security bot commented Aug 2, 2023

dpkp Aug 2, 2023

wbarnha Aug 2, 2023

wbarnha Aug 2, 2023

jeffwidman Aug 2, 2023

wbarnha Aug 2, 2023

jeffwidman commented Aug 2, 2023

wbarnha commented Aug 3, 2023

wbarnha Aug 3, 2023

wbarnha Aug 3, 2023

wbarnha Aug 3, 2023

wbarnha Aug 3, 2023

wbarnha Aug 3, 2023

wbarnha commented Aug 4, 2023 •

edited

Loading

wbarnha commented Aug 4, 2023

		all_topics = set(['t{}'.format(i) for i in range(1, n_topics + 1)])
		all_topics = sorted(['t{}'.format(i) for i in range(1, n_topics + 1)])

Transition CI/CD to GitHub Workflows #2378

Transition CI/CD to GitHub Workflows #2378

Conversation

wbarnha commented Aug 2, 2023 • edited by dpkp Loading

github-advanced-security bot commented Aug 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeffwidman commented Aug 2, 2023

wbarnha commented Aug 3, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wbarnha commented Aug 4, 2023 • edited Loading

wbarnha commented Aug 4, 2023

wbarnha commented Aug 2, 2023 •

edited by dpkp

Loading

wbarnha commented Aug 4, 2023 •

edited

Loading