Support for Python 3.11? #1452

thibaultbetremieux · 2022-10-25T13:16:02Z

Description

I tried installing confluent-kafka inside a conda environment with python=3.11. It did not work. It was working previously with python=3.10.6 and I'd like to use some features from 3.11 🙂 .

How to reproduce

Create conda environment with Python 3.11 (enter the commands one by one and accept prompts, this is not a script):

conda create -n test_confluent_kafka
conda activate test_confluent_kafka
conda install python=3.11 -c conda-forge

Attempt to install confluent-kafka inside the enviromnent

pip install confluent-kafka

Output

Collecting confluent-kafka Using cached confluent-kafka-1.9.2.tar.gz (109 kB) Preparing metadata (setup.py) ... done Building wheels for collected packages: confluent-kafka Building wheel for confluent-kafka (setup.py) ... error error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [53 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-311
creating build/lib.linux-x86_64-cpython-311/confluent_kafka
copying src/confluent_kafka/error.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka
copying src/confluent_kafka/serializing_producer.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka
copying src/confluent_kafka/deserializing_consumer.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka
copying src/confluent_kafka/init.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka
creating build/lib.linux-x86_64-cpython-311/confluent_kafka/avro
copying src/confluent_kafka/avro/error.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/avro
copying src/confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/avro
copying src/confluent_kafka/avro/load.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/avro
copying src/confluent_kafka/avro/init.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/avro
creating build/lib.linux-x86_64-cpython-311/confluent_kafka/admin
copying src/confluent_kafka/admin/_config.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/admin
copying src/confluent_kafka/admin/_acl.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/admin
copying src/confluent_kafka/admin/_resource.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/admin
copying src/confluent_kafka/admin/init.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/admin
creating build/lib.linux-x86_64-cpython-311/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/verifiable_client.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/init.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/kafkatest
creating build/lib.linux-x86_64-cpython-311/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/error.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/schema_registry_client.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/json_schema.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/protobuf.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/avro.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/init.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/schema_registry
creating build/lib.linux-x86_64-cpython-311/confluent_kafka/serialization
copying src/confluent_kafka/serialization/init.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/serialization
creating build/lib.linux-x86_64-cpython-311/confluent_kafka/avro/serializer
copying src/confluent_kafka/avro/serializer/message_serializer.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/avro/serializer
copying src/confluent_kafka/avro/serializer/init.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/avro/serializer
running build_ext
building 'confluent_kafka.cimpl' extension
creating build/temp.linux-x86_64-cpython-311
creating build/temp.linux-x86_64-cpython-311/tmp
creating build/temp.linux-x86_64-cpython-311/tmp/pip-install-hrf7miay
creating build/temp.linux-x86_64-cpython-311/tmp/pip-install-hrf7miay/confluent-kafka_24487a6a36c04ac3bd52ab375f05fcd5
creating build/temp.linux-x86_64-cpython-311/tmp/pip-install-hrf7miay/confluent-kafka_24487a6a36c04ac3bd52ab375f05fcd5/src
creating build/temp.linux-x86_64-cpython-311/tmp/pip-install-hrf7miay/confluent-kafka_24487a6a36c04ac3bd52ab375f05fcd5/src/confluent_kafka
creating build/temp.linux-x86_64-cpython-311/tmp/pip-install-hrf7miay/confluent-kafka_24487a6a36c04ac3bd52ab375f05fcd5/src/confluent_kafka/src
gcc -pthread -B /home/thibaultbetremieux/anaconda3/envs/test_confluent_kafka/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/thibaultbetremieux/anaconda3/envs/test_confluent_kafka/include -fPIC -O2 -isystem /home/thibaultbetremieux/anaconda3/envs/test_confluent_kafka/include -fPIC -I/home/thibaultbetremieux/anaconda3/envs/test_confluent_kafka/include/python3.11 -c /tmp/pip-install-hrf7miay/confluent-kafka_24487a6a36c04ac3bd52ab375f05fcd5/src/confluent_kafka/src/Admin.c -o build/temp.linux-x86_64-cpython-311/tmp/pip-install-hrf7miay/confluent-kafka_24487a6a36c04ac3bd52ab375f05fcd5/src/confluent_kafka/src/Admin.o
In file included from /tmp/pip-install-hrf7miay/confluent-kafka_24487a6a36c04ac3bd52ab375f05fcd5/src/confluent_kafka/src/Admin.c:17:
/tmp/pip-install-hrf7miay/confluent-kafka_24487a6a36c04ac3bd52ab375f05fcd5/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: librdkafka/rdkafka.h: No such file or directory
23 | #include <librdkafka/rdkafka.h>
| ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for confluent-kafka
Running setup.py clean for confluent-kafka
Failed to build confluent-kafka
Installing collected packages: confluent-kafka
Running setup.py install for confluent-kafka ... error
error: subprocess-exited-with-error

× Running setup.py install for confluent-kafka did not run successfully.
│ exit code: 1
╰─> [55 lines of output]
running install
/home/thibaultbetremieux/anaconda3/envs/test_confluent_kafka/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-311
creating build/lib.linux-x86_64-cpython-311/confluent_kafka
copying src/confluent_kafka/error.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka
copying src/confluent_kafka/serializing_producer.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka
copying src/confluent_kafka/deserializing_consumer.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka
copying src/confluent_kafka/init.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka
creating build/lib.linux-x86_64-cpython-311/confluent_kafka/avro
copying src/confluent_kafka/avro/error.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/avro
copying src/confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/avro
copying src/confluent_kafka/avro/load.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/avro
copying src/confluent_kafka/avro/init.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/avro
creating build/lib.linux-x86_64-cpython-311/confluent_kafka/admin
copying src/confluent_kafka/admin/_config.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/admin
copying src/confluent_kafka/admin/_acl.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/admin
copying src/confluent_kafka/admin/_resource.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/admin
copying src/confluent_kafka/admin/init.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/admin
creating build/lib.linux-x86_64-cpython-311/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/verifiable_client.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/kafkatest
copying src/confluent_kafka/kafkatest/init.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/kafkatest
creating build/lib.linux-x86_64-cpython-311/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/error.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/schema_registry_client.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/json_schema.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/protobuf.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/avro.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/schema_registry
copying src/confluent_kafka/schema_registry/init.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/schema_registry
creating build/lib.linux-x86_64-cpython-311/confluent_kafka/serialization
copying src/confluent_kafka/serialization/init.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/serialization
creating build/lib.linux-x86_64-cpython-311/confluent_kafka/avro/serializer
copying src/confluent_kafka/avro/serializer/message_serializer.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/avro/serializer
copying src/confluent_kafka/avro/serializer/init.py -> build/lib.linux-x86_64-cpython-311/confluent_kafka/avro/serializer
running build_ext
building 'confluent_kafka.cimpl' extension
creating build/temp.linux-x86_64-cpython-311
creating build/temp.linux-x86_64-cpython-311/tmp
creating build/temp.linux-x86_64-cpython-311/tmp/pip-install-hrf7miay
creating build/temp.linux-x86_64-cpython-311/tmp/pip-install-hrf7miay/confluent-kafka_24487a6a36c04ac3bd52ab375f05fcd5
creating build/temp.linux-x86_64-cpython-311/tmp/pip-install-hrf7miay/confluent-kafka_24487a6a36c04ac3bd52ab375f05fcd5/src
creating build/temp.linux-x86_64-cpython-311/tmp/pip-install-hrf7miay/confluent-kafka_24487a6a36c04ac3bd52ab375f05fcd5/src/confluent_kafka
creating build/temp.linux-x86_64-cpython-311/tmp/pip-install-hrf7miay/confluent-kafka_24487a6a36c04ac3bd52ab375f05fcd5/src/confluent_kafka/src
gcc -pthread -B /home/thibaultbetremieux/anaconda3/envs/test_confluent_kafka/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/thibaultbetremieux/anaconda3/envs/test_confluent_kafka/include -fPIC -O2 -isystem /home/thibaultbetremieux/anaconda3/envs/test_confluent_kafka/include -fPIC -I/home/thibaultbetremieux/anaconda3/envs/test_confluent_kafka/include/python3.11 -c /tmp/pip-install-hrf7miay/confluent-kafka_24487a6a36c04ac3bd52ab375f05fcd5/src/confluent_kafka/src/Admin.c -o build/temp.linux-x86_64-cpython-311/tmp/pip-install-hrf7miay/confluent-kafka_24487a6a36c04ac3bd52ab375f05fcd5/src/confluent_kafka/src/Admin.o
In file included from /tmp/pip-install-hrf7miay/confluent-kafka_24487a6a36c04ac3bd52ab375f05fcd5/src/confluent_kafka/src/Admin.c:17:
/tmp/pip-install-hrf7miay/confluent-kafka_24487a6a36c04ac3bd52ab375f05fcd5/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: librdkafka/rdkafka.h: No such file or directory
23 | #include <librdkafka/rdkafka.h>
| ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> confluent-kafka

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Checklist

Please provide the following information:

confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): confluent-kafka-=1.9.2, librdkafka=?
Apache Kafka broker version: ?
Client configuration: {...}
Operating system: Pop!_OS 22.04 LTS
Provide client logs (with 'debug': '..' as necessary)
Provide broker log excerpts
Critical issue

The text was updated successfully, but these errors were encountered:

ermakov-oleg · 2022-10-26T09:59:24Z

For python 3.10 you used wheel and did not need to compile from source.
Wheels for python 3.11 are not ready yet and you need to install librdkafka v1.9.0 or later and try to install it again.

@edenhill can you build a wheel for python 3.11?

Gr1N · 2022-10-31T16:29:50Z

Hey!

Any plans for a wheel release for Python 3.11?

pranavrth · 2022-11-02T11:41:58Z

We are planning to add a 3.11 wheel for our next release which is about a month out

Hawax · 2022-11-03T08:52:15Z

apt-get install -y --no-install-recommends gcc git libssl-dev g++ make && cd /tmp && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && ./configure --prefix=/usr && make && make install

works for me on 3.11 on ubuntu 18.04 😄

RichardDally · 2022-11-04T16:37:08Z

apt-get install -y --no-install-recommends gcc git libssl-dev g++ make && cd /tmp && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && ./configure --prefix=/usr && make && make install

works for me on 3.11 on ubuntu 18.04 😄

I've tried same steps in RH8 but repository is still using an old librdkafka so I had to compile it too. I would be very interested to compile confluent kafka wheel with a static version of librdkafka 1.9.0

nitin-karolla · 2022-11-17T23:17:58Z

Waiting for the 3.11 wheel release 🚀

maciejfic · 2022-11-18T14:03:52Z

Hello, do you have any estimate on when we can expect python 3.11 wheel? Thakns in advance!

Additional question: I was trying to build one myself, but no matter which version of librdkafka I used I got error cimpl.cpython-311-x86_64-linux-gnu.so: undefined symbol: rd_kafka_AclBinding_destroy. Any tips what I'm doing wrong?

reupugi · 2022-11-21T13:39:12Z

apt-get install -y --no-install-recommends gcc git libssl-dev g++ make && cd /tmp && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && ./configure --prefix=/usr && make && make install

works for me on 3.11 on ubuntu 18.04 smile

I tried your method - still get ModuleNotFoundError when I try to import confluent_kafka - Any suggestions?

Hawax · 2022-11-21T15:14:37Z

apt-get install -y --no-install-recommends gcc git libssl-dev g++ make && cd /tmp && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && ./configure --prefix=/usr && make && make install
works for me on 3.11 on ubuntu 18.04 smile

I tried your method - still get ModuleNotFoundError when I try to import confluent_kafka - Any suggestions?

Have you tried to install Kafka with pip after these steps?

reupugi · 2022-11-21T15:55:42Z

apt-get install -y --no-install-recommends gcc git libssl-dev g++ make && cd /tmp && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && ./configure --prefix=/usr && make && make install
works for me on 3.11 on ubuntu 18.04 smile

I tried your method - still get ModuleNotFoundError when I try to import confluent_kafka - Any suggestions?

Have you tried to install Kafka with pip after these steps?

yes I did this now and it works

sbv-csis · 2022-11-23T05:46:17Z

#1467 <--- getting there 👍

agates4 · 2022-11-28T14:44:23Z

Does the merge of #1467 mean this will be in the next release?

nitin-karolla · 2022-11-29T15:00:10Z

When is the next release? When can we expect the wheel for 3.11 to be available?

wbarnha · 2022-12-02T16:19:08Z

I've been running this in python:3.11-slim-bullseye while waiting for Confluent to release an update:

RUN apt-get update -y && apt-get install -y --no-install-recommends gcc git libssl-dev g++ make && cd /tmp && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && ./configure --prefix=/usr && make && make install
RUN python -m pip install --upgrade 'confluent-kafka @ git+https://github.com/confluentinc/confluent-kafka-python'

Hopefully this makes someone's life more convenient. It'll make the image size increase by 300-400 MB but if you're just verifying your code for Python 3.11, it'll do.

TheHenrick · 2022-12-07T21:45:09Z

I've been running this in python:3.11-slim-bullseye while waiting for Confluent to release an update:
RUN apt-get update -y && apt-get install -y --no-install-recommends gcc git libssl-dev g++ make && cd /tmp && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && ./configure --prefix=/usr && make && make install
RUN python -m pip install --upgrade 'confluent-kafka @ git+https://github.com/confluentinc/confluent-kafka-python'
Hopefully this makes someone's life more convenient. It'll make the image size increase by 300-400 MB but if you're just verifying your code for Python 3.11, it'll do.

Thanks, great for testing in the meantime.

giacomomiolo · 2022-12-30T08:27:30Z

Any updates on this? Thanks for the great work!

y3ti · 2023-01-02T11:55:34Z

If you want to build *.whl file:

Makefile:

.PHONY: build
build:
	DOCKER_BUILDKIT=1 docker build --output type=local,dest=./wheels .

Dockerfile:

###############################################################################
# BUILDER
###############################################################################
FROM quay.io/pypa/manylinux_2_24_x86_64 AS builder

ARG CONFLUENT_KAFKA_VERSION=1.9.2
ARG LIBRDKAFKA_VERSION=1.9.2
ENV PYBIN=/opt/python/cp311-cp311/bin/

RUN set -ex \
    &&  apt-get update \
    &&  apt-get install --no-install-recommends --yes \
            build-essential \
            git \
            libsasl2-dev \
            libssl-dev \
            libzstd-dev

# build and install librdkafka
WORKDIR /tmp/librdkafka
RUN set -ex \
    &&  git clone --branch v${LIBRDKAFKA_VERSION} https://github.com/edenhill/librdkafka.git . \
    &&  ./configure --prefix=/usr \
    &&  make \
    &&  make install


# build confluent-kafka python wheel
WORKDIR /tmp/confluent-kafka-python
RUN set -ex \
    &&  git clone --branch v${CONFLUENT_KAFKA_VERSION} https://github.com/confluentinc/confluent-kafka-python.git . \
    &&  mkdir -p dist /wheels \
    &&  ${PYBIN}/python3.11 setup.py bdist_wheel -d dist \
    &&  auditwheel repair dist/*.whl -w /wheels

###############################################################################
# WHEELS
###############################################################################
FROM scratch AS wheels

COPY --from=builder /wheels/*.whl .

Build the python wheel for 3.11:

$ make build
...
$ ls wheels/
confluent_kafka-1.9.2-cp311-cp311-manylinux_2_24_x86_64.whl

The wheel includes all required shared libraries:

$ unzip -l wheels/confluent_kafka-1.9.2-cp311-cp311-manylinux_2_24_x86_64.whl 
Archive:  wheels/confluent_kafka-1.9.2-cp311-cp311-manylinux_2_24_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
...
   113257  2023-01-02 11:51   confluent_kafka.libs/libsasl2-4427c928.so.2.0.25
   483537  2023-01-02 11:51   confluent_kafka.libs/libssl-6db5df62.so.1.1
 10700473  2023-01-02 11:51   confluent_kafka.libs/librdkafka-ddc374ef.so.1
  2833457  2023-01-02 11:51   confluent_kafka.libs/libcrypto-f5871c65.so.1.1

rdasari5 · 2023-01-03T21:12:16Z

Hi Team,

I am getting below error while trying to install locust plugins, please check if its related to this issue.

C:\Users\532720\AppData\Local\Temp\pip-install-xs29pl_m\confluent-kafka_5fb7b110405e498a8ff4d4598602445f\src\confluent_kafka\src\confluent_kafka.h(23): fatal error C1083: Cannot open include file: 'librdkafka/rdkafka.h': No such file or directory

error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.34.31933\bin\HostX86\x64\cl.exe' failed with exit code 2
[end of output]

stegayet · 2023-01-07T13:59:40Z

@pranavrth Do you have an ETA regarding next release by any chance ? Thanks a lot 👍

pranavrth · 2023-01-09T07:01:19Z

We are planning to have the next release after librdkafka is released. So around end of January or early February.

stegayet · 2023-01-09T12:26:17Z

Thanks for the follow-up 👍

; // FIXME: Enable support for python 3.11 when confluentinc/confluent-kafka-python#1452 is fixed This line was wrongly commented

ehn-lpbezerra · 2023-01-18T23:05:22Z

If you want to build *.whl file:

Makefile:

.PHONY: build
build:
	DOCKER_BUILDKIT=1 docker build --output type=local,dest=./wheels .

Dockerfile:

###############################################################################
# BUILDER
###############################################################################
FROM quay.io/pypa/manylinux_2_24_x86_64 AS builder

ARG CONFLUENT_KAFKA_VERSION=1.9.2
ARG LIBRDKAFKA_VERSION=1.9.2
ENV PYBIN=/opt/python/cp311-cp311/bin/

RUN set -ex \
    &&  apt-get update \
    &&  apt-get install --no-install-recommends --yes \
            build-essential \
            git \
            libsasl2-dev \
            libssl-dev \
            libzstd-dev

# build and install librdkafka
WORKDIR /tmp/librdkafka
RUN set -ex \
    &&  git clone --branch v${LIBRDKAFKA_VERSION} https://github.com/edenhill/librdkafka.git . \
    &&  ./configure --prefix=/usr \
    &&  make \
    &&  make install


# build confluent-kafka python wheel
WORKDIR /tmp/confluent-kafka-python
RUN set -ex \
    &&  git clone --branch v${CONFLUENT_KAFKA_VERSION} https://github.com/confluentinc/confluent-kafka-python.git . \
    &&  mkdir -p dist /wheels \
    &&  ${PYBIN}/python3.11 setup.py bdist_wheel -d dist \
    &&  auditwheel repair dist/*.whl -w /wheels

###############################################################################
# WHEELS
###############################################################################
FROM scratch AS wheels

COPY --from=builder /wheels/*.whl .

Build the python wheel for 3.11:

$ make build
...
$ ls wheels/
confluent_kafka-1.9.2-cp311-cp311-manylinux_2_24_x86_64.whl

The wheel includes all required shared libraries:

$ unzip -l wheels/confluent_kafka-1.9.2-cp311-cp311-manylinux_2_24_x86_64.whl 
Archive:  wheels/confluent_kafka-1.9.2-cp311-cp311-manylinux_2_24_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
...
   113257  2023-01-02 11:51   confluent_kafka.libs/libsasl2-4427c928.so.2.0.25
   483537  2023-01-02 11:51   confluent_kafka.libs/libssl-6db5df62.so.1.1
 10700473  2023-01-02 11:51   confluent_kafka.libs/librdkafka-ddc374ef.so.1
  2833457  2023-01-02 11:51   confluent_kafka.libs/libcrypto-f5871c65.so.1.1

How do I use this wheel file? I've managed to build the wheel following your instructions, but I don't know what to do with it. Any help would be much appreciated.

Thanks in advance!

y3ti · 2023-01-19T08:15:12Z

@ehn-lpbezerra just install it via pip3 install --no-index wheels/confluent_kafka-1.9.2-cp311-cp311-manylinux_2_24_x86_64.whl

pranavrth · 2023-01-24T13:37:47Z

We have released v2.0.2 with Python 3.11 wheels included.

RichardDally · 2023-01-27T21:45:26Z

OpenSSL 3 is required to use this wheel, right ?

axgkl · 2023-09-04T08:48:57Z

On conda (micromamba) a micromamba install librdkafka before installing this with pip did the trick.

mhowlett added investigate further It's unclear what the issue is at this time but there is enough interest to look into it MEDIUM labels Oct 25, 2022

joesmolsky mentioned this issue Nov 23, 2022

python 3.11 fails unit test SNEWS2/SNEWS_Publishing_Tools#62

Closed

Symbianx mentioned this issue Nov 28, 2022

Instrument newer versions open-telemetry/opentelemetry-python-contrib#1468

Merged

5 tasks

pranavrth mentioned this issue Dec 13, 2022

Windows Python311 Install Failing due to lack of Wheel #1480

Closed

6 tasks

vuongphamaioz mentioned this issue Jan 13, 2023

Installation fails SvenskaSpel/locust-plugins#110

Closed

pridhi-arora added a commit to pridhi-arora/opentelemetry-python-contrib that referenced this issue Jan 18, 2023

Modified tox.ini to fix the comment line

bb809a0

; // FIXME: Enable support for python 3.11 when confluentinc/confluent-kafka-python#1452 is fixed This line was wrongly commented

pranavrth closed this as completed Jan 25, 2023

mkmoisen mentioned this issue Aug 21, 2023

Python 3.12 Support #1629

Closed

upadhyeammit mentioned this issue Oct 6, 2023

Fixes #RHIROS-1357 - Upgrade to Python 3.11 RedHatInsights/ros-backend#381

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for Python 3.11? #1452

Support for Python 3.11? #1452

thibaultbetremieux commented Oct 25, 2022

ermakov-oleg commented Oct 26, 2022

Gr1N commented Oct 31, 2022

pranavrth commented Nov 2, 2022

Hawax commented Nov 3, 2022

RichardDally commented Nov 4, 2022

nitin-karolla commented Nov 17, 2022

maciejfic commented Nov 18, 2022

reupugi commented Nov 21, 2022

Hawax commented Nov 21, 2022

reupugi commented Nov 21, 2022

sbv-csis commented Nov 23, 2022

agates4 commented Nov 28, 2022

nitin-karolla commented Nov 29, 2022

wbarnha commented Dec 2, 2022 •

edited

Loading

TheHenrick commented Dec 7, 2022

giacomomiolo commented Dec 30, 2022

y3ti commented Jan 2, 2023 •

edited

Loading

rdasari5 commented Jan 3, 2023

stegayet commented Jan 7, 2023

pranavrth commented Jan 9, 2023

stegayet commented Jan 9, 2023

ehn-lpbezerra commented Jan 18, 2023

y3ti commented Jan 19, 2023

pranavrth commented Jan 24, 2023

RichardDally commented Jan 27, 2023

axgkl commented Sep 4, 2023

Support for Python 3.11? #1452

Support for Python 3.11? #1452

Comments

thibaultbetremieux commented Oct 25, 2022

Description

How to reproduce

Checklist

ermakov-oleg commented Oct 26, 2022

Gr1N commented Oct 31, 2022

pranavrth commented Nov 2, 2022

Hawax commented Nov 3, 2022

RichardDally commented Nov 4, 2022

nitin-karolla commented Nov 17, 2022

maciejfic commented Nov 18, 2022

reupugi commented Nov 21, 2022

Hawax commented Nov 21, 2022

reupugi commented Nov 21, 2022

sbv-csis commented Nov 23, 2022

agates4 commented Nov 28, 2022

nitin-karolla commented Nov 29, 2022

wbarnha commented Dec 2, 2022 • edited Loading

TheHenrick commented Dec 7, 2022

giacomomiolo commented Dec 30, 2022

y3ti commented Jan 2, 2023 • edited Loading

rdasari5 commented Jan 3, 2023

stegayet commented Jan 7, 2023

pranavrth commented Jan 9, 2023

stegayet commented Jan 9, 2023

ehn-lpbezerra commented Jan 18, 2023

y3ti commented Jan 19, 2023

pranavrth commented Jan 24, 2023

RichardDally commented Jan 27, 2023

axgkl commented Sep 4, 2023

wbarnha commented Dec 2, 2022 •

edited

Loading

y3ti commented Jan 2, 2023 •

edited

Loading