-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] Add s390x architecture #4777
base: master
Are you sure you want to change the base?
Conversation
API tests result: success ✅ |
Hello @buger, I've got good news :) To start off, it builds ok for s390x.
EDIT: I see that redis is needed for running |
@v1gnesh yes. The main issue is not a gw but its dependencies. For example current tests are failing because it can't make work Python plugins. So if we want this arch, maybe we will have to say that Python plugins not tested/supported, and disable it in tests. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm afraid this PR is incomplete. I would approach this by having a build validated on s390 and then approach adding s390 to the build pipeline.
I may be completely off on this one, please bear with me. This issue details GitLab's quest with And finally, IBM provides build instructions for In summary, one way is to build with Either way, this is quite a deep rabbit hole. Is any of this relevant or helpful? 😅 |
API tests result: success ✅ |
API tests result: success ✅ |
Is there an option of trying the python grpcio build with RHEL? grpcio Python module fails to build on IBM/z (s390x) machine #23797 For a bunch of these errors, it seems that installing
Alternatively, I came across a command-line option to point I've to say, I'm pretty out of depth now lol. |
@v1gnesh to be frank I'm thinking of just removing this from the docker grpc - it is optional and required for Python and Coprocess plugins. So I think it should be good compromise. |
On an unrelated note, is there an option of using plugins in Rust, if we're unable to work out Python? |
Okay. Another strange question - is there an option of using NATS for the inter-language connect? I do understand that RPC and messaging are different; just wondering if it makes sense for this platform or in some contexts, since you'll know more about both. |
It is a bit different context, yes. RPC is about real-time response, e..g request -> response. NATS more about async work, you send the message and wait for worker to process it (and not always expect response back). But if you have some specific use-case with it, it can be implemented using Go plugins. |
As for your question about Rust. GRPC plugins was designed for this case. And in general, maybe it will work without installing dependencies. In order for GRPC plugins to work, it require only Go codebase, and does not depend on Python deps. So if I remove Python support from this image, gRPC with plugin written in Rust still should be an option. |
Is there a significant (like maybe an order of magnitude) difference in response time when using NATS (async) for RPC (real-time)? |
That's good to hear. I wonder how much one can re-use stuff from axum. |
Here is the simple Rust gRPC plugin, written by ChatGPT :)
Have not tested, and probably not work out of the box, but you can get the idea, it is similar to another gRPC plugins like https://tyk.io/docs/plugins/supported-languages/rich-plugins/grpc/request-transformation-java/ |
💥 CI tests failed 🙈git-stateall ok Please look at the run or in the Checks tab. |
API tests result: success ✅ |
API tests result: success ✅ |
Hi @buger, I've managed to get access to s390x VMs through this IBM LinuxONE OSS Community Cloud form. To start off, I got a $ sudo dnf install gcc-c++ openssl-devel golang python3-devel python3-pip redis
$ pip install --upgrade pip setuptools wheel ... and $ GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 pip3 install grpcio
Defaulting to user installation because normal site-packages is not writeable
Collecting grpcio
Using cached grpcio-1.51.3.tar.gz (22.1 MB)
Preparing metadata (setup.py) ... done
Building wheels for collected packages: grpcio
Building wheel for grpcio (setup.py) ... done
Created wheel for grpcio: filename=grpcio-1.51.3-cp39-cp39-linux_s390x.whl size=78562413 sha256=eae086a6e53fade54c33ec1a6ccf0e00d1b369bdd56b9fe1ef3ce5cdb0fdae01
Stored in directory: /home/linux1/.cache/pip/wheels/a9/cd/20/1347f1406df5f8bec22089be8f7406b843870f7685cc3fcfe4
Successfully built grpcio
Installing collected packages: grpcio
Successfully installed grpcio-1.51.3 If you could sign up for an s390x VM (which is free for OSS projects) in the above link, you'll get access to a machine to build/test for s390x. |
That's great research! But also quite complex process. I would say it will depend on prioritisation. In its current state it works well, even with gRPC plugins (just Python plugins does not work). So for majoirty of users it can be not a big deal. If we will have some commercial lever for Python plugin support on this platform, then we will get deeper. Hope it makes sense! |
I understand. Do you think it's worth re-running the build process once, with these additional packages going in before grpcio in the pip install command. It's really just 3 commands. $ sudo dnf install gcc-c++ openssl-devel golang python3-devel python3-pip
$ pip install --upgrade pip setuptools wheel
$ GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 pip3 install grpcio On Ubuntu, I guess it'll be Adding a link to the original build failure logs for future reference - I'm sure you already know this, but having the whole thing working unlocks (for Tyk) new customers who run on a whole new platform. For a reasonably small amount of extra work, your product becomes an important & compelling answer to API management across domains, for any company w/ distributed & mainframe tech. If it was for Java plugins, I wouldn't care lol. A lot of people know Python (than Go for example), and having that piece of the puzzle work for this platform would be nice. |
One challenge with building grpcio from source as part of the release is the time it takes. It's not easy to cross-compile Python modules and it takes a long time to build under emulation; I've also found emulation of s390x and ppc64le through QEMU to be a bit buggy at times (especially when running locally on an M1 Mac). Self-hosted Github runners don't support many architectures at present; Gitlab runners are more flexible but that's a whole different topic. There's also the question of supporting 3rd party package compilation.. what happens when a new release of the package fails to compile on a given target architecture? There's probably many more questions I haven't thought of yet :) One approach might be to separate building of grpcio from the release of Tyk itself.. I've experimented with building the package as a wheel that is installed in a later build stage. This could be separated into a different build process altogether and once compiled the wheels could be pushed to a package repository rather than being built as part of the core Tyk release process: during the Tyk release the pre-compiled package for the target architecture would be installed. However, this also introduces a bunch of work/validation: Python wheels are sensitive to architecture, Python version and possibly pip version too. Not everybody uses Docker.. so an interim step might just be to document how to build/install the Python packages on architectures where pre-compiled versions are not available? |
Thank you for sharing the details Chris, always fun learning the thought processes behind decisions.
It might be worth taking a quick peek at https://earthly.dev/.
IBM provides s390x virtual machine(s) for open source projects, and sign-up is here:
You sharing your thought process here is great, everyone reading this can learn from it too!
True, I guess the only way to find out is to test the waters.
Yup, I haven't gone beyond working out EDIT: In addition to a 120-day free trial, IBM can also provide permanent set of machines, which can be worked out with them (no cost for OSS projects). |
Hey @chrisanderton, it seems you can also sign up for OpenShift on IBM/Marist's community cloud - https://linuxone.cloud.marist.edu/#/register?flag=OCP |
💥 CI tests failed 🙈git-stateall ok Please look at the run or in the Checks tab. |
I have added the proper cross compilation toolchain for given architectures to base golang-cross image, and it should resolve Go plugins issue. Soon we are migrating to Go 1.19, and this PR will migrate there too. Once 1.19 will be merged, we start planning merging of this PR as well. |
API tests result: success ✅ |
💥 CI tests failed 🙈git-stateall ok Please look at the run or in the Checks tab. |
You have successfully added a new SonarCloud configuration ``. As part of the setup process, we have scanned this repository and found no existing alerts. In the future, you will see all code scanning alerts on the repository Security tab. |
API tests result: success ✅ |
Thanks for your continued engagement @v1gnesh 👍
Exactly this. We're looking at this from all sides, both business and technical. The technical team are taking a closer look at including it in our build process so we can better evaluate the end to end situation. I expect you'll have seen some of the commits. From a business perspective having a customer agree to pay for a supported environment would obviously accelerate some of the thinking!
The good thing is that getting all the packages built without Python plugin support is quite straightforward. As before, including Python plugin support in the build has a dependency on compiling grpcio against OpenSSL; doing this in the core build process is out of scope for our initial investigation. I know from my own experiments that it's possible to build and install the grpcio Python package on both s390x and ppc64le.. so this isn't a technical problem to solve, more of a packaging challenge/topic.
Yep.. I have one of these on a trial basis and saw the other option for longer term use. The gap is not so much access to virtual environments, but more having a CI-system that supports remote runners on the architecture. We currently use Github actions and self-hosted runners are limited in architecture support at present. It's all part of the mix :) We're on an invite together from a few weeks back so if you ever want to drop me an email or talk about this on a call i'm available. We'll keep you posted on any progress in the meantime. |
IBM & others involved have been working a lot on bringing more open source to s390x. I found these recently - |
Kudos, SonarCloud Quality Gate passed! |
API tests result: success ✅ |
Description
Related Issue
Motivation and Context
How This Has Been Tested
Screenshots (if appropriate)
Types of changes
Checklist