Migrated debezium perf to process executors; can send 500k msg/s now. #2067
Conversation
…fined in .env files, instead of having them splintered all over the place.
| item_prices = None | ||
| def init_resources(kafka_only = False): | ||
| global producer, cursor, connection, item_prices | ||
| if not kafka_only: |
There was a problem hiding this comment.
There is still the cursor stuff on line 334 that I don't see gated by this.
There was a problem hiding this comment.
If you look at the initialization code for the executors, the purchases executor is initialized with init_resources(False), which means it will have a cursor. Line 334 using the global cursor is the purchases loop. So I believe this is fine; also, from output running there are no exceptions showing up for cursor being None.
| # With python 3.10 we will be able to use parenthesis around | ||
| # 'with' with multiple 'as', and get rid | ||
| # of the backslashes. | ||
| with \ |
There was a problem hiding this comment.
Do you need to have this gated on Kafka only?
There was a problem hiding this comment.
This is database initialization code in main; is independent of what the executors (which are separate processes) do.
The initialization code that gates on kafka is run on the forked sub-processes of the process pool executor.
I've changed the names of the methods and global variables that are executor specific to all start with executor_ in the hope that that makes it more clear.
No description provided.