feat: Multi queue consumption #1475
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ankush
force-pushed
the
memory_optimized_config
branch
from
f7a6ac5
to
1de4f4d
Compare
|
Kudos, SonarCloud Quality Gate passed!
|
github-actions bot
pushed a commit
that referenced
this pull request
Aug 8, 2023
# [5.17.0](v5.16.4...v5.17.0) (2023-08-08) ### Features * Multi queue consumption ([#1475](#1475)) ([5a2c052](5a2c052))
|
🎉 This PR is included in version 5.17.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
github-actions bot
pushed a commit
to Zawala/bench
that referenced
this pull request
May 26, 2024
# [5.9.0](v5.8.1...v5.9.0) (2024-05-26) ### Bug Fixes * (re)Install every Frappe app even if not installed to env ([ffae670](ffae670)) * add code removed via erroneous commit ([b28cfab](b28cfab)) * add customer background workers in group ([frappe#1439](https://github.com/Zawala/bench/issues/1439)) ([4ec73c3](4ec73c3)) * add jammy package for Ubuntu 22 ([frappe#1549](https://github.com/Zawala/bench/issues/1549)) ([5a0922d](5a0922d)) * add safety filter for untarring ([e8ea985](e8ea985)) * Add support for options on bench main group ([8a0b784](8a0b784)) * add useforsecurity with fallback for <3.9 ([319809d](319809d)) * added code removed via erroneous commit ([5b64175](5b64175)) * added code removed via erroneous commit ([7e28a3d](7e28a3d)) * allow bench init with git tags ([c943b30](c943b30)) * Archive app for different repo/app names ([3af6058](3af6058)) * bad URL in raw.githubusercontent.com ([frappe#1363](https://github.com/Zawala/bench/issues/1363)) ([589f7a4](589f7a4)) * **build:** pin hatchling version 2nd try ([b74ff54](b74ff54)) * **build:** pin hatchling version to prevent import error ([3a73059](3a73059)) * **build:** support pyproject version ([frappe#1502](https://github.com/Zawala/bench/issues/1502)) ([7c64aa3](7c64aa3)) * Bump Nodejs v12 => v14 ([cd0fccf](cd0fccf)) * calc md5 for app cache ([16f8906](16f8906)) * case-insensitive matching ([3f57616](3f57616)) * check current branch ([2287f3f](2287f3f)) * Check for unauthorized status code in response ([e9b0112](e9b0112)) * check for valid branch in resolve ([79765e6](79765e6)) * check frappe-deps during get-app ([d93656e](d93656e)) * check if installed FF supports use-cached ([d177d8f](d177d8f)) * check md5 when fetching from cache ([efb5171](efb5171)) * Check yarn files during install ([frappe#1533](https://github.com/Zawala/bench/issues/1533)) ([1a42175](1a42175)) * checking status code to determine rate limit ([dbb1904](dbb1904)) * coerce git_url to app_name ([54b514b](54b514b)) * **config:** Fix fail2ban filter ([frappe#1308](https://github.com/Zawala/bench/issues/1308)) ([94a25d3](94a25d3)) * correct python path for migrate_env ([61e0f4a](61e0f4a)) * del node_modules if frappe supports app_cache ([9988b83](9988b83)) * Delete folder if --no-backup is passed ([f7f7459](f7f7459)) * don't compile python files manually. ([frappe#1469](https://github.com/Zawala/bench/issues/1469)) ([036cb62](036cb62)) * Don't handle `BaseException`s ([ad000f1](ad000f1)) * dont attempt migrating to active virtualenv ([0e59159](0e59159)) * dont use glob pattern for `python` exec ([dc047a5](dc047a5)) * **DX:** make warnings go to stderr ([b6b9481](b6b9481)) * easy-install for letsencrypt tls ([frappe#1424](https://github.com/Zawala/bench/issues/1424)) ([eba0f7a](eba0f7a)) * **easy-install:** set default site to site1.localhost ([fde7a82](fde7a82)) * empty states json condition ([67df07b](67df07b)) * Fetch env python if exists before returning fallback ([e5fa4df](e5fa4df)) * FileNotFound bug ([frappe#1383](https://github.com/Zawala/bench/issues/1383)) ([965e178](965e178)) * fixed app states and checking/updating frappe version on install ([30a4ec3](30a4ec3)) * fixed app states and checking/updating frappe version on install ([42f4d74](42f4d74)) * fixed args to sync-states ([c667be9](c667be9)) * fixed frappe branch check ([6c15327](6c15327)) * fixed init in get-app and frappe versions ([88c2367](88c2367)) * fixed init in get-app and frappe versions ([e3bd34c](e3bd34c)) * Fixed order of install ([281d96a](281d96a)) * fixed org, branch for apps mounted on disk ([frappe#1293](https://github.com/Zawala/bench/issues/1293)) ([c8ef822](c8ef822)) * fixed states sync ([7f37488](7f37488)) * fixed update_app_states call ([frappe#1326](https://github.com/Zawala/bench/issues/1326)) ([45267fc](45267fc)) * get-app on existing apps ([e03f597](e03f597)) * Handle supervisor escalation better ([frappe#1438](https://github.com/Zawala/bench/issues/1438)) ([c690e8e](c690e8e)) * Handle tag fetching failures of disk mounted App inits ([117d456](117d456)) * Hide irrelevant stacks during exc raise ([8175239](8175239)) * ignore org check fail if using cache ([aa12aab](aa12aab)) * ignore supervisor restart failures where possible ([frappe#1400](https://github.com/Zawala/bench/issues/1400)) ([7a63f4f](7a63f4f)) * image replace and site creation ([f93f065](f93f065)) * increase tar file limit ([d8ce1b3](d8ce1b3)) * install one app at a time, frappe first ([4f0193c](4f0193c)) * install wheel in venv ([4996afb](4996afb)) * logging when in init-bench ([a26f3ce](a26f3ce)) * logging when in init-bench ([de315f2](de315f2)) * merge conflicts ([61a864d](61a864d)) * merge conflicts ([ca45305](ca45305)) * migrate_env silent failures ([7f1c5ad](7f1c5ad)) * minor fixes ([94105c0](94105c0)) * minor fixes ([e629ca6](e629ca6)) * Misc fixes ([frappe#1294](https://github.com/Zawala/bench/issues/1294)) ([eb71573](eb71573)) * missing installed branch ([af3c871](af3c871)) * more reliable gunicorn restart ([2ec0c0a](2ec0c0a)) * **nginx:** Allow specifying log_format setting ([bbf0169](bbf0169)) * **nginx:** set `proxy_http_version` to `1.1` ([7da62f9](7da62f9)) * no-git support for dependency resolver ([frappe#1368](https://github.com/Zawala/bench/issues/1368)) ([fcbc2ea](fcbc2ea)) * nodejs redis v4 interprets localhost as ipv6 address ([697939e](697939e)) * override X-Forwarded-For address in bench manager ([53a8fed](53a8fed)) * parsing error ([frappe#1364](https://github.com/Zawala/bench/issues/1364)) ([4bc3a08](4bc3a08)) * Pass abs path from get_env_cmd ([10473b6](10473b6)) * Pass cwd for cmd subprocess execution ([01b7759](01b7759)) * **patch:** Rename parent dir instead of moving individual dirs ([0bd0717](0bd0717)) * port number mismatch when having multiple benches ([frappe#1480](https://github.com/Zawala/bench/issues/1480)) ([3c8cbbc](3c8cbbc)) * prevent circular dependency ([7bcea60](7bcea60)) * Prevent Gunicorn child workers hangup ([36f2194](36f2194)) * Print failing command with CommandFailedError ([a1d2c52](a1d2c52)) * print help when no args passed ([228aeaf](228aeaf)) * print required by in output ([5bb3b52](5bb3b52)) * Remove backup command ([frappe#1345](https://github.com/Zawala/bench/issues/1345)) ([0cd5fca](0cd5fca)) * Remove broken import path ([2c5c28b](2c5c28b)) * Remove except-pass ValueError ([d9c8335](d9c8335)) * Remove frappe cmd caching in .bench.cmd ([f773529](f773529)) * remove rem check from app-cache --clear ([4e170a2](4e170a2)) * remove unused nodemodules before caching ([0e2e8b4](0e2e8b4)) * Remove virtualenv dependency ([e0f6b17](e0f6b17)) * **remove-app:** Use App.name instead of App.repo for consistent App name ([a3b859b](a3b859b)) * removed erroneous file push ([f2fe569](f2fe569)) * removed erroneous file push ([b7e46aa](b7e46aa)) * removed unnecessary requests while removing apps ([49900ce](49900ce)) * replace certbot-auto with certbot ([frappe#1303](https://github.com/Zawala/bench/issues/1303)) ([a889325](a889325)) * resolve filesystem app dependencies ([frappe#1450](https://github.com/Zawala/bench/issues/1450)) ([f3b55f7](f3b55f7)) * restart proc manager if set in config ([frappe#1391](https://github.com/Zawala/bench/issues/1391)) ([c59d1ed](c59d1ed)) * Revert usage of app_name for repo name ([f7fbee9](f7fbee9)) * reverted changes made by erroneous commit ([b8aa008](b8aa008)) * reverted erroneous commits ([ff98aaf](ff98aaf)) * RQ crashes on Mac ([frappe#1529](https://github.com/Zawala/bench/issues/1529)) ([f5ab045](f5ab045)) * Setup chdir patch earlier ([db165d1](db165d1)) * Setup exception handler for CommandFailedError only ([94a1b8c](94a1b8c)) * specify hatch version in release ([87b4b1f](87b4b1f)) * support for local and remote apps when looking for dependencies ([2f17da1](2f17da1)) * support single required app ([c8205e4](c8205e4)) * Switched to native GIT way to lookup branch names ([frappe#1553](https://github.com/Zawala/bench/issues/1553)) ([1751b2d](1751b2d)) * sync socketio port for old benches ([frappe#1481](https://github.com/Zawala/bench/issues/1481)) ([c2af169](c2af169)) * trigger release ([f374e80](f374e80)) * update check condition ([3e749ba](3e749ba)) * update install_app with using_cached flag ([10bb5a4](10bb5a4)) * update pre url coercion check ([6f074a7](6f074a7)) * updated apps.sync to support new app_name parameter ([7980709](7980709)) * Use == for dict comparison ([e80daf8](e80daf8)) * Use == to define had pinned requirement ([beac865](beac865)) * Use App.app_name over repo attr ([4f42392](4f42392)) * Use hardcoded python3 env cmd for Bench.python ([c9c6bf4](c9c6bf4)) * use os.remove instead of shutil.rmtree for symlinks ([frappe#1372](https://github.com/Zawala/bench/issues/1372)) ([9fd15f6](9fd15f6)) * Use python* to match any pattern in env ([24b9af6](24b9af6)) * use ref param instead of branch for get_required_deps (GitHub contents API) ([frappe#1362](https://github.com/Zawala/bench/issues/1362)) ([9131ba0](9131ba0)) * use specified python for venv ([df84c27](df84c27)) * Use subscript instead of method for regex ([e126ff2](e126ff2)) * Use venv module instead of virtualenv ([f60c2d0](f60c2d0)) * User messages on get-app actions ([d33d987](d33d987)) * using local clones if path is passed in get-app ([5f37135](5f37135)) * **utils:** Strip * from cmd via get_env_cmd ([6ae1997](6ae1997)) * version check backward compatibility ([frappe#1409](https://github.com/Zawala/bench/issues/1409)) ([23eede5](23eede5)) * version check before AbsoluteLinkError ([3502c77](3502c77)) * version check before data_filter import ([80f2e70](80f2e70)) * wrap tarfile with error handling ([30f301e](30f301e)) * wrong help string ([7a89ccd](7a89ccd)) ### Features * `bench validate-dependencies` command ([frappe#1534](https://github.com/Zawala/bench/issues/1534)) ([ab9b617](ab9b617)) * add `max_requests` to gunicorn args ([b57838f](b57838f)) * add bench app-cache helper command ([42ac015](42ac015)) * add commit hash and branch in `states.json` ([9004e1d](9004e1d)) * add GHA tests ([e76c7dc](e76c7dc)) * Add pre-commit checks ([b1b8d70](b1b8d70)) * add required and order of install ([9ac091b](9ac091b)) * Added app states ([fdab757](fdab757)) * Added app states ([549e8e2](549e8e2)) * Added app_name attribute to App class ([35504f8](35504f8)) * Added bench command for resolve and install ([cdaeea1](cdaeea1)) * Added bench command for resolve and install ([7a311a2](7a311a2)) * Added exception for invalid version ([ddd498e](ddd498e)) * Added support for backward compatibility in states ([a1a35ee](a1a35ee)) * added support for fetching dependencies from hooks.py ([58319a2](58319a2)) * added support for fetching dependencies from hooks.py ([7f678a3](7f678a3)) * Added support for mounted app in get_dependencies ([0b862c0](0b862c0)) * allow custom image in easy install ([8ba685c](8ba685c)) * Allow skipping supervisord config check in setup supervisor ([a3af905](a3af905)) * alternative request to validate branch ([8dd92c3](8dd92c3)) * cache get-app artifacts by commit_hash ([87d4aa3](87d4aa3)) * comment out unsupported typing ([23bd717](23bd717)) * **config:** Add IPv6 listener to nginx site configuration ([frappe#1312](https://github.com/Zawala/bench/issues/1312)) ([593c7b7](593c7b7)) * Exiting when app does not exist ([fc0be25](fc0be25)) * flag to setup bench with developer mode enabled ([f29e25a](f29e25a)) * get-app ln -s ([e69b612](e69b612)) * introduced simple resolver ([bfd69c3](bfd69c3)) * introduced simple resolver ([d41e05e](d41e05e)) * Made dynamic mount path if bench attr is passed to App class ([b36c072](b36c072)) * mimicking get-app behaviour when app is already installed ([bbca3e9](bbca3e9)) * Multi queue consumption ([frappe#1475](https://github.com/Zawala/bench/issues/1475)) ([5a2c052](5a2c052)) * New easy-install.py ([044e01b](044e01b)) * **nginx:** Allow logging settings as site/combined/none ([5370129](5370129)) * pass verbose to npm install ([00391bf](00391bf)) * pass verbose to yarn install ([346d08d](346d08d)) * printing resolved dependencies when using resolve-deps flag wih get-app ([95b0834](95b0834)) * printing resolved dependencies when using resolve-deps flag wih get-app ([bba4019](bba4019)) * **remove-app:** Options for no backup of app & force removal ([69e14e5](69e14e5)) * set `startretries` in supervisor configuration ([deadc7c](deadc7c)) * Setup dev-dependencies defined via pyproject ([75957a5](75957a5)) * slower alternative for get_required_deps ([d0bd294](d0bd294)) * **supervisor:** restart each group separately ([839bd7c](839bd7c)) * Support Frappe app distributions with pyproject.toml ([800394c](800394c)) * using gitpython for branch validation ([cb2b0b0](cb2b0b0)) * verbose env setup & checking for compatible versions ([d178b08](d178b08)) * verbose error messages ([19f9325](19f9325)) ### Performance Improvements * add Cache-Control header for assets ([8d3270e](8d3270e)) * Avoid use of glob for fetching python executable ([8db23dd](8db23dd)) * caching deps and orgs, repos to avoid time taking checks & added slower version to check org and repo ([bd105a6](bd105a6)) * single worker in development install ([frappe#1392](https://github.com/Zawala/bench/issues/1392)) ([9c80f5d](9c80f5d)) * Using github API ([b0ae3ae](b0ae3ae)) ### Reverts * Revert "fix: resolve filesystem app dependencies (frappe#1450)" (frappe#1452) ([082692c](082692c)), closes [frappe#1450](https://github.com/Zawala/bench/issues/1450) [frappe#1452](https://github.com/Zawala/bench/issues/1452)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refer last section here for more info: https://frappe.io/blog/engineering/reducing-memory-footprint-of-frappe-framework