-
-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Build process: use Celery handlers #8815
Commits on Jan 13, 2022
-
Build process: use Celery handlers
Initial migration/refactor of the build process to make usage of the Celery handlers (`on_success`, `on_failure`, etc)
Configuration menu - View commit details
-
Copy full SHA for 3b758a6 - Browse repository at this point
Copy the full SHA 3b758a6View commit details
Commits on Jan 17, 2022
-
Configuration menu - View commit details
-
Copy full SHA for b7e4683 - Browse repository at this point
Copy the full SHA b7e4683View commit details -
Configuration menu - View commit details
-
Copy full SHA for 61b5e8b - Browse repository at this point
Copy the full SHA 61b5e8bView commit details -
Test: LocalBuildEnvironment run(), __enter__ and __exit__ required
We use this environment a lot from our tests, which I don't like. However, for now, I'm keeping the because the change is not small. We should split unit-test from integration test and reduce the complexity of executing "real commands" when calling tests in exchange of more mocked data instead and unit tests for the logic.
Configuration menu - View commit details
-
Copy full SHA for 0f8cc23 - Browse repository at this point
Copy the full SHA 0f8cc23View commit details -
Configuration menu - View commit details
-
Copy full SHA for f80d2c7 - Browse repository at this point
Copy the full SHA f80d2c7View commit details -
Configuration menu - View commit details
-
Copy full SHA for d3f0dde - Browse repository at this point
Copy the full SHA d3f0ddeView commit details -
Configuration menu - View commit details
-
Copy full SHA for f1c960b - Browse repository at this point
Copy the full SHA f1c960bView commit details -
Configuration menu - View commit details
-
Copy full SHA for c72f02c - Browse repository at this point
Copy the full SHA c72f02cView commit details -
Test: avoid short-circuiting the task to test
Integration tests should not create the environment where to execute the commands nor passing weird attributes to the task. They should mock the responses of the API for the task to grab the correct data instead. This way, we avoid having "extra arguments" on the tasks just for test purposes and also avoid cheating the normal flow by overriding the environment with custom attributes. This creates lot of complexity in our code and we endup with multiple different flows that are not normal and won't ever happen in production but that we need to maintain just for the tests. In particular, we should avoid "integration tests" when they are not strictly required. In this particular case, we could create a unit-test that checks the logic for `SyncRepositoryMixin.validate_duplicate_reserved_versions` which is a lot easier to generate the context required and then an "integration test" with most things mocked (no `git` commands are executed at all) and only checks that function is being called.
Configuration menu - View commit details
-
Copy full SHA for bc8c4e8 - Browse repository at this point
Copy the full SHA bc8c4e8View commit details -
Cleanup: remove "Wipe" feature
We are always using `CLEAN_AFTER_BUILD=True`, so there is no environment to wipe anymore. This is a confusing message to send to the user since it has no effect. I'm also removing the code because it doesn't make sense anymore. If we ever need it again, we can re-implement it correctly and recover it from here.
Configuration menu - View commit details
-
Copy full SHA for 2c07948 - Browse repository at this point
Copy the full SHA 2c07948View commit details -
Configuration menu - View commit details
-
Copy full SHA for a5123cc - Browse repository at this point
Copy the full SHA a5123ccView commit details -
Cleanup: remove pip cache method helper
We are not caching pip packages anymore.
Configuration menu - View commit details
-
Copy full SHA for 640b3d3 - Browse repository at this point
Copy the full SHA 640b3d3View commit details -
Test: comments and notes about what how to migrate them
There are lot of tests not really accurate and very complex without reason. I'd like to use the same flow in production than in tests instead of short-circuiting/hacking the process in tests to behave as we want. We should use mocks to avoid executing commands when running tests instead, but making sure the "build process" is following exact the same flow in both environments. If we want to test something in particular outside the flow, we should create unit tests for those functions/methods.
Configuration menu - View commit details
-
Copy full SHA for 8df55fd - Browse repository at this point
Copy the full SHA 8df55fdView commit details
Commits on Jan 18, 2022
-
Configuration menu - View commit details
-
Copy full SHA for a6053d9 - Browse repository at this point
Copy the full SHA a6053d9View commit details -
clean_build: pass the version object
This way we avoid doing an aditional API call to get the version, since we already have it in memory.
Configuration menu - View commit details
-
Copy full SHA for 7b382dc - Browse repository at this point
Copy the full SHA 7b382dcView commit details -
Test: migrate
rtd_tests/test_builds.py
to new formatI wasn't able to mock the whole task as I wanted yet, but I follow a similar convention than the one used for the migrated test. However, I've simplify them a little.
Configuration menu - View commit details
-
Copy full SHA for 4f3e213 - Browse repository at this point
Copy the full SHA 4f3e213View commit details
Commits on Jan 19, 2022
-
There is no need to have a `base.py` and `BuildTaskBase` because it don't share code with sync repository task.
Configuration menu - View commit details
-
Copy full SHA for 38a005a - Browse repository at this point
Copy the full SHA 38a005aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5abb5a7 - Browse repository at this point
Copy the full SHA 5abb5a7View commit details -
Build: follow the same pattern for localmedia than the others
Skip moving the file if the old artifact path does not exist.
Configuration menu - View commit details
-
Copy full SHA for fdd884d - Browse repository at this point
Copy the full SHA fdd884dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4eaab60 - Browse repository at this point
Copy the full SHA 4eaab60View commit details -
Test: initial structure to mock the build completely
Ideally, we should be able to mock: * API calls * Commands run inside the environment * Git repository * Read the Docs config file This commit mocks most of them. The only missing here is the config file, but there should be code already for that. These mocks allow us to use the exact same flow than in production (e.g. hitting the API to get the Version/Build/Project object) and avoid short-circuiting it. Once the "mocked build" is finished, we can assert all the calls to them.
Configuration menu - View commit details
-
Copy full SHA for 354da75 - Browse repository at this point
Copy the full SHA 354da75View commit details -
Test: call the task with
.delay
That way, it executes the `on_success`, `on_failure`, etc.
Configuration menu - View commit details
-
Copy full SHA for 65d9d04 - Browse repository at this point
Copy the full SHA 65d9d04View commit details -
Test: check important calls when the build fails
New test to check that we are updating the build object via the API, we are sending notifications and more when the build fails.
Configuration menu - View commit details
-
Copy full SHA for d3e0bb7 - Browse repository at this point
Copy the full SHA d3e0bb7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 535834b - Browse repository at this point
Copy the full SHA 535834bView commit details -
Test: full test for successful build
It checks all the API calls done to the webs, and that the functions we expect to be called are effectively called.
Configuration menu - View commit details
-
Copy full SHA for d49ef65 - Browse repository at this point
Copy the full SHA d49ef65View commit details -
Test: migrate/update build tests
Move tests from `test_celery.py` to the "new system" and remove the invalid ones.
Configuration menu - View commit details
-
Copy full SHA for 2fba199 - Browse repository at this point
Copy the full SHA 2fba199View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2263a53 - Browse repository at this point
Copy the full SHA 2263a53View commit details -
Configuration menu - View commit details
-
Copy full SHA for 99ad8c1 - Browse repository at this point
Copy the full SHA 99ad8c1View commit details
Commits on Jan 20, 2022
-
Configuration menu - View commit details
-
Copy full SHA for a359fd6 - Browse repository at this point
Copy the full SHA a359fd6View commit details -
Configuration menu - View commit details
-
Copy full SHA for b7feb4d - Browse repository at this point
Copy the full SHA b7feb4dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0bf8171 - Browse repository at this point
Copy the full SHA 0bf8171View commit details -
Configuration menu - View commit details
-
Copy full SHA for 03e8bc2 - Browse repository at this point
Copy the full SHA 03e8bc2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1273a68 - Browse repository at this point
Copy the full SHA 1273a68View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5e94afb - Browse repository at this point
Copy the full SHA 5e94afbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3311bc3 - Browse repository at this point
Copy the full SHA 3311bc3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9841b2a - Browse repository at this point
Copy the full SHA 9841b2aView commit details -
Configuration menu - View commit details
-
Copy full SHA for cb74e02 - Browse repository at this point
Copy the full SHA cb74e02View commit details -
Configuration menu - View commit details
-
Copy full SHA for 503d10e - Browse repository at this point
Copy the full SHA 503d10eView commit details -
Configuration menu - View commit details
-
Copy full SHA for e77bd40 - Browse repository at this point
Copy the full SHA e77bd40View commit details
Commits on Jan 21, 2022
-
Configuration menu - View commit details
-
Copy full SHA for d2d6f8a - Browse repository at this point
Copy the full SHA d2d6f8aView commit details -
Build: remove the concept of "forcing a build"
It was used to "force a build" by omiting the environment cached on disk. However, this does not makes sense anymore since we are not caching anything (always CLEAN_BUILD=True) and we run just one build per instance.
Configuration menu - View commit details
-
Copy full SHA for 2e11886 - Browse repository at this point
Copy the full SHA 2e11886View commit details -
Build: remove the ability to (not)record a whole Build object
We always want to record a Build into our database. However, there are _particular_ commands that we don't want to record to not expose them to the user. The un-needed logic was removed to simplify the flow.
Configuration menu - View commit details
-
Copy full SHA for 41c4f2f - Browse repository at this point
Copy the full SHA 41c4f2fView commit details -
Configuration menu - View commit details
-
Copy full SHA for a4b1d70 - Browse repository at this point
Copy the full SHA a4b1d70View commit details -
Build: remove BuildEnvironmentWarning exception
This was only to communicate `BuildEnvironment.__exit__` that it should not log the exception as an error.
Configuration menu - View commit details
-
Copy full SHA for 0cc8a73 - Browse repository at this point
Copy the full SHA 0cc8a73View commit details -
Build: make sure we have
self.build_pk
when handling exceptionsIf the API fails when retriving the build, we should use just the `build_pk` passed when triggered the task to report it back. However, the API may fail again when hit, but at least it does not fails in our application code.
Configuration menu - View commit details
-
Copy full SHA for 41a6d1e - Browse repository at this point
Copy the full SHA 41a6d1eView commit details -
Configuration menu - View commit details
-
Copy full SHA for a45c703 - Browse repository at this point
Copy the full SHA a45c703View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4742600 - Browse repository at this point
Copy the full SHA 4742600View commit details -
Build: differentiate "user errors" from "app errors"
Show different messages depending on the error type: - user error: something the user should be able to solve by themselves (e.g. wrong RTD config file) - app error: nothing the user can do, it's an internal error that we known (e.g. can't connect to Docker API) - unknown error: we don't know what happened. Log the exception and show a generic error to the user
Configuration menu - View commit details
-
Copy full SHA for f6100be - Browse repository at this point
Copy the full SHA f6100beView commit details -
Cleanup: remove
BuildCommand.description
We are not using that field at all.
Configuration menu - View commit details
-
Copy full SHA for a5c8cc9 - Browse repository at this point
Copy the full SHA a5c8cc9View commit details -
Build: generic message for OOM/timeout when
Killed
in outputWhen the work `Killed` appears in the output it could be OOM or timeout. We are not sure. So, we insert a generic message in the build's output saying that could be one or the other. However, if we are _sure_ what the error was, the Exception risen will have the proper message and it will be shown correctly to the user.
Configuration menu - View commit details
-
Copy full SHA for 0ca908e - Browse repository at this point
Copy the full SHA 0ca908eView commit details
Commits on Jan 24, 2022
-
Build: save 'builder' when its known
Store the `Build.builder` in `before_start` so it's saved immediately next time the Build object is updated instead of at the end of the build process.
Configuration menu - View commit details
-
Copy full SHA for 5d41fed - Browse repository at this point
Copy the full SHA 5d41fedView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6299525 - Browse repository at this point
Copy the full SHA 6299525View commit details -
Configuration menu - View commit details
-
Copy full SHA for 00861bc - Browse repository at this point
Copy the full SHA 00861bcView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6426900 - Browse repository at this point
Copy the full SHA 6426900View commit details -
Configuration menu - View commit details
-
Copy full SHA for ce306aa - Browse repository at this point
Copy the full SHA ce306aaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1c2eb6a - Browse repository at this point
Copy the full SHA 1c2eb6aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 92e701b - Browse repository at this point
Copy the full SHA 92e701bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 40c373c - Browse repository at this point
Copy the full SHA 40c373cView commit details -
VCS support: normalize error reporting
`BaseVCS` class is raising a `RepositoryError` exception when the command executed in the environment fails. The logic is as follows: - Environment runs the command - Raises a BuildUserError if the command fails - `BaseVCS` detects this exception and re-raise a `RepositoryError` Each VCS Backend class, handle this `RepositoryError` nicely and return whatever is required in that case.
Configuration menu - View commit details
-
Copy full SHA for 2f6d89b - Browse repository at this point
Copy the full SHA 2f6d89bView commit details -
Configuration menu - View commit details
-
Copy full SHA for a530ad9 - Browse repository at this point
Copy the full SHA a530ad9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9e017c9 - Browse repository at this point
Copy the full SHA 9e017c9View commit details -
Merge branch 'master' of github.com:readthedocs/readthedocs.org into …
…humitos/celery-handlers
Configuration menu - View commit details
-
Copy full SHA for ceb3dcc - Browse repository at this point
Copy the full SHA ceb3dccView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4f06286 - Browse repository at this point
Copy the full SHA 4f06286View commit details -
Configuration menu - View commit details
-
Copy full SHA for aba7b1e - Browse repository at this point
Copy the full SHA aba7b1eView commit details -
Deploy: re-define the task under
readthedocs.projects.tasks
We need to re-define this task here to avoid old web instances trigger these tasks while new builders won't have it configured. Probably, we will want to deploy builders first.
Configuration menu - View commit details
-
Copy full SHA for d5a8ffb - Browse repository at this point
Copy the full SHA d5a8ffbView commit details
Commits on Jan 26, 2022
-
Since we are using only 1 process per builder, there is no need to have a mechanisim to check if there is another build running in the host and lock the version. This commit removes all the locking mechanisim to simplify the code and reduce complexity. We can come back to this if we need it again in the future.
Configuration menu - View commit details
-
Copy full SHA for 48c3507 - Browse repository at this point
Copy the full SHA 48c3507View commit details -
Configuration menu - View commit details
-
Copy full SHA for 91a2e9a - Browse repository at this point
Copy the full SHA 91a2e9aView commit details -
Cleanup: remove setting
Build
fields that are not usedThese fields were used long time ago and are not required anymore. We can go a little further in the future and remove the database fields as well.
Configuration menu - View commit details
-
Copy full SHA for d0a22e2 - Browse repository at this point
Copy the full SHA d0a22e2View commit details -
Build: improve copy for user facing error messages
Co-authored-by: Anthony <aj@ohess.org>
Configuration menu - View commit details
-
Copy full SHA for d27e487 - Browse repository at this point
Copy the full SHA d27e487View commit details -
Configuration menu - View commit details
-
Copy full SHA for a393ddb - Browse repository at this point
Copy the full SHA a393ddbView commit details -
Merge branch 'humitos/celery-handlers' of github.com:readthedocs/read…
…thedocs.org into humitos/celery-handlers
Configuration menu - View commit details
-
Copy full SHA for 0f8af38 - Browse repository at this point
Copy the full SHA 0f8af38View commit details -
Celery: add more renamed tasks
I thought they weren't required because, but they are. It could happen that `fileify` tasks get queued and while we are deploying webs. In that case, when new web instances spun up they won't have the old name defined. So, it's a good idea to define the old name for these tasks as well.
Configuration menu - View commit details
-
Copy full SHA for b3d7491 - Browse repository at this point
Copy the full SHA b3d7491View commit details -
Celery: make required arguments mandatory for build tasks
- sync_repository_task: must receive `version_id` - update_docs_task: must receive `version_id` and `build_id` and optionally a `build_commit`
Configuration menu - View commit details
-
Copy full SHA for 9fff24f - Browse repository at this point
Copy the full SHA 9fff24fView commit details -
Configuration menu - View commit details
-
Copy full SHA for ac24233 - Browse repository at this point
Copy the full SHA ac24233View commit details -
Configuration menu - View commit details
-
Copy full SHA for 928cabf - Browse repository at this point
Copy the full SHA 928cabfView commit details -
Configuration menu - View commit details
-
Copy full SHA for f48abbd - Browse repository at this point
Copy the full SHA f48abbdView commit details -
Configuration menu - View commit details
-
Copy full SHA for fb27227 - Browse repository at this point
Copy the full SHA fb27227View commit details -
Configuration menu - View commit details
-
Copy full SHA for d35d5c8 - Browse repository at this point
Copy the full SHA d35d5c8View commit details -
Configuration menu - View commit details
-
Copy full SHA for a3bc857 - Browse repository at this point
Copy the full SHA a3bc857View commit details -
Configuration menu - View commit details
-
Copy full SHA for 41216c8 - Browse repository at this point
Copy the full SHA 41216c8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 99c704d - Browse repository at this point
Copy the full SHA 99c704dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7534a84 - Browse repository at this point
Copy the full SHA 7534a84View commit details -
CircleCI: give more memory to elasticsearch
Hopefully with this it doesn't exit with `Exited with code 137`
Configuration menu - View commit details
-
Copy full SHA for 40a7750 - Browse repository at this point
Copy the full SHA 40a7750View commit details -
Merge branch 'master' of github.com:readthedocs/readthedocs.org into …
…humitos/celery-handlers
Configuration menu - View commit details
-
Copy full SHA for 21c3f10 - Browse repository at this point
Copy the full SHA 21c3f10View commit details -
Configuration menu - View commit details
-
Copy full SHA for 68f257f - Browse repository at this point
Copy the full SHA 68f257fView commit details -
Celery: re-register tasks with the old name
This allows new builds to execute tasks triggered by old web instances while doing the deploy. This code can be removed after that.
Configuration menu - View commit details
-
Copy full SHA for 9154e8d - Browse repository at this point
Copy the full SHA 9154e8dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1e9c724 - Browse repository at this point
Copy the full SHA 1e9c724View commit details
Commits on Feb 1, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 8b237a3 - Browse repository at this point
Copy the full SHA 8b237a3View commit details -
Celery: use an internal namespace to store build task's data
Use a `Task.data` (`readthedocs.projects.task.builds.TaskData` object) to store all the data the task needs to work instead of storing it directly using `self.`. This is to allow us a simpler way to perform a clean _before_ (and/or _after_) starting the execution of a new task and avoid potentially sharing state with a previous task executed that may not be able to perform the cleanup. The only thing we need to keep in mind is that when modifying these Celery tasks, we _always_ have to add any new value inside the `self.data.<my-new-attribute>` and not directly `self.<my-new-attribute>` to avoid this problem. In the future, we could implement this protection at a code level if we want to avoid this mistake. See #8815 (comment) See https://docs.celeryproject.org/en/master/userguide/tasks.html#instantiation
Configuration menu - View commit details
-
Copy full SHA for 174985a - Browse repository at this point
Copy the full SHA 174985aView commit details
Commits on Feb 3, 2022
-
Merge branch 'master' of github.com:readthedocs/readthedocs.org into …
…humitos/celery-handlers
Configuration menu - View commit details
-
Copy full SHA for 699fe96 - Browse repository at this point
Copy the full SHA 699fe96View commit details -
Merge branch 'humitos/celery-handlers' of github.com:readthedocs/read…
…thedocs.org into humitos/celery-handlers-self
Configuration menu - View commit details
-
Copy full SHA for e08c00e - Browse repository at this point
Copy the full SHA e08c00eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 017df68 - Browse repository at this point
Copy the full SHA 017df68View commit details -
Configuration menu - View commit details
-
Copy full SHA for a546f59 - Browse repository at this point
Copy the full SHA a546f59View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4c89df1 - Browse repository at this point
Copy the full SHA 4c89df1View commit details
Commits on Feb 7, 2022
-
Merge branch 'master' of github.com:readthedocs/readthedocs.org into …
…humitos/celery-handlers
Configuration menu - View commit details
-
Copy full SHA for fde6a1a - Browse repository at this point
Copy the full SHA fde6a1aView commit details -
Re-structure: move search utils tasks into
...tasks.search
I wasn't able to find nor solve the problem with the `cyclic-import` error reported by pylint. So, I tried moving the utils functions used by search tasks out from `readthedocs.projects.tasks.utils` into `readthedocs.projects.tasks.search` instead. This made `prospector` and `pylint` to pass locally without reporting any issue.
Configuration menu - View commit details
-
Copy full SHA for f8ab09e - Browse repository at this point
Copy the full SHA f8ab09eView commit details -
Merge branch 'humitos/celery-handlers' of github.com:readthedocs/read…
…thedocs.org into humitos/celery-handlers-self
Configuration menu - View commit details
-
Copy full SHA for 2491854 - Browse repository at this point
Copy the full SHA 2491854View commit details -
Configuration menu - View commit details
-
Copy full SHA for b68730d - Browse repository at this point
Copy the full SHA b68730dView commit details