Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Meta injection #366

Merged
merged 8 commits into from
Mar 3, 2023
Merged

Meta injection #366

merged 8 commits into from
Mar 3, 2023

Conversation

rodrigondec
Copy link
Contributor

Resume

The issue #37 points to save data using the arg meta from the method update_state, but the meta attribute is renamed to result when it is passed to self.backend.store_result.

As explained here in this comment #37 (comment).

https://github.com/celery/celery/blob/cb4c3256326e33f005c63d48520d0abb5e898151/celery/app/task.py#L982-L994

    def update_state(self, task_id=None, state=None, meta=None, **kwargs):
        """Update task state.

        Arguments:
            task_id (str): Id of the task to update.
                Defaults to the id of the current task.
            state (str): New state.
            meta (Dict): State meta-data.
        """
        if task_id is None:
            task_id = self.request.id
        self.backend.store_result(
            task_id, meta, state, request=self.request, **kwargs)

https://github.com/celery/celery/blob/cb4c3256326e33f005c63d48520d0abb5e898151/celery/backends/base.py#L513-L522

    def store_result(self, task_id, result, state,
                     traceback=None, request=None, **kwargs):
        """Update task state and result.

        if always_retry_backend_operation is activated, in the event of a recoverable exception,
        then retry operation with an exponential backoff until a limit has been reached.
        """
        result = self.encode_result(result, state)


        retries = 0

Problem

So the argued meta arg on store_result is not really arbitrary metadata. It's actually the result of the Task.

Solution

Use the TaskResult.meta to store arbitrary data.

How to inject?

I implemented the meta injection through the Task Celery Request. This way I don't mess around with too many implementations and function signatures.

@auvipy auvipy self-requested a review February 26, 2023 18:10
Copy link
Member

@auvipy auvipy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you please check the CI failures?

@rodrigondec
Copy link
Contributor Author

can you please check the CI failures?

Yes, I was awaiting the maintainer approval for the CI to run 😁

Copy link
Member

@auvipy auvipy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

anything else you want to add or improve in this PR?

@rodrigondec
Copy link
Contributor Author

anything else you want to add or improve in this PR?

No, for me it's good to go 🚀

@auvipy auvipy self-requested a review March 1, 2023 15:08
@auvipy auvipy merged commit 812909f into celery:main Mar 3, 2023
@rodrigondec
Copy link
Contributor Author

There's a roadmap with this PR on a release? There's a expected date for the new version?

@auvipy
Copy link
Member

auvipy commented Mar 6, 2023

if possible this week, if not in this month

@rodrigondec rodrigondec deleted the feature/meta-injection branch March 6, 2023 13:13
@cdevacc1
Copy link

@rodrigondec @auvipy I still have this problem with the latest version (2.5.1)

self.update_state(state='RUNNING', meta={'current': order_counter, 'total': total_order})

image

@ericswpark
Copy link

@cdevacc1 I have the same problem, see issue #367. In your screenshot though, I think it might be because RUNNING is not one of the task states defined within the library. If you look at my screenshot here, you can see PROGRESS printed next to the task ID, but the actual task state field is set to - like yours.

@cdevacc1
Copy link

@ericswpark Yes. I looked through models.py, status must be in celery.states.ALL_STATES
['FAILURE', 'PENDING', 'RECEIVED', 'RETRY', 'REVOKED', 'STARTED', 'SUCCESS']

status = models.CharField(
        max_length=50, default=states.PENDING,
        choices=TASK_STATE_CHOICES,
        verbose_name=_('Task State'),
        help_text=_('Current state of the task being run'))

ShouId I remove choices from status field to make it work with custom states?

@ericswpark
Copy link

@cdevacc1 that would be great :)

@peterbaumert
Copy link

peterbaumert commented Aug 18, 2023

For me the example from the "injecting-metadata" file is not working at all. No matter what i do, the information i put in request.meta or the update_state function, the info is nowhere to be found :(

"Result Data" only shows the pid and hostname. ( i dont write that in there, so must come from somewhere else ^^ )

ericswpark added a commit to ericswpark/django-celery-results that referenced this pull request Sep 3, 2023
From celery#366 (comment)

Thanks to @cdevacc1 for the idea. I just decided to write up the PR as
it only takes a couple of minutes. This should resolve the bug where
custom states are not shown at all in the admin panel.
@ericswpark
Copy link

@cdevacc1 decided to tackle this issue today so I just made a PR from your comment. See ^

ericswpark added a commit to ericswpark/django-celery-results that referenced this pull request Jun 19, 2024
From celery#366 (comment)

Thanks to @cdevacc1 for the idea. I just decided to write up the PR as
it only takes a couple of minutes. This should resolve the bug where
custom states are not shown at all in the admin panel.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants