-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Content Delivery API deadlocks when simultaneously requesting items referencing each other #15827
Comments
Hi there @diger74! Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better. We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.
We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions. Thanks, from your friendly Umbraco GitHub bot 🤖 🙂 |
Hi team, is there any progress on prioritizing and fixing this bug? I've done some investigation and I think I've found what causes the deadlock. Please have a look at these 2 lines of code: https://github.com/umbraco/Umbraco-CMS/blob/contrib/src/Umbraco.Core/DeliveryApi/PropertyRenderer.cs#L14-L17 Both methods set a lock during their execution. Now, imagine 2 Content Delivery API requests (for pages that are set as related to each other through content picker property) are hitting Umbraco almost at the same time, this is what happens:
Locally, I managed to avoid deadlocks by simply removing the lock here in a Property class: I've done a few tests and haven't noticed any regression side effects. It wasn't clear from the code, what exactly these locks are defending against. Also, this Property implementation is an internal class so I can't even go ahead and replace my implementation easily without going down a full custom Umbraco build path. I am desperate. The project goes live, the client's not particularly happy about this random page hanging situation, and I can't do anything about it. Help Me Obi-Wan Kenobi, You're My Only Hope. Regards, |
Closing this because of #15928 being merged in. |
Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)
12.3.7
Bug summary
When there are multiple items in the CMS tree which reference each other, e.g. assume Author content type with SimilarAuthors multinode treepicker property, and the following content:
When 2 or more of these items are requested through Content Delivery API simultaneously, it causes deadlocks in Umbraco. These requests (and all further requests to these items) hang infinitely until Memory Cache Reload or application restart. Requesting other items works just fine, only these colliding items hang.
This is only reproducible when these requested items are not yet in the Content Delivery API cache. They can be in nuCache though.
Specifics
We are implementing a website using Umbraco 12 Content Delivery API with NextJS 14.
This website contains a few thousand of products, many of which have Related Products (multinode treepicker) populated. Very often these related products form clusters of 3-5 products which are specified as related to each other.
By the description of this bug, you may think this issue occurs extremely rarely because it is hard to imagine that 2 Content Delivery API requests (with an average execution time of about 50ms) will collide at almost the same time. In our case this however happens very often during the build of a static site in NextJS, when the framework requests many pages to be built at once. With the number of pages we got in Umbraco CMS, and the fact the related products are present there very often, our NextJS builds are crushing 80% of the times with requests to Umbraco hanging in deadlocks.
Steps to reproduce
Please find a vanilla test solution prepared on Umbraco 12.3.7:
TestApiLanguages.zip
Within the CMS content tree it contains:
API requests to query this content:
To reproduce this issue, you need to be able to send the requests above at almost exactly the same time. Manually through Postman or some other tool, it is impossible to reproduce. Please find below the script implemented in k6 framework that tries to send these requests randomly in 100 concurrent threads. This approach has a pretty high probability of reproducing this issue. You need to install k6 to run this script, or implement something similar with any other tool.
Test case 1 (no deadlocks on warmed-up cache):
Restart Umbraco (works 100%) OR reload Memory Cache (works 70% of time)
Test case 2 (deadlocks):
Expected result / actual result
Test case 1:
You should NOT see any deadlocks with warmed up Content Delivery API cache.
k6 script run should show you some solid execution stats, on my machine there were ~40k successful calls to Content Delivery API in 10 seconds, returning the content back:
Test case 2:
You should experience deadlocks now, k6 script run should show something like this, with no executions, because all the requests were hanging waiting for the response from Umbraco:
Then, after running the script, if you try to request any of these 3 URLs via Postman or some other tool you should see infinite loading:
The text was updated successfully, but these errors were encountered: