Significant CPU Utilization Involving Both devenv.exe and ServiceHub.RoslynCodeAnalysisService.exe

[vsfeedback]

This issue has been moved from a ticket on Developer Community.

---

[severity:It's more difficult to complete my work]
This may be related to this issue:
https://developercommunity.visualstudio.com/t/Significant-CPU-Churn-with-MicrosoftCo/1564847

However, in this case the CPU utilization is higher for devenv.exe and also now includes ServiceHub.RoslynCodeAnalysisService.exe, reaching a total of 50-55% utilization in some cases between the two processes.

Additionally, this has been occurring for some time now and does not appear to be abiding as with the issue above.

What I am seeing in devenv.exe:

And ServiceHub.RoslynCodeAnalysisService.exe:

EDIT: After many many minutes, the CPU does indeed throttle down to nominal/expected usage.

---

Original Comments

Feedback Bot on 10/29/2021, 00:02 AM:

We have directed your feedback to the appropriate engineering team for further evaluation. The team will review the feedback and notify you about the next steps.

DragonSpark on 11/3/2021, 02:37 AM:

I was able to easily produce this issue with the attached solution.

1. Open solution.
2. Locate DragonSpark.Presentation.Components.ComponentBase and make it abstract
3. Add protected abstract void HelloWorld(); to the class. This will create all sorts of compile errors.
4. With each file you edit to repair, also add a new Razor Component to the directory (CTRL-ALT-C -> Component.razor). With this new razor component, add a @inherits DragonSpark.Presentation.Components.ComponentBase to the file and implement the abstract method.
5. Eventually you will land on Starbeam.Presentation.Components.ComponentBase<T> and Starbeam.Presentation.Components.ComponentBase. Make these abstract and add a new abstract method:

		protected abstract void HelloWorldAgain();
	<span class="hljs-function"><span class="hljs-keyword">protected</span> <span class="hljs-keyword">override</span> <span class="hljs-keyword">void</span> <span class="hljs-title">HelloWorld</span>(<span class="hljs-params"></span>)</span>
	{
		HelloWorldAgain();
	}  

Bonus step:

1. Change the signature of Starbeam.Presentation.Components.ComponentBase<T>.Initialize to Starbeam.Presentation.Components.ComponentBase<T>.Initialize(bool hello) and Starbeam.Presentation.Components.ComponentBase.Initialize to Starbeam.Presentation.Components.ComponentBase.Initialize(bool hello)

(The idea here is to make breaking changes to a base class that causes compile errors in lots of inheriting classes that upon compile force you to open and fix)

From here, with each new file that you edit, add a new Razor component file and have it inherit Starbeam.Presentation.Components.ComponentBase. This eventually results in many many files opened with many errors and eventually you will start to notice the system start to really take a hit, running into this issue.

---

Original Solutions

(no solutions)

[NTaylorMullen]

With latest 17.1-Preview3 updates this scenario has improved considerably; however the RoslynCodeANalysisService usage is still higher than we'd like. @Mike-E-angelo was kind enough to do a little investigation over in the developer community and found a similar use-case for latest bits: https://developercommunity.visualstudio.com/t/10-20-CPU-Churn-for-Over-90-Seconds-Aft/1629896

My thought is the fix here most likely invovles heavily caching assembly -> set of TagHelpers. Easier said than done given the implications buttttt for future onlookers :)

[cdanek]

After reading the above comment, I migrated over to the preview track and have noticed that 17.1P4 does in fact work a bit more smoothly, but still needs a lot of love. This comment is simply here to reiterate that the CPU and Memory utilization is a fairly major process blocker - requiring a restart of VS every hour or so after many code iterations and application restarts.

Are there any improvements/changes/configuration I can make to lessen this impact?

[NTaylorMullen]

Are there any improvements/changes/configuration I can make to lessen this impact?

@cdanek It would depend on what's actually resulting in your high CPU utilization. If you're willing to file a new issue in VS with a recording i can take a look to see what's plagueing your environment.

[Mike-E-angelo]

I unfortunately am still running into major CPU utilization simply having the SLN open in 17.2 Preview 1. It's getting to the point where I am having trouble doing simple development. :(

Reported here:
https://developercommunity.visualstudio.com/t/Perpetual-CPU-Churn-with-Solution-Opened/1670456

FWIW in two days, I will be announcing the application that I have been building with your technology for the past two years. Or I guess I should say have been trying to build, in spite of all the tooling challenges. 😅

[Mike-E-angelo]

So it looks like currently with 17.2 Preview 1 my CPU will oscillate between 25-75% utilization for 30-40 minutes and then finally idle down. :|

[Mike-E-angelo]

NM... a few keystrokes brings it right back. I feel like I have been fighting and reporting the same bug for over 18 months now and it's only getting worse. 😭

[NTaylorMullen]

@Mike-E-angelo ya we're looking at your logs and something super odd seems to be happening where the system thinks there's project chancges happening and then constantly trying to re-create project.razor.json's on your behalf (which it shouldn't). Would you be up to doing a screenshare / call to dig in person?

[Mike-E-angelo]

That's a good question, @NTaylorMullen ... I am not sure how feasible a screen share would be in such a condition, as my entire machine is being consumed by this process when it occurs. It's nearly impossible to get anything done when it is occurring. Debugging is especially challenging. That stated, after complaining about it, it seems to have stopped. 😅 I have been doing development throughout the day today and haven't had this issue. So there is definitely a condition that triggers it and will be looking out for it.

I am open to doing a screenshare, or really, anything that will slay this monster. I would prefer it, however, if I could have this reproducible and know the conditions to cause it before doing so (so that we're not wasting time). The CPU utilization is definitely a concern, as the last time this happened, I was lucky to get the IDE closed as I was getting the sense it was starting to BSOD. I mean, I haven't seen one of those in a very long while, but that was the sense I was getting, if that makes sense! In any case, we would have to strategize on how to do the screenshare because ~50% of my CPU is allocated to this issue when it occurs.

BTW/FWIW, today I finally deployed the project that is responsible for all the grief that I have been reporting these 18 months to you. 😁

https://blog.starbeam.one/2022/02/announcing-mvp-alpha-preview/

I have a shout-out to you and the entire team there for all your amazing efforts:
https://alpha.starbeam.one/about/acknowledgements

[NTaylorMullen]

That stated, after complaining about it, it seems to have stopped.

lmao welllll I have mixed feelings. Part of me is happy that things are working for you while the other half of me is sad it's not reproing anymore for us to easily get to the bottom of it.

if I could have this reproducible and know the conditions to cause it before doing so (so that we're not wasting time)

Ya I'd absolutely prefer that as well. Out of curiosity, is it possible you had VSCode open on the project simultaneously w/ VS? There's a known issue we have where VSCode + VS clash for understanding of the project system and result in this type of thrashing.

BTW/FWIW, today I finally deployed the project that is responsible for all the grief that I have been reporting these 18 months to you. 😁

Hey congrats! Holy cow what a journey that must have been and I know you're thanking us for our patience when in reality we should be thanking you for all of your help in digging. Can thank you enough for all that you've contributed back to the community here ❤️

Oh and if you end up finding what potentially causes this lets line something up immediately to chat / dig. Probably the area I'll be most curious to understand is why this is getting hit repeatedly, aka why the WithProjectWorkspaceState thinks things have changed

[Mike-E-angelo]

while the other half of me is sad it's not reproing anymore for us to easily get to the bottom of it.

Don't worry, bragging that it is no longer occurring is the first step in making it appear again. 😁

Out of curiosity, is it possible you had VSCode open on the project simultaneously w/ VS?

Not in this case no, I've never used VSCode and do not intend on doing so, especially after all this investment in VS. :)

Oh and if you end up finding what potentially causes this lets line something up immediately to chat / dig

Alright, I will keep an eye open, I have had my marketing hat on for the past month in getting videos and such done. Now I should start getting back to the coding and will be able to have a better idea on what is causing this.

Do note also that I am still seeing those "Disco Colors" ... they occur intermittently and do not persist as much as before, but they are still there.

My inclination is to say that this is due to the framework code that I mentioned above, but I was able to do some modifications to some files yesterday without any trouble. Don't worry, however. It's lurking in there and will present itself at some point.

[Mike-E-angelo]

Yep this is definitely a thing @NTaylorMullen ... although TBH I am not sure how a screen share will be any better than recording ETL/DMP file and sending it your way. I am basically opening a file that is used throughout my solution and pressing Enter. I have added another recording here:
https://developercommunity.visualstudio.com/t/Perpetual-CPU-Churn-with-Solution-Opened/1670456

As noted in the latest comment I posted, I started noticing this after opening and editing a _Layout.cshtml file which is essentially referenced everywhere.

I am definitely open to installing a special build with additional telemetry if that helps figure out what is going on here.

[NTaylorMullen]

I am not sure how a screen share will be any better than recording ETL/DMP file and sending it your way.

I understand the sentiment. I think the hard part here is that internally there's a few problems happening which is making it really difficult to look into your issues. First is that for some reason the dump files that VS captures are never valid, constantly met with "Invalid dump format" after trying to load 😢. Second is there's an ongoing issue where LogHub logs (the central VS logging system that we report information in) has been on the fritz recently and hasn't been adding logs to feedback tickets. Now all that said the ETL traces have been usable and we've garnered a lot of information from them. Currently those traces are showing that this call path is the one responsible for all of your pain. The tricky part is that call path should only ever be called if you change a project.razor.json file in your solutions workspace directory. Technically that should only ever happen if you add/remove/change a components structure, i.e. add a new Razor component, mutate a component's [Parameter]s or add/mutate a reference that brings in new components/TagHelpers. Basically it should rarely be called and only ever called a single time for each of those instances, not perpetually. That train of thought has led me to think that there's a part of your system that's triggering file change events far more often than it should.

In addition to that your ETL traces mentioned that you had some project.razor.json's in your VS install directory which was super odd (do you run VS in admin?). Out of curiosity, would you be able to share how your projects are setup / included in your solution (if you're able to upload just the sln file itself in your VSTS issue that'd also be helpful)? i.e. do you have something like

C://Projects/MyProject/Foo.sln
-> C:/SomeRandomProject.csproj
-> ://Projects/MyProject/Foo/AnotherProject.csproj
-> D:/ProjectOnAnotherDrive.csproj

I'm trying to understand how / why you could have the serialized state strewn throughout your machine. I'm in investigatory mode 😄

[Mike-E-angelo]

First is that for some reason the dump files that VS captures are never valid, constantly met with "Invalid dump format" after trying to load

This is beyond frustrating for me and starting to travel into infuriation. This has been happening for many many months now and is captured here for me:
https://developercommunity.visualstudio.com/t/Feedback-Tool-produces-0-byte-dump-file/1395889

Are you able to escalate this to the group responsible for fixing this and/or able to provide a little transparency into what is taking so long in fixing such a critical feature?

Second is there's an ongoing issue where LogHub logs (the central VS logging system that we report information in) has been on the fritz recently and hasn't been adding logs to feedback tickets

🤦‍♂️🤦‍♂️🤦‍♂️

That train of thought has led me to think that there's a part of your system that's triggering file change events far more often than it should

I would bet that it's my Framework submodule that is a part of my SLN. I have attached the SLN to that ticket for your review. If providing the codebase may help, I could also be interested in doing that. One thing I did back about a year ago was provide some code that was about a year prior to that. I actually do not have a problem providing code that I am using, just the code I am using right now. 😁 Maybe like 9 months or so ago? That should still be plenty (100 projects vs 150) that can help provide additional context around this.

I'm in investigatory mode 😄

Totally understood and appreciated. I wasn't aware that the tooling was failing so poorly here. Thank you for the update/context/insight and I will assist in any way that I can. 👍

[NTaylorMullen]

Are you able to escalate this to the group responsible for fixing this and/or able to provide a little transparency into what is taking so long in fixing such a critical feature?

Ya I've been escalating internally. Thankfully there's a dev looking at it but I'm not sure how soon it will actually happen

[NTaylorMullen]

@Mike-E-angelo I took a look at your sln file and I'm getting a little bit of insight into how your solution is setup. I think you're onto something in believing it's the Framework submodule. This isn't a scenario we typically exercise a lot so might lead to some insight. I'll paly around with adding your framework as a submodule to some test projects and see what I come up with. I'll report back.

[Mike-E-angelo]

Sounds good @NTaylorMullen. Please let me know if there is any further information I can provide to assist.

[NTaylorMullen]

@Mike-E-angelo as an update. I wasn't able to reproduce one aspect of your slowness (project.razor.json's being serialized continuously) BUT I was able to reproduce another.

For the issue that I couldn't reproduce, next time you notice slowness hogging your machine could you use procmon or some other utility to find out which processes are grabbing handles / writing to project.razor.json files? I'm starting to think there might be something external at play but it's just a guess.

For the latter, I'm currently investigating and early findings are pointing towards SourceGenerators being at fault. When typing in a Razor file the generated C# gets fed into every source generator into existence (expected) BUT as part of deciding if a source generator should run the entire C# syntax tree gets allocated (less expected) resulting in loads of allocations. Not entirely sure why your scenario is especially bad here but it's a starting point. Meeting next week with folks about digging into this more

[Mike-E-angelo]

Sounds good @NTaylorMullen thank you for your continued investigation into this.

I did encounter this just now and it is hundreds of perpetual new handles opened in procmon. Notice the path:

Does that seem right? Seems like it should be in the project it is opening in, correct?

[Mike-E-angelo]

So I took a splash and deleted that file, and that seems to have addressed this issue for now. So maybe a workaround here... depending on how often that file gets written there 😁

[sharwell]

@Mike-E-angelo do you run Visual Studio as administrator? If so, avoiding that in the future should prevent the problem from occurring again.

I frequently see issue reports suggesting users are running Visual Studio as administrator, even though it's been many years since I've seen a case where that was necessary.

[Mike-E-angelo]

Indeed I do, @sharwell. I have run into many issues in the past where administrator access (or lack thereof) was a contributing factor, usually after spending many many hours tracking down a weird condition/scenario. Call me damaged. 😁 Additionally, these days I use Developer PowerShell extensively and that runs under the principal context of the IDE if I understand correctly.

In any case, if I didn't run in Administrator mode, it would hide this issue, where it appears to be a bug writing to the wrong directory. I am not sure about you, but I would rather know about that than not. :)

[sharwell]

where it appears to be a bug writing to the wrong directory

Most of the time, when I've seen this it was the result of incorrectly authored project files and/or build customization. It does give us a starting point for the investigation though.

[Mike-E-angelo]

Most of the time, when I've seen this it was the result of incorrectly authored project files and/or build customization

That is understandable. Note that I have not seen this re-occur since having deleted that directory. I am apt to think that if it was due to my project/build configuration that it would have easily presented itself again by now with a new solution build.

In any case, I am keeping an eye on it and will update here accordingly.

[NTaylorMullen]

@sharwell is there an easy way @Mike-E-angelo would be able to get a bin log of what VS is doing if he encounters it again?

[Mike-E-angelo]

Gosh @NTaylorMullen thank you so much for that. I was actually feeling really silly after posting that, telling myself Stop Whining but now I am so glad I did because another part of me is actually questioning if there is anyone out there that cares as much as I do about this -- and that is an unequivocal YES! As you know, it's not this issue but the many before it as well, nearly 18 months that it feels like and the grief is real.

And yes, I am in full agreement on the perception that one thing is fixed and another is introduced. I believe I have mentioned it feels like I have been reporting the same bug for over a year and a half now under a different guise this entire time.

The really anxiety-producing part for me is that this gets worse with the more code that I write, which is a total inevitability. That's what I was gripping with today: I am writing more code and it's only getting worse. 😭 Plus it doesn't feel like anything has changed, but as you clearly list out there are some extreme challenges.

Anyways, I cannot thank you enough for taking the time to write all that out. It is SO very much appreciated. Like, TRULY. It's a huge shot in the arm for me, thank you. 🙏

[NTaylorMullen]

@Mike-E-angelo 17.2-P3 just released. Mind trying your scenarios there and seeing if things are as noisy? It doesn't have the "BIG" fix of source generators not allocating a ton but it does have the fix of not requesting colorization for non-visible documents which I believe to be a huge contributor in your scenario

[Mike-E-angelo]

Indeed @NTaylorMullen I have been noticing it is a bit snappier but have been hesitant to say anything as it seems that when I do so it's only then that the gremlins come out to attack and prove me wrong. 😁

I haven't yet done any major refactoring/reorganization but the tag formation/completion seems better. 👍

BTW/FWIW, I sent out a shout to you in my monthly standup here when discussing friction points:
https://youtu.be/1hXKq9xk_QQ?t=585

This was actually done last week, but since the entire episode is basically me talking about everything going wrong w/ my project ATM, it's not exactly something I am jumping with endless enthusiasm to promote. :P

[Mike-E-angelo]

FWIW/FYI @NTaylorMullen looks like the DMP file issue was finally figured out:
https://developercommunity.visualstudio.com/t/Feedback-Tool-produces-0-byte-dump-file/1395889#T-N10023075

[NTaylorMullen]

FWIW/FYI @NTaylorMullen looks like the DMP file issue was finally figured out: https://developercommunity.visualstudio.com/t/Feedback-Tool-produces-0-byte-dump-file/1395889#T-N10023075

yayyyy!! That's incredible! How has 17.2-PLatest been working for you recently? Things working better?

[Mike-E-angelo]

I've done a little development with 3.0 @NTaylorMullen and it seems better. 4.0 was released but I haven't been able to get too much into it. Unfortunately I have been bamboozled by Azure deployments for the past week (again!), but I think I finally got it right this time and will be getting back into development either tomorrow or the day after. I will for sure let you know anything that I find. 👍

[Mike-E-angelo]

Dang, Preview Preview 5.0 was released today... you all are on a tear @NTaylorMullen! Are there any Razor improvements in this one?

[NTaylorMullen]

@Mike-E-angelo Preview4 actually had a masssssive perf win. One of the SourceGenerator issues was resolved in that 😉 dotnet/roslyn#59818

[Mike-E-angelo]

Woohoo @NTaylorMullen that is good news! Looking forward to trying out the latest here and will let you know. 👍

[Mike-E-angelo]

Hmm... FWIW @NTaylorMullen still tracking down this issue and in recording it in Preview 5.0, it seems to have gotten worse. Still not sure if it's related to Razor tooling but since I seem to be the only one reporting it in a verifiably consistent manner with a solution that has Razor components, I have my suspicions. :)

[Mike-E-angelo]

Alright @NTaylorMullen after doing some refactoring here, Preview 5.0 does seem a bit better. I didn't experience any churn until starting to work in files that are referenced by the startup project. In pressing keys within Razor files contained in the startup project, CPU churn is only 6-7 seconds (expected), but when performing operations in Razor components within referenced projects by the startup project, CPU churn is 30-40 seconds.

Note also that adding files to the referenced project makes the CPU go bananas as well. I have attached a new recording of this for your review here:
https://developercommunity.visualstudio.com/t/Perpetual-CPU-Churn-with-Solution-Opened/1670456#T-N10028505

Despite that, everything else seems to be doing much better. 👍

EDIT: Yeah, nevermind. Seems like it's the same old story here, unfortunately. Things look good until actual earnest development occurs and the grief is overwhelming. 😭 Basic keyboard presses are generating 70-80% CPU utilization for many minutes at a time with general development, within referenced projects. 😭😭😭

[NTaylorMullen]

@Mike-E-angelo I'm looking into your trace that you provided about referenced Razor projects. Would you mind sharing an additional trace for your 70-80% utilization bit in 17.2-P6? I'm assuming that's source generator related but definitely want to verify

[Mike-E-angelo]

Thank you for checking in @NTaylorMullen. 🙏 Unfortunately, I have closed the instance since then, and while I currently cannot pull off any multi-minute grief ATM, I added a new recording with about 100 seconds of CPU churn with a few key-presses. My assumption has been that it's basically the same issue, but you're right I should have snapped a recording for you. 😭

[NTaylorMullen]

Thank you for checking in @NTaylorMullen. 🙏 Unfortunately, I have closed the instance since then, and while I currently cannot pull off any multi-minute grief ATM, I added a new recording with about 100 seconds of CPU churn with a few key-presses. My assumption has been that it's basically the same issue, but you're right I should have snapped a recording for you. 😭

Ahhh interesting, that's a clue though. So it doesn't happen regularly?

[Mike-E-angelo]

So it doesn't happen regularly?

The recordings I am submitting are constant in my world, or at least seem this way whenever I open/edit Razor files. I know, I know... I said things seemed better. But it's not definitive until I get into doing Razor files, and ones where I am editing in the framework (referenced) projects, and usually something that involves refactoring/reorganization of code/classes/namespaces. Once I start doing that, that's when the baddies really jump out, and have been trying to capture with the recent recordings.

(Not sure if you saw my Twitter post earlier, but you can see it in action here)

What seems to be different overall is the perpetual CPU utilization. Whereas before the CPU would churn forever without an end, it now stops after 1.5-3 minutes after the last keypress. So... progress? :)

[Mike-E-angelo]

Also @NTaylorMullen I'm curious (and keep forgetting to ask)... but has there been any luck/success/effort made to reproduce this on your end with the code and steps that I have provided?

[NTaylorMullen]

Also @NTaylorMullen I'm curious (and keep forgetting to ask)... but has there been any luck/success/effort made to reproduce this on your end with the code and steps that I have provided?

I actually was only able to reproduce it on my laptop.... BUT I was able to figure out what it was... ANd you're going to hate me. It's the dumbest mistake in the world: #6364

With this change the devenv CPU utilization basically acts as if it were idle for the entirety; however, the ServiceHub bits still has some decent utilization. Granted, this should reduce the ServiceHub utilization a little bit because cache hits means less serialization CPU usage.

There may be a follow up issue here where we need to do even better on the ServiceHub side but after digging through your specific project I thought it was worth mentioning that many of your projects look like they have a massive reference chain that brings in thousands and thousands of Components / TagHelpers. This means the ServiceHub bits spend huge amounts of time discovering all of those TagHelpers for every project that reference them even though I presume those references aren't really relevant to most. Cleaning up some references to not reference component libraries when not needed may help you. IF that's not an option and even after this fix things aren't good we can create another issue to build a per-assembly cache of Components. We don't have this today due to implications on the Razor compiler and it'd be quite a large task to undertake; however, it's something we can consider if this isn't enough 😄

[Mike-E-angelo]

ANd you're going to hate me

Not a chance! I'm fully in the better late than never camp here. :) Long-term minded, too, which I'll bring up in a bit here.

Cleaning up some references to not reference component libraries when not needed may help you

I may be helped here with an example. Note that all my "presentation" assemblies are referencing those base "presentation" assemblies. I would wager that I will (or am) not be the only person doing this either. I may be the only one doing it with such tenacity ATM, however. 😁

That brings me back to being long-term minded. I would definitely like to think that others will adopt down the road using the same strategy/approach, and this will become a more common problem in the future as adoption takes hold. However, you probably have more data around that, and I understand the apprehension about having to do more work.

I would say for the moment if there is something you can see immediately that would help address this as an example, please provide it, and I'll see if I can "get it" and take it from there to make some adjustments/improvements on my end. 👍

In any case, I very much appreciate you taking the time in figuring this out @NTaylorMullen! It's a total morale booster for me and it has really helped/sorted me out with how I feel about all my investment here... 2.33 years and counting now. :)

[NTaylorMullen]

I would say for the moment if there is something you can see immediately that would help address this as an example, please provide it, and I'll see if I can "get it" and take it from there to make some adjustments/improvements on my end. 👍

Took a closer look at your project structure and the dependency layout and you're right, all your presentation bits need it 😄. Seems like the component packages you reference are just especially heavy (not your fault at all) and you reference two of them. I'm sure you're not alone here though. Might just be a matter of time till this perf-issue becomes more mainstream as you mentioned

[Mike-E-angelo]

I'M JUST AHEAD OF MY TIME @NTaylorMullen!!! 😅😅😅😭😭😭