Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use our own app insights client to record test results #9715

Closed
wants to merge 3 commits into from

Conversation

amunger
Copy link
Contributor

@amunger amunger commented Apr 19, 2022

Fixes #9218

Pulled in a lot of the npm package from https://github.com/microsoft/vscode-extension-telemetry to be able to send telemetry during CI tests.
I did not pull in the web-based version, since I'm not sure where/how that would be used at this point.
I had to add a sleep call to let the reporter flush, even after awaiting the dispose which is supposed to do that. If I ran a single test without the sleep call, the event would never show up in Kusto.

@amunger amunger requested a review from a team as a code owner April 19, 2022 19:15
@amunger amunger force-pushed the dev/aamunger/appInsightsClient branch from 1f3bc90 to 1d752a8 Compare April 19, 2022 19:21
* An isolated wrapper for an Application Insights client that is specifically for sending telemetry during CI jobs.
* This won't run on a users machine, so there is no need to check for opt-in status.
*/
export class CiTelemetryReporter extends BaseTelemetryReporter {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of pulling in copies of the code vscode-extension-telemetry, could you just extend the TelemetryReporter in the extension-telemetry npm module?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not the way it's currently written - getTelemetryLevel is a utility function which can't be overridden, and is used as a guard on multiple levels

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I'm following. Sorry.

getTelemetryLevel isn't overridden here?

Did you change it in the BaseTelemetryReporter?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed getTelemetryLevel entirely since this will only collect telemetry in CI, but in the npm package, it's a global utility function that checks the vscode setting.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you monkey patch it in the npm code? Just remove it at run time?

I guess I'm not too sure about making a copy of the extension-telemetry code (with modifications).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Monkey patching in the test only code sounds safer to me than copying the code over.

@lramos15
Copy link
Member

This is a surprising amount of copied code. In theory since it's just perf objects you wouldn't need most of the code and could just call trackEvent on app insights directly. I don't think there's a need to copy over the nicely structured reporters or anything like that.

@DonJayamanne
Copy link
Contributor

This is a surprising amount of copied code. In theory since it's just perf objects you wouldn't need most of the code and could just call trackEvent on app insights directly. I don't think there's a need to copy over the nicely structured reporters or anything like that.

I believe the solution (as discussed yesterday) is to enable telemetry on our CI server and just send the telemetry events specific to the tests on CI servers.

@amunger
Copy link
Contributor Author

amunger commented Apr 20, 2022

enable telemetry on our CI server

vscode internally disables any telemetry during testing, so we can't just set a configuration to enable it. Monkey patching the function that checks if it's enabled might work, but that also might send additional telemetry from vscode as well

@lramos15
Copy link
Member

I will look into a module level solution for this and report back with my findings.

@amunger
Copy link
Contributor Author

amunger commented Apr 20, 2022

still waiting on Logan's thoughts, I just wanted to push up the work that I had done earlier

@codecov-commenter
Copy link

Codecov Report

Merging #9715 (5bccba2) into main (261afd9) will increase coverage by 0%.
The diff coverage is n/a.

@@          Coverage Diff          @@
##           main   #9715    +/-   ##
=====================================
  Coverage    63%     63%            
=====================================
  Files       204     204            
  Lines      9840    9854    +14     
  Branches   1556    1558     +2     
=====================================
+ Hits       6242    6261    +19     
+ Misses     3335    3092   -243     
- Partials    263     501   +238     
Impacted Files Coverage Δ
src/platform/common/utils/async.ts 49% <0%> (-1%) ⬇️
src/platform/logging/index.ts 73% <0%> (-1%) ⬇️
src/platform/api.ts 57% <0%> (ø)
src/platform/errors/types.ts 77% <0%> (ø)
src/platform/errors/errors.ts 62% <0%> (ø)
src/platform/common/utils.node.ts 57% <0%> (ø)
src/platform/common/utils/misc.ts 71% <0%> (ø)
src/platform/errors/errorUtils.ts 75% <0%> (ø)
src/platform/common/cancellation.ts 72% <0%> (ø)
src/platform/common/helpers.node.ts 60% <0%> (ø)
... and 43 more

@lramos15
Copy link
Member

Just an update that we have added this native functionality to the module so it should be much easier to implement this and involve no code copying.

@amunger amunger closed this Apr 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enable queries on test run history
5 participants