Add XUnitLogChecker to log libraries dumps, do not enable for NativeAOT tests #94868

carlossanlop · 2023-11-16T19:59:02Z

Second attempt of #93906 . That change had to be reverted because NativeAOT tests were broken in outerloop.

I added an extra commit that changes the NativeAOT condition per @MichalStrehovsky 's suggestion in this comment.

ghost · 2023-11-16T19:59:08Z

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

Second attempt of #93906 . That change had to be reverted because NativeAOT tests were broken in outerloop.

I added an extra commit that changes the NativeAOT condition per @MichalStrehovsky 's suggestion in this comment.

Author:	carlossanlop
Assignees:	carlossanlop
Labels:	`area-Infrastructure-libraries`
Milestone:	-

carlossanlop · 2023-11-16T19:59:14Z

/azp run

azure-pipelines · 2023-11-16T19:59:20Z

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

carlossanlop · 2023-11-16T19:59:45Z

/azp list

carlossanlop · 2023-11-16T20:00:33Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2023-11-16T20:00:44Z

Azure Pipelines successfully started running 1 pipeline(s).

hoyosjs

Lgtm as long as OuterLoop passes

carlossanlop · 2023-11-16T23:51:45Z

It failed again, but I see what's happening:

The Build step for NativeAOT jobs compile the product by passing the /p:TestNativeAot=true property, which ensures XUnitLogChecker won't be built (log).
- Building the product for outerloop NativeAOT tests is done here: eng/pipelines/coreclr/runtime-nativeaot-outerloop.yml#L67
But the Send to helix step does not pass either of the two properties that IsXUnitLogCheckerSupported checks (TestNativeAot or the freshly added RunNativeAotTestApps). So naturally, the IsXUnitLogCheckerSupported property will be true and the tests will fail because they try to run the tool but can't find it (log).
- Testing is done via /p:testRunNamePrefixSuffix=NativeAOT_$(BuildConfig) property is passed: eng/pipelines/coreclr/runtime-nativeaot-outerloop.yml#L71-L74

I'm thinking that one option would be to pass the /p:IsXUnitLogCheckerSupported=false property to the test command for NativeAOT tests. I need to determine if I can just pass any extra global custom properties to the postBuildSteps parameters.

eng/pipelines/coreclr/runtime-nativeaot-outerloop.yml

carlossanlop · 2023-11-17T14:58:57Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2023-11-17T14:59:13Z

Azure Pipelines successfully started running 1 pipeline(s).

carlossanlop · 2023-11-17T17:16:21Z

@MichalStrehovsky @hoyosjs The latest commit seems to have worked: The NativeAOT runs hit unrelated test failures, but they are printing the expected message, are not running XUnitLogChecker, and the final returned exit code is the one from the test run itself:

From the linux-arm64 Release NativeAOT_Libs log file:

   at System.Net.Sockets.SocketAsyncEngine.EventLoop() + 0xc4
./RunTests.sh: line 174:    23 Aborted                 (core dumped) ./System.Net.HttpListener.Tests -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing -xml testResults.xml $RSP_FILE
/root/helix/work/workitem/e
----- end Fri Nov 17 04:30:09 PM UTC 2023 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.
ulimit -c value: unlimited
cat /proc/sys/kernel/core_pattern: /home/helixbot/dotnetbuild/dumps/core.%u.%p
cat /proc/sys/kernel/core_uses_pid: 0
cat: /proc/sys/kernel/coredump_filter: No such file or directory
cat /proc/sys/kernel/coredump_filter:
Looking around for any Linux dumps...
dmesg: read kernel buffer failed: Operation not permitted
The '__IsXUnitLogCheckerSupported' env var is not set.
+ export _commandExitCode=134
+ python /root/helix/work/correlation/reporter/run.py https://dev.azure.com/dnceng-public/ public 10820448 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJMb2NhdGlvblNlcnZpY2UuQ29ubmVjdCBQaXBlbGluZUNhY2hlLlJlYWRXcml0ZVJvb3RBY2Nlc3MgUmVhZEFuZFB1Ymxpc2hUZXN0OmNiYjE4MjYxLWM0OGYtNGFiYi04NjUxLThjZGNiNTQ3NDY0OSBSZWFkQW5kVXBkYXRlQnVpbGRCeVVyaTpjYmIxODI2MS1jNDhmLTRhYmItODY1MS04Y2RjYjU0NzQ2NDkvZG90bmV0L3J1bnRpbWUvMjY1OkJ1aWxkL0J1aWxkLzQ3MzkzMiIsImF1aSI6ImRmZWY2ZTMyLTI3YzItNDEzZi1hNWNhLWIyNDllMTZmMjJhOCIsInNpZCI6ImIxODMwZWQ3LWE2MWQtNGU4Ni1hZGU1LTY1ZjI1ZGM5MDZiYSIsIkJ1aWxkSWQiOiJjYmIxODI2MS1jNDhmLTRhYmItODY1MS04Y2RjYjU0NzQ2NDk7NDczOTMyIiwiam9icmVmIjoiMDQxNTFlYjgtMTY4Ny00ODY0LTlhZTctOWI5ZWQ4MDA0MGZkOjFhOTAzYmVjLTU2YzEtNWY2OS03YjU3LWYwYWI5YzFlMjM4ZSIsInBwaWQiOiJ2c3RmczovLy9CdWlsZC9CdWlsZC80NzM5MzIiLCJvcmNoaWQiOiIwNDE1MWViOC0xNjg3LTQ4NjQtOWFlNy05YjllZDgwMDQwZmQuYnVpbGQuYnVpbGRfbGludXhfYXJtNjRfcmVsZWFzZV9uYXRpdmVhb3RfbGlicy5fX2RlZmF1bHQiLCJyZXBvSWRzIjoiIiwiaXNzIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbSIsImF1ZCI6ImFwcC52c3Rva2VuLnZpc3VhbHN0dWRpby5jb218dnNvOjZmY2M5MmU1LTczYTctNGY4OC04ZDEzLWQ5MDQ1YjQ1ZmIyNyIsIm5iZiI6MTcwMDIzMjU4OCwiZXhwIjoxNzAwMjUxNzg4fQ.OUYvf3FjAJ6KoM4hsVyLeehlu3cqL6OIY2w8hK4k7pdLADD2aRNPDJNFGbGHa9oeup1Zbhmf8fPoKu4Rcvsq1_2JqjaPlvrogj_SfMUxS3agyLVRu8piL6ObEKV2ED5aAlk3kH48nH_UMNBFPm8XXJxxEsm1DjmAkgJOScK1NteCUJSO-0SY_9dLgleD84oi2_YCAQMBkE3jExI59Xf80qKZkmR9HIjWDVHjgXnKWxlCb8iP3fWr5bsxDzPtuyKHW50QDO89hnHUI1rKsbIQcHWNk0zF41LhYIxzi2kPqglxecBTzjNUBEy06vhN1GjcXIZKZOuOQc8cEUQ_Wx8Iww
2023-11-17T16:30:09.997Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2023-11-17T16:30:09.998Z	INFO   	run.py	__init__(42)	read_results	Searching '/root/helix/work/workitem/e' for test results files
2023-11-17T16:30:09.998Z	INFO   	run.py	__init__(42)	read_results	Searching '/root/helix/work/workitem/uploads' for test results files
2023-11-17T16:30:09.998Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2023-11-17T16:30:09.998Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/root/helix/work/workitem/e/__test_report.json'
2023-11-17T16:30:09.998Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1812 bytes
+ python /root/helix/work/correlation/gen-debug-dump-docs.py -buildid 473932 -workitem System.Net.HttpListener.Tests -jobid d4beded8-3f54-46c2-b4f3-f693c4dfa1b9 -outdir /root/helix/work/workitem/uploads -templatedir /root/helix/work/correlation -dumpdir /home/helixbot/dotnetbuild/dumps -productver 9.0.0
gen-debug-dump-docs.py: read file: /root/helix/work/correlation/debug-dump-template.md
gen-debug-dump-docs.py: writing output file: /root/helix/work/workitem/uploads/how-to-debug-dump.md
gen-debug-dump-docs.py: done writing debug dump information
+ exit 134
+ export _commandExitCode=134
+ chmod -R 777 /home/helixbot/dotnetbuild/dumps
+ exit 134

[END EXECUTION]
Exit Code:134

From the linux_musl-x64 Release NativeAOT_Libs log file:

----- end Fri Nov 17 16:32:15 UTC 2023 ----- exit code 139 ----------------------------------------------------------
./RunTests.sh: line 174:    23 Segmentation fault      (core dumped) ./System.Linq.Expressions.Tests -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing -xml testResults.xml $RSP_FILE
/root/helix/work/workitem/e
----- end Fri Nov 17 16:32:15 UTC 2023 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.
ulimit -c value: unlimited
cat /proc/sys/kernel/core_pattern: /home/helixbot/dotnetbuild/dumps/core.%u.%p
cat /proc/sys/kernel/core_uses_pid: 1
cat: /proc/sys/kernel/coredump_filter: No such file or directory
cat /proc/sys/kernel/coredump_filter:
Looking around for any Linux dumps...
Looking for files matching core.* ...
dmesg: klogctl: Operation not permitted
The '__IsXUnitLogCheckerSupported' env var is not set.
+ export '_commandExitCode=139'
+ python /root/helix/work/correlation/reporter/run.py https://dev.azure.com/dnceng-public/ public 10820498 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJMb2NhdGlvblNlcnZpY2UuQ29ubmVjdCBQaXBlbGluZUNhY2hlLlJlYWRXcml0ZVJvb3RBY2Nlc3MgUmVhZEFuZFB1Ymxpc2hUZXN0OmNiYjE4MjYxLWM0OGYtNGFiYi04NjUxLThjZGNiNTQ3NDY0OSBSZWFkQW5kVXBkYXRlQnVpbGRCeVVyaTpjYmIxODI2MS1jNDhmLTRhYmItODY1MS04Y2RjYjU0NzQ2NDkvZG90bmV0L3J1bnRpbWUvMjY1OkJ1aWxkL0J1aWxkLzQ3MzkzMiIsImF1aSI6ImQzNWUyZjJkLTkxMzktNDQ0Yi1hMjI0LTQ2YzJmMjBiMjZjMCIsInNpZCI6IjIwMzg5NWZjLTM0NTMtNGIzZC1iOTE0LWY2ZDRiNzQzYTcyNSIsIkJ1aWxkSWQiOiJjYmIxODI2MS1jNDhmLTRhYmItODY1MS04Y2RjYjU0NzQ2NDk7NDczOTMyIiwiam9icmVmIjoiMDQxNTFlYjgtMTY4Ny00ODY0LTlhZTctOWI5ZWQ4MDA0MGZkOjIwZTZmNjczLTBiMGMtNWM4NS0zNzdjLTVkOTQxZDRjOGVlMyIsInBwaWQiOiJ2c3RmczovLy9CdWlsZC9CdWlsZC80NzM5MzIiLCJvcmNoaWQiOiIwNDE1MWViOC0xNjg3LTQ4NjQtOWFlNy05YjllZDgwMDQwZmQuYnVpbGQuYnVpbGRfbGludXhfbXVzbF94NjRfcmVsZWFzZV9uYXRpdmVhb3RfbGlicy5fX2RlZmF1bHQiLCJyZXBvSWRzIjoiIiwiaXNzIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbSIsImF1ZCI6ImFwcC52c3Rva2VuLnZpc3VhbHN0dWRpby5jb218dnNvOjZmY2M5MmU1LTczYTctNGY4OC04ZDEzLWQ5MDQ1YjQ1ZmIyNyIsIm5iZiI6MTcwMDIzMjYyMiwiZXhwIjoxNzAwMjUxODIyfQ.M1LiFYfZyXiO5O8or7rXoOXcdeIfYe6lYTDg9sE-eFqlBwAg1H0qBkAcQ2_XTNv_G3k-hMYGfFp6cjaJVDUWOsr7nxloZNaU7-0jW-1pPS4cfack8gUoEWXzt--f6Xcuhdgoq_4oaZ3hDQvJnISFb4cr5DpGByjktMw9u3Q96LDpl3uecjPF6wyP_ciG2Vc9hv4TsFLQtyZGSkx-ntvEyQyN_qeXVwwUr3PHrwHjPQSXTmlCXJspRNGQLoeQ7lKQ_sYo-2FB_5nXhO_6GcC3SxTsYoIYZRYOJiClmdItag6MZMpJD56rT7TjkXl1de8VvuftInbiCDVS1OHs5NfuMA
2023-11-17T16:32:16.370Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2023-11-17T16:32:16.371Z	INFO   	run.py	__init__(42)	read_results	Searching '/root/helix/work/workitem/e' for test results files
2023-11-17T16:32:16.371Z	INFO   	run.py	__init__(42)	read_results	Searching '/root/helix/work/workitem/uploads' for test results files
2023-11-17T16:32:16.371Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2023-11-17T16:32:16.372Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/root/helix/work/workitem/e/__test_report.json'
2023-11-17T16:32:16.372Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1816 bytes
+ python /root/helix/work/correlation/gen-debug-dump-docs.py -buildid 473932 -workitem System.Linq.Expressions.Tests -jobid 08d04944-d50b-4b38-b20f-46778845c041 -outdir /root/helix/work/workitem/uploads -templatedir /root/helix/work/correlation -dumpdir /home/helixbot/dotnetbuild/dumps -productver 9.0.0
gen-debug-dump-docs.py: read file: /root/helix/work/correlation/debug-dump-template.md
gen-debug-dump-docs.py: writing output file: /root/helix/work/workitem/uploads/how-to-debug-dump.md
gen-debug-dump-docs.py: done writing debug dump information
+ exit 139
+ export '_commandExitCode=139'
+ chmod -R 777 /home/helixbot/dotnetbuild/dumps
+ exit 139

[END EXECUTION]
Exit Code:139

agocke

This seems likely to break user workflows, to me. Now test scripts have to be passed an additional flag, where they didn't before.

I don't think it's a good idea to introduce a new system to pass extra information about the product to the test. We already do this via customizing the RunScript today. https://github.com/dotnet/runtime/compare/main...agocke:runtime:xunitlogchecker?expand=1 shows an example of how we could encode the environment variable into the script. I think that's simpler and more reliable than the additions in this PR.

MichalStrehovsky · 2023-11-20T06:53:16Z

Is the issue only that we don't have a workable dotnet.exe?

Instead of piping through a flag, could we detect this when we try to run it and emit message to the log that we tried, but we couldn't run XUnitLogChecker?

Libraries CI failures that generate dumps will now show the dump output in the console log itself.

- Avoid using yml extraHelixArguments. - Add the new embedded SetCommands optional section in runner scripts. - Set __IsXUnitLogCheckerSupported in tests.targets as SetScriptCommand items instead of directly in sendtohelixhelp.proj.

carlossanlop · 2023-11-28T03:13:09Z

I added a new commit based on the feedback by @agocke. I rebased and force pushed because there were conflicts.
@agocke can you please let me know if you agree with my latest commit?

agocke

This looks directionally right to me, I think there may just be some parameter confusion around the xunitwrapper binaries.

And no YAML changes! Great!

…hether it was built or not has already been decided before, and the runner scripts know when to execute it.

carlossanlop · 2023-11-28T21:44:09Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2023-11-28T21:44:18Z

Azure Pipelines successfully started running 1 pipeline(s).

carlossanlop · 2023-11-29T01:58:19Z

I manually triggered runtime-nativeaot-outerloop against my latest commit and it surfaced a build failure that consistently showed up in all the jobs. I don't think my change caused it. Unfortunately, the failure is preventing me from validating my changes, since a failure would show up after building. @dotnet/ilc-contrib any ideas on what the root cause could be?

Oh and I opened an issue to see if it could catch other hits besides the ones in my PR: #95367

MichalStrehovsky · 2023-11-29T08:28:56Z

I manually triggered runtime-nativeaot-outerloop against my latest commit and it surfaced a build failure that consistently showed up in all the jobs. I don't think my change caused it. Unfortunately, the failure is preventing me from validating my changes, since a failure would show up after building. @dotnet/ilc-contrib any ideas on what the root cause could be?

Oh and I opened an issue to see if it could catch other hits besides the ones in my PR: #95367

Thanks for reporting it! It's a JIT bug - I think I have a fix in #95383, but it's in the JIT codebase and I'm not a JIT dev so... we'll see.

MichalStrehovsky · 2023-11-29T14:52:15Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2023-11-29T14:52:30Z

Azure Pipelines successfully started running 1 pipeline(s).

carlossanlop · 2023-11-29T17:59:08Z

Thanks @MichalStrehovsky for the quick fix!

The newly triggered run did not show the build error and was able to execute tests. The failures are unrelated to this PR and prove that my change worked and is not blocking nativeaot outerloop anymore:

[HTTP3] [outerloop] Requesting HTTP version 3.0 with version policy RequestVersionExact while unable to establish HTTP/3 connection #95413
Test_EventSource_EtwManifestGeneration* tests failing in CI #48798
[outerloop] [NativeAOT] Segmentation fault in System.Runtime.InteropServices.Tests #95414

carlossanlop added the area-Infrastructure-libraries label Nov 16, 2023

carlossanlop requested review from agocke, ViktorHofer, ericstj, MichalStrehovsky, hoyosjs and ivdiazsa November 16, 2023 19:59

carlossanlop self-assigned this Nov 16, 2023

This comment was marked as resolved.

Sign in to view

hoyosjs approved these changes Nov 16, 2023

View reviewed changes

ViktorHofer reviewed Nov 17, 2023

View reviewed changes

eng/pipelines/coreclr/runtime-nativeaot-outerloop.yml Outdated Show resolved Hide resolved

carlossanlop mentioned this pull request Nov 17, 2023

XUnitLogChecker: Enable for NativeAOT tests #94722

Open

This was referenced Nov 17, 2023

Test failure - System.NullReferenceException in System.Threading.Lock.TryInitializeStatics #94728

Closed

DisabledRuntimeMarshalling.PInvokeAssemblyMarshallingEnabled.DelegatesFromExternalAssembly.StructWithDefaultNonBlittableFields() failing #94931

Closed

agocke requested changes Nov 18, 2023

View reviewed changes

ghost added the needs-author-action An issue or pull request that requires more info or actions from the author. label Nov 18, 2023

carlossanlop added 3 commits November 27, 2023 12:29

Add XUnitLogChecker to log libraries dumps (dotnet#93906)

719415d

Libraries CI failures that generate dumps will now show the dump output in the console log itself.

Change NativeAOT condition

7520d7f

Pass the global property set to false in the post build step.

bd0b435

Address suggestion:

a147fd5

- Avoid using yml extraHelixArguments. - Add the new embedded SetCommands optional section in runner scripts. - Set __IsXUnitLogCheckerSupported in tests.targets as SetScriptCommand items instead of directly in sendtohelixhelp.proj.

carlossanlop force-pushed the LibrariesXUnitLogChecker2 branch from 9eda395 to a147fd5 Compare November 28, 2023 03:11

ghost removed the needs-author-action An issue or pull request that requires more info or actions from the author. label Nov 28, 2023

carlossanlop requested a review from agocke November 28, 2023 03:13

build-analysis bot mentioned this pull request Nov 28, 2023

Failing test -System.IO.Tests.File_GetSetTimes_SafeFileHandle_Pathless.WritingShouldUpdateWriteTime_After_SetLastAccessTime on windows/x86 #95298

Closed

agocke approved these changes Nov 28, 2023

View reviewed changes

Only add XUnitLogChecker as HelixPayload if its directory is found. W…

1d513dd

…hether it was built or not has already been decided before, and the runner scripts know when to execute it.

This was referenced Nov 28, 2023

[outerloop] [NativeAOT] Code generation failed for method System.Linq.Tests.EmptyPartitionTests.SingleInstance() #95367

Closed

Test_EventSource_EtwManifestGeneration* tests failing in CI #48798

Closed

build-analysis bot mentioned this pull request Nov 29, 2023

WritingShouldUpdateWriteTime_After_SetLastAccessTime [FAIL] #95378

Closed

This was referenced Nov 29, 2023

[HTTP3] [outerloop] Requesting HTTP version 3.0 with version policy RequestVersionExact while unable to establish HTTP/3 connection #95413

Closed

[outerloop] [NativeAOT] Segmentation fault in System.Runtime.InteropServices.Tests #95414

Closed

carlossanlop merged commit 35b866c into dotnet:main Nov 29, 2023
199 of 205 checks passed

carlossanlop deleted the LibrariesXUnitLogChecker2 branch November 29, 2023 17:59

build-analysis bot mentioned this pull request Nov 29, 2023

Tracking issue for CI build timeouts #76454

Closed

janvorli mentioned this pull request Dec 7, 2023

Local running of libraries tests is broken #95762

Closed

github-actions bot locked and limited conversation to collaborators Dec 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add XUnitLogChecker to log libraries dumps, do not enable for NativeAOT tests #94868

Add XUnitLogChecker to log libraries dumps, do not enable for NativeAOT tests #94868

carlossanlop commented Nov 16, 2023

ghost commented Nov 16, 2023

carlossanlop commented Nov 16, 2023

azure-pipelines bot commented Nov 16, 2023

carlossanlop commented Nov 16, 2023

This comment was marked as resolved.

carlossanlop commented Nov 16, 2023

azure-pipelines bot commented Nov 16, 2023

hoyosjs left a comment

carlossanlop commented Nov 16, 2023

carlossanlop commented Nov 17, 2023

azure-pipelines bot commented Nov 17, 2023

carlossanlop commented Nov 17, 2023 •

edited

Loading

agocke left a comment

MichalStrehovsky commented Nov 20, 2023

carlossanlop commented Nov 28, 2023

agocke left a comment

carlossanlop commented Nov 28, 2023

azure-pipelines bot commented Nov 28, 2023

carlossanlop commented Nov 29, 2023 •

edited

Loading

MichalStrehovsky commented Nov 29, 2023

MichalStrehovsky commented Nov 29, 2023

azure-pipelines bot commented Nov 29, 2023

carlossanlop commented Nov 29, 2023

Add XUnitLogChecker to log libraries dumps, do not enable for NativeAOT tests #94868

Add XUnitLogChecker to log libraries dumps, do not enable for NativeAOT tests #94868

Conversation

carlossanlop commented Nov 16, 2023

ghost commented Nov 16, 2023

carlossanlop commented Nov 16, 2023

azure-pipelines bot commented Nov 16, 2023

carlossanlop commented Nov 16, 2023

This comment was marked as resolved.

carlossanlop commented Nov 16, 2023

azure-pipelines bot commented Nov 16, 2023

hoyosjs left a comment

Choose a reason for hiding this comment

carlossanlop commented Nov 16, 2023

carlossanlop commented Nov 17, 2023

azure-pipelines bot commented Nov 17, 2023

carlossanlop commented Nov 17, 2023 • edited Loading

agocke left a comment

Choose a reason for hiding this comment

MichalStrehovsky commented Nov 20, 2023

carlossanlop commented Nov 28, 2023

agocke left a comment

Choose a reason for hiding this comment

carlossanlop commented Nov 28, 2023

azure-pipelines bot commented Nov 28, 2023

carlossanlop commented Nov 29, 2023 • edited Loading

MichalStrehovsky commented Nov 29, 2023

MichalStrehovsky commented Nov 29, 2023

azure-pipelines bot commented Nov 29, 2023

carlossanlop commented Nov 29, 2023

carlossanlop commented Nov 17, 2023 •

edited

Loading

carlossanlop commented Nov 29, 2023 •

edited

Loading