Add secrets search and bump DotUtils.MsBuild.BinlogRedactor.SensitiveDataDetector version #829
Conversation
YuliiaKovalova
changed the title
|
Can this be added to list of sample searches? MSBuildStructuredLog/src/StructuredLogViewer/Controls/BuildControl.xaml.cs Lines 633 to 646 in fa3c09e 
|
| var haystack = file.Value; | ||
| var resultsInFile = haystack.Find(searchText); | ||
| if (resultsInFile.Count > 0) | ||
| if (!string.IsNullOrEmpty(searchText) && searchText.StartsWith("$secret")) |
There was a problem hiding this comment.
this condition will run for every file, and it doesn't change. How about we extract a bool above the foreach and just check it here?
|
Very nice! I left a couple small comments but otherwise it's good to go! |
|
Oh, I forgot. When there are no secrets found, we should return something, otherwise you type $secret and nothing happens. For a regular search we return |
| @@ -1023,24 +1022,19 @@ private object FindInFiles(string searchText, int maxResults, CancellationToken | |||
| { | |||
| var results = new List<(string, IEnumerable<(int, string)>)>(); | |||
|
|
|||
| NodeQueryMatcher notQueryMatcher = new NodeQueryMatcher(searchText); | |||
There was a problem hiding this comment.
did you mean nodeQueryMatcher?
| @@ -1023,24 +1022,19 @@ private object FindInFiles(string searchText, int maxResults, CancellationToken | |||
| { | |||
| var results = new List<(string, IEnumerable<(int, string)>)>(); | |||
|
|
|||
| NodeQueryMatcher notQueryMatcher = new NodeQueryMatcher(searchText); | |||
| bool isSecretsSearch = !string.IsNullOrEmpty(searchText) && searchText.StartsWith("$secret"); | |||
There was a problem hiding this comment.
you can also check for nodeQueryMatcher.TypeKeyword == "secret"
|
Works like a charm!
|
|
@YuliiaKovalova - do you want to promete the functionality on the viewer help? Probably here?: https://github.com/KirillOsenkov/MSBuildLog/blob/6f87e14be0eb1a7cfe7c40e912b96b9ce0cf0301/index.html#L185-L191 |
|
One todo would be to investigate adding parallelism to secret detection because right now it's really slow on real-life binlogs, even moderately sized. Takes over two minutes on a 5 MB binlog I use often. I think a judicious use of Task.Run() will really make things better here. You can maybe spawn one task per string, and one task per file, then just foreach the tasks and do task.Wait() on each of them. No need for async as only one thread will be blocked (the one where the computation happens) |
|
I filed #832 |
Fixes
#823
Context
This change adds a support of a new search key word "$secret" that allows to detect all the suspicious tree entries, based on the capabilities of this library MSBuild.BinlogRedactor.
MSBuild.BinlogRedactor is already used for secrets reduction functionality, but it this context it's possible to check the presence of the secrets on fly
$secret not(SensitiveDataKind) statement is supported here:
It's possible to use the new keyword on 2 tabs:
"Search Log"
"Find in Files"
Changes made
A new SecretsSearch class was implemented based on ISearchExtension interface.
To make this functionality available on "Find in Files" tab, the method was extended
MSBuildStructuredLog/src/StructuredLogViewer/Controls/BuildControl.xaml.cs
Line 1022 in 6e4fc91
msbuild_logWithFalseSecrets.zip