[JavaCallableWrappers] Parallel.ForEach per assembly #365
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Build logs of these changes: GenerateJavaStubs.zip |
jonpryor
reviewed
Sep 4, 2018
| var javaTypes = new BlockingCollection<TypeDefinition> (); | ||
|
|
||
| Parallel.ForEach (assemblies, assembly => { | ||
| var assm = resolver.GetAssembly (assembly); |
There was a problem hiding this comment.
Nope. Nopity nopity not going to do it NOPE.
Cecil isn't thread safe. This cannot work. See also: b1667a2.
There was a problem hiding this comment.
Hmm b1667a2, looks like it is parallelizing on the TypeDefinition and this is doing a Task per AssemblyDefinition.
Let me try writing a test like you have here, and see what happens.
Since we are looking at every type in each assembly, we can parallelize this work to make things a bit faster. However, there are a few concerns when doing this: - Made sure to add a *new* method `GetJavaTypesInParallel` so the caller can opt-in to this - The `cache` in `DirectoryAssemblyResolver` needs to now be a `ConcurrentDictionary` - The list of `javaTypes` returned by `JavaTypeScanner` needs to also use a `BlockingCollection` - Downstream in `xamarin-android`, we need to make sure `GenerateJavaStubs` is now an `AsyncTask`, so the "thread safe" logger methods are used. To time the improvements, I used this project in Xamarin.Forms: https://github.com/xamarin/Xamarin.Forms/tree/master/Xamarin.Forms.ControlGallery.Android Before I did anything, the times were: 2594 ms GenerateJavaStubs 2 calls After making things parallel: 2143 ms GenerateJavaStubs 2 calls And then after a few more LINQ improvements in the `GenerateJavaStubs` MSBuild task: 2104 ms GenerateJavaStubs 2 calls So we have close to a 500ms improvement for `GenerateJavaStubs` in this project, that will build a library project + application project. To see the full changes required downstream in `xamarin-android`: dotnet/android@master...jonathanpeppers:generatejavastubs-perf
Test fails with:
Expected: 6959
But was: 6958
So we should abandon this... 😢
jonathanpeppers
force-pushed
the
javatypescanner-parallel
branch
from
b012118 to
bf0b5a0
Compare
|
See bf0b5a0, an adventure in showing the issue with this idea... |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since we are looking at every type in each assembly, we can
parallelize this work to make things a bit faster.
However, there are a few concerns when doing this:
GetJavaTypesInParallelso thecaller can opt-in to this
cacheinDirectoryAssemblyResolverneeds to now be aConcurrentDictionaryjavaTypesreturned byJavaTypeScannerneeds to alsouse a
BlockingCollectionxamarin-android, we need to make sureGenerateJavaStubsis now anAsyncTask, so the "thread safe"logger methods are used.
To time the improvements, I used this project in Xamarin.Forms:
https://github.com/xamarin/Xamarin.Forms/tree/master/Xamarin.Forms.ControlGallery.Android
Before I did anything, the times were:
After making things parallel:
And then after a few more LINQ improvements in the
GenerateJavaStubsMSBuild task:
So we have close to a 500ms improvement for
GenerateJavaStubsinthis project, that will build a library project + application project.
To see the full changes required downstream in
xamarin-android:dotnet/android@master...jonathanpeppers:generatejavastubs-perf