Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dotnet] Figure out where to publish files in the app bundle. Fixes #12572. #13591

Merged
merged 61 commits into from
Jan 17, 2022

Conversation

rolfbjarne
Copy link
Member

@rolfbjarne rolfbjarne commented Dec 16, 2021

In .NET, all files that should be published (put into the final .app bundle) are put into the @(ResolvedFileToPublish) item group, and at the end of the build process, .NET will publish all the files in that item group. Which files are in this item group, and how they're put in there, is out of our control (it's just how the build process works in .NET), so we'll have to cope.

Additionally, publishing an app for Apple platforms is different than publishing other .NET apps, because we can't just put all the files in the a directory like .NET usually does, we have a fairly strict directory structure we need to follow, and it also differs between platforms (the structure is different between macOS and iOS for instance).

This means that for every file in the ResolvedFileToPublish item group, we have to figure out:

  • Should it be put into the app bundle in the first place?
  • If so, in which subdirectory (if any)?

This PR implements these changes. The exact details are explained in a document in the PR, but the general logic is:

  • We make an educated guess for some types of files we know about (assemblies, native libraries, images, etc).
  • We provide a way to set metadata on each item specifying which type of item it is (assembly, native library, image, etc), and we'll treat the item as such. This method can also be used to override the guess we made (for files that shouldn't be published for instance).
  • We warn if we run into files we're not educated enough to be able to guess about, and for which there's no custom metadata set.

Fixes #12572.

This PR is best reviewed commit-by-commit.

@rolfbjarne rolfbjarne added not-notes-worthy Ignore for release notes run-dotnet-tests Run all the .NET tests labels Dec 16, 2021
@rolfbjarne rolfbjarne changed the title [DRAFT] [dotnet] Figure out where to publish files in the app bundle. [DRAFT] [dotnet] Figure out where to publish files in the app bundle. Fixes #12572. Dec 16, 2021
@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

Use the right template variable in the template expansion to avoid duplicating variable names.
…rstand *.xcframework paths.

Previously we'd only support passing in a fake reference to a file inside the *.xcframework.

This makes some other code simpler later on.

Best viewed by ignoring whitespace, since this is mostly whitespace changes.
…function.

Best viewed by ignoring whitespace, since this is mostly whitespace changes.
So that the tables can be used in more places.
…esToBundle

We're soon going to use this task to copy other types of directories (such as plugins)
as well, and in that case the old target name would be misleading.
@rolfbjarne rolfbjarne self-assigned this Dec 22, 2021
Copy link
Contributor

@chamons chamons left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Epic PR. ❤️ the significant test coverage.

@@ -55,6 +56,20 @@ public void BuildNativeReferenceFlags (TaskLoggingHelper Log, ITaskItem[] Native
if (Path.GetExtension (path) != ".framework")
path = Path.GetDirectoryName (path);
Frameworks.Add (path);
} else if (kind == NativeReferenceKind.Dynamic) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be behind a platform being targeted check, so people don't use it on iOS and then later fail/complain on submission, or is letting them bounce on submission fine?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that letting them bounce on submission is fine (after all dylibs work fine during development, when it can be useful).

$(Q) $(MAKE) build BUILD_ARGUMENTS="/p:RuntimeIdentifier=ios-arm64 /p:MtouchUseLlvm=true /p:MtouchLink=SdkOnly /p:Configuration=Release-llvm /bl:@.binlog"

norm:
$(Q) $(MAKE) build BUILD_ARGUMENTS="/p:RuntimeIdentifier=ios-arm64 /p:MtouchLink=SdkOnly /p:Configuration=Release-norm /bl:@.binlog"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super nit - any reason for the whitespace here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to line arguments up in the text editor (makes it easier to see the difference between them), but in any case these targets are not needed, so I'll just remove them.

Predicate<string?> predicate = (v) => {
var fn = Path.GetFileName (v!);

switch (fn) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If/when the runtime adds more files to the bundle this test pat looks to break. Maybe add a comment here explaining to future us when to add more files to this list?

AddExpectedPlugInFiles (platform, expectedFiles, "PlugInA", isSigned); // PlugIns
AddExpectedPlugInFiles (platform, expectedFiles, "CompressedPlugInB", isSigned); // CompressedPlugIns
}
// UnknownI.bin: Unknown -- this should show a warning
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another weird spacing issue?

Copy link
Contributor

@tj-devel709 tj-devel709 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alt Text

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2
Copy link
Collaborator

❌ [PR Build] Tests failed on Build ❌

Tests failed on Build.

API diff

✅ API Diff from stable

View API diff
View dotnet API diff
View dotnet legacy API diff
View dotnet iOS-MacCatalayst API diff

API Current PR diff

ℹ️ API Diff (from PR only) (please review changes)

View API diff
View dotnet API diff
View dotnet legacy API diff
View dotnet iOS-MacCatalayst API diff

Generator diff

Generator Diff (no change)

GitHub pages

Results can be found in the following github pages (it might take some time to publish):

Test results

1 tests failed, 216 tests passed.

Failed tests

  • monotouch-test/Mac Catalyst [dotnet]/Debug [dotnet]: Failed (Test run failed.
    Tests run: 2706 Passed: 2510 Inconclusive: 11 Failed: 1 Ignored: 195)

Pipeline on Agent XAMBOT-1101.BigSur'
Merge 05ca39b into 5a45a3c

Copy link
Member

@mandel-macaque mandel-macaque left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor comments that could make the code simpler to understand.

Comment on lines +134 to +135
var items = entry.Value;
var item = new TaskItem (entry.Key);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can expand tuples inside the foreach.

Comment on lines 251 to 261
if (filename.EndsWith (".dll", StringComparison.OrdinalIgnoreCase)) {
return PublishFolderType.Assembly;
} else if (filename.EndsWith (".exe", StringComparison.OrdinalIgnoreCase)) {
return PublishFolderType.Assembly;
} else if (filename.EndsWith (".pdb", StringComparison.OrdinalIgnoreCase)) {
return PublishFolderType.Assembly;
} else if (filename.EndsWith (".dll.mdb", StringComparison.OrdinalIgnoreCase) || filename.EndsWith (".exe.mdb", StringComparison.OrdinalIgnoreCase)) {
return PublishFolderType.Assembly;
} else if (filename.EndsWith (".config", StringComparison.OrdinalIgnoreCase)) {
return PublishFolderType.Assembly;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The all return 'PublishFolderType.Assembly', could we have done this with a loop through he known extensions:

Suggested change
if (filename.EndsWith (".dll", StringComparison.OrdinalIgnoreCase)) {
return PublishFolderType.Assembly;
} else if (filename.EndsWith (".exe", StringComparison.OrdinalIgnoreCase)) {
return PublishFolderType.Assembly;
} else if (filename.EndsWith (".pdb", StringComparison.OrdinalIgnoreCase)) {
return PublishFolderType.Assembly;
} else if (filename.EndsWith (".dll.mdb", StringComparison.OrdinalIgnoreCase) || filename.EndsWith (".exe.mdb", StringComparison.OrdinalIgnoreCase)) {
return PublishFolderType.Assembly;
} else if (filename.EndsWith (".config", StringComparison.OrdinalIgnoreCase)) {
return PublishFolderType.Assembly;
}
foreach(var extension in new string [] {".dll", ".exe",".pdb", "dll.mdb", ".exe.mdb", ".config"}) {
if (filename.EndsWith (extension, StringComparison.OrdinalIgnoreCase))
return PublishFolderType.Assembly;
}

Or you can create a helper function that returns if it is an assembly checking the extension and do a simple if.

Comment on lines 272 to 277
// resources (png, jpg, ...?)
if (filename.EndsWith (".jpg", StringComparison.OrdinalIgnoreCase)) {
return PublishFolderType.Resource;
} else if (filename.EndsWith (".png", StringComparison.OrdinalIgnoreCase)) {
return PublishFolderType.Resource;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not using a single if and a ||

Comment on lines 280 to 284
if (filename.EndsWith (".framework.zip", StringComparison.OrdinalIgnoreCase)) {
return PublishFolderType.CompressedAppleFramework;
} else if (filename.EndsWith (".xcframework.zip", StringComparison.OrdinalIgnoreCase)) {
return PublishFolderType.CompressedAppleFramework;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same

Comment on lines +287 to +291
if (filename.EndsWith (".a", StringComparison.OrdinalIgnoreCase)) {
return PublishFolderType.StaticLibrary;
} else if (filename.EndsWith (".dylib", StringComparison.OrdinalIgnoreCase)) {
return PublishFolderType.DynamicLibrary;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This returns different values, so a single if and || won't work.

Log.LogWarning (MSBStrings.E7089 /* The file '{0}' does not specify a 'PublishFolderType' metadata, and a default value could not be calculated. The file will not be copied to the app bundle. */, item.ItemSpec);

return PublishFolderType.None;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I winder if we could simplify all this with a switch and a return using patterns matching with the strings, should make the method smaller and easier to understand.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic is a bit more complex in a few cases than a simple string matching a pattern.

@vs-mobiletools-engineering-service2
Copy link
Collaborator

❌ [PR Build] Tests failed on Build ❌

Tests failed on Build.

API diff

✅ API Diff from stable

View API diff
View dotnet API diff
View dotnet legacy API diff
View dotnet iOS-MacCatalayst API diff

API Current PR diff

ℹ️ API Diff (from PR only) (please review changes)

View API diff
View dotnet API diff
View dotnet legacy API diff
View dotnet iOS-MacCatalayst API diff

Generator diff

Generator Diff (no change)

GitHub pages

Results can be found in the following github pages (it might take some time to publish):

Test results

2 tests failed, 215 tests passed.

Failed tests

  • monotouch-test/Mac [dotnet]/Debug [dotnet]: TimedOut (Execution timed out after 1200 seconds.
    Test run crashed)
  • monotouch-test/Mac [dotnet]/Debug (static registrar) [dotnet]: TimedOut (Execution timed out after 1200 seconds.
    Test run crashed)

Pipeline on Agent XAMBOT-1101.BigSur
Merge 0bbe442 into 3a1b06e

The symlink might point to a file that has been updated, and if that file
needs more processing and the logic checks the symlink to check if the file is
up-to-date, then we need to reflect that in the symlink (for instance if a
framework binary is updated (which is a symlink on some platforms), it needs
to be resigned, even if the symlink itself didn't change).
…an app bundle.

This fixes the following problem:

* App with framework is built and signed.
* App is rebuilt, and the framework is copied in again.
* This time, the framework's executable's timestamp will be earlier than the
  timestamp when it was last signed, and as such it won't be signed again.

Fix this by touching all the copied files when copying a directory to the app bundle.
…de the zip files in two different tasks.

Split decompressing zip files and figuring out what was inside the zip files
in two different tasks, so that we do the second part even if the first part
isn't done (it could have been done in a previous build).

This is required for rebuilds to work correctly.
@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2
Copy link
Collaborator

❌ [PR Build] Tests failed on Build ❌

Tests failed on Build.

API diff

✅ API Diff from stable

View API diff
View dotnet API diff
View dotnet legacy API diff
View dotnet iOS-MacCatalayst API diff

API Current PR diff

ℹ️ API Diff (from PR only) (please review changes)

View API diff
View dotnet API diff
View dotnet legacy API diff
View dotnet iOS-MacCatalayst API diff

Generator diff

Generator Diff (no change)

GitHub pages

Results can be found in the following github pages (it might take some time to publish):

Test results

2 tests failed, 215 tests passed.

Failed tests

  • monotouch-test/Mac [dotnet]/Debug [dotnet]: Failed (Test run failed.
    Tests run: 2622 Passed: 2462 Inconclusive: 6 Failed: 1 Ignored: 159)
  • monotouch-test/Mac [dotnet]/Debug (static registrar) [dotnet]: Failed (Test run failed.
    Tests run: 2619 Passed: 2460 Inconclusive: 6 Failed: 1 Ignored: 158)

Pipeline on Agent XAMBOT-1099.BigSur'
Merge 6564841 into 9894605

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
not-notes-worthy Ignore for release notes run-dotnet-tests Run all the .NET tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

.NET: What to do about files in ResolvedFileToPublish.
7 participants