Module does not load on PowerShell Core/Linux

[fireflycons]

Note that it does load on PSCore/Windows

Testing using docker image microsoft/powershell image in Docker for Windows with powershell-yaml 0.4.0

PS /usr/local/src> import-module powershell-yaml -force
Add-Type : (7,36): error CS0012: The type 'Object' is defined in an assembly that is not referenced. You must add a reference to assembly 'mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.
public class StringQuotingEmitter: ChainedEventEmitter {
                                   ^
At /root/.local/share/powershell/Modules/powershell-yaml/0.4.0/powershell-yaml.psm1:297 char:5
+     Add-Type -TypeDefinition $stringQuotingEmitterSource -ReferencedA ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidData: ((7,36): error CS001\u2026=b77a5c561934e089'.:CSDiagnostic) [Add-Type], Exception
+ FullyQualifiedErrorId : SOURCE_CODE_ERROR,Microsoft.PowerShell.Commands.AddTypeCommand

..and a couple of hundred more lines, mostly complaining about the same missing reference.

Whether this is an issue with the Add-Type or is an underlying issue in the .net standard build of YamlDotNet, I don't know at this time.

[gabriel-samfira]

Hi,

Will debug this today. Apologies for the late reply.

[gabriel-samfira]

So after a short test, here are my results:

[gabriel@rossak ~ ]$ docker pull microsoft/powershell
Using default tag: latest
latest: Pulling from microsoft/powershell
898c46f3b1a1: Pull complete 
63366dfa0a50: Pull complete 
041d4cd74a92: Pull complete 
6e1bee0f8701: Pull complete 
87f96cbca9ae: Pull complete 
01359b119872: Pull complete 
Digest: sha256:65c203786b382796eeb66e6173964b6e2d830c9b6623823dacd40c7c9eb20622
Status: Downloaded newer image for microsoft/powershell:latest
[gabriel@rossak ~ ]$ docker image list
REPOSITORY                TAG                 IMAGE ID            CREATED             SIZE
microsoft/powershell      latest              6a978be224cc        6 days ago          373MB
[gabriel@rossak ~ ]$ docker run -it 6a978be224cc bash
root@06d8b067f1dc:/# pwsh
PowerShell 6.2.0
Copyright (c) Microsoft Corporation. All rights reserved.

https://aka.ms/pscore6-docs
Type 'help' to get help.

PS /> install-module powershell-yaml -Force -Confirm:$false
PS /> import-module powershell-yaml
PS /> get-module powershell-yaml       

ModuleType Version    Name                                ExportedCommands
---------- -------    ----                                ----------------
Script     0.4.0      powershell-yaml                     {ConvertFrom-Yaml, ConvertTo-Yaml, cfy, cty}

PS /> @{"hello"="world"} | cty
hello: world

Do you have a sample dockerfile with which I could reproduce this?

[gabriel-samfira]

Aditional info:

PS /> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      6.2.0
PSEdition                      Core
GitCommitId                    6.2.0
OS                             Linux 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

PS /> [System.AppDomain]::CurrentDomain.GetAssemblies() | where-object Location -Match YamlDotNet.dll 

GAC    Version        Location
---    -------        --------
False  v4.0.30319     /root/.local/share/powershell/Modules/powershell-yaml/0.4.0/lib/netstandard1.3/YamlDotNet.dll

PS /> [System.AppDomain]::CurrentDomain.GetAssemblies() | where-object Location -Match YamlDotNet.dll | fl *

DefinedTypes        : {YamlDotNet.StandardRegexOptions, YamlDotNet.ReflectionExtensions, YamlDotNet.CultureInfoAdapter, YamlDotNet.PropertyInfoExtensions…}
ExportedTypes       : {YamlDotNet.Serialization.BuilderSkeleton`1[TBuilder], YamlDotNet.Serialization.WrapperFactory`2[TComponentBase,TComponent], 
                      YamlDotNet.Serialization.WrapperFactory`3[TArgument,TComponentBase,TComponent], YamlDotNet.Serialization.Deserializer…}
CodeBase            : file:///root/.local/share/powershell/Modules/powershell-yaml/0.4.0/lib/netstandard1.3/YamlDotNet.dll
EntryPoint          : 
FullName            : YamlDotNet, Version=5.0.0.0, Culture=neutral, PublicKeyToken=null
ImageRuntimeVersion : v4.0.30319
IsDynamic           : False
Location            : /root/.local/share/powershell/Modules/powershell-yaml/0.4.0/lib/netstandard1.3/YamlDotNet.dll
ReflectionOnly      : False
IsFullyTrusted      : True
CustomAttributes    : {[System.Runtime.CompilerServices.ExtensionAttribute()], [System.Runtime.CompilerServices.CompilationRelaxationsAttribute((Int32)8)], 
                      [System.Runtime.CompilerServices.RuntimeCompatibilityAttribute(WrapNonExceptionThrows = True)], 
                      [System.Diagnostics.DebuggableAttribute((System.Diagnostics.DebuggableAttribute+DebuggingModes)2)]…}
EscapedCodeBase     : file:///root/.local/share/powershell/Modules/powershell-yaml/0.4.0/lib/netstandard1.3/YamlDotNet.dll
ManifestModule      : YamlDotNet.dll
Modules             : {YamlDotNet.dll}
GlobalAssemblyCache : False
HostContext         : 0
SecurityRuleSet     : None

[gabriel-samfira]

I ran this on a linux machine. Installing docker for windows as well and giving this another try.

[fireflycons]

Interesting.
I just ran all the above and got the same result.
Let me try to figure out how I broke it.

[gabriel-samfira]

You may be using an older image. On the machine experiencing the issues, try to update the microsoft/powershell image. See if a docker pull helps.

[fireflycons]

The above was indeed with a newer image, i.e. they have updated it in the last 11 days or so. If it is a .net core bug fix, then I hope that's made it as far as the Appveyor images!

I'll re-enable loading of powershell-yaml in my PSCloudFormation repo and see if it builds now (locally first).

[fireflycons]

Gets better!
Are you able to see these build results? https://ci.appveyor.com/project/fireflycons/pscloudformation
I was using powershell-yaml 0.3.5 in windows, and none in Ubuntu. Enabled 0.4.0 in both and boom in both cases!

On my windows workstation, I'm getting the same with 0,4,0, but works in the latest powershell docker image.

[gabriel-samfira]

Interesting. Would you mind adding the following to your build.ps1 right before you import powershell-yaml:

$PSVersionTable

$assemblies = [System.AppDomain]::CurrentDomain.GetAssemblies()
$yamlDotNet = $assemblies | where-object Location -Match YamlDotNet.dll
if ($yamlDotNet) {
    $yamlDotNet | fl *
}
$assemblies

I am curious if there is already a version of YamlDotNet installed inside the image itself. The LoadAssemblies.ps1 script looks for the YamlDotNet assemblies, and if present just returns. So an older version of YamlDotNet may cause a problem.

[fireflycons]

"Curiouser and curiouser" said Alice.

Committed this fireflycons/PSCloudFormation@a2a43ee

Now on the AppVeyor build, Windows is suddenly working, however Linux still fails.

Worth noting that the Appveyor image is running PS 6.1.1 and the latest docker image is on 6.2.0

[fireflycons]

Just committed a change to appveyor.yml to see if I can force apt to upgrade powershell before it's used.

[gabriel-samfira]

You need to add:

echo "msodbcsql msodbcsql/accept_eula select true" | sudo debconf-set-selections

to your build script :). Such is life when we have to deal with non free software.

[gabriel-samfira]

To be honest, if the build server is an actual VM, and not a docker container, you could even do a:

sudo snap install --classic powershell

Then you can invoke the binary from /snap/bin/pwsh. This way you can skip the whole apt-get dist-upgrade and dpkg song and dance.

[fireflycons]

Had to leave that going earlier - kids to feed, put to bed etc!

I believe Appveyor use VMs (cloud instances) rather than containers. If I try the snap route, I may also need to replace the existing pwsh binary with a link to the snap one as appveyor's build service is going to invoke the one it expects to be installed.

[gabriel-samfira]

Completely understand. I am on bottle duty currently.

Does appveyor allow you to specify the interpretor path via a config option in the yaml?

[fireflycons]

Got this far

powershell 6.2.0 from 'microsoft-powershell' installed
/snap/bin/pwsh -Command "$PSVersionTable"
mkdir: cannot create directory ‘/run/user/1001’: Permission denied

Assuming that 1001 is the uid of the appveyor build service, and that I'm going to have to create and chown this directory after the snap install.

Doesn't look like I can configure an alternative interpreter path.

This is beginning to smell like an issue with the underlying .NET core framework that is no longer an issue in PS 6.2.0 which is targeting .netcore 2.1.8

[gabriel-samfira]

hardcore. To be fair, you could probably get away with:

mkdir -p /run/user/1001
chown 1001:1001 /run/user/1001

But at this point, I think it's easier to just run:

echo "msodbcsql msodbcsql/accept_eula select true" | sudo debconf-set-selections

before doing a:

sudo apt-get update
sudo apt-get -q -y dist-upgrade

[gabriel-samfira]

To be extra paranoid, use:

DEBIAN_FRONTEND=noninteractive apt-get -y -qq \
    -o Dpkg::Options::="--force-confdef" \
    -o Dpkg::Options::="--force-confold" dist-upgrade

[fireflycons]

Found something.

FullName            : YamlDotNet, Version=3.6.0.0, Culture=neutral, PublicKeyToken=ec19458f3c15af5e
ImageRuntimeVersion : v2.0.50727
EscapedCodeBase     : file:///home/appveyor/.local/share/powershell/Modules/PSDeploy/1.0.1/Private/PSYaml/Lib/YamlDotNet.dll```

I'm importing that before powershell-yaml
Might need to talk to @RamblingCookieMonster
Changing the import order to see if it makes a difference

[fireflycons]

That's nailed it. And it works on PS 6.1.1. CI now all green!
Much appreciated the time you've spent working though this with me today - two heads are better than one!
Might be worth putting a check in your .PSM1 file for an unsupported YamlDotNet.dll being present in the app domain so that you can throw a meaningful exception :-)
I guess this is always going to be an issue when projects depend on another project's output and not everyone is up to date with their dependencies.

[gabriel-samfira]

Awesome! Great catch! :)

Glad you managed to find this. This is why I asked you print the assemblies before importing powershell-yaml :D. It felt like a YamlDotNet version mismatch.

I need to update LoadAssemblies.ps1 anyway. There is another bug that needs to be fixed in the same script. Will fail with a more meaningful message if the loaded version of YamlDotNet does not match what we expect.

Closing this issue for now. Feel free to open it again if you feel it should stay open.

[gabriel-samfira]

Changed LoadAssemblies.ps1 to validate that, if YamlDotNet is already loaded, it has required types. That way, if the user has an assembly of a different version, but it has what we need, we will just use it, but if not, we error out.

If no assembly is present, we load the bundled one.

[fireflycons]

Nice.
Let me know as and when you push a new version to the gallery.

Cheers