Discussion: Provide an easier way to add MetaDataReferences from Types

[RickStrahl]

Background and Motivation

I hope this is in the right place for a discussion.

I have a small scripting library that is built ontop of Roslyn compiler libraries to dynamically compile C# code into assemblies/types.

One huge sticking point in usage of this tooling is the complexity of adding MetaDataReferences. There are a number of issues but there's at least one scenario that I have no solution for:

• Dynamically compile a C# class and get back an assembly and type
• Type instantiates fine at this point - all good
• Now create a second dynamic C# class and compile
• How do I get the first type's meta data reference into the second compilation?

This came up from a user who was trying to use multiplate compilations and then use the output from prior compilations in another new compilation.

What doesn't work:

• Trying to get a MetaDataReference from an in-memory generated type

What does work (hack):

• Get a MetaDataReference from an on-disk compiled type

The issue here is that currently I use the assembly location to load the MetaDataReference:

var systemReference = MetadataReference.CreateFromFile(type.Assembly.Location);

This works with on-disk assemblies, but doesn't with in-memory assemblies (which actually live on disk?) which don't return a type.Assembly.Location. I could not figure out a way to create a MetaDataReference from the in-memory type (or assembly).

Specific question: Is there a way to get a MetaDataReference off an in-memory compiled type?

To give some context - this is roughly the code the user provided as a scenario. He basically is using multiple compilations of types with type 1 compiled and instantiated and then trying to pass this type 1 to a compilation of type 2.

The code fails because it can't resolve the MetaDataRefernce from the first type (in-memory compiled) as type.Assembly.Location which for the in-memory dll is blank. If I uncomment the line to explicitly write out an assembly to disk (script.OutputAssembly = "...") the code below works.

        [TestMethod]
        public void TwoDynamicClassesTest()
        {
            var class1Code = @"
using System;

namespace Test1 {
    public class Person
    {
        public string Name {get; set; } = ""Rick"";
        public string Description {get; set; } = ""Testing"";
    } 
}
";

            var class2Code = @"
using System;
using Test1;

namespace Test
{

    public class Customer
    {
        public Test1.Person CustomerInfo {get; set; } = new Test1.Person();
        public string CustomerNumber  { get; set; }         
    } 
}
";

            var script = new CSharpScriptExecution();
            script.AddLoadedReferences();
            script.SaveGeneratedCode = true;
            script.GeneratedClassName = "__person";
//            script.OutputAssembly = @"c:\temp\person.dll";    // if I do this it works

            var personType = script.CompileClassToType(class1Code);
            var person = Activator.CreateInstance(personType);


            Assert.IsNotNull(person, "Person should not be null. " + script.ErrorMessage + "\n" + script.GeneratedClassCodeWithLineNumbers);
            Console.WriteLine("Location: " + personType.Assembly.Location);
            
            //script = new CSharpScriptExecution();
            //script.AddDefaultReferencesAndNamespaces(); //AddLoadedReferences();
            //script.AddAssembly(script.OutputAssembly);
            
            script.SaveGeneratedCode = true;
            script.GeneratedClassName = "__customer";
            script.OutputAssembly = null;
            script.AddAssembly(personType);     // MetaDataReference.FromFile() in there
            var customerType = script.CompileClassToType(class2Code);

            Assert.IsNotNull(customerType, "Customer should not be null. " + script.ErrorMessage + "\n" + script.GeneratedClassCodeWithLineNumbers);
            Console.WriteLine(customerType);
            Console.WriteLine(customerType.Assembly.Location);

            dynamic customer = Activator.CreateInstance(customerType);

            
            Assert.IsNotNull(customer.CustomerInfo.Name, "Customer should not be null");
            Console.WriteLine(customer.CustomerInfo.Name);
        }

code link in Westwind.Scripting repo

Proposal

Overall creating and providing of MetaDataReferences is a pain in the butt currently. If you want to avoid loading the reference libraries runtime references of the .NET core libraries there's almost no rhyme or reason to what's included and what's not. The problem with the reference. Also these are not officially provided by Microsoft which seems odd (Jared Parsons maintains those).

It would be really helpful at minimum if MetaDataReference had some additional methods for loading a reference directly from a type so it can work of dynamically created types.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[tmat]

You should be able to use TryGetRawMetadata API and MetadataReference.CreateFromImage as a workaround for now.

[RickStrahl]

Thanks for that - that looks promising but... I had no luck getting this to work.

Basically what I'm now doing is this:

/// <summary>
/// Adds an assembly reference from an existing type
/// </summary>
/// <param name="type">any .NET type that can be referenced in the current application</param>
public bool AddAssembly(Type type)
{
    try
    {
        // *** TODO: need a better way to identify for in memory dlls that don't have location
        if (References.Any(r => r.FilePath == type.Assembly.Location))
            return true;

        if (string.IsNullOrEmpty(type.Assembly.Location))
        {
#if NETCORE
            unsafe
            {
                bool result = type.Assembly.TryGetRawMetadata(out byte* metaData, out int size);
                var bytes = new byte[size];
                for (int i = 0; i < size; i++)
                {
                    bytes[i] = metaData[i + 1];
                }
                var ref1 = MetadataReference.CreateFromImage(bytes);
                References.Add(ref1);
            }
#else
            return false;
#endif
        }
        else
        {
            var systemReference = MetadataReference.CreateFromFile(type.Assembly.Location);
            References.Add(systemReference);
        }
        
    }
    catch
    {
        return false;
    }

    return true;
}

References is a HashTable<PortableExecutableReference> I'm adding refs to, then pull them out during compilation.

When this code triggers with an in-memory assembly the unsafe code seems to work to retrieve a metadata referncee (I get true and buffer and I can add it to my list), but the compiler is not happy with this reference and gives me:

error CS0009: Metadata file '<in-memory assembly>' could not be opened -- PE image doesn't contain managed metadata.

I suppose it's possible I need to add this when I compile the assembly but not sure what triggers that.

[tmat]

I see, we don't have a readily available factory that takes only the metadata blob -- CreateFromImage takes PE image (the entire dll/exe file), but TryGetRawMetadata only returns the metadata section of that file.

You'll need to construct AssemblyMetadata object and then call GetReference on it. First, use ModulMetadata.CreateFromMetadata to create ModuleMetadata and pass it to AssemblyMetadata.Create.

[RickStrahl]

Thank you... this gets me a little closer now:

 if (string.IsNullOrEmpty(type.Assembly.Location))
        {
#if NETCORE
            unsafe
            {
                bool result = type.Assembly.TryGetRawMetadata(out byte* metaData, out int size);
                var moduleMetaData = ModuleMetadata.CreateFromMetadata( (nint) metaData, size);
                var assemblyMetaData = AssemblyMetadata.Create(moduleMetaData);
                References.Add(assemblyMetaData.GetReference());
            }
#else
            return false;
#endif
        }
        else
        {
            var systemReference = MetadataReference.CreateFromFile(type.Assembly.Location);
            References.Add(systemReference);
        }

Roslyn can now resolve the reference and compile the code which I guess solves this particular problem...

It would be nice if there was a more obvious way to do this - like an overload that can use an active type to create the metadata reference (presumably using similar code).

Thank you...

The next problem though is that the runtime activation can't find the assembly when instantiating the top level type with Activator.CreateInstance():

Could not load file or assembly '__person.dll, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.

A bit odd because the type is already loaded since it was used to access the meta data reference and in the sample code even has an active instance.

But... that's a separate issue somewhat unrelated to Roslyn and MetaDataReferences.