New approach: link section

[rambip]

This PR contains all my work to implement the linker-based approach for next major version of manganis.

A lot of ideas are taken from linkme

General idea

In rust, you can add an attribute to a static to make rust store it in a specific section of the object file.

#[used]
#[link_section = "foo_section"]
static MY_DATA : [u8; 2] = [42, 1];

Then if you look at the object files generated by rustc, you will see that the "foo_section" indeed contains some data.

The idea is then to generate these special static variables containing the description of the assets, ensure rust will store them for each dependency, and ensure the linker will merge all these sections in the final executable.

Then, the job of manganis-cli-support will be to read from the executable, retrieve the asset descriptions and use that to add the assets

Macro

The MANIFEST = mg!(file("Cargo.toml")) macro invocation will generate something like

static MANIFEST: &str = {
    // used for lsp completion, I didn't change it
    const _: &dyn manganis::ForMgMacro = {
        use manganis::*;
        &file("Cargo.toml")
    };
    // where the magic happens
    #[link_section = "manganis"]
    #[used]
    static ASSET: [u8; 193usize] = *
        // json description of the asset
        b"{\"File\":{\"location\":{\"unique_name\":\"Cargotoml47085c5f397cc4d4.toml\",\"source\":{\"Local\":\"/home/rambip/proj/dioxus-test/Cargo.toml\"}},\"options\":{\"Other\":{\"extension\":\"toml\"}},\"url_encoded\":false}}";
    
    // and return the hashed path
    "/assetsCargotoml47085c5f397cc4d4.toml"
};

The macro still checks the existence of the path, but it doesn't do anything more complicated. In particular it don't touch the filesystem except for logs

Cli support

The new methods now are:

/// An extension trait CLI support for the asset manifest
pub trait AssetManifestExt {
    /// Load a manifest from the assets in the executable at the given path
    /// The asset descriptions are stored inside the binary, in the link-section
    fn load(executable: &Path) -> Self;
    /// Optimize and copy all assets in the manifest to a folder
    fn copy_static_assets_to(&self, location: impl Into<PathBuf>) -> anyhow::Result<()>;
    /// Collect all tailwind classes and generate string with the output css
    fn collect_tailwind_css(
        &self,
        include_preflight: bool,
        warnings: &mut Vec<TailwindWarning>
    ) -> String;
}

The AssetManifest type also changed, it is now a simple vec of AssetType.

The load function is much simpler than before:

...
        let manganis_data = // get raw string from the section

        let deserializer = serde_json::Deserializer::from_str(&manganis_data);
        let assets = deserializer
            .into_iter::<AssetType>()
            .map(|x| x.unwrap())
            .collect();

        Self::new(assets)
}

Init

To force the linker to include all the sections in the final executable, the most robust and straight-forward way seeems to be:

// in the file where the assets are used

extern "Rust" {
    #[link_name = "__start_manganis"]
    static MANGANIS_START: u8;
}

fn main(){
    unsafe {
        assert!(MANGANIS_START != 0)
    }
}

It makes the linker see that it needs "__start_manganis", so it will keep the manganis section of the binary and of every nested dependency

This function is defined at the end of src/lib.rs in the main manganis crate

Linker

The new file common/src/linker.rs defines the section names and link names for different platforms, like linkme does.

Caveats

There are some things to improve:

• test this for every platform
• improve the library usage documentation
• remove the data from the binary once the assests are copied to reduce privacy risks.
• explain in the documentation that you need manganis::init

[Andrew15-5]

Then if you look at the object files generated by rustc, you will see that the "foo_section" indeed contains some data.

The idea is then to generate these special static variables containing the description of the assets, ensure rust will store them for each dependency, and ensure the linker will merge all these sections in the final executable.

This opens a possibility of making a truly single-file server binary that not only doesn't depend on shared libraries (but I think this is not currently possible for glibc-based OSes), but also contains all of the assets within. Despite a very nice feature of not having any problems with relative asset paths, this will heavily increase the binary size for medium/big projects. But for blog post-like websites this is probably still an interesting idea.

// in the file where the assets are used

extern "Rust" {
    #[link_name = "__start_manganis"]
    static MANGANIS_START: u8;
}

fn main(){
    unsafe {
        assert!(MANGANIS_START != 0)
    }
}

Is this what will be in the last version of this PR? Am I the only one who thinks that sprinkling unsafe here and there is not the way to go?