[Question] [Extensions] Use existing (non-annotated) classes as extension

[EotT123]

Like Manifold, Lombok also supports extensions (in a different way). I'm in favor of the Manifold-way, but I've found a case that I think isn't possible with Manifold (or is it?).

With Lombok, an ExtensionMethod annotation is added for each class where you would use the extension methods. It is possible to add this: @ExtensionMethod({ StringUtils.class }). This will make all methods from the (in my case) apache commons StringUtils class available as extension methods, even though all methods in the StringUtils class are just normal methods, without any annotations.
I'm not sure if something like this is possible with Manifold. I would like to use those methods as extension methods, without having to create an extension class myself, which delegates all methods to the StringUtils class. (However, I'm not sure what would happen with name clashes).

Another useful class which could be used as an extension is java.nio.file.Files.

It would be nice if we could configure somewhere that we want to use a class as an extension.
In the Files class, not all methods are applicable to Paths (some have an equivalent with a String parameter) , so maybe only the methods for which the first parameter is the same as a specified type could be taken into account.

This might be a long shot and not be possible at all.

[rsmckinney]

This is possible with manifold, however it is not so readily possible.

You can generate extension classes based on other resources and classes. See Generating extension classes.

Sometimes the contents of an extension class reflect metadata from other resources. In this case rather than painstakingly writing such classes by hand it's easier and less error-prone to produce them via type manifold. To facilitate this use-case, your type manifold must implement the IExtensionClassProducer interface so that the ExtensionManifold can discover information about the classes your type manifold produces. For the typical use case your type manifold should extend AbstractExtensionProducer.

See the manifold-ext-producer-sample module for a sample type manifold implementing IExtensionClassProvider.

This is basically saying you can write a type manifold that generates extension classes for just about any scenario, including the one you're proposing.

But I'm thinking this should be a more directly supported feature. Something like this:

package myproject.extensions.java.nio.file.Paths;

@Extension(source=java.nio.file.Files.class)
public class MyPathsExt {
}

Here we support the feature with an optional source parameter indicating a class having static methods to transform into extension methods on the extended class.

Nice example too, the Paths, Files, etc. nio stuff is confusing.

Great suggestion!

[EotT123]

This seems a bit more complex than the proposed solution. However, I'm going to try to make this work. Nice to see that so much is possible! Thanks for pointing out how to do it.

[rsmckinney]

This seems a bit more complex than the proposed solution.

It reflects the difference in extension architecture between manifold and Lombok. With manifold extensions apply to modules and “just work”, with Lombok extensions must be manually included in each file that uses them. There are pros and cons to both techniques. In my experience I almost always want them to “just work” particularly in the IDE. Having to include them manually is a PITA when nearly 100% of the time the extension is intended to be accessible across the entire module. Again, this relates to my own experience and preferences.

Of course, these techniques are not mutually exclusive, so I understand your proposition is meant to combine them. My proposal is just meant to augment the existing architecture with the means to also support static libraries as extension classes. After that is finished a discussion can be had about scoping extensions at the file level similar to Lombok, C#, etc.

[rsmckinney]

However, I'm going to try to make this work.

I won't stop you ;)

But as you probably know I am rather particular about pull requests, they are heavily scrutinized and unless they fit like a glove I tend not to accept them as PRs. Not because they aren't well crafted, but primarily because I didn't write them. While that may sound off putting, it comes down to the time required to thoroughly digest the code, the collaboration, the testing, the perf, the IDE plugin impact, the future maintenance burden, etc. By the time I've finished all of that, I may as well have written it myself, probably in less time.

Normally, I wouldn't take such a guarded stance with a project. But most projects are far easier to grok and maintain. Manifold is insane on many levels, but it has to be.

Anyhow, I appreciate your attention and contributions to this project. Your feedback is priceless! I don't want to discourage you, I want you to keep it coming, including going forward with this proposal. I'm just setting expectations. Hopefully, you're still on board :)

[EotT123]

No problem at all. I didn't intend to create a PR just yet, but I enjoy diving deeper into the code. I'm just scratching the surface and currently have only a very basic overview of it.
You can count on me to continue reporting any bugs I encounter (hopefully, this won't happen too often ;-)).

[EotT123]

I'm almost there I think.

I can annotate a class to generate the needed methods:

When processing the annotation, a new java source file is generated in target/generated-sources/annotations:

Which gets then compiled:

So far, so good. I can use the generated methods by calling the static methods directly (e.g. PathExtension1719255870629.copy(...). However, I can't get it to work as an extension method.

Adding a new method to the PathExtension.java extension class works fine:

So I'm not sure why methods in PathExtension.java are working, but the generated ones in PathExtension1719255870629 aren't. I'm using a separate module to generated the files, and (try to) use them in another module.

The relevant parts of my pom.xml:

<dependencies>
    <!-- dependency containing the annotation processor & new annotation -->
    <dependency>
        <groupId>test</groupId>
        <artifactId>test</artifactId>
        <version>1.0-SNAPSHOT</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>systems.manifold</groupId>
        <artifactId>manifold-ext-rt</artifactId>
        <version>${manifold.version}</version>
    </dependency>
</dependencies>
<build>
    <plugins>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>build-helper-maven-plugin</artifactId>
            <version>3.6.0</version>
            <executions>
                <execution>
                    <id>add-source</id>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>add-source</goal>
                    </goals>
                    <configuration>
                        <sources>
                            <source>target/generated-sources/annotations/</source>
                        </sources>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <fork>true</fork>
                <compilerArgs>
                    <arg>-Xplugin:Manifold</arg>
                    <arg>-verbose</arg>
                </compilerArgs>
                <annotationProcessorPaths>
                    <path>
                        <groupId>systems.manifold</groupId>
                        <artifactId>manifold-ext</artifactId>
                        <version>${manifold.version}</version>
                    </path>
                    <!-- dependency containing the annotation processor -->
                    <path>
                        <groupId>test</groupId>
                        <artifactId>test</artifactId>
                        <version>1.0-SNAPSHOT</version>
                    </path>
                </annotationProcessorPaths>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <configuration>
                <archive>
                    <manifestEntries>
                        <!--class files as source must be available for extension method classes-->
                        <Contains-Sources>java,class</Contains-Sources>
                    </manifestEntries>
                </archive>
            </configuration>
        </plugin>
    </plugins>
</build>

Any ideas? If needed, I can provided the source code, but it's still very ugly at the moment (I have a lot of cleaning up to do.)

[EotT123]

@rsmckinney After further testing, I can conclude that it is working. Compilation is successful. However, IntelliJ cannot resolve the methods, nor is there any content assist. My guess is that the source files in the target folder are not being picked up by the Manifold-IJ plugin, even though the target folder containing the generated source file is marked as a source folder.

[rsmckinney]

Hey @EotT123 sorry for not getting back sooner, been busy with other stuff, coming up for air now...

We can talk about why the files aren't picked up, however my preferred way of achieving this feature would not involve creating a conventional annotation processor and generating intermediate source files. Instead, why not just process @ExtensionClass directly along with @Extension and generate methods the same way? I don't see the need to create intermediate files for this. Just my two cents.

[EotT123]

Thanks for the answer! I've changed it to what you suggested (also using the AbstractExtensionProducer way, which I wasn't using in the previous post, as I couldn't get it working).
However, it still not working.

I have a java source file containing an extension and the extension class:

This is compiled to:

No intermediate jar files, nor other class files are created.

When looking at the javac output, I see the following:

• Generated java file for all methods in Files.java
  
• Also the combined interface file containing the extension method I defined in the above class, together with all methods from Files.java.
  

However, only the test method is usable as an extension method (using one of the other methods doesn't compile).

[rsmckinney]

I'd need to examine your code to figure out what's happening/not happening.

Perhaps a simpler solution is to just follow extension producer sample more closely. Just create a resource file type, say with a .extmap extension, that maps the classes to be extended to the classes having static methods:

../resources/org/example/MyExtensionMap.extmap

java.nio.file.Path: java.nio.file.Files
java.lang.String: your.favorite.StringUtil
java.lang.String: some.other.StringUtil
. . .

Pretty straightforward if you follow along with that sample project and its test module. I think I like this approach most.

[EotT123]

I managed to make it work, there was a very small bug in my code, and it took me a long time to figure it out (many hours debugging the code, going through all the Manifold code that handles most of the processing).
I'm very happy it's functioning correctly now. I can add the annotation, and both compilation and content assist work perfectly. Thanks for the help!

@ExtensionClass(Files.class) // creates extension methods of all methods defined in Files.class
@Extension
public class PathExtension {

    // other extension methods
}

I created a separate annotation because I wanted to use it as an add-on to the Manifold framework without modifying the Manifold code.

[rsmckinney]

Nice!

[EotT123]

Created a PR which implements this functionality: #653

[EotT123]

Created a new PR with new implementation: #662