Cpp modular layers

[hgaiser]

Hey,

Something that I found frustrating is that everyone seems to have their own fork of Caffe because of a few layers that needed to be implemented or changed (https://github.com/rbgirshick/py-faster-rcnn for example).

I like the idea of the Python layer in Caffe, where you only define the parameters and the module to use and the Python layer does the rest. I was thinking of the same thing but for c++. The idea I have is analogous to that of the Python layer. You would define a layer in prototxt (CppLayer for instance) and a name of the library to dynamically load. This dynamic library would expose a function which initializes and returns a layer (through a function called createLayer for instance, the name of this function can also be overridden in the prototxt definition if necessary). This custom layer would simply inherit from the Layer class and therefore have all the expected functions (forward / backward / etc.). Parameters can be defined in the prototxt the same way as is done for the Python layer (a yaml string). These libraries can be found by searching a determined environment variable (CAFFE_MODULE_PATH for instance, or perhaps an executable flag, up for discussion?).

In this way, developers and researchers can define their layers in their own packages by including and linking against upstream Caffe, reference these libraries in their prototxt and simply run using 'vanilla' caffe. An additional benefit is that projects like py-faster-rcnn are more future proof because they are dynamically loaded by the latest version of Caffe. Currently py-faster-rcnn cannot be used on the latest versions of CUDA and CuDNN because the Caffe they forked isn't up-to-date (this can be easily fixed by merging with upstream caffe and fixing some issues here and there, but thats besides the point).

I am interested in working out this idea, but I would like to have input from the Caffe developers before I do. Are there certain concerns or considerations I should take into account? What do you think of the idea, would it work? Would Caffe benefit from such functionality?

Caffe claims to aim for modularity, so I say let's add modular layers ;)

ps. I didn't think this'd belong in the caffe-users group, as it is a discussion for a Caffe improvement, not a Caffe useage question.

[willyd]

I believe this idea was already suggested (at least for datalayers) in this PR #3955.

[hgaiser]

Right. It is indeed the same idea, except for data layers. That PR seems to be a bit dead though. I could make a new PR to support compute layers.

Maybe it is even possible to support both data and compute layers with the same plugin loader layer. Although if caffe treats data layers specially that might be slightly more complicated than supporting compute and data layers separately. I'm not sure if this is the case though.

[cypof]

That would be great to have, and data layers should work the same way. You might be able to use the existing layer registry mechanism. I haven't looked in detail but layers already register themselves through a macro, so it might not be very different to have this run when loading a library.

[de-vri-es]

The macros end up using a singleton to register the layer. I think the libraries might use their own private singleton if they're linked without the rest of caffe. So it may be necessary to explicitly pass the registry to a function from the library.

[willyd]

If caffe is built as a shared library the singleton LayerRegistry should be shared among the plugin libraries and the main caffe.[so|dll]. However, it may be required to move the implementation of LayerRegistry to a .cpp file as we did in the windows branch.

[shelhamer]

See #1896 for related discussion. A shared object/dynamic loading layer could make sense as a workaround for further layer modularity.

[hgaiser]

I see @shelhamer, that looks very similar indeed. In my opinion having third party layers through dynamically loaded libraries makes more sense compared to dynamically generating unique IDs for all registered layers.

[de-vri-es]

The dynamically loaded library could register their layer(s) with the global layer registry with a name chosen by the plugin. However, that makes it possible that multiple layers from different plugin writers use the same name. Especially when people tweak layers from others this is actually not that far fetched.

The alternative proposed here is to have one DynamicLibraryLayer (or something along those lines) which loads another layer from a library and delegates all its members to that layer, I believe. This avoids the potential for layers in different libraries trying to claim the same name, so I like this approach more.

[hgaiser]

I agree with @de-vri-es that registering the dynamically loaded libraries seems unwise, specifically for the reason mentioned where multiple libraries can claim the same name.

The solution proposed here is completely analogous to the solution used in the Python layer, but for shared libraries as opposed to python modules.

Advantage of registering the classes would be that the current layers of caffe can be split up into other repositories such that the core of caffe is lighter and possibly fewer dependencies. You would simply choose the modules of layers you need and dynamically load these by name. Registering by name is advantageous here because I think it makes the prototxt more readable (otherwise all layers would basically be of type DynamicLibraryLayer or whatever it'd be called).

My personal preference would be to not register these classes and instead treat them similar to Python layers.

[de-vri-es]

Maybe it would be even nicer not to wrap the created layer inside another layer, but instead write a specific creator function which loads the layer from the library and returns it directly. For that to work the creator function must have access to the layer configuration (to read the library and symbol name to load), but I believe that is the case.

Biggest advantage is that RTTI won't be broken, so dynamic_cast will still work as expected. It could also mean that the actual work required is pretty minimal, since it boils down to making a fancy creator function.

[hgaiser]

Closed due to submission of PR #5294 .