[question] Declaring dependencies for consumed packages

[MartyMcFlyInTheSky]

What is your question?

In our company we're working with lapack and our project A defines a dependency on it:

    def requirements(self):
        self.requires("lapack/3.8.0@<mycompany>/production")

But whenever I configure project A I get this:

CMake Error at build/Release/generators/cmakedeps_macros.cmake:39 (message):
  Library 'gfortran' not found in package.  If 'gfortran' is a system
  library, declare it with 'cpp_info.system_libs' property
Call Stack (most recent call first):
  build/Release/generators/LAPACK-Target-release.cmake:24 (conan_package_library_targets)
  build/Release/generators/LAPACKTargets.cmake:26 (include)
  build/Release/generators/LAPACKConfig.cmake:16 (include)
  CMakeLists.txt:39 (find_package)

gfortran is supposed to be available as a system dependency, however as I understand it lapack doesn't configure a system_libs dependency on gfortran (which it should) and therefore the lookup fails since the system fallback is not used. Now what options do I have?

• Is there a conan specific way to specify system libs dependencies for packages consumed by my project?
• Could I alter the conanfile.txt of the lapack library (since it's packaged by us) in a way to specify this dependency?
• Something completely different

lapack conanfile.txt

[settings]
    arch=x86_64
    build_type=Release
    compiler=clang
    compiler.libcxx=libstdc++11
    compiler.version=14
    os=Linux

[requires]
    zlib/1.Y.Z

[options]
    fPIC=True
    shared=False

[full_settings]
    arch=x86_64
    build_type=Release
    compiler=clang
    compiler.libcxx=libstdc++11
    compiler.version=14
    os=Linux

[full_requires]
    zlib/1.2.13:24d596ecc3e7cfef35630620092c5615473ba82a

[full_options]
    fPIC=True
    shared=False
    zlib:fPIC=True
    zlib:shared=False

[recipe_hash]
    252d109bf46a76f791e6da86564cd4a6

[env]
    CC=/usr/bin/clang
    CXX=/usr/bin/clang++

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @MartyMcFlyInTheSky

Thanks for your question

gfortran is supposed to be available as a system dependency, however as I understand it lapack doesn't configure a system_libs dependency on gfortran (which it should) and therefore the lookup fails since the system fallback is not used. Now what options do I have?

In this case, the issue is that gfortran is not really a library, it is a compiler isn't it? So it shouldn't have a dependency requires to it. It would be good to see the conanfile.py of your lapack/3.8.0 to understand the relationship.

At the moment, gfortran in ConanCenter is considered as a compiler and as other compilers it is considered a system tool, that should be installed in the system (as gcc, clang, and other compilers). Most likely, this is the simplest recommended approach.

Is there a conan specific way to specify system libs dependencies for packages consumed by my project?
Could I alter the conanfile.txt of the lapack library (since it's packaged by us) in a way to specify this dependency?

Not the conanfile.txt, it has to be a conanfile.py, because .txt files do not really create packages, you need a .py for it. But you can definitely add to your lapack recipe a self.tool_requires("gfortran/version") if you create a package for gfortran itself, we know there are many users doing similar things.
But it shouldn't be a system_lib, because it is not a library.

lapack conanfile.txt

this looks like a Conan 1.X conaninfo.txt from inside the package, not a conanfile.txt, isn't it?
It is very strongly recommended to upgrade to Conan 2, it will soon be 2 years from its release and Conan 1 is considered legacy.

[MartyMcFlyInTheSky]

Thanks for your answer @memsharded ! Yes we recently stumbled across the announced freeze of conan1 packages on conancenter. We consider this the first step to move towards conan2, because we have some internal dependency problems that requires some change in the setup anyway, that's why I decided to start packaging one of our libraries (on that note what would be the recommended approach to include a git submodule not with cmake's add_directory but as a conan package?). I believe from conan 1.61 to 2 it would not be a huge leap of faith, but we're even still using the old cmake_paths generator! So in changing that and to a conanfile.py it was that I experienced above problems.

To answer your questions: This is our current lapack/3.8.0 conanfile.py:

from conans import ConanFile, CMake, tools
from conans.errors import ConanException
from conans.model.version import Version

from io import StringIO
import os


# This recipe is a combination of:
# https://github.com/conan-io/conan-center-index/issues/4509
# https://github.com/conan-io/conan-center-index/pull/15556/files#diff-0e59ad1de05978bb06d99be8b439b055cac7e3f58f61d82055e9e0a137cdb2ca
# https://jgsogo.es/conan-community-web/conan-lapack/detail/testing-3-8-0_conanfile.html 
# 
# also OpenBLAS was used for inspiration:
# https://github.com/conan-io/conan-center-index/blob/5e30b91bd37f87337a98eb159ea529ae9091ab13/recipes/openblas/all/conanfile.py
#
# Note: the lapack relies on the gfortran being linked against, and this
# dependency can't be easily pulled from conan (at the moment of creation of the
# recipe), also see note down about dependencies
# for linux: gfortran is expected to be installed as a system dependency
# for macOS: gfortran is expected to be installed with brew
class LapackConan(ConanFile):
    name = "lapack"
    license = "BSD-3-Clause"
    homepage = "https://github.com/Reference-LAPACK/lapack"
    description = "Fortran subroutines for solving problems in numerical linear algebra"
    topics = "lapack"
    url = "https://github.com/conan-io/conan-center-index"
    settings = "os", "arch", "compiler", "build_type"
    options = {"shared": [True, False], "fPIC": [True, False]}
    default_options = {"shared": False, "fPIC": True}

    exports_sources = ["CMakeLists.txt"]
    generators = "cmake"

    _cmake = None
    _build_subfolder = "build_subfolder"

    def config_options(self):
        if self.settings.os == "Windows":
            del self.options.fPIC

    def configure(self):
        if self.options.shared:
            self.options.rm_safe("fPIC")
    
    # def layout(self):
    #     tools.cmake_layout(self)
    # or in conan2.0?
    #     tools.layout.basic_layout(self, src_folder="source")

    def source(self):
        tools.get(**self.conan_data["sources"][self.version], strip_root=True)

    def requirements(self):
        self.requires("zlib/1.2.13")
    
    def system_requirements(self):
        # Note: dependencies - the below will forcefully install gfrotran, if
        # that's the intention then it can be uncommented, however, it is better
        # if the end-user does it willingly
        #
        # installer = tools.SystemPackageTool()
        # if tools.os_info.is_linux:
        #     if tools.os_info.with_pacman or \
        #         tools.os_info.with_yum:
        #         installer.install("gcc-fortran")
        #     else:
        #         installer.install("gfortran")
        #         versionfloat = Version(self.settings.compiler.version.value)
        #         if self.settings.compiler == "gcc":
        #             if versionfloat < "5.0":
        #                 installer.install("libgfortran-{}-dev".format(versionfloat))
        #             else:
        #                 installer.install("libgfortran-{}-dev".format(int(versionfloat)))
        # if tools.os_info.is_macos and Version(self.settings.compiler.version.value) > "7.3":
        #     try:
        #         installer.install("gcc", update=True, force=True)
        #     except Exception:
        #         self.output.warn("brew install gcc failed. Tying to fix it with 'brew link'")
        #         self.run("brew link --overwrite gcc")        
        pass

    def _configure_cmake(self):
        if self._cmake:
            return self._cmake
        self._cmake = CMake(self)
        self.output.warn("Building lapack requires a Fortran compiler.")
        if self.settings.os == "Windows":
            self.output.error("Can't build for Windows.")

        self._cmake.definitions["CMAKE_GNUtoMS"] = self.settings.os == "Windows"
        self._cmake.definitions["BUILD_TESTING"] = False
        self._cmake.definitions["BUILD_SHARED_LIBS"] = self.options.shared
        self._cmake.definitions["LAPACKE"] = True
        self._cmake.definitions["CBLAS"] = True

        self._cmake.configure(build_folder=self._build_subfolder)
        return self._cmake

    def build(self):
        cmake = self._configure_cmake()
        for target in ["blas", "cblas", "lapack", "lapacke"]:
            cmake.build(target=target)

    def package(self):
        # self.copy(self, pattern="LICENSE", dst="licenses", src="source")
        cmake = self._configure_cmake()
        cmake.install()
        tools.rmdir(os.path.join(self.package_folder, "lib", "pkgconfig"))
        tools.rmdir(os.path.join(self.package_folder, "lib", "cmake"))

    def package_info(self):
        self.cpp_info.set_property("cmake_file_name", "LAPACK")
        self.cpp_info.set_property("cmake_target_name", "LAPACK::LAPACK")
        self.cpp_info.names["cmake_find_package"] = "LAPACK"
        self.cpp_info.names["cmake_find_package_multi"] = "LAPACK"
        # the order is important for static builds
        self.cpp_info.libs = ["lapacke", "lapack", "blas", "cblas", "gfortran", "quadmath"]
        if self.settings.os in ("Linux", "FreeBSD"):
            self.cpp_info.libs.append("m")
        self.cpp_info.libdirs = ["lib"]
        if tools.os_info.is_macos:
            brewout = StringIO()
            try:
                self.run("gfortran --print-file-name libgfortran.dylib", output=brewout)
            except Exception as error:
                raise ConanException("Failed to run command: {}. Output: {}".format(error, brewout.getvalue()))
            lib = os.path.dirname(os.path.normpath(brewout.getvalue().strip()))
            self.cpp_info.libdirs.append(lib)

I believe that gfortran also comes with a library. At least on my system I have libfortran installed in /usr/lib/gcc/x86_64-linux-gnu/9/libgfortran.a. I'm not sure though if it actually links against it though. It seems to though since if I define gfortran as system-dependency it stops complaining about gfortran but then continues with quadmath (which I also have installed). I'm a bit surprised that this was never an issue with the cmake_paths generator 😅

[memsharded]

Thanks for the feedback. I see the system_requirements is basically commented out:

def system_requirements(self):
        # Note: dependencies - the below will forcefully install gfrotran, if
        # that's the intention then it can be uncommented, however, it is better
        # if the end-user does it willingly

I agree with the comments. The new new system.package_manager (see https://docs.conan.io/2/reference/tools/system/package_manager.html, https://docs.conan.io/en/1.65/reference/conanfile/tools/system/package_manager.html for Conan1.X) defaulted to this: it will not install anything in the system unless explicitly activated by the user via a conf. It allows to do checks on installed dependencies without installing and raising accordingly. This package_manager also support reporting and collecting system dependencies in the graph json files in Conan 2.

I think this might be a good first step towards Conan 2 compatibility.

installer.install("gcc", update=True, force=True)

On the other hand, I'd never install gcc from system_requirements().

Then gfortran doesn't look like a packaged library, it is not built by the package, but installed as system-requirements, so most likely it seems that it needs to go to self.cpp_info.system_libs = ["gfortran"], the initial diagnostics seems were correct.

self.cpp_info.libs.append("m")

Same, I'd say the m needs to go to self.cpp_info.system_libs = ["m"] instead

Quick question: is this recipe supposed to be able to work in the open? If you want to put it in a Github repo, then I might try to collaborate in the update of the recipe.

[MartyMcFlyInTheSky]

Oh thanks a lot @memsharded! I got it up and running now, setting 'm', 'quadmath' and 'gfortran' as a system dependency!

To answer your question: No for the moment we don't plan to use this in the open or make it public.

One more question: Say this package B is now a dependency of another package A, but package A lives in its repository with package B checked out as a git submodule, how should package A require this dependency? Should there be a statement like self.requires(..) with a preliminary create . executed in the submodule? Or does conan(1 still) have a dedicated workflow for this case?

[memsharded]

One more question: Say this package B is now a dependency of another package A, but package A lives in its repository with package B checked out as a git submodule, how should package A require this dependency? Should there be a statement like self.requires(..) with a preliminary create . executed in the submodule? Or does conan(1 still) have a dedicated workflow for this case?

Using a dependency as a git submodule is an opposite and alternative approach to using it from an external package.
Often not as important being it a git submodule, but instead the way to consume it.

Very often, when something is a submodule, the root CMakeLists.txt will contain a add_subdirectory(). This will not work while trying to use it as a Conan package, because it is "hardcoding" that it wants to use it from source from that folder, not from an external package.

If it doesn't contain a add_subdirectory(), but contains a regular find_package() then, the flow could be go to the subfolder and do conan create . there. Very often this will not be necessary if that was already created somewhere and uploaded to the server, which is the recommended flow. For an even better and automated flow, Conan has the conan editable feature. This allows to put the subfolder (or any other folder) as "editable" and let the other consumers use it from there instead of using it from the Conan cache. This is a quite advanced use case, intended for users that need to develop simultaneously on the 2 packages, the dependency and the consumer. Still, it is not as "native" and integrated as a single build that is achieved with add_subdirectory(). Read more about editables here https://docs.conan.io/2/tutorial/developing_packages/editable_packages.html

In the general case when using packages, the recommended flow is to decouple it, remove the submodule and use it always from an external dependency with conan install and retrieving it from the server.

[MartyMcFlyInTheSky]

I fear that we actually need packages in editable mode, since we're usually working on both at the same time. Thanks for pointing out that feature! Where is this information about editable references stored btw? Is there a way that I can forward this to my colleagues that are using the package as default?

[memsharded]

Where is this information about editable references stored btw? Is there a way that I can forward this to my colleagues that are using the package as default?

The information is stored in the Conan home folder, but it is not recommended or supported to share that information, because it will contain absolute paths that change from machine to machine and for different users.

This kind of information will be part of the workspace feature, that we plan to resume for Conan 2 as soon as possible, it will use some local file to the project to define the editable packages. This is the ticket you should track to see progress: #15992

[MartyMcFlyInTheSky]

Interesting idea and greatly appreciated! Thanks for your help. I'll close this ticket then.