Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

macOS Cross-Compile release_29: ffs::ffs missing in adios2-config.cmake #3716

Closed
ax3l opened this issue Jul 26, 2023 · 17 comments
Closed

macOS Cross-Compile release_29: ffs::ffs missing in adios2-config.cmake #3716

ax3l opened this issue Jul 26, 2023 · 17 comments
Assignees

Comments

@ax3l
Copy link
Contributor

ax3l commented Jul 26, 2023

I am trying to build release_29 due to the many patches that are needed to get our all-static dependency wheels to build.

I am currently blocked by:

...
    -- Found ADIOS2: /usr/local/lib/cmake/adios2/adios2-config.cmake (found suitable version "2.9.0", minimum required is "2.7.0") found components: C CXX
    CMake Error at CMakeLists.txt:444 (find_package):
      Found package configuration file:
  
        /usr/local/lib/cmake/adios2/adios2-config.cmake
  
      but it set ADIOS2_FOUND to FALSE so package "ADIOS2" is considered to be
      NOT FOUND.  Reason given by package:
  
      The following imported targets are referenced, but are missing: ffs::ffs

openPMD/openPMD-api#1387

when cross-compiling for macOS arm64 (from x86).

cc @vicentebolea @eisenhauer

Note that ADIOS2 itself builds & installs fine, it is the installed adios2-config.cmake package that seems to forget to define one of the used targets.

@ax3l ax3l changed the title release_29: ffs not found release_29: ffs::ffs not found macOS Cross-Compile Jul 26, 2023
@ax3l ax3l changed the title release_29: ffs::ffs not found macOS Cross-Compile macOS Cross-Compile release_29: ffs::ffs missing in adios2-config.cmake Jul 26, 2023
@vicentebolea
Copy link
Collaborator

@ax3l thanks for reporting. Can i get the full configure/build output.

@ax3l
Copy link
Contributor Author

ax3l commented Jul 26, 2023

@vicentebolea
Yes, let me download and link the details from the cibuildwheels setup in openPMD/openPMD-api#1387. library_builders.sh has the build logic in this link.

Build is for:

  • arm64 wheel on macos-11
  • host system is macos-11 on x86-64

Complete logs are: macos-xcompile.zip

@vicentebolea vicentebolea self-assigned this Jul 26, 2023
@vicentebolea
Copy link
Collaborator

I see is it possible to configure openpmd with --trace-expand and write the the output. I have seen this error before and as I recall it was something to do with ADIOS2_DIR must be specified as a env variable rather than a cmake flag.

@ax3l
Copy link
Contributor Author

ax3l commented Jul 27, 2023

Thanks, ok let me see how I can pipe that eloquently through cibuildwheels through pip through setup.py back into CMake 😅
--trace-expand has no environment variable equivalent, does it?

@ax3l
Copy link
Contributor Author

ax3l commented Jul 27, 2023

@vicentebolea ok, got it here:
cmake-trace.zip

And /usr/local/lib/cmake/adios2/adios2-config.cmake seems to contain something like:

2023-07-27T01:14:40.8017740Z /Users/runner/work/openPMD-api/openPMD-api/src
2023-07-27T01:14:40.8018560Z cmake_minimum_required(VERSION 3.1)
2023-07-27T01:14:40.8019170Z 
2023-07-27T01:14:40.8019800Z set(_ADIOS2_CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH})
2023-07-27T01:14:40.8020670Z list(INSERT CMAKE_MODULE_PATH 0 "${CMAKE_CURRENT_LIST_DIR}")
2023-07-27T01:14:40.8021290Z 
2023-07-27T01:14:40.8022750Z if(NOT OFF)
2023-07-27T01:14:40.8024300Z   if(NOT )
2023-07-27T01:14:40.8025360Z     set(EVPath_DIR ${CMAKE_CURRENT_LIST_DIR}/thirdparty)
2023-07-27T01:14:40.8026110Z   endif()
2023-07-27T01:14:40.8026650Z 
2023-07-27T01:14:40.8027120Z   if(NOT OFF)
2023-07-27T01:14:40.8027850Z     set(atl_DIR ${CMAKE_CURRENT_LIST_DIR}/thirdparty)
2023-07-27T01:14:40.8028530Z   endif()
2023-07-27T01:14:40.8029070Z 
2023-07-27T01:14:40.8029540Z   if(NOT OFF)
2023-07-27T01:14:40.8030260Z     set(dill_DIR ${CMAKE_CURRENT_LIST_DIR}/thirdparty)
2023-07-27T01:14:40.8030960Z   endif()
2023-07-27T01:14:40.8031500Z 
2023-07-27T01:14:40.8031940Z   if(NOT OFF)
2023-07-27T01:14:40.8032670Z     set(ffs_DIR ${CMAKE_CURRENT_LIST_DIR}/thirdparty)
2023-07-27T01:14:40.8033360Z   endif()
2023-07-27T01:14:40.8033890Z 
2023-07-27T01:14:40.8034380Z   if(NOT OFF)
2023-07-27T01:14:40.8035070Z     set(enet_DIR ${CMAKE_CURRENT_LIST_DIR}/thirdparty)
2023-07-27T01:14:40.8035870Z   endif()
2023-07-27T01:14:40.8036550Z endif()
2023-07-27T01:14:40.8037100Z 
2023-07-27T01:14:40.8037670Z set(${CMAKE_FIND_PACKAGE_NAME}_CONFIG "${CMAKE_CURRENT_LIST_FILE}")
2023-07-27T01:14:40.8038690Z include("${CMAKE_CURRENT_LIST_DIR}/adios2-config-common.cmake")
2023-07-27T01:14:40.8039380Z 
2023-07-27T01:14:40.8039920Z set(CMAKE_MODULE_PATH ${_ADIOS2_CMAKE_MODULE_PATH})
2023-07-27T01:14:40.8040800Z unset(_ADIOS2_CMAKE_MODULE_PATH)

@ax3l
Copy link
Contributor Author

ax3l commented Jul 27, 2023

Side question: FFS is a mandatory ADIOS2 dependency or can it be disabled? (I did not find an option for it.)

@eisenhauer
Copy link
Member

Side question: FFS is a mandatory ADIOS2 dependency or can it be disabled? (I did not find an option for it.)

The options to disable it are indirect. It is required both for SST and for BP5. If you disable both of those FFS will not be built. (But BP5 is now the default file format. If you disable it there are some tests that won't pass, functionality will be lost, etc.)

@ax3l
Copy link
Contributor Author

ax3l commented Jul 27, 2023

Thanks Greg, yes that makes sense then to not ship without it.

@eisenhauer @vicentebolea does the detailed CMake traceback above provide you with the info we need?
I also checked again in conda-forge/adios2-feedstock#74 and saw not FFS specific work-around there.

@vicentebolea
Copy link
Collaborator

vicentebolea commented Jul 27, 2023

I have found the culprit but not the reason of this happening.

Looking at the trace, /usr/local/lib/cmake/adios2/adios2-config.cmake(7) should be if (NOT OFF) since the OFF is ADIOS2_USE_EXTERNAL_EVPATH, however it is for some reason undef.


2023-07-27T01:15:15.9040940Z 
2023-07-27T01:15:15.9041520Z   /usr/local/lib/cmake/adios2/adios2-config.cmake(3):  set(_ADIOS2_CMAKE_MODULE_PATH /Users/runner/work/openPMD-api/openPMD-api/src/share/openPMD/cmake )
2023-07-27T01:15:15.9042260Z   /usr/local/lib/cmake/adios2/adios2-config.cmake(4):  list(INSERT CMAKE_MODULE_PATH 0 /usr/local/lib/cmake/adios2 )
2023-07-27T01:15:15.9042820Z   /usr/local/lib/cmake/adios2/adios2-config.cmake(6):  if(NOT OFF )
2023-07-27T01:15:15.9043330Z   /usr/local/lib/cmake/adios2/adios2-config.cmake(7):  if(NOT )
2023-07-27T01:15:15.9043840Z   /usr/local/lib/cmake/adios2/adios2-config.cmake(11):  if(NOT OFF )
2023-07-27T01:15:15.9044440Z   /usr/local/lib/cmake/adios2/adios2-config.cmake(12):  set(atl_DIR /usr/local/lib/cmake/adios2/thirdparty )
2023-07-27T01:15:15.9045010Z   /usr/local/lib/cmake/adios2/adios2-config.cmake(15):  if(NOT OFF )
2023-07-27T01:15:15.9045620Z   /usr/local/lib/cmake/adios2/adios2-config.cmake(16):  set(dill_DIR /usr/local/lib/cmake/adios2/thirdparty )
2023-07-27T01:15:15.9046510Z   /usr/local/lib/cmake/adios2/adios2-config.cmake(19):  if(NOT OFF )
2023-07-27T01:15:15.9047120Z   /usr/local/lib/cmake/adios2/adios2-config.cmake(20):  set(ffs_DIR /usr/local/lib/cmake/adios2/thirdparty )
2023-07-27T01:15:15.9047680Z   /usr/local/lib/cmake/adios2/adios2-config.cmake(23):  if(NOT OFF )
2023-07-27T01:15:15.9048280Z   /usr/local/lib/cmake/adios2/adios2-config.cmake(24):  set(enet_DIR /usr/local/lib/cmake/adios2/thirdparty )
2023-07-27T01:15:15.9048960Z   /usr/local/lib/cmake/adios2/adios2-config.cmake(28):  set(ADIOS2_CONFIG /usr/local/lib/cmake/adios2/adios2-config.cmake )
2023-07-27T01:15:15.9049670Z   /usr/local/lib/cmake/adios2/adios2-config.cmake(29):  include(/usr/local/lib/cmake/adios2/adios2-config-common.cmake )

Here is the source file of adios2-config.cmake:

if(NOT @BUILD_SHARED_LIBS@)
  if(NOT @ADIOS2_USE_EXTERNAL_EVPATH@)
    set(EVPath_DIR ${CMAKE_CURRENT_LIST_DIR}/thirdparty)
  endif()

  if(NOT @ADIOS2_USE_EXTERNAL_ATL@)
    set(atl_DIR ${CMAKE_CURRENT_LIST_DIR}/thirdparty)
  endif()

  if(NOT @ADIOS2_USE_EXTERNAL_DILL@)
    set(dill_DIR ${CMAKE_CURRENT_LIST_DIR}/thirdparty)
  endif()

  if(NOT @ADIOS2_USE_EXTERNAL_FFS@)
    set(ffs_DIR ${CMAKE_CURRENT_LIST_DIR}/thirdparty)
  endif()

  if(NOT @ADIOS2_USE_EXTERNAL_ENET@)
    set(enet_DIR ${CMAKE_CURRENT_LIST_DIR}/thirdparty)
  endif()
endif()

It is located at cmake/adios2-config-install.cmake.in. we need set(EVPath_DIR ${CMAKE_CURRENT_LIST_DIR}/thirdparty) since later it will find_dependency(EVPATH) which will load all the other (ffs, atl, dill, enet) thirdparty cmake config files

Note that this is only happening with static build

@vicentebolea
Copy link
Collaborator

@ax3l can you also run the --trace-expand for the configuration of Adios2, this might bring some light on why this is happening

@ax3l
Copy link
Contributor Author

ax3l commented Jul 28, 2023

@vicentebolea no problem, here is the log output, including --trace-expand for the ADIOS2 build step.
cmake-adios2-trace.zip

@ax3l
Copy link
Contributor Author

ax3l commented Jul 28, 2023

@vicentebolea I think the part this script above is not expecting is in

if(NOT @BUILD_SHARED_LIBS@)

I do build all-static for wheels, so the most outer if becomes

if(NOT FALSE)

@ax3l
Copy link
Contributor Author

ax3l commented Jul 28, 2023

@vicentebolea is the if(NOT @BUILD_SHARED_LIBS@) generally the issue here? Don't we need to define the CMake targets either way, independent if we use them as shared or static targets?

@vicentebolea
Copy link
Collaborator

The issue happens because EVPATH dependency gets properly exported when using SST, however, even without SST we use ffs which gets exposed through EVPATH. @ax3l the workaround should be to enable SST or disable BP5.

Things to do:

  • Properly export ffs in adios2-config when SST is disabled.

@ax3l
Copy link
Contributor Author

ax3l commented Jul 28, 2023

Ah, but I need BP5 and was bit able to cross-compile SST in the past (#3116 - fixed in 2.8.0+).
Let me try to turn SST on again for ADIOS 2.9.0.

Update: that works for now 🎉
I just need to disable SST here and there for some platforms, so #3724 will be appreciated 🙏

@vicentebolea
Copy link
Collaborator

I believe that this has been resolved in #3760

@vicentebolea
Copy link
Collaborator

Closing this since we cannot reproduce this anymore. Please re-open if you observe this again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants