Fails to build on Intel Mac

[nezda]

Hello, All -

It isn't clear to me if this is supposed to work on Intel Macs or not, but I can't get this project to build cleanly on my late 2019 Intel-based MacBook Pro running macOS 13.5. Compilation appears to get fairly close when using brew install gcc (13.2.0), export CC=/usr/local/opt/gcc/bin/gcc-13, export CXX=/usr/local/opt/gcc/bin/g++-13. pip install -v . I'm left with 2 kinds of errors one of which I maybe worked around?

First error type

• intel-extension-for-transformers/intel_extension_for_transformers/llm/library/jblas/jblas/kernel_jit.h

  Line 1231 in 43e30bc

  db(reinterpret_cast<uintptr_t>(nullptr), sizeof(intptr_t)); // case 0 should never occur

  causes error

  /Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/jblas/kernel_jit.h:1231:7: error: call of overloaded 'db(uintptr_t, long unsigned int)' is ambiguous
   1231 |     db(reinterpret_cast<uintptr_t>(nullptr), sizeof(intptr_t));  // case 0 should never occur
        |     ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  In file included from /Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/jblas/jit_base.h:19,
                   from /Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/jblas/jit_blas_storage.h:15:
  /Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/jblas/xbyak/xbyak.h:1133:14: note: candidate: 'void Xbyak::CodeArray::db(uint64_t, size_t)'
   1133 |         void db(uint64_t code, size_t codeSize)
        |              ^~
  /Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/jblas/xbyak/xbyak.h:1129:14: note: candidate: 'void Xbyak::CodeArray::db(const uint8_t*, size_t)'
   1129 |         void db(const uint8_t *code, size_t codeSize)
        |              ^~
  ninja: build stopped: subcommand failed.

• changing this to db(nullptr, sizeof(intptr_t)); // case 0 should never occur seems to compile

Second error type

• 5 instances of error SYS_arch_prctl is not declared, e.g.,

  [1/7] Building CXX object dispatcher/CMakeFiles/jblas_dispatcher.dir/src/dispatcher_utils.cpp.o
  FAILED: dispatcher/CMakeFiles/jblas_dispatcher.dir/src/dispatcher_utils.cpp.o
  /usr/local/opt/gcc/bin/g++-13 -DUSE_C10D_GLOO -DUSE_DISTRIBUTED -DUSE_RPC -DUSE_TENSORPIPE -I/Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas -isystem /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/torch/include -isystem /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -fconcepts -O3 -DNDEBUG -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk -fPIC -fopenmp -MD -MT dispatcher/CMakeFiles/jblas_dispatcher.dir/src/dispatcher_utils.cpp.o -MF dispatcher/CMakeFiles/jblas_dispatcher.dir/src/dispatcher_utils.cpp.o.d -o dispatcher/CMakeFiles/jblas_dispatcher.dir/src/dispatcher_utils.cpp.o -c /Users/nezda/code/itrex/intel_extension_for_transformers/llm/operator/csrc/dispatcher/src/dispatcher_utils.cpp
  In file included from /Users/nezda/code/itrex/intel_extension_for_transformers/llm/operator/csrc/dispatcher/src/../include/dispatcher_utils.hpp:19,
                   from /Users/nezda/code/itrex/intel_extension_for_transformers/llm/operator/csrc/dispatcher/src/dispatcher_utils.cpp:14:
  /Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/jblas/jit_blas_utils.h: In function 'void jblas::utils::request_perm_xtile_data()':
  /Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/jblas/jit_blas_utils.h:375:16: error: 'SYS_arch_prctl' was not declared in this scope; did you mean 'SYS_csrctl'?
    375 |   rc = syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, XFEATURE_XTILEDATA);
        |                ^~~~~~~~~~~~~~
        |                SYS_csrctl

I wasn't sure what to do for that one.

Thoughts?
- Luke

[DDEle]

Although we don't have plan to support Intel Mac, I can try to give you some personal suggestions. For your second error type, I guess that you can just change L369 of jit_blas_utils.h to something like #if !defined(_WIN32) && !defined(__APPLE__). That function is for AMX which is currently not available on client products.

[nezda]

That definitely got me further - I made a draft PR in fork of this repo #1085 . tldr build now fails with

 -- Build files have been written to: /Users/nezda/code/itrex/build/qbits
  /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/cmake/data/bin/cmake --build . -j
  [1/7] Building CXX object dispatcher/CMakeFiles/jblas_dispatcher.dir/src/dispatcher_utils.cpp.o
  [2/7] Building CXX object dispatcher/CMakeFiles/jblas_dispatcher.dir/src/jblas_gemm_dispatcher.cpp.o
  [3/7] Building CXX object CMakeFiles/qbits.dir/src/dropout.cpp.o
  [4/7] Building CXX object CMakeFiles/qbits.dir/qbits.cpp.o
  [5/7] Building CXX object dispatcher/CMakeFiles/jblas_dispatcher.dir/src/jblas_weightonly_dispatcher.cpp.o
  [6/7] Linking CXX static library dispatcher/libjblas_dispatcher.a
  [7/7] Linking CXX shared library /Users/nezda/code/itrex/build/lib.macosx-13-x86_64-cpython-310/intel_extension_for_transformers/libqbits.dylib
  FAILED: /Users/nezda/code/itrex/build/lib.macosx-13-x86_64-cpython-310/intel_extension_for_transformers/libqbits.dylib
  : && /usr/local/opt/gcc/bin/g++-13 -O3 -DNDEBUG -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk -dynamiclib -Wl,-headerpad_max_install_names -L/usr/local/opt/llvm/lib -L/usr/local/opt/libomp/lib -o /Users/nezda/code/itrex/build/lib.macosx-13-x86_64-cpython-310/intel_extension_for_transformers/libqbits.dylib -install_name @rpath/libqbits.dylib CMakeFiles/qbits.dir/qbits.cpp.o CMakeFiles/qbits.dir/src/dropout.cpp.o -L/lib/intel64   -L/lib/intel64_win   -L/lib/win-x64 -Wl,-rpath,/lib/intel64 -Wl,-rpath,/lib/intel64_win -Wl,-rpath,/lib/win-x64 -Wl,-rpath,/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/torch/lib  dispatcher/libjblas_dispatcher.a  /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/torch/lib/libtorch.dylib  /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.dylib  /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/torch/lib/libc10.dylib  /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/torch/lib/libc10.dylib  /usr/local/Cellar/gcc/13.2.0/lib/gcc/current/libgomp.dylib && :
  ld: warning: directory not found for option '-L/lib/intel64'
  ld: warning: directory not found for option '-L/lib/intel64_win'
  ld: warning: directory not found for option '-L/lib/win-x64'
  Undefined symbols for architecture x86_64:
    "__ZN3c1010TensorImpl17set_autograd_metaESt10unique_ptrINS_21AutogradMetaInterfaceESt14default_deleteIS2_EE", referenced from:
        __ZN5torch8autograd13make_variableEN2at6TensorEbb in dropout.cpp.o
        __ZN5torch8autograd13make_variableEN2at6TensorEbb in libjblas_dispatcher.a(jblas_weightonly_dispatcher.cpp.o)
    "__ZN3c106detail14torchCheckFailEPKcS2_jRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE", referenced from:
        __ZN10jblas_gemm19dispatch_jblas_gemmEPNS_22jblas_gemm_runtime_ctxE in libjblas_dispatcher.a(jblas_gemm_dispatcher.cpp.o)
        __ZN3c1019fromIntArrayRefSlowENS_8ArrayRefIxEE in libjblas_dispatcher.a(jblas_weightonly_dispatcher.cpp.o)
        __ZN3woq12parse_weightILNS_8WOQ_TASKE1EN5jblas4gemm22ICoreRowNAmxint8KBlockILi48ELi16EEEEEvPNS_16woq_config_paramEPNS_15woq_runtime_ctxE in libjblas_dispatcher.a(jblas_weightonly_dispatcher.cpp.o)
        __ZN3woq12parse_weightILNS_8WOQ_TASKE1EN5jblas4gemm25ICoreRowNAvx512vnniKBlockILi48ELi4EEEEEvPNS_16woq_config_paramEPNS_15woq_runtime_ctxE in libjblas_dispatcher.a(jblas_weightonly_dispatcher.cpp.o)
        __ZN3woq12parse_weightILNS_8WOQ_TASKE1EN5jblas4gemm22ICoreRowNAvxvnniKBlockILi48ELi2EEEEEvPNS_16woq_config_paramEPNS_15woq_runtime_ctxE in libjblas_dispatcher.a(jblas_weightonly_dispatcher.cpp.o)
        __ZN3woq12parse_weightILNS_8WOQ_TASKE1EN5jblas4gemm16SCoreRowNAvx512fILi48ELi8EEEEEvPNS_16woq_config_paramEPNS_15woq_runtime_ctxE in libjblas_dispatcher.a(jblas_weightonly_dispatcher.cpp.o)
        __ZN3woq12parse_weightILNS_8WOQ_TASKE1EN5jblas4gemm13SCoreRowNAvx2ILi48ELi2EEEEEvPNS_16woq_config_paramEPNS_15woq_runtime_ctxE in libjblas_dispatcher.a(jblas_weightonly_dispatcher.cpp.o)
        ...
    "__ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE", referenced from:
        __ZN3c104impl31make_boxed_from_unboxed_functorINS0_6detail31WrapFunctionIntoRuntimeFunctor_IPFvRKN2at6TensorERS5_bRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESG_SG_EvNS_4guts8typelist8typelistIJS7_S8_bSG_SG_SG_EEEEELb0EE4callEPNS_14OperatorKernelERKNS_14OperatorHandleENS_14DispatchKeySetEPSt6vectorINS_6IValueESaISW_EE in qbits.cpp.o
        __ZN3c104impl31make_boxed_from_unboxed_functorINS0_6detail31WrapFunctionIntoRuntimeFunctor_IPFvRKN2at6TensorES7_S7_RS5_xbRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESG_SG_EvNS_4guts8typelist8typelistIJS7_S7_S7_S8_xbSG_SG_SG_EEEEELb0EE4callEPNS_14OperatorKernelERKNS_14OperatorHandleENS_14DispatchKeySetEPSt6vectorINS_6IValueESaISW_EE in qbits.cpp.o
        __ZN3c104impl31make_boxed_from_unboxed_functorINS0_6detail31WrapFunctionIntoRuntimeFunctor_IPFN2at6TensorERKS5_bxRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESF_SF_ES5_NS_4guts8typelist8typelistIJS7_bxSF_SF_SF_EEEEELb0EE4callEPNS_14OperatorKernelERKNS_14OperatorHandleENS_14DispatchKeySetEPSt6vectorINS_6IValueESaISV_EE in qbits.cpp.o
    "__ZN5torch3jit17parseSchemaOrNameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE", referenced from:
        __ZN5torch6detail21constructSchemaOrNameEPKc in qbits.cpp.o
    "__ZN5torch7LibraryC1ENS0_4KindENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEN3c108optionalINS8_11DispatchKeyEEEPKcj", referenced from:
        __ZN5torch6detail16TorchLibraryInitC1ENS_7Library4KindEPFvRS2_EPKcN3c108optionalINS9_11DispatchKeyEEES8_j in qbits.cpp.o
  ld: symbol(s) not found for architecture x86_64
  collect2: error: ld returned 1 exit status
  ninja: build stopped: subcommand failed.

Thanks in advance @DDEle!

[nezda]

This may be related... torch==2.1.0+cpu doesn't seem to work with my python setup (created 3.10.13 venv for this (would've preferred 3.11)): when i run pip install -r requirements.txt I get error

ERROR: Could not find a version that satisfies the requirement torch==2.1.0+cpu (from versions: 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2)
ERROR: No matching distribution found for torch==2.1.0+cpu

so I hacked the torch requirement to just torch==2.1.0.

[DDEle]

The torch==2.1.0 should not be the problem as long as the error message is not about Nvidia. We haven't tried to build extension for pytorch on Mac and I am not familiar with the torch part. If you are only interested in the LLM Runtime (Efficient LLM Inference on CPUs), maybe try only building it and see how far we can go?

You can disable the build of qbits and the deprecated executor as following

BTW, thanks for your interest of ITREX and it would be nice if we can have some limited support for Intel Mac users with help of community.

[nezda]

Hi @DDEle -

I did what you said and another thing and made more progress - changes pushed to #1085 . Error now is below - I plan to look at it again in the morning. I'm interested to see what I can get out of this Intel-based machine for my LLM work - thanks for your help!

Kind regards,
- Luke

[130/191] Linking CXX shared library /Users/nezda/code/itrex/build/lib.macosx-13-x86_64-cpython-310/intel_extension_for_transformers/llm/runtime/graph/libne_layers.dylib
  FAILED: /Users/nezda/code/itrex/build/lib.macosx-13-x86_64-cpython-310/intel_extension_for_transformers/llm/runtime/graph/libne_layers.dylib
  : && /usr/local/opt/gcc/bin/g++-13 -O3 -DNDEBUG -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk -dynamiclib -Wl,-headerpad_max_install_names -L/usr/local/opt/llvm/lib -L/usr/local/opt/libomp/lib -o /Users/nezda/code/itrex/build/lib.macosx-13-x86_64-cpython-310/intel_extension_for_transformers/llm/runtime/graph/libne_layers.dylib -install_name @rpath/libne_layers.dylib core/CMakeFiles/ne_layers.dir/ne_layers.c.o core/CMakeFiles/ne_layers.dir/layers/conv.cpp.o core/CMakeFiles/ne_layers.dir/layers/inner_product.cpp.o core/CMakeFiles/ne_layers.dir/layers/ip_fusion_ffn.cpp.o core/CMakeFiles/ne_layers.dir/layers/ip_fusion_qkv.cpp.o core/CMakeFiles/ne_layers.dir/layers/jblas_gemm.cpp.o core/CMakeFiles/ne_layers.dir/layers/memory.cpp.o core/CMakeFiles/ne_layers.dir/layers/mha_dense.cpp.o core/CMakeFiles/ne_layers.dir/layers/ne_jblas.cpp.o  lib/libne_vec.a  -lrt  /usr/local/Cellar/gcc/13.2.0/lib/gcc/current/libgomp.dylib  lib/libcpu_vec.a  -lemutls_w && :
  ld: library not found for -lrt
  collect2: error: ld returned 1 exit status
  ninja: build stopped: subcommand failed.

[nezda]

Removed OpenMP::OpenMP_C from

intel-extension-for-transformers/intel_extension_for_transformers/llm/library/jblas/CMakeLists.txt

Line 33 in 65af04e

target_link_libraries(${PROJECT_NAME} INTERFACE OpenMP::OpenMP_CXX OpenMP::OpenMP_C)

and clang compiled with brew install libomp installed and inference works, very slowly despite apparently using all the cores. I think this is not a supported clang configuration https://github.com/oneapi-src/oneDNN/blob/7f2419ddc5c296cc9395ca2da11780c2657595c1/README.md?plain=1#L205 because is linking to https://openmp.llvm.org/ (not Intel or GCC ?). Is this the best this machine can do performance wise?

[DDEle]

Glad to know that you finally made it run!!!

• For clang / libomp related questions, I'm not familiar about this but I agree that you can try to install he Intel omp as said by OneDNN to see if you can get better performance.

• BTW, from your log:

    -- The C compiler identification is AppleClang 14.0.3.14030022
    -- The CXX compiler identification is GNU 13.2.0
  

  Seems you are using clang for C and but for C++?

• Don't worry about the NE_WITH_AVX2 warnings, this flag is only used in the deprecated executor and does not affect performance of CPP graph.

• For the performance, how slow did you mean by "very slowly"? As a reference, in a 10c20t Windows desktop 10900 (no avx512), I had ~1000ms first-token and ~300ms next-token (slower than what we can get on Linux and troubleshooting is on the way). So, what we can expect firstly should be at least 1000ms next-token performance.

• To debug the performance problem, I think what we should first confirm that it really uses multiple threads. You can change your "single thread hack" to the number of threads and the number of cores (usually it is the half of thread numbers), and compare its performance.

• In addition, we can check the profiling.

  If you want to debug the performance further, you can compile (pip install -v .) with CMAKE_ARGS="-DNE_PROFILING=ON" and run python with ENGINE_PROFILING=1, and show us the profiling log so that we may have some clues.

[nezda]

Seems you are using clang for C and but for C++?

Oof, another good catch. I wonder how the heck I even did that!? When I do

export CC=/usr/bin/gcc
export CXX=/usr/bin/g++

naturally it uses clang for c++ (The CXX compiler identification is AppleClang 14.0.3.14030022)

and the build fails 🤦 - how "lucky" am I!? To have ever seen it work I accidentally failed to unset CXX leaving behind my previous export CXX=/usr/local/opt/gcc/bin/g++-13 (have confirmed this hypothesis).

itrex$ pip install -v --editable .
Using pip 23.3.2 from /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/pip (python 3.10)
Obtaining file:///Users/nezda/code/itrex
  Running command python setup.py egg_info
  /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/__init__.py:80: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
  !!

          ********************************************************************************
          Requirements should be satisfied by a PEP 517 installer.
          If you are using pip, you can try `pip install --use-pep517`.
          ********************************************************************************

  !!
    dist.fetch_build_eggs(dist.setup_requires)
  WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
   --- Trying to initialize submodules
   --- Submodule initialization took 0.58 sec
  running egg_info
  creating /private/var/folders/q0/qzzpn5bd4wj8_mxbsv23wjq00000gn/T/pip-pip-egg-info-g7ybu5xd/intel_extension_for_transformers.egg-info
  writing /private/var/folders/q0/qzzpn5bd4wj8_mxbsv23wjq00000gn/T/pip-pip-egg-info-g7ybu5xd/intel_extension_for_transformers.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/q0/qzzpn5bd4wj8_mxbsv23wjq00000gn/T/pip-pip-egg-info-g7ybu5xd/intel_extension_for_transformers.egg-info/dependency_links.txt
  writing entry points to /private/var/folders/q0/qzzpn5bd4wj8_mxbsv23wjq00000gn/T/pip-pip-egg-info-g7ybu5xd/intel_extension_for_transformers.egg-info/entry_points.txt
  writing requirements to /private/var/folders/q0/qzzpn5bd4wj8_mxbsv23wjq00000gn/T/pip-pip-egg-info-g7ybu5xd/intel_extension_for_transformers.egg-info/requires.txt
  writing top-level names to /private/var/folders/q0/qzzpn5bd4wj8_mxbsv23wjq00000gn/T/pip-pip-egg-info-g7ybu5xd/intel_extension_for_transformers.egg-info/top_level.txt
  writing manifest file '/private/var/folders/q0/qzzpn5bd4wj8_mxbsv23wjq00000gn/T/pip-pip-egg-info-g7ybu5xd/intel_extension_for_transformers.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  adding license file 'LICENSE'
  writing manifest file '/private/var/folders/q0/qzzpn5bd4wj8_mxbsv23wjq00000gn/T/pip-pip-egg-info-g7ybu5xd/intel_extension_for_transformers.egg-info/SOURCES.txt'
  Preparing metadata (setup.py) ... done
Requirement already satisfied: packaging in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (23.2)
Requirement already satisfied: numpy in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (1.26.2)
Requirement already satisfied: schema in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (0.7.5)
Requirement already satisfied: pyyaml in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (6.0.1)
Requirement already satisfied: neural_compressor in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (2.4)
Requirement already satisfied: transformers in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (4.36.2)
Requirement already satisfied: deprecated>=1.2.13 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (1.2.14)
Requirement already satisfied: opencv-python-headless in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (4.8.1.78)
Requirement already satisfied: pandas in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (2.1.4)
Requirement already satisfied: Pillow in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (10.1.0)
Requirement already satisfied: prettytable in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (3.9.0)
Requirement already satisfied: psutil in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (5.9.7)
Requirement already satisfied: py-cpuinfo in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (9.0.0)
Requirement already satisfied: requests in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (2.31.0)
Requirement already satisfied: scikit-learn in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (1.3.2)
Requirement already satisfied: pycocotools in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (2.0.7)
Requirement already satisfied: contextlib2>=0.5.5 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from schema->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (21.6.0)
Requirement already satisfied: filelock in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from transformers->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (3.13.1)
Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from transformers->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (0.20.1)
Requirement already satisfied: regex!=2019.12.17 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from transformers->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (2023.12.25)
Requirement already satisfied: tokenizers<0.19,>=0.14 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from transformers->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (0.15.0)
Requirement already satisfied: safetensors>=0.3.1 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from transformers->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (0.4.1)
Requirement already satisfied: tqdm>=4.27 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from transformers->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (4.66.1)
Requirement already satisfied: wrapt<2,>=1.10 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from deprecated>=1.2.13->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (1.16.0)
Requirement already satisfied: fsspec>=2023.5.0 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (2023.10.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (4.9.0)
Requirement already satisfied: python-dateutil>=2.8.2 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from pandas->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from pandas->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from pandas->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (2023.3)
Requirement already satisfied: wcwidth in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from prettytable->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (0.2.12)
Requirement already satisfied: matplotlib>=2.1.0 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from pycocotools->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (3.8.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from requests->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from requests->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from requests->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (2.1.0)
Requirement already satisfied: certifi>=2017.4.17 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from requests->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (2023.11.17)
Requirement already satisfied: scipy>=1.5.0 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from scikit-learn->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (1.11.4)
Requirement already satisfied: joblib>=1.1.1 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from scikit-learn->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (1.3.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from scikit-learn->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (3.2.0)
Requirement already satisfied: contourpy>=1.0.1 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from matplotlib>=2.1.0->pycocotools->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (1.2.0)
Requirement already satisfied: cycler>=0.10 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from matplotlib>=2.1.0->pycocotools->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from matplotlib>=2.1.0->pycocotools->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (4.47.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from matplotlib>=2.1.0->pycocotools->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (1.4.5)
Requirement already satisfied: pyparsing>=2.3.1 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from matplotlib>=2.1.0->pycocotools->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (3.1.1)
Requirement already satisfied: six>=1.5 in /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->neural_compressor->intel-extension-for-transformers==1.4.dev36+g05b66dbef5.d20240107) (1.16.0)
Installing collected packages: intel-extension-for-transformers
  Attempting uninstall: intel-extension-for-transformers
    Found existing installation: intel-extension-for-transformers 1.4.dev36+g05b66dbef5.d20240106
    Uninstalling intel-extension-for-transformers-1.4.dev36+g05b66dbef5.d20240106:
      Removing file or directory /Users/nezda/.virtualenvs/billex_playground.3.10/bin/neural_engine
      Removing file or directory /Users/nezda/.virtualenvs/billex_playground.3.10/bin/neuralchat
      Removing file or directory /Users/nezda/.virtualenvs/billex_playground.3.10/bin/neuralchat_client
      Removing file or directory /Users/nezda/.virtualenvs/billex_playground.3.10/bin/neuralchat_server
      Removing file or directory /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/intel-extension-for-transformers.egg-link
      Removing pth entries from /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/easy-install.pth:
      Removing entry: /Users/nezda/code/itrex
      Successfully uninstalled intel-extension-for-transformers-1.4.dev36+g05b66dbef5.d20240106
  Running setup.py develop for intel-extension-for-transformers
    Running command python setup.py develop
    /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/__init__.py:80: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
    !!

            ********************************************************************************
            Requirements should be satisfied by a PEP 517 installer.
            If you are using pip, you can try `pip install --use-pep517`.
            ********************************************************************************

    !!
      dist.fetch_build_eggs(dist.setup_requires)
    WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
     --- Trying to initialize submodules
     --- Submodule initialization took 0.40 sec
    running develop
    /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.
    !!

            ********************************************************************************
            Please avoid running ``setup.py`` and ``easy_install``.
            Instead, use pypa/build, pypa/installer or other
            standards-based tools.

            See https://github.com/pypa/setuptools/issues/917 for details.
            ********************************************************************************

    !!
      easy_install.initialize_options(self)
    /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
    !!

            ********************************************************************************
            Please avoid running ``setup.py`` directly.
            Instead, use pypa/build, pypa/installer or other
            standards-based tools.

            See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
            ********************************************************************************

    !!
      self.initialize_options()
    running egg_info
    writing intel_extension_for_transformers.egg-info/PKG-INFO
    writing dependency_links to intel_extension_for_transformers.egg-info/dependency_links.txt
    writing entry points to intel_extension_for_transformers.egg-info/entry_points.txt
    writing requirements to intel_extension_for_transformers.egg-info/requires.txt
    writing top-level names to intel_extension_for_transformers.egg-info/top_level.txt
    reading manifest template 'MANIFEST.in'
    adding license file 'LICENSE'
    writing manifest file 'intel_extension_for_transformers.egg-info/SOURCES.txt'
    running build_ext
    Re-run cmake no build system arguments
    -- The C compiler identification is AppleClang 14.0.3.14030022
    -- The CXX compiler identification is AppleClang 14.0.3.14030022
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/gcc - skipped
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/g++ - skipped
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    -- Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)
    -- Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES)
    -- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND)
    -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
    -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
    -- Found Threads: TRUE
    -- pybind11 v2.12.0 dev1
    CMake Warning (dev) at application/third_party/pybind11/tools/FindPythonLibsNew.cmake:98 (find_package):
      Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
      are removed.  Run "cmake --help-policy CMP0148" for policy details.  Use
      the cmake_policy command to set the policy and suppress this warning.

    Call Stack (most recent call first):
      application/third_party/pybind11/tools/pybind11Tools.cmake:50 (find_package)
      application/third_party/pybind11/tools/pybind11Common.cmake:188 (include)
      application/third_party/pybind11/CMakeLists.txt:211 (include)
    This warning is for project developers.  Use -Wno-dev to suppress it.

    -- Found PythonInterp: /Users/nezda/.virtualenvs/billex_playground.3.10/bin/python (found suitable version "3.10.13", minimum required is "3.6")
    -- Found PythonLibs: /usr/local/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib/libpython3.10.dylib
    -- Performing Test HAS_FLTO
    -- Performing Test HAS_FLTO - Success
    -- Performing Test HAS_FLTO_THIN
    -- Performing Test HAS_FLTO_THIN - Success
    -- Configuring done (4.6s)
    CMake Error at /Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/CMakeLists.txt:33 (target_link_libraries):
      The link interface of target "jblas" contains:

        OpenMP::OpenMP_CXX

      but the target was not found.  Possible reasons include:

        * There is a typo in the target name.
        * A find_package call is missing for an IMPORTED target.
        * An ALIAS target is missing.



    -- Generating done (0.1s)
    CMake Warning:
      Manually-specified variables were not used by the project:

        DNNL_CPU_RUNTIME
        NE_VERSION_STRING
        NE_WITH_AVX2
        NE_WITH_TESTS


    CMake Generate step failed.  Build files cannot be regenerated correctly.
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "<pip-setuptools-caller>", line 34, in <module>
      File "/Users/nezda/code/itrex/setup.py", line 256, in <module>
        setup(
      File "/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/__init__.py", line 103, in setup
        return distutils.core.setup(**attrs)
      File "/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
        return run_commands(dist)
      File "/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
        dist.run_commands()
      File "/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
        self.run_command(cmd)
      File "/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/dist.py", line 1001, in run_command
        super().run_command(command)
      File "/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
        cmd_obj.run()
      File "/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/command/develop.py", line 34, in run
        self.install_for_development()
      File "/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/command/develop.py", line 109, in install_for_development
        self.run_command('build_ext')
      File "/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
        self.distribution.run_command(command)
      File "/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/dist.py", line 1001, in run_command
        super().run_command(command)
      File "/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
        cmd_obj.run()
      File "/Users/nezda/code/itrex/setup.py", line 222, in run
        return super().run()
      File "/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 88, in run
        _build_ext.run(self)
      File "/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
        self.build_extensions()
      File "/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
        self._build_extensions_serial()
      File "/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
        self.build_extension(ext)
      File "/Users/nezda/code/itrex/setup.py", line 196, in build_extension
        subprocess.run(config_command, cwd=build_dir, check=True, env=my_env)
      File "/usr/local/Cellar/python@3.10/3.10.13_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 526, in run
        raise CalledProcessError(retcode, process.args,
    subprocess.CalledProcessError: Command '['/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/cmake/data/bin/cmake', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/Users/nezda/code/itrex/build/lib.macosx-13-x86_64-cpython-310/intel_extension_for_transformers/llm/runtime/graph/', '-DCMAKE_RUNTIME_OUTPUT_DIRECTORY=/Users/nezda/code/itrex/build/lib.macosx-13-x86_64-cpython-310/intel_extension_for_transformers/llm/runtime/graph/', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_DEBUG=/Users/nezda/code/itrex/build/lib.macosx-13-x86_64-cpython-310/intel_extension_for_transformers/llm/runtime/graph/', '-DCMAKE_RUNTIME_OUTPUT_DIRECTORY_DEBUG=/Users/nezda/code/itrex/build/lib.macosx-13-x86_64-cpython-310/intel_extension_for_transformers/llm/runtime/graph/', '-DPYTHON_EXECUTABLE=/Users/nezda/.virtualenvs/billex_playground.3.10/bin/python', '-DCMAKE_BUILD_TYPE=Debug', '-DNE_VERSION_STRING=1.4.dev36+g05b66dbef5.d20240107', '-DDNNL_CPU_RUNTIME=OMP', '-DNE_WITH_AVX2=ON', '-DNE_WITH_TESTS=OFF', '-DNE_PYTHON_API=ON', '-DCMAKE_MAKE_PROGRAM:FILEPATH=/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/ninja/data/bin/ninja', '-GNinja', '/Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph']' returned non-zero exit status 1.
    /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/cmake/data/bin/cmake -DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/Users/nezda/code/itrex/build/lib.macosx-13-x86_64-cpython-310/intel_extension_for_transformers/llm/runtime/graph/ -DCMAKE_RUNTIME_OUTPUT_DIRECTORY=/Users/nezda/code/itrex/build/lib.macosx-13-x86_64-cpython-310/intel_extension_for_transformers/llm/runtime/graph/ -DCMAKE_LIBRARY_OUTPUT_DIRECTORY_DEBUG=/Users/nezda/code/itrex/build/lib.macosx-13-x86_64-cpython-310/intel_extension_for_transformers/llm/runtime/graph/ -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_DEBUG=/Users/nezda/code/itrex/build/lib.macosx-13-x86_64-cpython-310/intel_extension_for_transformers/llm/runtime/graph/ -DPYTHON_EXECUTABLE=/Users/nezda/.virtualenvs/billex_playground.3.10/bin/python -DCMAKE_BUILD_TYPE=Debug -DNE_VERSION_STRING=1.4.dev36+g05b66dbef5.d20240107 -DDNNL_CPU_RUNTIME=OMP -DNE_WITH_AVX2=ON -DNE_WITH_TESTS=OFF -DNE_PYTHON_API=ON -DCMAKE_MAKE_PROGRAM:FILEPATH=/Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/ninja/data/bin/ninja -GNinja /Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph
    error: subprocess-exited-with-error

    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> See above for output.

    note: This error originates from a subprocess, and is likely not a problem with pip.
    full command: /Users/nezda/.virtualenvs/billex_playground.3.10/bin/python -c '
    exec(compile('"'"''"'"''"'"'
    # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
    #
    # - It imports setuptools before invoking setup.py, to enable projects that directly
    #   import from `distutils.core` to work with newer packaging standards.
    # - It provides a clear error message when setuptools is not installed.
    # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
    #   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
    #     manifest_maker: standard file '"'"'-c'"'"' not found".
    # - It generates a shim setup.py, for handling setup.cfg-only projects.
    import os, sys, tokenize

    try:
        import setuptools
    except ImportError as error:
        print(
            "ERROR: Can not execute `setup.py` since setuptools is not available in "
            "the build environment.",
            file=sys.stderr,
        )
        sys.exit(1)

    __file__ = %r
    sys.argv[0] = __file__

    if os.path.exists(__file__):
        filename = __file__
        with tokenize.open(__file__) as f:
            setup_py_code = f.read()
    else:
        filename = "<auto-generated setuptools caller>"
        setup_py_code = "from setuptools import setup; setup()"

    exec(compile(setup_py_code, filename, "exec"))
    '"'"''"'"''"'"' % ('"'"'/Users/nezda/code/itrex/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' develop --no-deps
    cwd: /Users/nezda/code/itrex/
  Rolling back uninstall of intel-extension-for-transformers
  Moving to /Users/nezda/.virtualenvs/billex_playground.3.10/bin/neural_engine
   from /private/var/folders/q0/qzzpn5bd4wj8_mxbsv23wjq00000gn/T/pip-uninstall-xyjv8iqc/neural_engine
  Moving to /Users/nezda/.virtualenvs/billex_playground.3.10/bin/neuralchat
   from /private/var/folders/q0/qzzpn5bd4wj8_mxbsv23wjq00000gn/T/pip-uninstall-xyjv8iqc/neuralchat
  Moving to /Users/nezda/.virtualenvs/billex_playground.3.10/bin/neuralchat_client
   from /private/var/folders/q0/qzzpn5bd4wj8_mxbsv23wjq00000gn/T/pip-uninstall-xyjv8iqc/neuralchat_client
  Moving to /Users/nezda/.virtualenvs/billex_playground.3.10/bin/neuralchat_server
   from /private/var/folders/q0/qzzpn5bd4wj8_mxbsv23wjq00000gn/T/pip-uninstall-xyjv8iqc/neuralchat_server
  Moving to /Users/nezda/.virtualenvs/billex_playground.3.10/lib/python3.10/site-packages/intel-extension-for-transformers.egg-link
   from /private/var/folders/q0/qzzpn5bd4wj8_mxbsv23wjq00000gn/T/pip-uninstall-ckkp9wux/intel-extension-for-transformers.egg-link
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Tried being silly and taking out OpenMP::OpenMP_CXX, in addition to already currently deleted OpenMP::OpenMP_C, leaving nothing which of course didn't compile. Will rummage around to see if I can figure out how to install Intel OMP in my macos setup to get this code to properly compile with clang everywhere...

---

With my current, and only remotely successful configuration with the frankenstein compiler setup (C = clang C++ = g++) ...

For the performance, how slow did you mean by "very slowly"? As a reference, in a 10c20t Windows desktop 10900 (no avx512), I had ~1000ms first-token and ~300ms next-token (slower than what we can get on Linux and troubleshooting is on the way). So, what we can expect firstly should be at least 1000ms next-token performance.

Here's my test script:

from transformers import AutoTokenizer, TextStreamer
from intel_extension_for_transformers.transformers import AutoModelForCausalLM
model_name = "Mistral-7B-v0.1.git"
model_name = "neural-chat-7b-v3-1.git"

prompt = "Once upon a time, there existed a little girl,"
tokenizer = AutoTokenizer.from_pretrained (model_name, trust_remote_code=True)
inputs = tokenizer(prompt, return_tensors="pt").input_ids
model = AutoModelForCausalLM.from_pretrained(model_name, load_in_4bit=True)
streamer = TextStreamer(tokenizer)
outputs = model.generate(inputs, streamer=streamer, max_new_tokens=300)

Running it with python test.py (where model is previously quantized) it is pretty slow: I started a stopwatch on my phone and the first token arrived after about 47 seconds (following tokens also allow sufficient time to get bored by the whole experience: ~17s); all cores appear to be pinned at 100% based on Activity Monitor. If I artificially set "threads" to 8 (number of physical cores I think: https://www.techpowerup.com/cpu-specs/core-i9-9880h.c2289) seems to go a bit faster (~30s to first token, ~10s per following token): maybe memory bandwidth a little less destroyed? (... tried 4 threads too: little better - ~23s to first token, ~9s per following token).

In your professional opinion does my processor have any chance to catching up with yours for this task?

[DDEle]

Sorry for the late reply. If you or someone are still interested in running quantizeds model on Intel Mac:

does my processor have any chance to catching up with yours for this task?

I think so. Your i9-9880H should not be that slower than 10c20t desktop 10900, especially for next-token, which is typically memory bound. However, as we don't have a similar device as 2019 Intel MacBook Pro, it is difficult to locate performance issues for you. Personally, I think ~10s to first token, ~3s per following token should be an achievable goal.

[nezda]

All good @DDEle . I am still interested in this. I kinda forget where we left off - did you have a recommended next step for me?