[Bug]: MacOS Import Errors (mach-o file & incompatible architecture)

[Roblouisck]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What happened?

Getting multiple errors specifying:

(mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

Full:

Successfully installed PySocks-1.7.1 beautifulsoup4-4.11.1 gdown-4.5.3 psutil-5.9.4 soupsieve-2.3.2.post1
To make your changes take effect please reactivate your environment
=============================================
=============================================
==============MORE INFORMATION===============
=============================================
=============================================
If you want to run the web UI again, you can run the following command:
./stable-diffusion-webui/run_webui_mac.sh
or
cd stable-diffusion-webui && ./run_webui_mac.sh
=============================================
=============================================
=============================================
=============================================
Traceback (most recent call last):
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 872, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/transformers/tokenization_utils.py", line 26, in <module>
    from .tokenization_utils_base import (
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 73, in <module>
    from tokenizers import AddedToken
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/tokenizers/__init__.py", line 79, in <module>
    from .tokenizers import (
ImportError: dlopen(/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/tokenizers/tokenizers.cpython-310-darwin.so, 0x0002): tried: '/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/tokenizers/tokenizers.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/tokenizers/tokenizers.cpython-310-darwin.so' (no such file), '/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/tokenizers/tokenizers.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 872, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/transformers/models/__init__.py", line 19, in <module>
    from . import (
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/transformers/models/layoutlm/__init__.py", line 28, in <module>
    from .configuration_layoutlm import LAYOUTLM_PRETRAINED_CONFIG_ARCHIVE_MAP, LayoutLMConfig
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/transformers/models/layoutlm/configuration_layoutlm.py", line 19, in <module>
    from transformers import PretrainedConfig, PreTrainedTokenizer, TensorType
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 862, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 874, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.tokenization_utils because of the following error (look up to see its traceback):
dlopen(/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/tokenizers/tokenizers.cpython-310-darwin.so, 0x0002): tried: '/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/tokenizers/tokenizers.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/tokenizers/tokenizers.cpython-310-darwin.so' (no such file), '/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/tokenizers/tokenizers.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/robert/stable-diffusion-webui/webui.py", line 13, in <module>
    from modules import devices, sd_samplers, upscaler, extensions, localization
  File "/Users/robert/stable-diffusion-webui/modules/sd_samplers.py", line 11, in <module>
    from modules import prompt_parser, devices, processing, images
  File "/Users/robert/stable-diffusion-webui/modules/processing.py", line 14, in <module>
    import modules.sd_hijack
  File "/Users/robert/stable-diffusion-webui/modules/sd_hijack.py", line 10, in <module>
    import modules.textual_inversion.textual_inversion
  File "/Users/robert/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 13, in <module>
    from modules import shared, devices, sd_hijack, processing, sd_models, images
  File "/Users/robert/stable-diffusion-webui/modules/shared.py", line 15, in <module>
    import modules.sd_models
  File "/Users/robert/stable-diffusion-webui/modules/sd_models.py", line 14, in <module>
    from modules.sd_hijack_inpainting import do_inpainting_hijack, should_hijack_inpainting
  File "/Users/robert/stable-diffusion-webui/modules/sd_hijack_inpainting.py", line 6, in <module>
    import ldm.models.diffusion.ddpm
  File "/Users/robert/stable-diffusion-webui/repositories/stable-diffusion/ldm/models/diffusion/ddpm.py", line 12, in <module>
    import pytorch_lightning as pl
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/pytorch_lightning/__init__.py", line 34, in <module>
    from pytorch_lightning.callbacks import Callback  # noqa: E402
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/pytorch_lightning/callbacks/__init__.py", line 14, in <module>
    from pytorch_lightning.callbacks.callback import Callback
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/pytorch_lightning/callbacks/callback.py", line 25, in <module>
    from pytorch_lightning.utilities.types import STEP_OUTPUT
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/pytorch_lightning/utilities/types.py", line 28, in <module>
    from torchmetrics import Metric
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/torchmetrics/__init__.py", line 14, in <module>
    from torchmetrics import functional  # noqa: E402
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/torchmetrics/functional/__init__.py", line 77, in <module>
    from torchmetrics.functional.text.bleu import bleu_score
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/torchmetrics/functional/text/__init__.py", line 30, in <module>
    from torchmetrics.functional.text.bert import bert_score  # noqa: F401
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/torchmetrics/functional/text/bert.py", line 24, in <module>
    from torchmetrics.functional.text.helper_embedding_metric import (
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/torchmetrics/functional/text/helper_embedding_metric.py", line 26, in <module>
    from transformers import AutoModelForMaskedLM, AutoTokenizer, PreTrainedModel, PreTrainedTokenizerBase
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 862, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 874, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.models.auto because of the following error (look up to see its traceback):
Failed to import transformers.tokenization_utils because of the following error (look up to see its traceback):
dlopen(/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/tokenizers/tokenizers.cpython-310-darwin.so, 0x0002): tried: '/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/tokenizers/tokenizers.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/tokenizers/tokenizers.cpython-310-darwin.so' (no such file), '/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/tokenizers/tokenizers.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))
(web-ui) robert@Roberts-MacBook-Pro ~ % 

Steps to reproduce the problem

1. Go to ....
2. Press ....
3. ...

What should have happened?

continue w/ install as normal

Commit where the problem happens

4bbe535

What platforms do you use to access UI ?

MacOS

What browsers do you use to access the UI ?

No response

Command Line Arguments

No response

Additional information, context and logs

No response

[0xdevalias]

The full error is this:

ImportError: dlopen(/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/tokenizers/tokenizers.cpython-310-darwin.so, 0x0002): tried: '/Users/robert/miniconda/envs/web-ui/lib/python3.10/site-packages/tokenizers/tokenizers.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

This issue seems relevant, and seems to include solutions:

• wrong architecture tokenizers.cpython-39-darwin.so (x86_64) when installing on apple silicon (arm64) huggingface/tokenizers#712

---

Hi there !

I've manually build binaries for tokenizers on arm m1 and released them for tokenizers 0.11.6.

We'll try our best to keep building those by hand while waiting for actions/runner#805.

Expect some delay between normal releases and m1 releases for now :)

Have a great day !

Originally posted by @McPatate in huggingface/tokenizers#712 (comment)

tokenizers==0.13.0 should now be built automatically for M1.

Originally posted by @Narsil in huggingface/tokenizers#712 (comment)