kikuchipy and hyperspy with pyinstaller #581

Erlendos12 · 2022-12-03T12:51:32Z

Erlendos12
Dec 3, 2022

Hi!
Background:
We are a group of stuents currently developing a graphical user interface using kikutchipy and hyperspy, and want to package our project using pyinstaller into a single bundle with an executable that can run on Windows and MacOS. However pyinstaller has some problems when packagining hyperspy and the kikutchipy libraries, think it might have something to do with what pyinstaller calls hidden imports: https://pyinstaller.org/en/stable/when-things-go-wrong.html#listing-hidden-imports.

Previous problems:
For example when running the created executable, we first get an uncaught exception that module kikutchipy.io._io does not exist. We work around this by importing it into our main.py script, so pyinstaller will import it. However more missing module exceptions appear, also from hyperspy, e.g. hyperspy._components.eels_arctan. Therefor we have added a lot of different imports to our main.py, and in the end the missing module exceptions are no longer. Here is a list of the extra imports (not sure that every import is needed):

import hyperspy
import hyperspy._components.eels_arctan
import hyperspy._components.bleasdale
import hyperspy._components.doniach
import hyperspy._components.eels_double_power_law
import hyperspy._components.eels_arctan
import hyperspy._components.eels_cl_edge
import hyperspy._components.error_function
import hyperspy._components.exponential
import hyperspy._components.expression
import hyperspy._components.gaussian
import hyperspy._components.gaussianhf
import hyperspy._components.heaviside
import hyperspy._components.logistic
import hyperspy._components.lorentzian
import hyperspy._components.offset
import hyperspy._components.pes_core_line_shape
import hyperspy._components.pes_voigt
import hyperspy._components.polynomial_deprecated
import hyperspy._components.polynomial
import hyperspy._components.power_law
import hyperspy._components.rc
import hyperspy._components.pes_see
import hyperspy._components.scalable_fixed_pattern
import hyperspy._components.skew_normal
import hyperspy._components.split_voigt
import hyperspy._components.eels_vignetting
import hyperspy._components.pes_voigt
import hyperspy._components.volume_plasmon_drude
import hyperspy._components.expression
import hyperspy._components.gaussian2d

import kikuchipy 
import kikuchipy.generators.virtual_bse_generator
import kikuchipy.generators._transfer_axes
import kikuchipy.io._io
import kikuchipy._util
import kikuchipy.io._util
import kikuchipy.data._data
import kikuchipy.indexing._dictionary_indexing
import kikuchipy.indexing._merge_crystal_maps
import kikuchipy.indexing._orientation_similarity_map
import kikuchipy.indexing._refinement
import kikuchipy.pattern._pattern
import kikuchipy.simulations._kikuchi_pattern_features
import kikuchipy.simulations._kikuchi_pattern_simulation
import kikuchipy.detectors.ebsd_detector
import kikuchipy.signals._kikuchipy_signal
from kikuchipy.signals._kikuchipy_signal import KikuchipySignal2D, LazyKikuchipySignal2D
import kikuchipy.signals._kikuchi_master_pattern
import kikuchipy.signals.ebsd
import kikuchipy.signals.ebsd_master_pattern
import kikuchipy.signals.ecp_master_pattern
import kikuchipy.signals.virtual_bse_image

After this the final bundle is still missing hyperspy/hyperspy_extension.yaml and kikuchipy/hyperspy_extension.yaml. We can however add these to the installer and place them correctly. We use auto-py-to-exe which is just a graphical user interface for pyinstaller, making this process easier.

NOW our application runs!

Current problem:
We have created an interactive python interpreter in our application where we can type python code and access the running environment. The following command gives the following output.

> hyperspy.utils.print_known_signal_types()

+--------------------+---------------------+--------------------+----------+
|    signal_type     |       aliases       |     class name     | package  |
+--------------------+---------------------+--------------------+----------+
| DielectricFunction | dielectric function | DielectricFunction | hyperspy |
|      EDS_SEM       |                     |   EDSSEMSpectrum   | hyperspy |
|      EDS_TEM       |                     |   EDSTEMSpectrum   | hyperspy |
|        EELS        |       TEM EELS      |    EELSSpectrum    | hyperspy |
|      hologram      |                     |   HologramImage    | hyperspy |
+--------------------+---------------------+--------------------+----------+

So it appears that in our executable, hyperspy does not recognize signals from kikutchipy, however it does when running it in our virtual python environment used when developing. This creates errors in our executable, when trying to run some kikutchipy functions that rely on hyperspy, mainly when we try to run s.remove_static_background() we get the following callstack with an error.

Call stack:
  File "main.py", line 384, in <module>

  File "main.py", line 140, in <lambda>

  File "main.py", line 230, in selectProcessing

  File "scripts\pattern_processing.py", line 62, in <lambda>

  File "scripts\pattern_processing.py", line 113, in preview_processing

  File "scripts\pattern_processing.py", line 94, in remove_static

  File "kikuchipy\signals\ebsd.py", line 383, in remove_static_background

  File "hyperspy\signal.py", line 4904, in map

  File "hyperspy\signal.py", line 5081, in _map_iterate

  File "kikuchipy\signals\_kikuchipy_signal.py", line 399, in _assign_subclass

  File "hyperspy\signal.py", line 5700, in _assign_subclass

  File "hyperspy\io.py", line 651, in assign_signal_subclass

  File "logging\__init__.py", line 1489, in warning

  File "logging\__init__.py", line 1624, in _log

  File "logging\__init__.py", line 1634, in handle

  File "logging\__init__.py", line 1696, in callHandlers

  File "logging\__init__.py", line 968, in handle

  File "logging\__init__.py", line 1108, in emit

Message: "`signal_type='EBSD'` not understood. See `hs.print_known_signal_types()` for a list of installed signal types or https://github.com/hyperspy/hyperspy-extensions-list for the list of all hyperspy extensions providing signals."
Arguments: ()
Traceback (most recent call last):
  File "scripts\pattern_processing.py", line 62, in <lambda>
  File "scripts\pattern_processing.py", line 113, in preview_processing
  File "scripts\pattern_processing.py", line 94, in remove_static
  File "kikuchipy\signals\ebsd.py", line 394, in remove_static_background
AttributeError
:
'LazySignal2D' object has no attribute '_set_custom_properties'

So it seems that after running map, hyperspy is not able to return the kikutchipy signal type, which first produces a warning. Therefor it is not able to call _set_custom_properties() on the returned signal, because it is a hyperspy signal instead of kikutchipy signal.

Is there a workaround for this, or a different way of packaging hyperspy and kikutchipy with pyinstaller, at least so that kikutchipy signals are available in hyperspy?

Our repo is avialable at: https://github.com/htrellin/EBSD-GUI/tree/dev

Answered by hakonanes

Dec 4, 2022

Hi @Erlendos12,

thank you for starting this discussion here. I'm not familiar with PyInstaller, so I'm afraid you have to debug this yourself. But, I can try to diagnose the problem, at least. And perhaps identify some possible solutions.

The AttributeError in the error message most likely results from the EBSD.remove_static_background() method calling the LazySignal2D.map() method, which does not return a LazyEBSD instance but a LazySignal2D instance. This is because HyperSpy doesn't recognize the kikuchipy signals as subclasses of its classes.

The kikuchipy/hyperspy_extension.yaml file must be present when installing kikuchipy, otherwise HyperSpy will not know of kikuchipy's subclasses.…

View full answer

hakonanes · 2022-12-04T11:31:27Z

hakonanes
Dec 4, 2022
Maintainer

Hi @Erlendos12,

thank you for starting this discussion here. I'm not familiar with PyInstaller, so I'm afraid you have to debug this yourself. But, I can try to diagnose the problem, at least. And perhaps identify some possible solutions.

The AttributeError in the error message most likely results from the EBSD.remove_static_background() method calling the LazySignal2D.map() method, which does not return a LazyEBSD instance but a LazySignal2D instance. This is because HyperSpy doesn't recognize the kikuchipy signals as subclasses of its classes.

The kikuchipy/hyperspy_extension.yaml file must be present when installing kikuchipy, otherwise HyperSpy will not know of kikuchipy's subclasses. Might it be that this file is not present when PyInstaller runs its "Analysis"? After reading a bit of PyInstaller's docs (thank you for linking to the relevant page) and a similar issue pyinstaller/pyinstaller#4984 reported in their issue tracker, it seems like listing the .yaml file in a hook file could be a solution? It seems like you can write specific hook files for these "hidden imports" as well (see their docs on their hook files).

For completeness, I should mention that HyperSpy and kikuchipy uses "lazy imports", which might be why PyInstaller's "Analysis" fails to find some imports. This is a feature of later Python versions, documented in PEP 562 (Python Enhancement Proposal number 562). The import list in the kikuchipy/kikuchipy/signals/ebsd.py file is long, and can take 5-7 seconds to import on my five year old Linux machine. Lazy importing means that these imports are not loaded when we do import kikuchipy as kp (less than 1 second on my machine), only once kp.signals.EBSD is directly accessed (the expected 5-7 seconds).

6 replies

Erlendos12 Dec 4, 2022
Author

Got it finally working!! I wrote hooks for both kikuchipy and hyperspy.

I do however get a logging error, when hyperspy is printing that it is enabling kikuchipy.

--- Logging error --- 
Traceback (most recent call last):    
File "logging\__init__.py", line 1103, in emit
  AttributeError: 'NoneType' object has no attribute 'write'  Call stack:   
File "main.py", line 332, in <module>   
File "main.py", line 88, in <lambda>    
File "main.py", line 174, in selectProcessing    
File "scripts\pattern_processing.py", line 39, in __init__    
File "kikuchipy\__init__.py", line 92, in __getattr__     return getattr(importlib.import_module(import_path), name)    
File "importlib\__init__.py", line 126, in import_module    
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import    
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load    
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked    
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked    
File "PyInstaller\loader\pyimod02_importers.py", line 499, in exec_module    
File "kikuchipy\io\_io.py", line 23, in <module>     from hyperspy.io_plugins import hspy    
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load    
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked    
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked    
File "PyInstaller\loader\pyimod02_importers.py", line 499, in exec_module    
File "hyperspy\io_plugins\__init__.py", line 22, in <module>     from hyperspy.io_plugins import (    
File "<frozen importlib._bootstrap>", line 1078, in _handle_fromlist    
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed    
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load    
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked    
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked    
File "PyInstaller\loader\pyimod02_importers.py", line 499, in exec_module    
File "hyperspy\io_plugins\blockfile.py", line 36, in <module>     from hyperspy.misc.utils import dummy_context_manager    
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load    
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked    
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked    
File "PyInstaller\loader\pyimod02_importers.py", line 499, in exec_module    
File "hyperspy\misc\utils.py", line 36, in <module>     from hyperspy.misc.signal_tools import broadcast_signals    
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load    
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked   
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked    
File "PyInstaller\loader\pyimod02_importers.py", line 499, in exec_module    
File "hyperspy\misc\signal_tools.py", line 24, in <module>     from hyperspy.misc.axis_tools import check_axes_calibration    
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load    
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked    
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked    
File "PyInstaller\loader\pyimod02_importers.py", line 499, in exec_module    
File "hyperspy\misc\axis_tools.py", line 20, in <module>     from hyperspy.api_nogui import _ureg    
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load    
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked    
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked    
File "PyInstaller\loader\pyimod02_importers.py", line 499, in exec_module    
File "hyperspy\api_nogui.py", line 28, in <module>     from hyperspy.defaults_parser import preferences    
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load    
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked    
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked    
File "PyInstaller\loader\pyimod02_importers.py", line 499, in exec_module    
File "hyperspy\defaults_parser.py", line 29, in <module>     from hyperspy.ui_registry import add_gui_method    
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load    
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked    
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked    
File "PyInstaller\loader\pyimod02_importers.py", line 499, in exec_module    
File "hyperspy\ui_registry.py", line 34, in <module>     from hyperspy.extensions import ALL_EXTENSIONS    
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load    
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked    
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked    
File "PyInstaller\loader\pyimod02_importers.py", line 499, in exec_module    
File "hyperspy\extensions.py", line 44, in <module>     _logger.info("Enabling extension %s" % _external_extension.name)    
File "logging\__init__.py", line 1477, in info    
File "logging\__init__.py", line 1624, in _log    
File "logging\__init__.py", line 1634, in handle    
File "logging\__init__.py", line 1696, in callHandlers    
File "logging\__init__.py", line 968, in handle    
File "logging\__init__.py", line 1108, in emit  

Message: 'Enabling extension kikuchipy' 
Arguments: ()

Any idea why it is giving this error, or is it possible to disable these messages from hyperspy? print() statements usually does not produce a logging error in our console.

hakonanes Dec 4, 2022
Maintainer

Hm, can you try setting HyperSpy's logging level to DEBUG?

>>> import hyperspy.api as hs
>>> hs.set_log_level("DEBUG")

Erlendos12 Dec 4, 2022
Author

Thanks! Setting it to "CRITICAL" did the trick.

hakonanes Dec 4, 2022
Maintainer

Glad you got it working. Let us know here if you have any other issues related to kikuchipy combined with HyperSpy!

If you find my answer satisfactory, please mark it as an answer (:

hakonanes Dec 4, 2022
Maintainer

By "here" I mean in another issue or discussion.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kikuchipy and hyperspy with pyinstaller #581

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

kikuchipy and hyperspy with pyinstaller #581

Erlendos12 Dec 3, 2022

Replies: 1 comment · 6 replies

hakonanes Dec 4, 2022 Maintainer

Erlendos12 Dec 4, 2022 Author

hakonanes Dec 4, 2022 Maintainer

Erlendos12 Dec 4, 2022 Author

hakonanes Dec 4, 2022 Maintainer

hakonanes Dec 4, 2022 Maintainer

Erlendos12
Dec 3, 2022

Replies: 1 comment 6 replies

hakonanes
Dec 4, 2022
Maintainer

Erlendos12 Dec 4, 2022
Author

hakonanes Dec 4, 2022
Maintainer

Erlendos12 Dec 4, 2022
Author

hakonanes Dec 4, 2022
Maintainer

hakonanes Dec 4, 2022
Maintainer