Issues serializing a model

[dnoliver]

Hi,

I am having the following issue serializing the pipeline when I use the OpenVINOGenerator.
The following code reproduce the issue:

from fastrag.generators.openvino import OpenVINOGenerator
from haystack import Pipeline

# Create OpenVINO generator
generator = OpenVINOGenerator(
    model="microsoft/Phi-3-mini-4k-instruct",
    compressed_model_dir="OpenVINO/Phi-3-mini-4k-instruct-int4-ov",
    device_openvino="GPU",
    task="text-generation",
    generation_kwargs={
        "max_new_tokens": 100,
    }
)

# Create a pipeline
pipeline = Pipeline()

# Add the generator to the pipeline
pipeline.add_component("generator", generator)

# Run the pipeline
results = pipeline.run({"generator": {"prompt": "Who is the best American actor?"}})

# Print the results
print(results)

# Serialize the pipeline into a file
with open("pipeline.yml", "w") as file:
    pipeline.dump(file)

The model runs fine, but the serialization part fails (pipeline.dump(file)). This is the error message:

(haystack) PS C:\Users\Nicolas Oliver\Downloads\haystack-101> python .\main.py
{'generator': {'replies': ['\n\n# Answer\nDetermining the "best" American actor is subjective and can vary based on personal preferences, acting styles, and the genres of movies or plays one enjoys. However, some actors are widely recognized for their exceptional talent and have received critical acclaim and numerous awards throughout their careers. Here are a few actors who are often celebrated for their acting prowess:\n\n1. **Meryl Streep** - Known for']}}

Traceback (most recent call last):
  File "C:\Users\Nicolas Oliver\Downloads\haystack-101\main.py", line 29, in <module>
    pipeline.dump(file)
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\haystack\core\pipeline\pipeline.py", line 213, in dump
    fp.write(marshaller.marshal(self.to_dict()))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\haystack\marshal\yaml.py", line 9, in marshal
    return yaml.dump(dict_)
           ^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\__init__.py", line 253, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\__init__.py", line 241, in dump_all
    dumper.represent(data)
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 27, in represent
    node = self.represent_data(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 342, in represent_object
    return self.represent_mapping(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Nicolas Oliver\.conda\envs\haystack\Lib\site-packages\yaml\representer.py", line 317, in represent_object
    reduce = data.__reduce_ex__(2)
             ^^^^^^^^^^^^^^^^^^^^^
TypeError: cannot pickle 'Model' object

The intention of serializing the pipeline is to get it deployed with HayHooks. Any hints why this might be happening? is it an issue with this library or with the HayStack framework?

Thanks!

[danielfleischer]

We'll look into it.

[danielfleischer]

Please try now, and serialize the pipeline before running it, which initializes the model.

[dnoliver]

Yes, that change makes it work. I tried fastrag built from source at d9097e5, and installed with pip. The one built from source produces a yaml file.

One note: v3.0.1 is still present in init.py at

fastRAG/fastrag/__init__.py

Line 4 in d9097e5

__version__ = "3.0.1"

, so you get a version 3.0.1 when built from source.

So, for this input (with changed models to Phi3.5 instead of Phi3):

# Create OpenVINO generator
generator = OpenVINOGenerator(
    model="microsoft/Phi-3.5-mini-instruct",
    compressed_model_dir="lokinfey/Phi-3.5-mini-instruct-ov-int4",
    device_openvino="GPU",
    task="text-generation",
    generation_kwargs={
        "max_new_tokens": 100,
    }
)

I get this Pipeline YAML:

components:
  generator:
    init_parameters:
      generation_kwargs:
        max_new_tokens: 100
        return_full_text: false
      huggingface_pipeline_kwargs:
        device: cpu
        model: microsoft/Phi-3.5-mini-instruct
        task: text-generation
      stop_words: null
      streaming_callback: null
      token:
        env_vars:
        - HF_API_TOKEN
        strict: false
        type: env_var
    type: fastrag.generators.openvino.OpenVINOGenerator
connections: []
max_loops_allowed: 100
metadata: {}

The YAML file doesn't have compressed_model_dir and device_openvino parameters set in the OpenVINOGenerator. And trying to load the pipeline with hayhooks fails:

# In one terminal: 
hayhooks run

# In another terminal:
hayhooks deploy test.yml 

# Hayhooks output is:
INFO:     Pipelines dir set to: pipelines.d
INFO:     Started server process [26620]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://localhost:1416 (Press CTRL+C to quit)
INFO:     ::1:50254 - "POST /deploy HTTP/1.1" 409 Conflict

# Deploy output is
Error deploying pipeline: Unable to parse Haystack Pipeline test: Component 'fastrag.generators.openvino.OpenVINOGenerator' not imported.

I think the deployment error is unrelated to the serialization problem. Because hayhooks fails before actually parsing the file. There is a reported issue related to this problem at deepset-ai/hayhooks#36, This comment deepset-ai/hayhooks#36 (comment) says you have to decorate it with @component to make it work? => Guess this is a different issue

So in summary:

1. Can't serialize => Solved
2. Parameters set to OpenVINOGenerator not present in serialized output => Open
3. OpenVINOGenerator component can't be imported by hayhooks => Open (but another issue I guess)

[danielfleischer]

Please, check out the latest code and see if the issues are resolved.

[dnoliver]

Yes, latest commit produces the following yaml

components:
  generator:
    init_parameters:
      compressed_model_dir: OpenVINO/Phi-3-mini-4k-instruct-int4-ov
      device_openvino: GPU
      generation_kwargs:
        max_new_tokens: 100
        return_full_text: false
      huggingface_pipeline_kwargs:
        device: cpu
        model: microsoft/Phi-3-mini-4k-instruct
        task: text-generation
      model: microsoft/Phi-3-mini-4k-instruct
      ov_config:
        CACHE_DIR: ''
        NUM_STREAMS: '1'
        PERFORMANCE_HINT: LATENCY
      stop_words: null
      streaming_callback: null
      token:
        env_vars:
        - HF_API_TOKEN
        strict: false
        type: env_var
    type: fastrag.generators.openvino.OpenVINOGenerator
connections: []
max_loops_allowed: 100
metadata: {}

Deployment on hayhooks==0.0.17 works, and exercising the pipeline also works fine!

The only thing I wanted to double check was about streaming callbacks. I did a quick test, with streaming_callback=lambda text: print(text),, I get a streaming_callback: __main__.<lambda> in the yml file, and then the pipeline becomes undeployable. But I wasn't expecting that callback to be serializable. By looking at https://github.com/deepset-ai/haystack/blob/main/haystack/utils/callable_serialization.py, I think there is more work to do on my side to get my callback to be serializable,deserializable (perhaps in a custom component or something like that)

So, all good now! Closing this issue as fixed.

[danielfleischer]

Great! if you have some insights regarding serialization of callbacks, please feel free to open an issue or even a PR.