On demand feature view not supporting objects

[roy651]

Expected Behavior

@on_demand_feature_view(
   inputs={
       'driver_hourly_stats': driver_hourly_stats_view
   },
   features=[
     Feature(name='conv_rate_plus_val1', dtype=ValueType.DOUBLE),
     Feature(name='conv_rate_plus_val2', dtype=ValueType.DOUBLE),
     Feature(name='conv_rate_plus_val3', dtype=ValueType.STRING)
   ]
)
def transformed_conv_rate(inputs: pd.DataFrame) -> pd.DataFrame:
    df = pd.DataFrame()
    df['conv_rate_plus_val1'] = (inputs['conv_rate'] + 1)
    df['conv_rate_plus_val2'] = (inputs['conv_rate'] + 1)
    df['conv_rate_plus_val3'] = str(inputs['conv_rate'])
    return df

Should process fine when applying.

Current Behavior

Traceback (most recent call last):
  File "/Users/user/anaconda3/envs/feast/bin/feast", line 8, in <module>
    sys.exit(cli())
  File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/feast/cli.py", line 390, in apply_total_command
    apply_total(repo_config, repo, skip_source_validation)
  File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/feast/usage.py", line 269, in wrapper
    return func(*args, **kwargs)
  File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/feast/repo_operations.py", line 260, in apply_total
    diff = store.apply(all_to_apply, objects_to_delete=all_to_delete, partial=False)
  File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/feast/usage.py", line 280, in wrapper
    raise exc.with_traceback(traceback)
  File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/feast/usage.py", line 269, in wrapper
    return func(*args, **kwargs)
  File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/feast/feature_store.py", line 579, in apply
    odfv.infer_features()
  File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/feast/on_demand_feature_view.py", line 226, in infer_features
    name=f, dtype=python_type_to_feast_value_type(f, type_name=str(dt))
  File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/feast/type_map.py", line 162, in python_type_to_feast_value_type
    assert value
AssertionError

Steps to reproduce

Get the on demand feature view demo from: https://github.com/feast-dev/feast-demo/blob/main/feature_repo/features.py
Add the 2 lines as noted in the snippet above for the new STRING feature: conv_rate_plus_val3

Specifications

• Version: 0.17.0
• Platform: Mac
• Subsystem: 12.0.1

Possible Solution

I might be doing something wrong or expecting something which shouldn't be supported. Would love to hear feedback.
Incase this turns out to be a bug, this might be a possible solution --
The type map defined at:

feast/sdk/python/feast/type_map.py

Line 94 in 9dc9e60

type_map = {

Should possibly include an entry for pandas obj dtype?

[pyalex]

Hi @roy651. Thanks for reporting this. I understand that exception information is very confusing. We should add some details to the assertion or better raise proper exceptions.

That being said, this line is not correct

df['conv_rate_plus_val3'] = str(inputs['conv_rate'])

If you need to convert column to string the correct way would be:

df['conv_rate_plus_val3'] = inputs['conv_rate'].astype(str)

Otherwise, you create a column with dtype object, which is intentionally not supported by Feast.

[roy651]

Thanks @pyalex for the guidance to use astype instead of the plain casting, however, in practice it yields the same result.
Using this code:

@on_demand_feature_view(
   inputs={
       'driver_hourly_stats': driver_hourly_stats_view
   },
   features=[
     Feature(name='conv_rate_plus_val1', dtype=ValueType.DOUBLE),
     Feature(name='conv_rate_plus_val2', dtype=ValueType.DOUBLE),
     Feature(name='conv_rate_plus_val3', dtype=ValueType.STRING)
   ]
)
def transformed_conv_rate(inputs: pd.DataFrame) -> pd.DataFrame:
    df = pd.DataFrame()
    df['conv_rate_plus_val1'] = (inputs['conv_rate'] + 1)
    df['conv_rate_plus_val2'] = (inputs['conv_rate'] + 1)
    df['conv_rate_plus_val3'] = inputs['conv_rate'].astype("string")
    print(f"###df.dtypes:\n{df.dtypes}")
    return df

Crashes similarly with:

###df.dtypes:
conv_rate_plus_val1    float64
conv_rate_plus_val2    float64
conv_rate_plus_val3     object
dtype: object
>>>type_name:object
Traceback (most recent call last):
  File "/Users/user/anaconda3/envs/feast_wo/bin/feast", line 8, in <module>
    sys.exit(cli())
  File "/Users/user/anaconda3/envs/feast_wo/lib/python3.10/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/user/anaconda3/envs/feast_wo/lib/python3.10/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/user/anaconda3/envs/feast_wo/lib/python3.10/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/user/anaconda3/envs/feast_wo/lib/python3.10/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/user/anaconda3/envs/feast_wo/lib/python3.10/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/user/anaconda3/envs/feast_wo/lib/python3.10/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/user/anaconda3/envs/feast_wo/lib/python3.10/site-packages/feast/cli.py", line 390, in apply_total_command
    apply_total(repo_config, repo, skip_source_validation)
  File "/Users/user/anaconda3/envs/feast_wo/lib/python3.10/site-packages/feast/usage.py", line 269, in wrapper
    return func(*args, **kwargs)
  File "/Users/user/anaconda3/envs/feast_wo/lib/python3.10/site-packages/feast/repo_operations.py", line 260, in apply_total
    diff = store.apply(all_to_apply, objects_to_delete=all_to_delete, partial=False)
  File "/Users/user/anaconda3/envs/feast_wo/lib/python3.10/site-packages/feast/usage.py", line 280, in wrapper
    raise exc.with_traceback(traceback)
  File "/Users/user/anaconda3/envs/feast_wo/lib/python3.10/site-packages/feast/usage.py", line 269, in wrapper
    return func(*args, **kwargs)
  File "/Users/user/anaconda3/envs/feast_wo/lib/python3.10/site-packages/feast/feature_store.py", line 579, in apply
    odfv.infer_features()
  File "/Users/user/anaconda3/envs/feast_wo/lib/python3.10/site-packages/feast/on_demand_feature_view.py", line 226, in infer_features
    name=f, dtype=python_type_to_feast_value_type(f, type_name=str(dt))
  File "/Users/user/anaconda3/envs/feast_wo/lib/python3.10/site-packages/feast/type_map.py", line 163, in python_type_to_feast_value_type
    assert value
AssertionError

Note that I've added a couple of debug prints, the last of which comes from within type_map.py.

A quick search reveals that Pandas relies upon Numpy for typing and, in short, because Numpy treats strings as arrays they become objects. There is an alternative approach, starting from Pandas 1.0.0, but it supports the use of string type and not str as explained in the Pandas documentation.

I saw that the type_map.py already has a newer version (than 0.17.0) with a different flow for the type check but I believe it will fail in a similar manner as the type map still doesn't contain string or obj.

[roy651]

@pyalex Just to clarify the point - This is obviously just an example, but as it stands, I couldn't get any strings to work in the transformed ODFV.
Hence the (previous) title which I gave the issue.

[pyalex]

@roy651, you're right. This is my mistake. It always will be dtype object unless we use special pandas.StringType. But we currently are not converting either, because even pandas.StringType will have the name string, which is not in the list

I will increase priority and return the bug label. We're gonna fix it ASAP.