OneHotEncoder can accidentally create columns with same name

[lars-reimann]

Describe the bug

The OneHotEncoder uses the schema <old_column_name>_<value> to name the created columns. This can lead to conflicts, however.

To Reproduce

Run this program:

from safeds.data.tabular.containers import Table
from safeds.data.tabular.transformation import OneHotEncoder

if __name__ == '__main__':
    table = Table.from_dict({"a_b": ["c"], "a": ["b_c"]})
    transformed_table = OneHotEncoder().fit_and_transform(table)

    print(transformed_table)

It raises an exception:

ValueError: Length mismatch: Expected axis has 2 elements, new values have 1 elements

The issue is that two columns with the same name (a_b_c) get created.

Expected behavior

No exception. The names of all created columns should be unique. They should also not conflict with existing columns in the Table. This can be done by detecting conflicts between two created columns or between a created column and an existing, unchanged column and appending a suffix _<counter> to the names of the created columns (e.g. a_b_c_1 vs. a_b_c_2).

Screenshots (optional)

No response

Additional Context (optional)

No response

[lars-reimann]

For better readability I'd suggest to use the schema <column_name>__<value>(#<counter>)? (e.g. color__blue or color__red#2) for the names of the columns created by the OneHotEncoder. Double underscores should be rare so in many cases we won't even need a counter. And it makes it easier for users to figure out what is column name and what is value if either contain single underscores.

Only the names of duplicates need to have counter. The first occurrence needn't be changed. Counting should start at two. Example:

• color__red
• color__red#2
• color__red#3
• ...

[lars-reimann]

Also useful: https://docs.python.org/3/library/collections.html#collections.Counter

[zzril]

We decided that we will implement the OneHotEncoder ourselfves, instead of using the one from scikit-learn.

We should also add performance tests to verify that our implementation is as effecient the one in scikit-learn. The tests should be performant on several large datasets.
(These tests do not need to be run by pytest automatically.)

[lars-reimann]

🎉 This issue has been resolved in version 0.12.0 🎉

The release is available on:

• v0.12.0
• GitHub release

Your semantic-release bot 📦🚀