GreaterThan constraint crashing with NaN values

[fealho]

As reported in issue #590 and #589, the GreaterThan seems to be failing in the presence of NaN values. The cause seems to be the following line:

SDV/sdv/metadata/table.py

Line 593 in 061d69b

if not constraint.is_valid(data).all():

Basically, since NaN values are not comparable, this line is returning false for the comparison between NaN's, which causes the code to crash.

We should work on a fix, seeing as this error happens with our demo dataset (student_placements).

[npatki]

FYI same thing is happening with ColumnFormula -- raised in #593

Under the hood, the is_valid methods for each constraint check for validity. In this case, GreaterThan seems to be using np.greater or np.greater_equal. These will output False whenever they encounter NaN values.

I think we should just ignore NaN values since they clearly don't apply to a GreaterThan constraint.

[katxiao]

@npatki I have a follow-up to the proposal of ignoring NaN values.

If we have a GreaterThan constraint on two columns a and b (a > b), I think it makes sense to ignore a row where both a and b are NaN, and return True for that row. However, if a is non-null and b is null, that row seems invalid and I think we should return False in that case.

What do you think?

[npatki]

@katxiao To me, the most straightforward interpretation of GreaterThan is: a > b if and only if a is present and b is present. We should return True whenever the > operator doesn't really apply.

1. Both values are missing or
2. One of the values is missing

Otherwise, I'm not sure how the user would handle the situation where certain values may be missing randomly.

[npatki]

One motivating example to the above: In the Loan dataset on Kaggle, there is a constraint that the loans paid_off_time > effective_date. However, not all loans are paid off. So if paid_off_time is missing, then I don't expect the > constraint to apply (return True)