NaN comparison in merge

[jseabold]

Parking this here so I don't forget about it. I assume that the NaNs here are not comparing equal, so we get "duplicate" rows on a merge, which I think is a bug.

data = [[1950, "A", 1.5],
        [1950, "B", 1.5],
        [1955, "B", 1.5],
        [1960, "B", np.nan],
        [1970, "B", 4.],
        [1950, "C", 4.],
        [1960, "C", np.nan],
        [1965, "C", 3.],
        [1970, "C", 4.],
        ]
frame = pandas.DataFrame(data, columns=["year", "panel", "data"])

other_data = [[1960, 'A', np.nan],
 [1970, 'A', np.nan],
 [1955, 'A', np.nan],
 [1965, 'A', np.nan],
 [1965, 'B', np.nan],
 [1955, 'C', np.nan]]
other = pandas.DataFrame(other_data, columns=['year', 'panel', 'data'])

together = frame.merge(other, how="outer")

Worked around this by filling in the NaNs before the merge, but then I have to convert them back to NaNs later.

[wesm]

Hm, I was fairly surprised by this. Having a look under the hood

[wesm]

Thanks for catching this one. Proper NA handling in merges really flew under the radar for a while (I had TODO notes sprinkled throughout pandas/tools/merge.py about it). Turned out to not be very difficult