Create unique id per row

[rosnfeld]

Currently using pandas index id for the master frame, which isn't stable upon data updates and thus manual exclusions would break.

Maybe create some sort of hash on certain columns? (shouldn't be all the columns, because they will likely add columns at some point) Maybe whatever can be used to identify a manual exclusion - most likely the free text columns along with year, donor, recipient.

[rosnfeld]

I'm not sure if this is even possible, using

id_columns = ['Year', 'donorcode', 'agencycode', 'recipientcode', 'crsid', 'projecttitle', 'shortdescription', 'longdescription', 'usd_disbursement']

still yields (a few) duplicates.

[rosnfeld]

Well, using all columns as the key there are no duplicates.