refactor: Add a level of record abstraction

[AndrewCiambrone]

Summary of Changes

RFC: amundsen-io/rfcs#5

Tests

I modified all the model tests to incorporate the new serializable functionality.

Documentation

What documentation did you add or modify and why? Add any relevant links then remove this line

CheckList

Make sure you have checked all steps below to ensure a timely review.

• PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.
• PR includes a summary of changes.
• PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
• PR passes make test

[feng-tao]

hey @AndrewCiambrone , thanks for the pr, has been a busy week..Do you think you could fix the conflict and CI? I would like to use the branch to do a quick lyft staging test. thanks a lot!

[feng-tao]

ping @AndrewCiambrone , I think this generic is super useful if we would like to support neptune in databuilder. Let us know if you have the chance to update the pr and I would like to do a quick test in our staging as well. cc @allisonsuarez in case I don't have a chance to test it.

[AndrewCiambrone]

Hey @feng-tao sorry its been a busy week. I will pull in the latest changes tomorrow.

[feng-tao]

thanks for the rebase!

[feng-tao]

just did a quick staging test with the pr with a few tasks and it runs fine so far :) I will look at the pr in more detail tomorrow and may ask questions as well, thanks for the great work!

[feng-tao]

I try to go through every line, but overall, I think the change lgtm and it is well written! Once the conflict is fixed, we could merge it ! thanks.

[feng-tao]

hey @AndrewCiambrone , I think we still need to keep this? or i miss the fix somewhere else?

[feng-tao]

nvm, I see the latter part

[feng-tao]

I end up to fix the merge conflict

[feng-tao]

@AndrewCiambrone given the change is complete, could you remove the WIP? or let me know if there are any models that haven't done the refactors.

For amundsen-io/rfcs#5, do you plan to implement the Neptune serializer as part of the same RFC?

[AndrewCiambrone]

If no new models were added in the last rebase I believe every model is accounted for.

As for the RFC I felt like they were two separate topics so I was planning on writing a second rfc for the Neptune databuilder.

[feng-tao]

thanks @AndrewCiambrone !

[dorianj]

When I run sample_data_loader.py with this changeset applied, I get this error:

Traceback (most recent call last):
  File "example/scripts/sample_data_loader.py", line 285, in <module>
    'databuilder.models.table_stats.TableColumnStats')
  File "example/scripts/sample_data_loader.py", line 115, in run_csv_job
    publisher=Neo4jCsvPublisher()).launch()
  File "/home/nada/paymob/data_discovery/data-discovery-env/lib/python3.7/site-packages/amundsen_databuilder-4.0.3-py3.7.egg/databuilder/job/job.py", line 77, in launch
  File "/home/nada/paymob/data_discovery/data-discovery-env/lib/python3.7/site-packages/amundsen_databuilder-4.0.3-py3.7.egg/databuilder/job/job.py", line 67, in launch
  File "/home/nada/paymob/data_discovery/data-discovery-env/lib/python3.7/site-packages/amundsen_databuilder-4.0.3-py3.7.egg/databuilder/task/task.py", line 65, in run
  File "/home/nada/paymob/data_discovery/data-discovery-env/lib/python3.7/site-packages/amundsen_databuilder-4.0.3-py3.7.egg/databuilder/loader/file_system_neo4j_csv_loader.py", line 119, in load
  File "/home/nada/anaconda3/lib/python3.7/csv.py", line 155, in writerow
    return self.writer.writerow(self._dict_to_list(rowdict))
  File "/home/nada/anaconda3/lib/python3.7/csv.py", line 151, in _dict_to_list
    + ", ".join([repr(x) for x in wrong_fields]))
ValueError: dict contains fields not in fieldnames: 'stat_val'

I haven't been able to debug it further, any ideas?