Msiega/log unexpected fields

[mfsiega-airbyte]

What

Log when we see fields that aren't in the catalog.

How

• Make a map of stream -> fields in the catalog.
• For each record, check if any of the fields are missing from the catalog; save these.
• At the end of the sync, log any unexpected fields.

[github-actions]

Airbyte Code Coverage

File	Coverage [90.15%]	🍏
DefaultReplicationWorker.java	90.15%	🍏

Total Project Coverage	23.99%	❌

[benmoriceau]

is there a posibility to have a valid non object here?

[malikdiarra]

I mean it should not happen, but if that happen something is going to break in the code in the if block, better to test and throw a more specific exception.
We could use a more specific error message though "Expecting object in record data, got '%s' instead" or something like this.

[mfsiega-airbyte]

(@malikdiarra you had a comment on the same block)

According to the Airbyte Protocol the data property of a record message must be a JSON of object type, so in theory no.

However, on second thought I don't think we should be throwing here in any case - that will basically crash the whole sync, and that's why we have the validation step (which doesn't throw, but rather builds a list of errors).

[malikdiarra]

This looks good to me. How easy is it to test this block of code before we release it?

[malikdiarra]

I mean it should not happen, but if that happen something is going to break in the code in the if block, better to test and throw a more specific exception.
We could use a more specific error message though "Expecting object in record data, got '%s' instead" or something like this.