Source Facebook Marketing: SAT fails on schema validation

[keu]

The problem with schema is that FB wraps numeric values into strings.
The schema says it is null or numeric but in the record it is "1234".

Steps to Reproduce

Run SAT on the latest FB connector
UPDATE: use this comment for instructions #4827 (comment)

Possible fix

1. update schemas - set type = ['null', 'string'], this is less desirable solution because destination will loose any type info.
2. do transform while reading inside streams, according to schema type. i.e. 'numeric' -> int, float

[Zirochkaa]

I can't reproduce this bug because of the following error:

The above error was also occurring in these 3 failed builds:

[Zirochkaa]

I had to change few things in order to run tests and all tests passed, see screenshot. Also I linked a PR with changes I made to tests. So I can't reproduce this issue.

Lonk to zira.

[Zirochkaa]

I commented out validate_schema: no line in acceptance-test-config.yml file and still wasn't able to reproduce this issue.

[keu]

Ok, it seems our config bypass the dates with records, so all problematic streams are empty. To reproduce this use the following config:

{
  "start_date": "2021-02-01T00:00:00Z",
  "account_id": "your_account",
  "access_token": "your_token",
  "include_deleted": false,
  "insights_lookback_window": 14,
  "insights_days_per_job": 1,
}

ERROR    root:test_core.py:136 The ad_sets stream has the following schema errors:
'0' is not of type 'null', 'number'

Failed validating 'type' in schema['properties']['daily_budget']:
    {'exclusiveMaximum': True,
     'exclusiveMinimum': True,
     'maximum': 100000000000000000000000000000000,
     'minimum': -100000000000000000000000000000000,
     'multipleOf': 1e-06,
     'type': ['null', 'number']}

On instance['daily_budget']:
    '0'
--------------------------------------------------------------------------------
'5000' is not of type 'null', 'number'

Failed validating 'type' in schema['properties']['budget_remaining']:
    {'exclusiveMaximum': True,
     'exclusiveMinimum': True,
     'maximum': 100000000000000000000000000000000,
     'minimum': -100000000000000000000000000000000,
     'multipleOf': 1e-06,
     'type': ['null', 'number']}

On instance['budget_remaining']:
    '5000'
--------------------------------------------------------------------------------

in the end

        if streams_errors:
>           pytest.fail(f"Please check your json_schema in selected streams {tuple(streams_errors.keys())}.")
E           Failed: Please check your json_schema in selected streams ('ad_sets', 'ads_insights', 'ads_insights_age_and_gender').

The tests passed when I set validate_schema: no
@Zirochkaa

[keu]

@Zirochkaa please check this PR to get the idea
#4994

[Zirochkaa]

@keu which config do we have in GitHub secrets for Facebook Marketing source?

[Zirochkaa]

We need to:

• Create transform() function and apply it on each record (~10h)
• Check that transform() works for 3 above mentioned streams (~2h)

[Zirochkaa]

We decided to use this issue as an experiment to write a generic class for performing transformations of record so it'll require more time than the estimate above.

[Zirochkaa]

With latest changes from master branch SAT tests are failing:

[Zirochkaa]

Blocked on #5592.