🐛 Source Linkedin Ads: connector assumes that api page with size smaller than maximum page size is last page

[FVidalCarneiro]

Connector Name

LinkedIn Ads

Connector Version

0.6.4

What step the error happened?

During the sync

Relevant information

We are facing an issue with the Linkedin Ads connector regarding the creatives endpoint. As you can identify in the associated pull request, currently, the Linkedin Ads connector assumes that it is ingesting the last Linkedin creatives API page when the number of records returned by the API is smaller than the maximum size of the page, in this case, 100 elements/records. This is a reasonable assumption, if all creatives are returned by the API per page. In our case, we have identified a rare, but possible BUG with some Linkedin creatives that are broken and cannot be deleted (we are in contact with the Linkedin Technical support team to understand the potential source of such bug). In our case, we have one creative that suffers from such a BUG, and as a consequence, the second page of the creatives API response (which should contain this creative), responds with 99 elements instead of 100. As a consequence, for the LinkedIn Ads account containing the bugged creative, airbyte assumes this is the last page and stops making calls. However, in our case, there were 4 more pages to consume. The consequence is that we have a lot of missing creatives. Below, we show you the equivalent postman API call that returns only 99 records (notice through the url that this is only the second page and that the total number of elements is 599).

Below a screenshot of what a broken Linkedin creative looks like from the UI.

Our solution is therefore to change the criteria of when to stop requesting next page from Linkedin API. Instead of stopping when the number of records returned from an API page is the smaller than the total allowed, we stop when an API call returned no records. We are aware that this would imply one useless extra API call per Linkedin Account, but avoid the above mentioned issue and making the connector more robust.

Relevant log output

No response

Contribute

• Yes, I want to contribute

[FVidalCarneiro]

Attempted resolution in related PR - #34166

[kev-datams]

Issue solved via:

• 🐛 Source Linkedin Ads: fix changing next_page_token stopping criteria #34166
• 🐛 Source Linkedin Ads: fix changing next_page_token stopping criteria (test copy for #34166) #37421
  isn't it ?