Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🎉 Source Facebook Marketing: improve sleeps time in rate limit handler #10698

Merged
merged 17 commits into from
Mar 25, 2022
Merged

🎉 Source Facebook Marketing: improve sleeps time in rate limit handler #10698

Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
e56e28a
update: improve sleeps time in rate limit handle caller
vladimir-remar Feb 28, 2022
4a24396
update: improve sleep time calculation
vladimir-remar Mar 7, 2022
e924de1
Merge branch 'master' into source_facebook_marketing_improve_rate_lim…
vladimir-remar Mar 7, 2022
d17c6e8
update connector version in source_definitions and docs
vladimir-remar Mar 7, 2022
67e83b2
fix conflicts
vladimir-remar Mar 10, 2022
3dba30e
fix conflicts
vladimir-remar Mar 14, 2022
5e90b3c
update: improve handle_call_rate_limit
vladimir-remar Mar 15, 2022
3505db1
fix conflitcs
vladimir-remar Mar 15, 2022
0f5a9e4
update Dockerfile
vladimir-remar Mar 15, 2022
778bf52
fix: formating
vladimir-remar Mar 15, 2022
6f24d62
fix: sleep time in seconds
vladimir-remar Mar 16, 2022
905e556
Merge branch 'master' into source_facebook_marketing_improve_rate_lim…
alafanechere Mar 21, 2022
c768b2c
unit tests
alafanechere Mar 21, 2022
1fa8aa6
fix _compute_pause_interval add comparison with time interval
vladimir-remar Mar 21, 2022
913abac
change unit test according to change
alafanechere Mar 21, 2022
29a8330
Merge branch 'master' into source_facebook_marketing_improve_rate_lim…
alafanechere Mar 25, 2022
263922b
update source_specs and source_definitions
alafanechere Mar 25, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 19 additions & 6 deletions airbyte-integrations/connectors/source-facebook-marketing/source_facebook_marketing/api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,9 @@ class MyFacebookAdsApi(FacebookAdsApi):
"""Custom Facebook API class to intercept all API calls and handle call rate limits"""

call_rate_threshold = 95 # maximum percentage of call limit utilization
min_call_rate_threslold = 90 # minimum percentage of call limit utilization
pause_interval_minimum = pendulum.duration(minutes=1) # default pause interval if reached or close to call rate limit
pause_interval_minimum_at_max_usage = pendulum.duration(minutes=5)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
call_rate_threshold = 95 # maximum percentage of call limit utilization
min_call_rate_threslold = 90 # minimum percentage of call limit utilization
pause_interval_minimum = pendulum.duration(minutes=1) # default pause interval if reached or close to call rate limit
pause_interval_minimum_at_max_usage = pendulum.duration(minutes=5)
MAX_RATE, MAX_PAUSE_INTERVAL = (95, pendulum.duration(minutes=5))
MIN_RATE, MAX_PAUSE_INTERVAL = (90, pendulum.duration(minutes=1)


@dataclass
class Throttle:
Expand Down Expand Up @@ -87,6 +89,12 @@ def _parse_call_rate_header(headers):

return usage, pause_interval

@staticmethod
def sleep_time_handler(usage, pause_interval: pendulum.duration):

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the definition of the custom pause interval according to usage should be determined in this function and not in handle_call_rate_limit.
I'd rename it to compute_pause_interval and make it return the total seconds to sleep. Then I'd call sleep with this value from handle_call_rate_limit. I think it would greatly improve the readability and maintainability of this logic.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alafanechere Thanks, I did some changes, tell me what you think.

logger.warning(f"Utilization is too high ({usage})%, pausing for {pause_interval}")
sleep(pause_interval.total_seconds())

def handle_call_rate_limit(self, response, params):
if "batch" in params:
max_usage = 0
Expand All @@ -104,17 +112,22 @@ def handle_call_rate_limit(self, response, params):
max_usage = max(max_usage, usage)
vladimir-remar marked this conversation as resolved.
Show resolved Hide resolved
max_pause_interval = max(max_pause_interval, pause_interval)
vladimir-remar marked this conversation as resolved.
Show resolved Hide resolved

if max_usage > self.call_rate_threshold:
if max_usage >= self.call_rate_threshold:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if max_usage >= self.call_rate_threshold:
max_rate, max_sleep_duration = self. min_max_rate_sleep_time["max"]
min_rate, min_sleep_duration = self. min_max_rate_sleep_time["min"]
if max_usage >= max_rate:

max_pause_interval = max(max_pause_interval, self.pause_interval_minimum_at_max_usage)
self.sleep_time_handler(usage=max_usage, pause_interval=max_pause_interval)
elif self.min_call_rate_threslold <= max_usage < self.call_rate_threshold:
max_pause_interval = max(max_pause_interval, self.pause_interval_minimum)
logger.warning(f"Utilization is too high ({max_usage})%, pausing for {max_pause_interval}")
sleep(max_pause_interval.total_seconds())
self.sleep_time_handler(usage=max_usage, pause_interval=max_pause_interval)

else:
headers = response.headers()
usage, pause_interval = self._parse_call_rate_header(headers)
if usage > self.call_rate_threshold or pause_interval:
if usage >= self.call_rate_threshold or pause_interval:
pause_interval = max(pause_interval, self.pause_interval_minimum_at_max_usage)
self.sleep_time_handler(usage=usage, pause_interval=pause_interval)
elif self.min_call_rate_threslold <= usage < self.call_rate_threshold:
pause_interval = max(pause_interval, self.pause_interval_minimum)
logger.warning(f"Utilization is too high ({usage})%, pausing for {pause_interval}")
sleep(pause_interval.total_seconds())
self.sleep_time_handler(usage=usage, pause_interval=pause_interval)

def _update_insights_throttle_limit(self, response: FacebookResponse):
"""
Expand Down