Skip to content

Commit

Permalink
Fix GCPLogsCollector paging issue (#919)
Browse files Browse the repository at this point in the history
* Fix result paging issue

* Updates
  • Loading branch information
jleaniz authored Oct 8, 2024
1 parent 5637f40 commit af85148
Showing 1 changed file with 6 additions and 4 deletions.
10 changes: 6 additions & 4 deletions dftimewolf/lib/collectors/gcp_logging.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,18 +72,20 @@ def ListPages(self, logging_client: Any) -> Any:
Args:
logging_client: A GCP Cloud Logging client
Returns:
results.pages: Query result pages generator
Yields:
results: Query result entries generator
"""
results = logging_client.list_entries(
order_by=logging.DESCENDING,
filter_=self._filter_expression,
page_size=1000)
return results.pages

# list_entries() returns a Generator object
yield results

def ProcessPages(self, pages: Any, backoff_multiplier: int,
output_file: Any, output_path: str) -> str:
"""Iterates through a generator or pages and saves logs to disk.
"""Iterates through a generator and saves logs to disk.
Can optionally perform exponential backoff if query API limits are exceeded.
Args:
Expand Down

0 comments on commit af85148

Please sign in to comment.