2024-05-20 21:07:11 [scrapy.utils.log] INFO: Scrapy 2.11.0 started (bot: gazette)
2024-05-20 21:07:11 [scrapy.utils.log] INFO: Versions: lxml 4.9.3.0, libxml2 2.10.3, cssselect 1.2.0, parsel 1.8.1, w3lib 2.1.2, Twisted 22.10.0, Python 3.11.5 (main, Aug 25 2023, 13:19:53) [GCC 9.4.0], pyOpenSSL 23.2.0 (OpenSSL 3.1.3 19 Sep 2023), cryptography 41.0.4, Platform Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
2024-05-20 21:07:11 [ba_itaquara_2019] INFO: Collecting data from 2022-01-04 to 2024-05-20.
2024-05-20 21:07:11 [scrapy.addons] INFO: Enabled addons:
[]
2024-05-20 21:07:11 [py.warnings] WARNING: /home/claromes/development/okbr/querido-diario/.venv/lib/python3.11/site-packages/scrapy/utils/request.py:254: ScrapyDeprecationWarning: '2.6' is a deprecated value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting.
It is also the default value. In other words, it is normal to get this warning if you have not defined a value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting. This is so for backward compatibility reasons, but it will change in a future version of Scrapy.
See the documentation of the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting for information on how to handle this deprecation.
return cls(crawler)
2024-05-20 21:07:11 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
2024-05-20 21:07:11 [scrapy.extensions.telnet] INFO: Telnet Password: 70647bc6bbc693d0
2024-05-20 21:07:11 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats',
'spidermon.contrib.scrapy.extensions.Spidermon',
'gazette.extensions.StatsPersist']
2024-05-20 21:07:11 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'gazette',
'COMMANDS_MODULE': 'gazette.commands',
'DOWNLOAD_TIMEOUT': 360,
'FILES_STORE_S3_ACL': 'public-read',
'LOG_FILE': 'log_ba_itaquara_2019_2022-01-04.txt',
'NEWSPIDER_MODULE': 'gazette.spiders',
'SPIDER_MODULES': ['gazette.spiders'],
'TEMPLATES_DIR': 'templates',
'USER_AGENT': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) '
'Gecko/20100101 Firefox/108.0'}
2024-05-20 21:07:11 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy_zyte_smartproxy.ZyteSmartProxyMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2024-05-20 21:07:11 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2024-05-20 21:07:11 [scrapy.middleware] INFO: Enabled item pipelines:
['gazette.pipelines.GazetteDateFilteringPipeline',
'gazette.pipelines.DefaultValuesPipeline',
'gazette.pipelines.QueridoDiarioFilesPipeline',
'spidermon.contrib.scrapy.pipelines.ItemValidationPipeline',
'gazette.pipelines.SQLDatabasePipeline']
2024-05-20 21:07:11 [scrapy.core.engine] INFO: Spider opened
2024-05-20 21:07:11 [gazette.database.models] INFO: Populating 'territories' table - Please wait!
2024-05-20 21:07:12 [gazette.database.models] INFO: Populating 'territories' table - Done!
2024-05-20 21:07:12 [gazette.database.models] INFO: Populating 'querido_diario_spider' table - Please wait!
2024-05-20 21:07:12 [gazette.database.models] INFO: Populating 'querido_diario_spider' table - Done!
2024-05-20 21:07:12 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-05-20 21:07:12 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2024-05-20 21:07:12 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None)
2024-05-20 21:07:14 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:14 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:14 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2022/02/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:14 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2022/03/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:15 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:15 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2022/05/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:15 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:15 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2022/06/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:16 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:16 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2022/08/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:16 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:16 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:16 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2022/04/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:16 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2022/07/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:17 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:17 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2022/09/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:17 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:17 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:17 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2022/10/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:17 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2022/11/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:17 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:17 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2022/12/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:18 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:18 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2023/01/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:18 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:18 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2023/02/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:19 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:19 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2023/03/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:19 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:19 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2023/04/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:19 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:20 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2023/05/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:20 [scrapy.core.engine] DEBUG: Crawled (200) (referer: http://pmitaquaraba.imprensaoficial.org/2022/01/)
2024-05-20 21:07:20 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:20 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2023/06/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:20 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:21 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2023/07/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:21 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:21 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2023/08/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:22 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:22 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2023/09/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:22 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:22 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2023/10/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:22 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:22 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2023/11/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:22 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None)
2024-05-20 21:07:22 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in
2024-05-20 21:07:23 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:23 [scrapy.core.scraper] DEBUG: Scraped from <200 http://pmitaquaraba.imprensaoficial.org/2022/01/04/edicao-no-2677/>
{'date': '2022-01-04',
'file_urls': ['http://www.imprensaoficial.org/pdf/baixar.php?arquivo=../pub/prefeituras/ba/itaquara/2022/proprio/2677.pdf'],
'files': [{'checksum': '52e81cd5597078fa432f53fa16ea49c4',
'path': '2916708/2022-01-04/a862506ff5b688d900ee49d0040abe1af5f59faf.pdf',
'status': 'downloaded',
'url': 'http://www.imprensaoficial.org/pdf/baixar.php?arquivo=../pub/prefeituras/ba/itaquara/2022/proprio/2677.pdf'}],
'is_extra_edition': False,
'power': 'executive',
'scraped_at': '2024-05-21T00:07:20.349063Z',
'territory_id': '2916708'}
2024-05-20 21:07:23 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2023/12/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:23 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:23 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2024/01/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:23 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:23 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2024/02/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:24 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:24 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2024/03/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:24 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:24 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2024/04/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:25 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2024-05-20 21:07:25 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 http://pmitaquaraba.imprensaoficial.org/2024/05/>: HTTP status code is not handled or not allowed
2024-05-20 21:07:25 [scrapy.core.engine] INFO: Closing spider (finished)
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] ------------------------------ MONITORS ------------------------------
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] Comparison Between Executions/Days without gazettes... OK
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] Requests/Items Ratio/Ratio of requests over items scraped count... FAIL
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] Error Count Monitor/test_stat_monitor... SKIPPED (Unable to find 'log_count/ERROR' in job stats.)
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] Finish Reason Monitor/Should have the expected finished reason(s)... OK
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] Item Validation Monitor/test_stat_monitor... SKIPPED (Unable to find 'spidermon/validation/fields/errors' in job stats.)
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] ----------------------------------------------------------------------
2024-05-20 21:07:25 [ba_itaquara_2019] ERROR: [Spidermon]
======================================================================
FAIL: Requests/Items Ratio/Ratio of requests over items scraped count
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/claromes/development/okbr/querido-diario/data_collection/gazette/monitors.py", line 30, in test_requests_items_ratio
self.assertLessEqual(
AssertionError: 31.0 not less than or equal to 5 : 3100.0% is greater than the allowed 500%
ratio of requests over items scraped.
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] 5 monitors in 0.005s
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] FAILED (failures=1, skipped=2)
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] -------------------------- FINISHED ACTIONS --------------------------
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] ----------------------------------------------------------------------
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] 0 actions in 0.000s
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] OK
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] --------------------------- PASSED ACTIONS ---------------------------
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] ----------------------------------------------------------------------
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] 0 actions in 0.000s
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] OK
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] --------------------------- FAILED ACTIONS ---------------------------
2024-05-20 21:07:25 [spidermon.contrib.actions.discord] INFO: *ba_itaquara_2019* finished
- Finish time: *2024-05-21 00:07:25.138225+00:00*
- Gazettes scraped: *1*
- 🔥 1 failures 🔥
===== FAILURES =====
Requests/Items Ratio/Ratio of requests over items scraped count: 31.0 not less than or equal to 5 : 3100.0% is greater than the allowed 500%
ratio of requests over items scraped.
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] CustomSendDiscordMessage... OK
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] ----------------------------------------------------------------------
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] 1 action in 0.000s
2024-05-20 21:07:25 [ba_itaquara_2019] INFO: [Spidermon] OK
2024-05-20 21:07:25 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 8939,
'downloader/request_count': 31,
'downloader/request_method_count/GET': 31,
'downloader/response_bytes': 2766107,
'downloader/response_count': 31,
'downloader/response_status_count/200': 3,
'downloader/response_status_count/404': 28,
'elapsed_time_seconds': 12.87516,
'file_count': 1,
'file_status_count/downloaded': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2024, 5, 21, 0, 7, 25, 138225, tzinfo=datetime.timezone.utc),
'httperror/response_ignored_count': 28,
'httperror/response_ignored_status_count/404': 28,
'item_scraped_count': 1,
'log_count/DEBUG': 34,
'log_count/ERROR': 1,
'log_count/INFO': 65,
'log_count/WARNING': 1,
'memusage/max': 128036864,
'memusage/startup': 128036864,
'request_depth_max': 1,
'response_received_count': 31,
'scheduler/dequeued': 30,
'scheduler/dequeued/memory': 30,
'scheduler/enqueued': 30,
'scheduler/enqueued/memory': 30,
'spidermon/validation/fields': 7,
'spidermon/validation/items': 1,
'spidermon/validation/validators': 1,
'spidermon/validation/validators/item/jsonschema': True,
'start_time': datetime.datetime(2024, 5, 21, 0, 7, 12, 263065, tzinfo=datetime.timezone.utc)}
2024-05-20 21:07:25 [scrapy.core.engine] INFO: Spider closed (finished)