2024-06-17 18:03:43 [scrapy.utils.log] INFO: Scrapy 2.11.0 started (bot: gazette) 2024-06-17 18:03:43 [scrapy.utils.log] INFO: Versions: lxml 4.9.3.0, libxml2 2.10.3, cssselect 1.2.0, parsel 1.8.1, w3lib 2.1.2, Twisted 22.10.0, Python 3.10.7 (main, May 29 2023, 13:51:48) [GCC 12.2.0], pyOpenSSL 23.2.0 (OpenSSL 3.1.3 19 Sep 2023), cryptography 41.0.4, Platform Linux-5.19.0-46-generic-x86_64-with-glibc2.36 2024-06-17 18:03:43 [pr_castro] INFO: Collecting data from 2024-05-01 to 2024-05-31. 2024-06-17 18:03:43 [scrapy.addons] INFO: Enabled addons: [] 2024-06-17 18:03:43 [py.warnings] WARNING: /home/marcos/Documentos/querido-diario/venv/lib/python3.10/site-packages/scrapy/utils/request.py:254: ScrapyDeprecationWarning: '2.6' is a deprecated value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting. It is also the default value. In other words, it is normal to get this warning if you have not defined a value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting. This is so for backward compatibility reasons, but it will change in a future version of Scrapy. See the documentation of the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting for information on how to handle this deprecation. return cls(crawler) 2024-06-17 18:03:43 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor 2024-06-17 18:03:43 [scrapy.extensions.telnet] INFO: Telnet Password: 05dbedfbd31e4fff 2024-06-17 18:03:43 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats', 'spidermon.contrib.scrapy.extensions.Spidermon', 'gazette.extensions.StatsPersist'] 2024-06-17 18:03:43 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'gazette', 'COMMANDS_MODULE': 'gazette.commands', 'DOWNLOAD_TIMEOUT': 360, 'FILES_STORE_S3_ACL': 'public-read', 'LOG_FILE': 'log_pr_castro_intervalo_maio_24.txt', 'NEWSPIDER_MODULE': 'gazette.spiders', 'SPIDER_MODULES': ['gazette.spiders'], 'TEMPLATES_DIR': 'templates', 'USER_AGENT': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) ' 'Gecko/20100101 Firefox/108.0'} 2024-06-17 18:03:43 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy_zyte_smartproxy.ZyteSmartProxyMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2024-06-17 18:03:43 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2024-06-17 18:03:43 [scrapy.middleware] INFO: Enabled item pipelines: ['gazette.pipelines.GazetteDateFilteringPipeline', 'gazette.pipelines.DefaultValuesPipeline', 'gazette.pipelines.QueridoDiarioFilesPipeline', 'spidermon.contrib.scrapy.pipelines.ItemValidationPipeline', 'gazette.pipelines.SQLDatabasePipeline'] 2024-06-17 18:03:43 [scrapy.core.engine] INFO: Spider opened 2024-06-17 18:03:43 [gazette.database.models] INFO: Populating 'querido_diario_spider' table - Please wait! 2024-06-17 18:03:43 [gazette.database.models] INFO: Populating 'querido_diario_spider' table - Done! 2024-06-17 18:03:43 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2024-06-17 18:03:43 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2024-06-17 18:03:44 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:44 [tzlocal] DEBUG: /etc/timezone found, contents: America/Sao_Paulo 2024-06-17 18:03:44 [tzlocal] DEBUG: /etc/localtime found 2024-06-17 18:03:44 [tzlocal] DEBUG: 2 found: {'/etc/timezone': 'America/Sao_Paulo', '/etc/localtime is a symlink to': 'America/Sao_Paulo'} 2024-06-17 18:03:44 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%221%22%7D%7D) 2024-06-17 18:03:45 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:45 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:45 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%222%22%7D%7D> {'date': '2024-05-23', 'edition_number': '2919', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222695%22%2C%22hash%22%3A%22DBA9289D2F52583E5D4884DEA74167270455945F%22%7D&cidade=padrao'], 'files': [{'checksum': 'f42091a0d2509bda367463360c901c5d', 'path': '4104907/2024-05-23/52b0a0bfc7a0afd7c240fb47d347dedff1669712.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222695%22%2C%22hash%22%3A%22DBA9289D2F52583E5D4884DEA74167270455945F%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.840905Z', 'territory_id': '4104907'} 2024-06-17 18:03:45 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:45 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:45 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%221%22%7D%7D> {'date': '2024-05-29', 'edition_number': '2924', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222700%22%2C%22hash%22%3A%22C4C79E56AB4D03E9EB13177C3E73C7D156E3B0EF%22%7D&cidade=padrao'], 'files': [{'checksum': '0e624a893c9896066a1eb92212bdc89e', 'path': '4104907/2024-05-29/f547374aaeac5d32814a4c97b5314c003a795660.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222700%22%2C%22hash%22%3A%22C4C79E56AB4D03E9EB13177C3E73C7D156E3B0EF%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.341532Z', 'territory_id': '4104907'} 2024-06-17 18:03:45 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:45 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:45 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:45 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:45 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%222%22%7D%7D> {'date': '2024-05-21', 'edition_number': '2917', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222693%22%2C%22hash%22%3A%22F9D0D539BAD7F32A6DE5DC04F2FF846BD7372EFF%22%7D&cidade=padrao'], 'files': [{'checksum': '2e867a4ef936c2e531a8dd5ef22bfed8', 'path': '4104907/2024-05-21/13569537dbbf27b4cbe918a434c7099a08859e1d.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222693%22%2C%22hash%22%3A%22F9D0D539BAD7F32A6DE5DC04F2FF846BD7372EFF%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.852594Z', 'territory_id': '4104907'} 2024-06-17 18:03:46 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%221%22%7D%7D> {'date': '2024-05-23', 'edition_number': '2920', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222696%22%2C%22hash%22%3A%22BDF264779E424FB3148B9CD323B559317B43935C%22%7D&cidade=padrao'], 'files': [{'checksum': '417c3fb518f886086cbdca80ef2f55b0', 'path': '4104907/2024-05-23/d7412355ef2e8fadc0de7260ff27ffc8583df511.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222696%22%2C%22hash%22%3A%22BDF264779E424FB3148B9CD323B559317B43935C%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.348775Z', 'territory_id': '4104907'} 2024-06-17 18:03:46 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:46 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:46 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:46 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:46 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%221%22%7D%7D> {'date': '2024-05-24', 'edition_number': '2921', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222697%22%2C%22hash%22%3A%22D66FDD13A227143BD67C0EA6A29B7F06CD8B44D1%22%7D&cidade=padrao'], 'files': [{'checksum': '1242dd9b020140c5b753639d6f5f7a8a', 'path': '4104907/2024-05-24/ad9d0427dff3e57166c572030dcfbd6643bf58a7.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222697%22%2C%22hash%22%3A%22D66FDD13A227143BD67C0EA6A29B7F06CD8B44D1%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.347016Z', 'territory_id': '4104907'} 2024-06-17 18:03:46 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%222%22%7D%7D> {'date': '2024-05-20', 'edition_number': '2916', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222692%22%2C%22hash%22%3A%220B3963C4CB7EE526CB2BEBD0A15321F6EE425ADF%22%7D&cidade=padrao'], 'files': [{'checksum': 'c4033830a0bfceb39922bbdd2a072c54', 'path': '4104907/2024-05-20/081d4ce30fb9b404c15a86f34f047b87fc2a0b9e.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222692%22%2C%22hash%22%3A%220B3963C4CB7EE526CB2BEBD0A15321F6EE425ADF%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.855310Z', 'territory_id': '4104907'} 2024-06-17 18:03:46 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:46 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:46 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%222%22%7D%7D> {'date': '2024-05-16', 'edition_number': '2914', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222690%22%2C%22hash%22%3A%22A52F9557DC318D1B53B7F9B4ABB7DFB9D5E1FB72%22%7D&cidade=padrao'], 'files': [{'checksum': 'c84dd8418500441cd56fbf30eedb1590', 'path': '4104907/2024-05-16/87f82c65a798a08470cd35881d7b95284f1f58d2.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222690%22%2C%22hash%22%3A%22A52F9557DC318D1B53B7F9B4ABB7DFB9D5E1FB72%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.858103Z', 'territory_id': '4104907'} 2024-06-17 18:03:47 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:47 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:47 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%222%22%7D%7D> {'date': '2024-05-16', 'edition_number': '2913', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222689%22%2C%22hash%22%3A%223BC9D182DC30BB36D074E07716BF9A1BD0EE859E%22%7D&cidade=padrao'], 'files': [{'checksum': 'e8993e61499a6e10a7dfeb2801e6051e', 'path': '4104907/2024-05-16/703d6b8f57267e97f1807b34864325cbf41fa028.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222689%22%2C%22hash%22%3A%223BC9D182DC30BB36D074E07716BF9A1BD0EE859E%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.859219Z', 'territory_id': '4104907'} 2024-06-17 18:03:47 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:47 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:47 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:47 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:47 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%222%22%7D%7D> {'date': '2024-05-15', 'edition_number': '2913', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222688%22%2C%22hash%22%3A%227492A6FE0C3D9482EC4243D8875DB9725F0A5E35%22%7D&cidade=padrao'], 'files': [{'checksum': 'c40f623b4c5eea4acf2d9fe273cdd477', 'path': '4104907/2024-05-15/b8a3014e647adb4ebbd71270ba20ac0a85ac1291.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222688%22%2C%22hash%22%3A%227492A6FE0C3D9482EC4243D8875DB9725F0A5E35%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.860313Z', 'territory_id': '4104907'} 2024-06-17 18:03:47 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%222%22%7D%7D> {'date': '2024-05-14', 'edition_number': '2912', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222687%22%2C%22hash%22%3A%223EE3CF0D46DEC632385C6197B01419552859CF95%22%7D&cidade=padrao'], 'files': [{'checksum': 'c40f623b4c5eea4acf2d9fe273cdd477', 'path': '4104907/2024-05-14/db83f30c3564f6be7ed20d0e21b360b0f9cb6e6a.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222687%22%2C%22hash%22%3A%223EE3CF0D46DEC632385C6197B01419552859CF95%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.861403Z', 'territory_id': '4104907'} 2024-06-17 18:03:48 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:48 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:48 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:48 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%221%22%7D%7D> {'date': '2024-05-28', 'edition_number': '2923', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222699%22%2C%22hash%22%3A%223C7509DF6B6EA1ACD7804E30CAC9B21B2379982C%22%7D&cidade=padrao'], 'files': [{'checksum': '5ec7780c5f5b1b782ddcdf11e280aa30', 'path': '4104907/2024-05-28/14c75c6c79d31f8984e664950fbc65a8d3d05a82.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222699%22%2C%22hash%22%3A%223C7509DF6B6EA1ACD7804E30CAC9B21B2379982C%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.343499Z', 'territory_id': '4104907'} 2024-06-17 18:03:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%222%22%7D%7D> {'date': '2024-05-17', 'edition_number': '2915', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222691%22%2C%22hash%22%3A%22E132899FBB9336019463650AFE99042079040DF9%22%7D&cidade=padrao'], 'files': [{'checksum': '29e453d4d4b224be8ec1f55e5b8a9198', 'path': '4104907/2024-05-17/8e3dfef644e5f7467d78417d464b5fdb8846ac60.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222691%22%2C%22hash%22%3A%22E132899FBB9336019463650AFE99042079040DF9%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.856594Z', 'territory_id': '4104907'} 2024-06-17 18:03:48 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:48 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%222%22%7D%7D> {'date': '2024-05-22', 'edition_number': '2918', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222694%22%2C%22hash%22%3A%22F57FEDFAEE566A9C93407746CF0BB18CBBE75BF4%22%7D&cidade=padrao'], 'files': [{'checksum': '60df3d6215d03bd27f884d92331562c4', 'path': '4104907/2024-05-22/b35c3b818344b6cd7f2e473a10e2465b4904e436.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222694%22%2C%22hash%22%3A%22F57FEDFAEE566A9C93407746CF0BB18CBBE75BF4%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.848690Z', 'territory_id': '4104907'} 2024-06-17 18:03:48 [scrapy.core.engine] DEBUG: Crawled (200) (referer: https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%222%22%7D%7D) 2024-06-17 18:03:48 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:48 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:48 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%222%22%7D%7D> {'date': '2024-05-09', 'edition_number': '2909', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222684%22%2C%22hash%22%3A%22F2714B24E689D346FF423612C0E5AE0937525CCD%22%7D&cidade=padrao'], 'files': [{'checksum': 'f3d57521a39acb720433821b7353e3a8', 'path': '4104907/2024-05-09/3189d5313d23693f2656239753c54892b8d0cfc7.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222684%22%2C%22hash%22%3A%22F2714B24E689D346FF423612C0E5AE0937525CCD%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.864721Z', 'territory_id': '4104907'} 2024-06-17 18:03:49 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:49 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%222%22%7D%7D> {'date': '2024-05-06', 'edition_number': '2906', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222681%22%2C%22hash%22%3A%22AB4D8B854716973829A1ED6D630CBA0C1520ED33%22%7D&cidade=padrao'], 'files': [{'checksum': 'adb849bf5b72d7f80e9d15ad3ecc6252', 'path': '4104907/2024-05-06/7da0d778b2d97b1df5c8640fd405bb6e81b2cc85.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222681%22%2C%22hash%22%3A%22AB4D8B854716973829A1ED6D630CBA0C1520ED33%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.868040Z', 'territory_id': '4104907'} 2024-06-17 18:03:49 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:49 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%222%22%7D%7D> {'date': '2024-05-10', 'edition_number': '2910', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222685%22%2C%22hash%22%3A%226C487D757C2D764167AA485348F7C6678A7E4CB7%22%7D&cidade=padrao'], 'files': [{'checksum': '6468076f1e3a833f2e769b10a5bcdceb', 'path': '4104907/2024-05-10/65b7ed6fc846e83ee0de10b2775d98f65b196b01.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222685%22%2C%22hash%22%3A%226C487D757C2D764167AA485348F7C6678A7E4CB7%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.863563Z', 'territory_id': '4104907'} 2024-06-17 18:03:49 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:49 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:49 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:49 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%222%22%7D%7D> {'date': '2024-05-07', 'edition_number': '2907', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222682%22%2C%22hash%22%3A%2221539EB1B2643898351E5478E06CFB28E9D78269%22%7D&cidade=padrao'], 'files': [{'checksum': '53c3c5bd390e43d06b3bbe992145fc5d', 'path': '4104907/2024-05-07/56565cde83bbde67d8484720f1bb4926b684319a.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222682%22%2C%22hash%22%3A%2221539EB1B2643898351E5478E06CFB28E9D78269%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.866943Z', 'territory_id': '4104907'} 2024-06-17 18:03:49 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%223%22%7D%7D> {'date': '2024-05-03', 'edition_number': '2905', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222680%22%2C%22hash%22%3A%22F6B8BF3FCEBB1816879E50FA2076B48E9D969112%22%7D&cidade=padrao'], 'files': [{'checksum': '61a7d788b8a48a28fd4b40ab790c9332', 'path': '4104907/2024-05-03/adeacaae747ce7e1e7b5eacadcb7ad7ff87edf83.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222680%22%2C%22hash%22%3A%22F6B8BF3FCEBB1816879E50FA2076B48E9D969112%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:48.619967Z', 'territory_id': '4104907'} 2024-06-17 18:03:50 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:50 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:50 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:50 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:50 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:50 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:50 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%223%22%7D%7D> {'date': '2024-05-02', 'edition_number': '2904', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222679%22%2C%22hash%22%3A%220F6448F5B36526950E4B4B82F8BCF9B6292967B5%22%7D&cidade=padrao'], 'files': [{'checksum': '177bd5e4714ec11c4d40d9ba35451451', 'path': '4104907/2024-05-02/00e4c5e2039bc63df0a72adb9f302ce10a200756.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222679%22%2C%22hash%22%3A%220F6448F5B36526950E4B4B82F8BCF9B6292967B5%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:48.654950Z', 'territory_id': '4104907'} 2024-06-17 18:03:50 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%222%22%7D%7D> {'date': '2024-05-08', 'edition_number': '2908', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222683%22%2C%22hash%22%3A%229F86AAEBDBBDF54C076EF4D8A93942ED0DE690EE%22%7D&cidade=padrao'], 'files': [{'checksum': 'ed429a2f2e1e137df2459685139954df', 'path': '4104907/2024-05-08/deaca1fede391c9007a531b8cc6b9f94b5e14b78.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222683%22%2C%22hash%22%3A%229F86AAEBDBBDF54C076EF4D8A93942ED0DE690EE%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.865844Z', 'territory_id': '4104907'} 2024-06-17 18:03:50 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%221%22%7D%7D> {'date': '2024-05-27', 'edition_number': '2922', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222698%22%2C%22hash%22%3A%2207181DDC6288295AC364C27E714D99E9131FE1A6%22%7D&cidade=padrao'], 'files': [{'checksum': 'f88154ba58f882202adfd760d3b42dd6', 'path': '4104907/2024-05-27/5945d064d4ae0d6fbb6e2ea6b96a11db8d9443b0.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222698%22%2C%22hash%22%3A%2207181DDC6288295AC364C27E714D99E9131FE1A6%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.345343Z', 'territory_id': '4104907'} 2024-06-17 18:03:50 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) 2024-06-17 18:03:50 [scrapy.pipelines.files] DEBUG: File (downloaded): Downloaded file from referred in 2024-06-17 18:03:50 [scrapy.core.scraper] DEBUG: Scraped from <200 https://castro.atende.net/diariooficial/edicao/pagina/atende.php?rot=54015&aca=101&ajax=t&processo=loadPluginDiarioOficial¶metro=%7B%22codigoPlugin%22%3A1%2C%22filtroPlugin%22%3A%7B%22pagina%22%3A%222%22%7D%7D> {'date': '2024-05-13', 'edition_number': '2911', 'file_urls': ['https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222686%22%2C%22hash%22%3A%22C0E996ECA4FCF1E7D477C5234B30772BDCF76A56%22%7D&cidade=padrao'], 'files': [{'checksum': 'c23c30fc15e99a7b979fcf806028d105', 'path': '4104907/2024-05-13/593c864fbff6c967036a4a87b863bd2d256fb95d.pdf', 'status': 'downloaded', 'url': 'https://castro.atende.net/atende.php?rot=54002&aca=737&processo=download¶metro=%7B%22codigo%22%3A%222686%22%2C%22hash%22%3A%22C0E996ECA4FCF1E7D477C5234B30772BDCF76A56%22%7D&cidade=padrao'}], 'is_extra_edition': False, 'power': 'executive_legislative', 'scraped_at': '2024-06-17T21:03:44.862483Z', 'territory_id': '4104907'} 2024-06-17 18:03:50 [scrapy.core.engine] INFO: Closing spider (finished) 2024-06-17 18:03:50 [scrapy.extensions.feedexport] INFO: Stored csv feed (22 items) in: pr_castro_intervalo_maio_24.csv 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] ------------------------------ MONITORS ------------------------------ 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] Comparison Between Executions/Days without gazettes... OK 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] Requests/Items Ratio/Ratio of requests over items scraped count... OK 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] Error Count Monitor/test_stat_monitor... SKIPPED (Unable to find 'log_count/ERROR' in job stats.) 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] Finish Reason Monitor/Should have the expected finished reason(s)... OK 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] Item Validation Monitor/test_stat_monitor... SKIPPED (Unable to find 'spidermon/validation/fields/errors' in job stats.) 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] ---------------------------------------------------------------------- 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] 5 monitors in 0.012s 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] OK (skipped=2) 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] -------------------------- FINISHED ACTIONS -------------------------- 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] ---------------------------------------------------------------------- 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] 0 actions in 0.000s 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] OK 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] --------------------------- PASSED ACTIONS --------------------------- 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] ---------------------------------------------------------------------- 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] 0 actions in 0.000s 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] OK 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] --------------------------- FAILED ACTIONS --------------------------- 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] ---------------------------------------------------------------------- 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] 1 action in 0.000s 2024-06-17 18:03:50 [pr_castro] INFO: [Spidermon] OK (skipped=1) 2024-06-17 18:03:50 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 12606, 'downloader/request_count': 25, 'downloader/request_method_count/GET': 25, 'downloader/response_bytes': 35782839, 'downloader/response_count': 25, 'downloader/response_status_count/200': 25, 'elapsed_time_seconds': 6.616991, 'feedexport/success_count/FileFeedStorage': 1, 'file_count': 22, 'file_status_count/downloaded': 22, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2024, 6, 17, 21, 3, 50, 440529, tzinfo=datetime.timezone.utc), 'httpcompression/response_bytes': 71069, 'httpcompression/response_count': 3, 'item_scraped_count': 22, 'log_count/DEBUG': 73, 'log_count/INFO': 34, 'log_count/WARNING': 1, 'memusage/max': 123269120, 'memusage/startup': 123269120, 'request_depth_max': 2, 'response_received_count': 25, 'scheduler/dequeued': 3, 'scheduler/dequeued/memory': 3, 'scheduler/enqueued': 3, 'scheduler/enqueued/memory': 3, 'spidermon/validation/fields': 176, 'spidermon/validation/items': 22, 'spidermon/validation/validators': 1, 'spidermon/validation/validators/item/jsonschema': True, 'start_time': datetime.datetime(2024, 6, 17, 21, 3, 43, 823538, tzinfo=datetime.timezone.utc)} 2024-06-17 18:03:50 [scrapy.core.engine] INFO: Spider closed (finished)