Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(deps): update patch/minor dependencies #2450

Merged
merged 2 commits into from
May 14, 2024
Merged

Conversation

renovate[bot]
Copy link
Contributor

@renovate renovate bot commented May 6, 2024

Mend Renovate

This PR contains the following updates:

Package Change Age Adoption Passing Confidence Type Update
@playwright/browser-chromium (source) 1.43.1 -> 1.44.0 age adoption passing confidence devDependencies minor
@playwright/browser-firefox (source) 1.43.1 -> 1.44.0 age adoption passing confidence devDependencies minor
@playwright/browser-webkit (source) 1.43.1 -> 1.44.0 age adoption passing confidence devDependencies minor
node (source) 20.12.2 -> 20.13.1 age adoption passing confidence volta minor
playwright (source) 1.43.1 -> 1.44.0 age adoption passing confidence devDependencies minor
puppeteer (source) 22.7.1 -> 22.8.1 age adoption passing confidence devDependencies minor
yarn (source) 4.2.1 -> 4.2.2 age adoption passing confidence packageManager patch
yarn (source) 4.2.1 -> 4.2.2 age adoption passing confidence volta patch

Release Notes

microsoft/playwright (@​playwright/browser-chromium)

v1.44.0

Compare Source

New APIs

Accessibility assertions

  • expect(locator).toHaveAccessibleName() checks if the element has the specified accessible name:

    const locator = page.getByRole('button');
    await expect(locator).toHaveAccessibleName('Submit');
  • expect(locator).toHaveAccessibleDescription() checks if the element has the specified accessible description:

    const locator = page.getByRole('button');
    await expect(locator).toHaveAccessibleDescription('Upload a photo');
  • expect(locator).toHaveRole() checks if the element has the specified ARIA role:

    const locator = page.getByTestId('save-button');
    await expect(locator).toHaveRole('button');

Locator handler

  • After executing the handler added with page.addLocatorHandler(), Playwright will now wait until the overlay that triggered the handler is not visible anymore. You can opt-out of this behavior with the new noWaitAfter option.
  • You can use new times option in page.addLocatorHandler() to specify maximum number of times the handler should be run.
  • The handler in page.addLocatorHandler() now accepts the locator as argument.
  • New page.removeLocatorHandler() method for removing previously added locator handlers.
const locator = page.getByText('This interstitial covers the button');
await page.addLocatorHandler(locator, async overlay => {
  await overlay.locator('#close').click();
}, { times: 3, noWaitAfter: true });
// Run your tests that can be interrupted by the overlay.
// ...
await page.removeLocatorHandler(locator);

Miscellaneous options

  • multipart option in apiRequestContext.fetch() now accepts FormData and supports repeating fields with the same name.

    const formData = new FormData();
    formData.append('file', new File(['let x = 2024;'], 'f1.js', { type: 'text/javascript' }));
    formData.append('file', new File(['hello'], 'f2.txt', { type: 'text/plain' }));
    context.request.post('https://example.com/uploadFiles', {
      multipart: formData
    });
  • expect(callback).toPass({ intervals }) can now be configured by expect.toPass.inervals option globally in testConfig.expect or per project in testProject.expect.

  • expect(page).toHaveURL(url) now supports ignoreCase option.

  • testProject.ignoreSnapshots allows to configure per project whether to skip screenshot expectations.

Reporter API

  • New method suite.entries() returns child test suites and test cases in their declaration order. suite.type and testCase.type can be used to tell apart test cases and suites in the list.
  • Blob reporter now allows overriding report file path with a single option outputFile. The same option can also be specified as PLAYWRIGHT_BLOB_OUTPUT_FILE environment variable that might be more convenient on CI/CD.
  • JUnit reporter now supports includeProjectInTestName option.

Command line

  • --last-failed CLI option for running only tests that failed in the previous run.

    First run all tests:

    $ npx playwright test
    
    Running 103 tests using 5 workers
    ...
    2 failed
      [chromium] › my-test.spec.ts:8:5 › two ─────────────────────────────────────────────────────────
      [chromium] › my-test.spec.ts:13:5 › three ──────────────────────────────────────────────────────
    101 passed (30.0s)

    Now fix the failing tests and run Playwright again with --last-failed option:

    $ npx playwright test --last-failed
    
    Running 2 tests using 2 workers
      2 passed (1.2s)

Browser Versions

  • Chromium 125.0.6422.14
  • Mozilla Firefox 125.0.1
  • WebKit 17.4

This version was also tested against the following stable channels:

  • Google Chrome 124
  • Microsoft Edge 124
nodejs/node (node)

v20.13.1: 2024-05-09, Version 20.13.1 'Iron' (LTS), @​marco-ippolito

Compare Source

2024-05-09, Version 20.13.1 'Iron' (LTS), @​marco-ippolito

Revert "tools: install npm PowerShell scripts on Windows"

Due to a regression in the npm installation on Windows, this commit reverts the change that installed npm PowerShell scripts on Windows.

Commits
  • [b7d80802cc] - Revert "tools: install npm PowerShell scripts on Windows" (marco-ippolito) #​52897

v20.13.0

Compare Source

puppeteer/puppeteer (puppeteer)

v22.8.1: puppeteer: v22.8.1

Compare Source

Miscellaneous Chores
  • puppeteer: Synchronize puppeteer versions
Dependencies
  • The following workspace dependencies were updated
    • dependencies
      • puppeteer-core bumped from 22.8.0 to 22.8.1

v22.8.0: puppeteer: v22.8.0

Compare Source

Miscellaneous Chores
  • puppeteer: Synchronize puppeteer versions
Dependencies
  • The following workspace dependencies were updated
    • dependencies
      • puppeteer-core bumped from 22.7.1 to 22.8.0
yarnpkg/berry (yarn)

v4.2.2

Compare Source


Configuration

📅 Schedule: Branch creation - "every weekday" (UTC), Automerge - At any time (no schedule defined).

🚦 Automerge: Enabled.

Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.


  • If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate. View repository job log here.

Copy link
Contributor Author

renovate bot commented May 6, 2024

Branch automerge failure

This PR was configured for branch automerge. However, this is not possible, so it has been raised as a PR instead.


  • Branch has one or more failed status checks

@renovate renovate bot force-pushed the renovate/all-non-major branch 3 times, most recently from 4ebc764 to f0b7669 Compare May 7, 2024 00:31
@renovate renovate bot changed the title chore(deps): update dependency puppeteer to v22.8.0 chore(deps): update patch/minor dependencies May 7, 2024
@renovate renovate bot force-pushed the renovate/all-non-major branch 14 times, most recently from f709f86 to 3243077 Compare May 13, 2024 14:07
@renovate renovate bot force-pushed the renovate/all-non-major branch from 3243077 to e0b0927 Compare May 14, 2024 09:06
@renovate renovate bot force-pushed the renovate/all-non-major branch from e0b0927 to 62fb500 Compare May 14, 2024 10:58
@B4nan B4nan merged commit 41ee5ef into master May 14, 2024
16 checks passed
@B4nan B4nan deleted the renovate/all-non-major branch May 14, 2024 12:11
gitworkflows pushed a commit to threatcode/crawlee that referenced this pull request May 25, 2024
* chore(deps): lock file maintenance

* chore(deps): lock file maintenance

* chore(deps): lock file maintenance

* chore(deps): lock file maintenance

* ci: test on node 22 (apify#2438)

* chore: use node 20 in templates

* chore(deps): update yarn to v4.2.1

* chore(deps): lock file maintenance

* fix: return true when robots.isAllowed returns undefined (apify#2439)

`undefined` means that there is no explicit rule for the requested
route. No rules means no disallow, therefore it's allowed.

Fixes apify#2437

---------

Co-authored-by: Jan Buchar <Teyras@gmail.com>

* chore(deps): update patch/minor dependencies to v3.3.0

* chore(deps): update patch/minor dependencies to v3.3.2

* chore(deps): lock file maintenance

* chore(deps): lock file maintenance

* docs: Should be "Same Domain" not "Same Subdomain" (apify#2445)

The docs appear to be a bit misleading. If people want "Same Subdomain"
they should actually use "Same Hostname".

![image](https://github.com/apify/crawlee/assets/10026538/2b5452c5-e313-404b-812d-811e0764bd2d)

* chore(docker): update docker state [skip ci]

* docs: fix two typos (array or requests -> array of requests, no much -> not much) (apify#2451)

* fix: sitemap `content-type` check breaks on `content-type` parameters (apify#2442)

According to the
[RFC1341](https://www.w3.org/Protocols/rfc1341/4_Content-Type.html), the
Content-type header can contain additional string parameters.

* chore(docker): update docker state [skip ci]

* chore(deps): lock file maintenance

* fix(core): fire local `SystemInfo` events every second (apify#2454)

During local development, we are firing events for the AutoscaledPool
about current system resources like memory or CPU. We were firing them
once a minute by default, but we remove those snapshots older than 30s,
so we never had anything to compare and always used only the very last
piece of information.

This PR changes the interval to 1s, aligning this with how the Apify
platform fires events.

* chore(deps): lock file maintenance

* chore(deps): lock file maintenance

* chore(deps): lock file maintenance

* chore(deps): lock file maintenance

* chore(deps): lock file maintenance

* chore(deps): lock file maintenance

* chore(deps): update dependency linkedom to ^0.18.0 (apify#2457)

* chore(docker): update docker state [skip ci]

* perf: optimize adding large amount of requests via `crawler.addRequests()` (apify#2456)

This PR resolves three main issues with adding large amount of requests
into the queue:
- Every requests added to the queue was automatically added to the LRU
requests cache, which has a size of 1 million items. this makes sense
for enqueuing a few items, but if we try to add more than the limit, we
end up with overloading the LRU cache for no reason. Now we only add the
first 1000 requests to the cache (plus any requests added via separate
calls, e.g. when doing `enqueueLinks` from inside a request handler,
again with a limit of the first 1000 links).
- We used to validate the whole requests array via `ow`, and since the
shape can vary, it was very slow (e.g. 20s just for the `ow`
validation). Now we use a tailored validation for the array that does
the same but resolves within 100ms or so.
- We always created the `Request` objects out of everything, which had a
significant impact on memory usage. Now we skip this completely and let
the objects be created later when needed (when calling
`RQ.addRequests()` which only receives the actual batch and not the
whole array)

Related: https://apify.slack.com/archives/C0L33UM7Z/p1715109984834079

* perf: improve scaling based on memory (apify#2459)

We only allowed to use 70% of the available memory, this PR changes the
limit to 90%. Tested with a low memory options and it did not have any
effect, while it allows to use more memory on the large memory setups -
where the 30% could mean 2gb or so, we dont need such a huge buffer.

Also increases the scaling steps to 10% instead of 5% so speed up the
scaling.

Related:
[apify.slack.com/archives/C0L33UM7Z/p1715109984834079](https://apify.slack.com/archives/C0L33UM7Z/p1715109984834079)

* feat: make `RequestQueue` v2 the default queue, see more on [Apify blog](https://blog.apify.com/new-apify-request-queue/) (apify#2390)

Closes apify#2388

---------

Co-authored-by: drobnikj <drobnik.j@gmail.com>
Co-authored-by: Martin Adámek <banan23@gmail.com>

* fix: do not drop statistics on migration/resurrection/resume (apify#2462)

This fixes a bug that was introduced with
apify#1844 and
apify#2083 - we reset the persisted
state for statistics and session pool each time a crawler is started,
which prevents their restoration.

---------

Co-authored-by: Martin Adámek <banan23@gmail.com>

* chore(deps): update patch/minor dependencies (apify#2450)

* chore(docker): update docker state [skip ci]

* fix: double tier decrement in tiered proxy (apify#2468)

* docs: scrapy-vs-crawlee blog (apify#2431)

Co-authored-by: Saurav Jain <sauain@SauravApify.local>
Co-authored-by: davidjohnbarton <41335923+davidjohnbarton@users.noreply.github.com>

* perf: optimize `RequestList` memory footprint (apify#2466)

The request list now delays the conversion of the source items into the
`Request` objects, resulting in a significantly less memory footprint.

Related: https://apify.slack.com/archives/C0L33UM7Z/p1715109984834079

* fix: `EnqueueStrategy.All` erroring with links using unsupported protocols (apify#2389)

This changes `EnqueueStrategy.All` to filter out non-http and non-https
URLs (`mailto:` links were causing the crawler to error).

Let me know if there's a better fix or if you want me to change
something.

Thanks!


```
Request failed and reached maximum retries. Error: Received one or more errors
    at _ArrayValidator.handle (/path/to/project/node_modules/@sapphire/shapeshift/src/validators/ArrayValidator.ts:102:17)
    at _ArrayValidator.parse (/path/to/project/node_modules/@sapphire/shapeshift/src/validators/BaseValidator.ts:103:2)
    at RequestQueueClient.batchAddRequests (/path/to/project/node_modules/@crawlee/src/resource-clients/request-queue.ts:340:36)
    at RequestQueue.addRequests (/path/to/project/node_modules/@crawlee/src/storages/request_provider.ts:238:46)
    at RequestQueue.addRequests (/path/to/project/node_modules/@crawlee/src/storages/request_queue.ts:304:22)
    at attemptToAddToQueueAndAddAnyUnprocessed (/path/to/project/node_modules/@crawlee/src/storages/request_provider.ts:302:42)
    at RequestQueue.addRequestsBatched (/path/to/project/node_modules/@crawlee/src/storages/request_provider.ts:319:37)
    at RequestQueue.addRequestsBatched (/path/to/project/node_modules/@crawlee/src/storages/request_queue.ts:309:22)
    at enqueueLinks (/path/to/project/node_modules/@crawlee/src/enqueue_links/enqueue_links.ts:384:2)
    at browserCrawlerEnqueueLinks (/path/to/project/node_modules/@crawlee/src/internals/browser-crawler.ts:777:21)
```

* fix(core): use createSessionFunction when loading Session from persisted state (apify#2444)

Changes SessionPool's new Session loading behavior in the core module to
utilize the configured createSessionFunction if specified. This ensures
that new Sessions are instantiated using the custom session creation
logic provided by the user, improving flexibility and adherence to user
configurations.

* fix(core): conversion between tough cookies and browser pool cookies (apify#2443)

Fixes the conversion from tough cookies to browser pool cookies and vice
versa, by correctly handling cookies where the domain has a leading dot
versus when it doesn't.

* test: fix e2e tests for zero concurrency

* chore(deps): update dependency puppeteer to v22.8.2

* chore(docker): update docker state [skip ci]

* docs: fixes (apify#2469)

@B4nan  minor fixes

* chore(deps): update dependency puppeteer to v22.9.0

* feat: implement ErrorSnapshotter for error context capture (apify#2332)

This commit introduces the ErrorSnapshotter class to the crawlee
package, providing functionality to capture screenshots and HTML
snapshots when an error occurs during web crawling.

This functionality is opt-in, and can be enabled via the crawler
options:

```ts
const crawler = new BasicCrawler({
  // ...
  statisticsOptions: {
    saveErrorSnapshots: true,
  },
});
```

Closes apify#2280

---------

Co-authored-by: Martin Adámek <banan23@gmail.com>

* test: fix e2e tests for error snapshotter

* feat: add `FileDownload` "crawler" (apify#2435)

Adds a new package `@crawlee/file-download`, which overrides the
`HttpCrawler`'s MIME type limitations and allows the users to download
arbitrary files.

Aside from the regular `requestHandler`, this crawler introduces
`streamHandler`, which passes a `ReadableStream` with the downloaded
data to the user handler.

---------

Co-authored-by: Martin Adámek <banan23@gmail.com>
Co-authored-by: Jan Buchar <jan.buchar@apify.com>

* chore(release): v3.10.0

* chore(release): update internal dependencies [skip ci]

* chore(docker): update docker state [skip ci]

* docs: add v3.10 snapshot

* docs: fix broken link for a moved content

* chore(deps): lock file maintenance

* docs: improve crawlee seo ranking (apify#2472)

* chore(deps): lock file maintenance

* refactor: Remove redundant fields from `StatisticsPersistedState` (apify#2475)

Those fields are duplicated in the base class anyway.

* chore(deps): lock file maintenance

* fix: provide URLs to the error snapshot (apify#2482)

This will respect the Actor SDK override automatically since importing
the SDK will fire this side effect:

https://github.com/apify/apify-sdk-js/blob/master/packages/apify/src/key_value_store.ts#L25

* docs: update keywords (apify#2481)

Co-authored-by: Saurav Jain <sauain@SauravApify.local>

* docs: add feedback from community. (apify#2478)

Co-authored-by: Saurav Jain <sauain@SauravApify.local>
Co-authored-by: Martin Adámek <banan23@gmail.com>
Co-authored-by: davidjohnbarton <41335923+davidjohnbarton@users.noreply.github.com>

* chore: use biome for code formatting (apify#2301)

This takes ~50ms on my machine 🤯 

- closes apify#2366 
- Replacing spaces with tabs won't be done right here, right now.
- eslint and biome are reconciled
- ~biome check fails because of typescript errors - we can either fix
those or find a way to ignore it~

* chore(docker): update docker state [skip ci]

* test: Check if the proxy tier drops after an amount of successful requests (apify#2490)

* chore: ignore docker state when checking formatting (apify#2491)

* chore: remove unused eslint ignore directives

* chore: fix formatting

* chore: run biome as a pre-commit hook (apify#2493)

* fix: adjust `URL_NO_COMMAS_REGEX` regexp to allow single character hostnames (apify#2492)

Closes apify#2487

* fix: investigate and temp fix for possible 0-concurrency bug in RQv2 (apify#2494)

* test: add e2e test for zero concurrency with RQ v2

* chore: update biome

* chore(docker): update docker state [skip ci]

* chore(deps): lock file maintenance (apify#2495)

* chore(release): v3.10.1

* chore(release): update internal dependencies [skip ci]

* chore(docker): update docker state [skip ci]

* chore: add undeclared dependency

* chore(deps): update patch/minor dependencies to v1.44.1

* chore(deps): lock file maintenance

* chore(docker): update docker state [skip ci]

* feat: Loading sitemaps from string (apify#2496)

- closes apify#2460

* docs: fix homepage gradients (apify#2500)

* fix: Autodetect sitemap filetype from content (apify#2497)

- closes apify#2461

* chore(deps): update dependency puppeteer to v22.10.0

* chore(deps): lock file maintenance

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Martin Adámek <banan23@gmail.com>
Co-authored-by: Gigino Chianese <Sajito@users.noreply.github.com>
Co-authored-by: Jan Buchar <Teyras@gmail.com>
Co-authored-by: Connor Adams <connorads@users.noreply.github.com>
Co-authored-by: Apify Release Bot <noreply@apify.com>
Co-authored-by: Jiří Spilka <jiri.spilka@apify.com>
Co-authored-by: Jindřich Bär <jindrichbar@gmail.com>
Co-authored-by: Vlad Frangu <kingdgrizzle@gmail.com>
Co-authored-by: drobnikj <drobnik.j@gmail.com>
Co-authored-by: Jan Buchar <jan.buchar@apify.com>
Co-authored-by: Saurav Jain <souravjain540@gmail.com>
Co-authored-by: Saurav Jain <sauain@SauravApify.local>
Co-authored-by: davidjohnbarton <41335923+davidjohnbarton@users.noreply.github.com>
Co-authored-by: Stefan Sundin <git@stefansundin.com>
Co-authored-by: Gustavo Silva <silva95gustavo@gmail.com>
Co-authored-by: Hamza Alwan <ihamzaalwan@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants