TypeScript builds fail when generated from 'npx crawlee create'

[gustavfridell]

Which package is this bug report for? If unsure which one to select, leave blank

None

Issue description

Using "npx crawlee create my-crawler" and then choosing a TypeScript template results in a project that fails to build.

To reproduce:

1. Run npx crawlee create my-crawler
2. Choose a TypeScript template
3. cd to the project
4. run 'npm build'

Error log:

3 /// <reference types="node_modules/got/dist/source/core/timed-out" />
                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

node_modules/@crawlee/http/internals/http-crawler.d.ts:328:85 - error TS2694: Namespace '"/Users/username/Developer/my-crawler/node_modules/got-scraping/dist/index"' has no exported member 'RequestFunction'.

328         request?: import("got-scraping", { with: { "resolution-mode": "import" } }).RequestFunction | undefined;
                                                                                        ~~~~~~~~~~~~~~~

node_modules/@crawlee/http/internals/http-crawler.d.ts:329:83 - error TS2694: Namespace '"/Users/username/Developer/my-crawler/node_modules/got-scraping/dist/index"' has no exported member 'Agents'.

329         agent?: import("got-scraping", { with: { "resolution-mode": "import" } }).Agents | undefined;
                                                                                      ~~~~~~

node_modules/@crawlee/http/internals/http-crawler.d.ts:332:85 - error TS2694: Namespace '"/Users/username/Developer/my-crawler/node_modules/got-scraping/dist/index"' has no exported member 'Delays'.

332         timeout?: import("got-scraping", { with: { "resolution-mode": "import" } }).Delays | undefined;
                                                                                        ~~~~~~

node_modules/@crawlee/http/internals/http-crawler.d.ts:338:87 - error TS2694: Namespace '"/Users/username/Developer/my-crawler/node_modules/got-scraping/dist/index"' has no exported member 'PromiseCookieJar'.

338         cookieJar?: import("got-scraping", { with: { "resolution-mode": "import" } }).PromiseCookieJar | import("got-scraping", { with: { "resolution-mode": "import" } }).ToughCookieJar | undefined;
                                                                                          ~~~~~~~~~~~~~~~~

node_modules/@crawlee/http/internals/http-crawler.d.ts:338:172 - error TS2694: Namespace '"/Users/username/Developer/my-crawler/node_modules/got-scraping/dist/index"' has no exported member 'ToughCookieJar'.

338         cookieJar?: import("got-scraping", { with: { "resolution-mode": "import" } }).PromiseCookieJar | import("got-scraping", { with: { "resolution-mode": "import" } }).ToughCookieJar | undefined;
                                                                                                                                                                               ~~~~~~~~~~~~~~

node_modules/@crawlee/http/internals/http-crawler.d.ts:341:99 - error TS2694: Namespace '"/Users/username/Developer/my-crawler/node_modules/got-scraping/dist/index"' has no exported member 'SearchParameters'.

341         searchParams?: string | import("got-scraping", { with: { "resolution-mode": "import" } }).SearchParameters | URLSearchParams | undefined;
                                                                                                      ~~~~~~~~~~~~~~~~

node_modules/@crawlee/http/internals/http-crawler.d.ts:360:85 - error TS2694: Namespace '"/Users/username/Developer/my-crawler/node_modules/got-scraping/dist/index"' has no exported member 'Headers'.

360         headers?: import("got-scraping", { with: { "resolution-mode": "import" } }).Headers | undefined;
                                                                                        ~~~~~~~

node_modules/@crawlee/http/internals/http-crawler.d.ts:362:96 - error TS2694: Namespace '"/Users/username/Developer/my-crawler/node_modules/got-scraping/dist/index"' has no exported member 'DnsLookupIpVersion'.

362         dnsLookupIpVersion?: import("got-scraping", { with: { "resolution-mode": "import" } }).DnsLookupIpVersion;
                                                                                                   ~~~~~~~~~~~~~~~~~~

node_modules/@crawlee/http/internals/http-crawler.d.ts:363:87 - error TS2694: Namespace '"/Users/username/Developer/my-crawler/node_modules/got-scraping/dist/index"' has no exported member 'ParseJsonFunction'.

363         parseJson?: import("got-scraping", { with: { "resolution-mode": "import" } }).ParseJsonFunction | undefined;
                                                                                          ~~~~~~~~~~~~~~~~~

node_modules/@crawlee/http/internals/http-crawler.d.ts:364:91 - error TS2694: Namespace '"/Users/username/Developer/my-crawler/node_modules/got-scraping/dist/index"' has no exported member 'StringifyJsonFunction'.

364         stringifyJson?: import("got-scraping", { with: { "resolution-mode": "import" } }).StringifyJsonFunction | undefined;
                                                                                              ~~~~~~~~~~~~~~~~~~~~~

node_modules/@crawlee/http/internals/http-crawler.d.ts:367:94 - error TS2694: Namespace '"/Users/username/Developer/my-crawler/node_modules/got-scraping/dist/index"' has no exported member 'CreateConnectionFunction'.

367         createConnection?: import("got-scraping", { with: { "resolution-mode": "import" } }).CreateConnectionFunction | undefined;
                                                                                                 ~~~~~~~~~~~~~~~~~~~~~~~~

node_modules/@crawlee/http/internals/http-crawler.d.ts:368:90 - error TS2694: Namespace '"/Users/username/Developer/my-crawler/node_modules/got-scraping/dist/index"' has no exported member 'CacheOptions'.

368         cacheOptions?: import("got-scraping", { with: { "resolution-mode": "import" } }).CacheOptions | undefined;
                                                                                             ~~~~~~~~~~~~

node_modules/@crawlee/http/internals/http-crawler.d.ts:369:83 - error TS2694: Namespace '"/Users/username/Developer/my-crawler/node_modules/got-scraping/dist/index"' has no exported member 'HttpsOptions'.

369         https?: import("got-scraping", { with: { "resolution-mode": "import" } }).HttpsOptions | undefined;
                                                                                      ~~~~~~~~~~~~

node_modules/@crawlee/http/internals/http-crawler.d.ts:373:90 - error TS2694: Namespace '"/Users/username/Developer/my-crawler/node_modules/got-scraping/dist/index"' has no exported member 'ResponseType'.

373         responseType?: import("got-scraping", { with: { "resolution-mode": "import" } }).ResponseType | undefined;
                                                                                             ~~~~~~~~~~~~

node_modules/@crawlee/http/internals/http-crawler.d.ts:374:88 - error TS2694: Namespace '"/Users/username/Developer/my-crawler/node_modules/got-scraping/dist/index"' has no exported member 'PaginationOptions'.

374         pagination?: import("got-scraping", { with: { "resolution-mode": "import" } }).PaginationOptions<unknown, unknown> | undefined;
                                                                                           ~~~~~~~~~~~~~~~~~

node_modules/@crawlee/http/internals/http-crawler.d.ts:379:91 - error TS2694: Namespace '"/Users/username/Developer/my-crawler/node_modules/got-scraping/dist/index"' has no exported member 'Hooks'.

379         hooks?: Partial<import("got-scraping", { with: { "resolution-mode": "import" } }).Hooks> | undefined;
                                                                                              ~~~~~

node_modules/@crawlee/http/internals/http-crawler.d.ts:380:91 - error TS2694: Namespace '"/Users/username/Developer/my-crawler/node_modules/got-scraping/dist/index"' has no exported member 'RetryOptions'.

380         retry?: Partial<import("got-scraping", { with: { "resolution-mode": "import" } }).RetryOptions> | undefined;
                                                                                              ~~~~~~~~~~~~

node_modules/@crawlee/jsdom/internals/jsdom-crawler.d.ts:3:23 - error TS2688: Cannot find type definition file for 'node_modules/got/dist/source/core/timed-out'.

3 /// <reference types="node_modules/got/dist/source/core/timed-out" />
                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

node_modules/@crawlee/linkedom/internals/linkedom-crawler.d.ts:3:23 - error TS2688: Cannot find type definition file for 'node_modules/got/dist/source/core/timed-out'.

3 /// <reference types="node_modules/got/dist/source/core/timed-out" />
                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

node_modules/@crawlee/utils/internals/sitemap.d.ts:1:32 - error TS7016: Could not find a declaration file for module 'sax'. '/Users/username/Developer/my-crawler/node_modules/sax/lib/sax.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/sax` if it exists or add a new declaration (.d.ts) file containing `declare module 'sax';`

1 import type { SAXStream } from 'sax';
                                 ~~~~~


Found 21 errors in 5 files.

Errors  Files
     1  node_modules/@crawlee/cheerio/internals/cheerio-crawler.d.ts:3
    17  node_modules/@crawlee/http/internals/http-crawler.d.ts:328
     1  node_modules/@crawlee/jsdom/internals/jsdom-crawler.d.ts:3
     1  node_modules/@crawlee/linkedom/internals/linkedom-crawler.d.ts:3
     1  node_modules/@crawlee/utils/internals/sitemap.d.ts:1

Code sample

No response

Package version

3.7.0

Node.js version

21.4.0

Operating system

macOS

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[Andrzej-Marek]

Have the same issue

[rubberduck203]

Can we get any clarity on what version this is and isn't affecting?

[vladfrangu]

This is available in crawlee@next, and will also be available in 3.7.1 whenever we release (probably after new years but don't quote me on it!)

[rubberduck203]

Thanks @vladfrangu!

FWIW to anyone reading this, downgrading to 3.6.x worked as a temporary solution.

[marcplouhinec]

As a workaround, try adding "skipLibCheck": true in your tsconfig.json file, like this:

{
    "extends": "@apify/tsconfig",
    "compilerOptions": {
        "module": "NodeNext",
        "moduleResolution": "NodeNext",
        "target": "ES2022",
        "outDir": "dist",
        "noUnusedLocals": false,
        "lib": [
            "DOM",
            "ES2021.String"
        ],
        "skipLibCheck": true
    },
    "include": [
        "./src/**/*"
    ]
}