How to display stack traces?

[zvictor]

TLDR: I am looking for a way to display full stack traces when an error occurs during the scraping process. Is there a configuration option or a method in the API that allows for this? If not, I believe this would be a valuable feature to consider for future updates.

---

I have read the How to analyze and fix errors when scraping a website and the Best practices when writing scrapers (#Error handling), besides looking into the Configuration, but I can't figure out how one can display stack traces.

Instead, I found this:

(...) these are usually ugly messages with a stack trace that only the experts will understand.

You can overcome this by adding try/catch blocks into your code. In the catch block, explain what happened and re-throw the error (so the request is automatically retried)

Besides being surprised that a tool made for developers (and with great DX, by the way ❤️) calls stack traces something only the experts will understand, I must express my concern over the decision to hide stack traces from developers. While I understand the intention might be to simplify error messages, it's important to note that stack traces are not just "ugly messages" that "only the experts will understand". They are a fundamental tool for debugging and understanding the flow of execution, especially when an error occurs.

In fact, hiding stack traces and swallowing errors is generally considered an anti-pattern in software development. It makes it significantly harder to identify the root cause of issues and can lead to a lot of wasted time and frustration.

I strongly believe that developers using Crawlee should have the option to see full stack traces when an error occurs. If there's a way to enable this that I've missed, I'd appreciate your guidance. If not, I urge you to reconsider this design decision and provide a way to display full stack traces in future updates.

In the meantime, any suggestions or workarounds for accessing stack traces would be greatly appreciated.

[B4nan]

Stack traces are hidden only when the request will be retried, you should get the full stack trace for the final error message (once all retries fail). Or are we talking about something else?

You can force the stack traces on each retry as well via CRAWLEE_VERBOSE_LOG env var.

[zvictor]

Yes! CRAWLEE_VERBOSE_LOG is what I was needing. I must have missed it from the config docs, I am so sorry.

So, I have set CRAWLEE_VERBOSE_LOG and I have also defined maxRequestRetries: 0 as they fit me better in my dev environment. Something like EXIT_ON_FAIL would be even better for that, but it's fine as it is. Thanks!

[zvictor]

The CRAWLEE_VERBOSE_LOG is good but not perfect as Crawlee still changes the final error object .
For a little more context, I am using an external database (FaunaDB) to store the data I am crawling.

Error displayed by Crawlee:

ERROR CheerioCrawler: Request failed and reached maximum retries. QueryCheckError: The query failed 1 validation check
    at _Client.getServiceError_fn (/node_modules/.pnpm/fauna@0.9.4/node_modules/fauna/src/client.ts:325:18)
    at _Client.getError_fn (/node_modules/.pnpm/fauna@0.9.4/node_modules/fauna/src/client.ts:279:16)
    at _Client.query_fn (/node_modules/.pnpm/fauna@0.9.4/node_modules/fauna/src/client.ts:423:15)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async concat (file:///src/store.ts:70:10)
    at async file:///src/routes.ts:82:3
    at async wrap (/node_modules/.pnpm/@apify+timeout@0.3.0/node_modules/@apify/src/index.ts:77:27) {"id":"5AB7pDtgu0SJQIi","url":"https://listazap.net/","method":"GET","uniqueKey":"https://listazap.net/"}

Actual Error (running outside of Crawlee):

QueryCheckError: The query failed 1 validation check
    at Object.getServiceError_fn (/node_modules/.pnpm/fauna@0.9.4/node_modules/fauna/dist/node/index.js:1367:16)
    at Object.getError_fn (/node_modules/.pnpm/fauna@0.9.4/node_modules/fauna/dist/node/index.js:1334:74)
    at Object.query_fn (/node_modules/.pnpm/fauna@0.9.4/node_modules/fauna/dist/node/index.js:1440:59)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at concat (/src/store.ts:70:10) {
  httpStatus: 400,
  code: 'invalid_query',
  queryInfo: {
    txn_ts: 1690631525282105,
    summary: 'error: Expected identifier\n' +
      'at *query*:4:23\n' +
      '  |\n' +
      '4 |     let content = doc.<value>\n' +
      '  |                       ^\n' +
      '  |',
    query_tags: undefined,
    stats: {
      compute_ops: 1,
      read_ops: 0,
      write_ops: 0,
      query_time_ms: 1,
      contention_retries: 0,
      storage_bytes_read: 0,
      storage_bytes_write: 0,
      rate_limits_hit: []
    }
  },
  constraint_failures: undefined
}

The original error gives me data that allows me to fix my FQL query, while the error wrapped by Crawlee only gives me a rough direction (/src/store.ts:70:10).

[B4nan]

I dont think so, this seems more like a logging issue.