Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Future work #11

Open
5 of 21 tasks
stevenwdv opened this issue Oct 6, 2022 · 0 comments
Open
5 of 21 tasks

Future work #11

stevenwdv opened this issue Oct 6, 2022 · 0 comments

Comments

@stevenwdv
Copy link
Owner

stevenwdv commented Oct 6, 2022 •
edited
Loading

  • Make summary of existing JSON file?
  • Check leaks on existing JSON file?
  • Allow specifying all CLI options via a config file, clean up CLI options
  • Add config & version to .crawl-state
  • Better break up & test main.ts
  • Log finished URLs and make it possible to continue stopped batch crawl
  • Source maps for stack traces of request leaks & sniffs
  • Option to simulate show password add-on (change type to text)
  • Screen recordings (CDP Page.startScreencast)
  • Detect password logged to console
  • Option to update block lists
  • Recursive link clicking
  • Automatically detect filled values in headed mode, removing the need to fill the exact provided values
  • Click field again after filling? Some trackers collect then
  • Crawler won't fill fields if final URL has a different domain than initial URL; base check on final domain instead?
  • Bias to filling form with password field, to avoid filling just newsletter form
  • Max retries for URL in case of interrupt (out-of-memory error)
  • Detect PII in URL (GET form)
  • Split and clean up FieldsCollector code
  • Generate HTML leak report
  • Also capture all initial web requests, if possible
@stevenwdv stevenwdv pinned this issue Jan 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@stevenwdv