The service uses Puppeteer, a headless Chrome, to scrape webpages. Currently it's only purpose is to provide information when a user suggests a new source. The scraper can find the icon, rss feed, name, and other relevant information for every page.
- Node v16.20.0 (a
.nvmrc
is presented for nvm users). - NPM for managing dependencies.
- Fastify as the web framework
__tests__
- There you can find all the tests and fixtures. Tests are written usingjest
.helm
- The home of the service helm chart for easily deploying it to kubernetes.src
- This is obviously the place where you can find the source files.scrape
- Stores many utility functions to scrape information from a webpage.
Daily Scraper requires nothing to run. It doesn't need any database or a service.
.env is used to set the required environment variables. It is loaded automatically by the project.
Finally run npm run dev
to run the service and listen to port 5001
.
So you want to contribute to Daily Scraper and make an impact, we are glad to hear it. 😍
Before you proceed we have a few guidelines for contribution that will make everything much easier. We would appreciate if you dedicate the time and read them carefully: https://github.com/dailydotdev/.github/blob/master/CONTRIBUTING.md