Scraper Documentation

Function Calls

1. setup()
   1. When enabled is true in the scrapers config, it will run createTimerInterval() after 5 seconds.
2. createTimerInterval()
   1. Runs timeIntervalBody() in a loop, dependent on getScrapingInterval(), which per default reads interval_milliseconds from the scrapers config.
3. timeIntervalBody()
   1. Runs getScrapingUrl(), by default reads scraping_url from scrapers config.
   2. Runs requestWebsite() with the given scraping URL.
   3. Runs parseWebsiteContentToJSON() with the data given to generate an array of JSON objects from the website content.
   4. Runs filterNewContent() with the generated JSON array, marking old content, updating or saving new.
   5. Runs getSortingFunction() to sort content after specified arguments.
   6. Runs getEmbed() to generate Discord embed for each generated JSON object.
   7. Runs filterEmbedLength() to filter length of generated embeds according to Discord's policies.
   8. Runs sendEmbedMessages() to send and update new content to subscribed users and guilds, when global_send_embeds and send_embeds are true in configs.

scrapeInfo Object

{
    url: String,                    // generated from getScrapingUrl()
    response: AxiosResponse,        // returned from requestWebsite()
    content: [
        {
            json: parsed JSON data, // parsed from parseWebsiteContentToJSON()
            newData: Boolean,       // determined by filterNewContent()
            embed: DiscordEmbed     // generated by getEmbed()
        },
        ...
    ]
}