A webcrawler which crawls a web page for titles, keywords, description and urls.
The web crawler works basically by passing url to the start variable and it sends back all the links and url on the webpage to the pages.json
file in the folder.
THis Webcrawler is built using PHP7, Object Oriented Programming in PHP, Prior knowledge of Google UserAgent.
Happy coding! :)