Skip to content

A web scrapper to extract the URLs embedded on the Website.

Notifications You must be signed in to change notification settings

thevillagehacker/urlscrapper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

34 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

urlscrapper

A web scrapper to extract the URLs embedded on the Website.

Installation

go install -v github.com/thevillagehacker/urlscrapper@latest

or

git clone https://github.com/thevillagehacker/urlscrapper.git
cd urlscrapper
go build

urlscrapper binary will be created and Move the binary to the required folder and add the path to the environment variables.

Usage

go run scrapper.go -u https://example.com

or

urlscrapper -u https://example.com

Update

  • Can now output the scrapped data into a file using -o flag.
  • Can now print HTTP status code of the scrapped URLs using -sc flag.

Example

Print the status code data

urlscrapper -u https://example.com -sc

img

Print the results to the output file

urlscrapper -u https://example.com -o out.txt

Print the results with status code to the output file

urlscrapper -u https://example.com -sc -o out.txt

About

A web scrapper to extract the URLs embedded on the Website.

Topics

Resources

Stars

Watchers

Forks

Packages

No packages published

Languages