Skip to content

Commit

Permalink
chore: -
Browse files Browse the repository at this point in the history
  • Loading branch information
@enenumxela
enenumxela committed Aug 2, 2022
1 parent 9019c34 commit d31321f
Show file tree
Hide file tree
Showing 9 changed files with 87 additions and 112 deletions.
18 changes: 9 additions & 9 deletions .github/workflows/build.yml
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: build
name: build 🔨

on:
push:
Expand All @@ -12,16 +12,16 @@ jobs:
runs-on: ubuntu-latest
steps:
-
name: Set up Go
uses: actions/setup-go@v2
name: Check out code
uses: actions/checkout@v3
with:
go-version: 1.17

fetch-depth: 0
-
name: Check out code
uses: actions/checkout@v2

name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '>=1.17.0'
-
name: Build
run: go build -v .
working-directory: cmd/sigrawl3r/
working-directory: ./cmd/hqcrawl3r
24 changes: 11 additions & 13 deletions .github/workflows/release.yml
View file
Original file line number Diff line number Diff line change
@@ -1,28 +1,26 @@
name: release
name: release 🎉

on:
create:
tags:
- v*
- v*.*.*
workflow_dispatch:

jobs:
release:
name: release
runs-on: ubuntu-latest
steps:
-
name: Set up Go
uses: actions/setup-go@v2
with:
go-version: 1.17

-
name: "Check out code"
uses: actions/checkout@v2
name: Check out code
uses: actions/checkout@v3
with:
fetch-depth: 0

-
name: Set up Go
uses: actions/setup-go@v3
with:
go-version: '>=1.17.0'
-
name: Get dependencies
run: |
Expand All @@ -31,12 +29,12 @@ jobs:
curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
dep ensure
fi
-
name: "Create release on GitHub"
uses: goreleaser/goreleaser-action@v2
uses: goreleaser/goreleaser-action@v3
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
with:
args: "release --rm-dist"
version: latest
distribution: goreleaser
4 changes: 2 additions & 2 deletions .goreleaser.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
builds:
-
binary: sigrawl3r
main: cmd/sigrawl3r/main.go
binary: hqcrawl3r
main: cmd/hqcrawl3r/main.go
goos:
- linux
- windows
Expand Down
39 changes: 20 additions & 19 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# sigrawl3r
# hqcrawl3r

[![release](https://img.shields.io/github/release/signedsecurity/sigrawl3r?style=flat&color=0040ff)](https://github.com/signedsecurity/sigrawl3r/releases) [![maintenance](https://img.shields.io/badge/maintained%3F-yes-0040ff.svg)](https://github.com/signedsecurity/sigrawl3r) [![open issues](https://img.shields.io/github/issues-raw/signedsecurity/sigrawl3r.svg?style=flat&color=0040ff)](https://github.com/signedsecurity/sigrawl3r/issues?q=is:issue+is:open) [![closed issues](https://img.shields.io/github/issues-closed-raw/signedsecurity/sigrawl3r.svg?style=flat&color=0040ff)](https://github.com/signedsecurity/sigrawl3r/issues?q=is:issue+is:closed) [![license](https://img.shields.io/badge/license-MIT-gray.svg?colorB=0040FF)](https://github.com/signedsecurity/sigrawl3r/blob/master/LICENSE) [![twitter](https://img.shields.io/badge/twitter-@signedsecurity-0040ff.svg)](https://twitter.com/signedsecurity)
[![release](https://img.shields.io/github/release/hueristiq/hqcrawl3r?style=flat&color=0040ff)](https://github.com/hueristiq/hqcrawl3r/releases) [![maintenance](https://img.shields.io/badge/maintained%3F-yes-0040ff.svg)](https://github.com/hueristiq/hqcrawl3r) [![open issues](https://img.shields.io/github/issues-raw/hueristiq/hqcrawl3r.svg?style=flat&color=0040ff)](https://github.com/hueristiq/hqcrawl3r/issues?q=is:issue+is:open) [![closed issues](https://img.shields.io/github/issues-closed-raw/hueristiq/hqcrawl3r.svg?style=flat&color=0040ff)](https://github.com/hueristiq/hqcrawl3r/issues?q=is:issue+is:closed) [![license](https://img.shields.io/badge/license-MIT-gray.svg?colorB=0040FF)](https://github.com/hueristiq/hqcrawl3r/blob/master/LICENSE) [![twitter](https://img.shields.io/badge/twitter-@itshueristiq-0040ff.svg)](https://twitter.com/itshueristiq)

A fast web crawler.

Expand All @@ -25,19 +25,19 @@ A fast web crawler.
## Usage

```bash
sigrawl3r -h
hqcrawl3r -h
```

```text
_ _ _____
___(_) __ _ _ __ __ ___ _| |___ / _ __
/ __| |/ _` | '__/ _` \ \ /\ / / | |_ \| '__|
\__ \ | (_| | | | (_| |\ V V /| |___) | |
|___/_|\__, |_| \__,_| \_/\_/ |_|____/|_| v1.1.0
|___/
_ _ _____
| |__ __ _ ___ _ __ __ ___ _| |___ / _ __
| '_ \ / _` |/ __| '__/ _` \ \ /\ / / | |_ \| '__|
| | | | (_| | (__| | | (_| |\ V V /| |___) | |
|_| |_|\__, |\___|_| \__,_| \_/\_/ |_|____/|_| v1.1.0
|_|
USAGE:
sigrawl3r [OPTIONS]
hqcrawl3r [OPTIONS]
OPTIONS:
-c, --concurrency Maximum concurrent requests for matching domains (default: 5)
Expand All @@ -47,7 +47,8 @@ OPTIONS:
--headless If true the browser will be displayed while crawling
Note: Requires '-r, --render' flag
Note: Usage to show browser: '--headless=false' (default true)
-H, --headers Custom headers separated by two semi-colons. E.g. -h 'Cookie: foo=bar;;Referer: http://example.com/'
-H, --headers Custom headers separated by two semi-colons.
E.g. -h 'Cookie: foo=bar;;Referer: http://example.com/'
--include-subs Extend scope to include subdomains (default: false)
--no-color Enable no color mode (default: false)
-p, --proxy Proxy URL (e.g: http://127.0.0.1:8080)
Expand All @@ -68,26 +69,26 @@ OPTIONS:

#### From Binary

You can download the pre-built binary for your platform from this repository's [releases](https://github.com/signedsecurity/sigrawl3r/releases/) page, extract, then move it to your `$PATH`and you're ready to go.
You can download the pre-built binary for your platform from this repository's [releases](https://github.com/hueristiq/hqcrawl3r/releases/) page, extract, then move it to your `$PATH`and you're ready to go.

#### From Source

sigrawl3r requires **go1.17+** to install successfully. Run the following command to get the repo
hqcrawl3r requires **go1.17+** to install successfully. Run the following command to get the repo

```bash
go install github.com/signedsecurity/sigrawl3r/cmd/sigrawl3r@latest
go install github.com/hueristiq/hqcrawl3r/cmd/hqcrawl3r@latest
```

#### From Github

```bash
git clone https://github.com/signedsecurity/sigrawl3r.git && \
cd sigrawl3r/cmd/sigrawl3r/ && \
git clone https://github.com/hueristiq/hqcrawl3r.git && \
cd hqcrawl3r/cmd/hqcrawl3r/ && \
go build . && \
mv sigrawl3r /usr/local/bin/ && \
sigrawl3r -h
mv hqcrawl3r /usr/local/bin/ && \
hqcrawl3r -h
```

## Contribution

[Issues](https://github.com/signedsecurity/sigrawl3r/issues) and [Pull Requests](https://github.com/signedsecurity/sigrawl3r/pulls) are welcome!
[Issues](https://github.com/hueristiq/hqcrawl3r/issues) and [Pull Requests](https://github.com/hueristiq/hqcrawl3r/pulls) are welcome!
9 changes: 5 additions & 4 deletions cmd/sigrawl3r/main.go → cmd/hqcrawl3r/main.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@ import (
"sync"

"github.com/enenumxela/urlx/pkg/urlx"
"github.com/hueristiq/hqcrawl3r/internal/configuration"
"github.com/hueristiq/hqcrawl3r/internal/crawler"
"github.com/hueristiq/hqcrawl3r/internal/utils/io"
"github.com/logrusorgru/aurora/v3"
"github.com/signedsecurity/sigrawl3r/internal/configuration"
"github.com/signedsecurity/sigrawl3r/internal/crawler"
"github.com/signedsecurity/sigrawl3r/internal/utils/io"
)

var (
Expand Down Expand Up @@ -60,7 +60,7 @@ func init() {
displayBanner()

h := "USAGE:\n"
h += " sigrawl3r [OPTIONS]\n"
h += " hqcrawl3r [OPTIONS]\n"

h += "\nOPTIONS:\n"
h += fmt.Sprintf(" -c, --concurrency Maximum concurrent requests for matching domains (default: %d)\n", configuration.DefaultConcurrency)
Expand All @@ -86,6 +86,7 @@ func init() {
h += " `web` for a random web user-agent\n"
h += " `mobile` for a random mobile user-agent\n"
h += " or you can set your special user-agent\n"
h += "\n"

fmt.Fprint(os.Stderr, h)
}
Expand Down
30 changes: 15 additions & 15 deletions go.mod
View file
Original file line number Diff line number Diff line change
@@ -1,37 +1,37 @@
module github.com/signedsecurity/sigrawl3r
module github.com/hueristiq/hqcrawl3r

go 1.17
go 1.18

require (
github.com/chromedp/chromedp v0.8.0
github.com/chromedp/chromedp v0.8.3
github.com/enenumxela/urlx v0.0.0-20220330083946-3fddfff8acee
github.com/gocolly/colly/v2 v2.1.0
github.com/logrusorgru/aurora/v3 v3.0.0
github.com/oxffaa/gopher-parse-sitemap v0.0.0-20191021113419-005d2eb1def4
)

require (
github.com/PuerkitoBio/goquery v1.8.0 // indirect
github.com/andybalholm/cascadia v1.3.1 // indirect
github.com/antchfx/htmlquery v1.2.4 // indirect
github.com/antchfx/xmlquery v1.3.10 // indirect
github.com/antchfx/xpath v1.2.0 // indirect
github.com/chromedp/cdproto v0.0.0-20220321060548-7bc2623472b3 // indirect
github.com/PuerkitoBio/goquery v1.5.1 // indirect
github.com/andybalholm/cascadia v1.2.0 // indirect
github.com/antchfx/htmlquery v1.2.3 // indirect
github.com/antchfx/xmlquery v1.2.4 // indirect
github.com/antchfx/xpath v1.1.8 // indirect
github.com/chromedp/cdproto v0.0.0-20220725225757-5988d9195a6c // indirect
github.com/chromedp/sysutil v1.0.0 // indirect
github.com/gobwas/glob v0.2.3 // indirect
github.com/gobwas/httphead v0.1.0 // indirect
github.com/gobwas/pool v0.2.1 // indirect
github.com/gobwas/ws v1.1.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
github.com/golang/protobuf v1.4.2 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/kennygrant/sanitize v1.2.4 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/saintfish/chardet v0.0.0-20120816061221-3af4cd4741ca // indirect
github.com/temoto/robotstxt v1.1.2 // indirect
github.com/temoto/robotstxt v1.1.1 // indirect
golang.org/x/net v0.0.0-20220325170049-de3da57026de // indirect
golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e // indirect
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8 // indirect
golang.org/x/text v0.3.7 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/protobuf v1.28.0 // indirect
google.golang.org/appengine v1.6.6 // indirect
google.golang.org/protobuf v1.24.0 // indirect
)
Loading

0 comments on commit d31321f

Please sign in to comment.