robots

Package robots implements robots.txt file parsing based on Google's specification:

Documentation

For installation, usage and description, please see the documentation:

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 48 Commits
testdata		testdata
v2		v2
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
example_test.go		example_test.go
go.mod		go.mod
go.sum		go.sum
interface.go		interface.go
lex.go		lex.go
parse.go		parse.go
robots_test.go		robots_test.go
types.go		types.go
types_test.go		types_test.go