robotstxt

Here are 5 public repositories matching this topic...

Go robots.txt parser

Go language library for parsing Sitemaps

go golang sitemap robots-txt sitemap-xml sitemaps robotstxt sitemap-xml-gz sitemap-parser sitemapxml

Extract endpoints marked as Allow and Disallow in robots.txt

robotstxt

Stream decoders for sitemap.xml data and link feeds.

go golang robotstxt sitemapxml

RobotsTXT parser reads and interprets robots.txt files into a struct.

crawler robotstxt

Add a description, image, and links to the robotstxt topic page so that developers can more easily learn about it.

To associate your repository with the robotstxt topic, visit your repo's landing page and select "manage topics."