Robots sensitive crawls

boogheta released this 25 Oct 15:09

· 184 commits to master since this release

ChangeLog:

Optional support of robots.txt respect by crawls (added by @stijn-uva #376 #421)
Minor fixes (sigma.js upgrade to v2.0beta, #370, #395, #423, #284, dba5721, ...)

Contributors

stijn-uva

Assets 2