revert changes where bots were recommented to avoid crawling the enti…

[pameyer]

…re site, to the previously commented out list of exclusions

RFI Checklist

Before submitting the pull request, fill out sections (1.) Related Issues and (2.) Pull Request Checklist.

1. Related Issues

directly related: #3666 , #2274

indirectly related: #2243 #2717 #1393

• [#### + Title HERE]

---

2. Pull Request Checklist

• Functionality completed as described in FRD
• Dependencies, risks, assumptions in FRD addressed
• Unit tests completed
• Deployment requirements identified (e.g., SQL scripts, indexing)
• Documentation completed
• All code checkins completed

---

3. Review Checklist

After the pull request has been submitted, fill out this section.

• Code review completed or waived
• Testing requirements completed
• Usability testing completed or waived
• Support testing completed or waived
• Merged with develop branch and resolved conflicts

Connects to #3666

[coveralls]

Coverage remained the same at 9.601% when pulling 92e5797 on pameyer:exp_robotstxt into 603bd0c on IQSS:develop.

[coveralls]

Coverage remained the same at 9.601% when pulling 3505ac4 on pameyer:exp_robotstxt into 603bd0c on IQSS:develop.

[sekmiller]

Looks OK to me. Moving to QA.

[kcondon]

Closing pull request since what we want is locked down by default but perhaps provide a production-ready example in the guides.