Spider Plugin Middleware

Spider Plugin Middleware

The spider Plugin module is designed to automatically create a robots.txt file. This is done by adding attributes to either a controller class or individual methods within the controller class, like:

`[Route("api/DemoApi")]
[DenySpider]
[ApiController]
public class DemoApiController : ControllerBase
{
}`

When initialised, the Spider plugin will read all routes within the application, building a list of denied paths for the robots.txt file.

UserSessionMiddleware

If the User Session middleware plugin is also loaded, the Spider Plugin will return a 403 forbidden response, should a bot attempt to access a denied route.