Create
robots.txt
for your Gridsome site.
This gridsome plugin is ported from a gatsby plugin by Marat Dreizin https://github.com/mdreizin/gatsby-plugin-robots-txt
yarn add gridsome-plugin-robots
or
npm install --save gridsome-plugin-robots
gridsome.config.js
module.exports = {
siteUrl: process.env.SITE_URL || 'https://www.example.com',
plugins: ['gridsome-plugin-robots']
};
This plugin uses generate-robotstxt
to generate content of robots.txt
and it has the following options:
Name | Type | Default | Description |
---|---|---|---|
host |
String |
${siteMetadata.siteUrl} |
Host of your site |
sitemap |
String / String[] |
${siteMetadata.siteUrl}/sitemap.xml |
Path(s) to sitemap.xml |
policy |
Policy[] |
[] |
List of Policy rules |
configFile |
String |
undefined |
Path to external config file |
output |
String |
/robots.txt |
Path where to create the robots.txt |
gridsome.config.js
module.exports = {
plugins: [
{
use: 'gridsome-plugin-robots',
options: {
host: 'https://www.example.com',
sitemap: 'https://www.example.com/sitemap.xml',
policy: [{ userAgent: '*', allow: '/' }]
}
}
]
};
gridsome.config.js
module.exports = {
plugins: [
{
resolve: 'gridsome-plugin-robots',
options: {
host: 'https://www.example.com',
sitemap: 'https://www.example.com/sitemap.xml',
env: {
development: {
policy: [{ userAgent: '*', disallow: ['/'] }]
},
production: {
policy: [{ userAgent: '*', allow: '/' }]
}
}
}
}
]
};
The env
key will be taken from process.env.SITE_ENV
first (see [Site hosting environment](It is custom environment), falling back to process.env.NODE_ENV
. When this is not available then it defaults to development
.
You can resolve the env
key by using resolveEnv
function:
gridsome.config.js
module.exports = {
plugins: [
{
resolve: 'gridsome-plugin-robots',
options: {
host: 'https://www.example.com',
sitemap: 'https://www.example.com/sitemap.xml',
resolveEnv: () => process.env.WEBSITE_ENV_TYPE,
env: {
development: {
policy: [{ userAgent: '*', disallow: ['/'] }]
},
production: {
policy: [{ userAgent: '*', allow: '/' }]
}
}
}
}
]
};
If you would like to disable crawlers for deploy-previews you can use the following snippet:
gridsome.config.js
const {
NODE_ENV,
URL: NETLIFY_SITE_URL = 'https://www.example.com',
DEPLOY_PRIME_URL: NETLIFY_DEPLOY_URL = NETLIFY_SITE_URL,
CONTEXT: NETLIFY_ENV = NODE_ENV
} = process.env;
const isNetlifyProduction = NETLIFY_ENV === 'production';
const siteUrl = isNetlifyProduction ? NETLIFY_SITE_URL : NETLIFY_DEPLOY_URL;
module.exports = {
siteUrl,
plugins: [
{
resolve: 'gridsome-plugin-robots',
options: {
resolveEnv: () => NETLIFY_ENV,
env: {
production: {
policy: [{ userAgent: '*' }]
},
'branch-deploy': {
policy: [{ userAgent: '*', disallow: ['/'] }],
sitemap: null,
host: null
},
'deploy-preview': {
policy: [{ userAgent: '*', disallow: ['/'] }],
sitemap: null,
host: null
}
}
}
}
]
};