make robots.txt restrictive by default

[ruslandoga]

Changes

This PR makes robots.txt restrictive by default in self-hosted setups.

Related:

• Enabling the robots.txt #1798 (reply in thread)
• https://3.basecamp.com/5308029/buckets/29267832/card_tables/cards/6961830456

Tests

• Automated tests have been added (I'll add some tests to make sure it works as expected in full build vs small build)

Changelog

• Entry has been added to changelog

Documentation

• Docs have been updated

Dark mode

• This PR does not change the UI

[ruslandoga]

https://chat.openai.com/share/aa80093e-3f71-43af-9f48-87d7c622311d

I wonder what the solution should be for the Cloud version. Is omitting robots.txt OK or should we COPY different files when building the container image?

Cloud (to keep the previous behavior)

# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-agent: *
# Disallow: /

Self-hosted

User-agent: *
Disallow: /

[ukutaht]

The current robots.txt isn't event used in prod because plausible/website serves its own one which the reverse proxy prefers: https://plausible.io/robots.txt

It should be fine to omit. But we should add a comment/doc line for some clarity. We never want CE to be indexed but in Cloud we do want pages like public dashboards and live demo to be indexed. This is controlled via X-Robots-Tag header rather than robots.txt file.

[ruslandoga]

Then instead of omitting, would it be OK to simply use

User-agent: *
Disallow: /

for both? With a comment, of course.

[ruslandoga]

Like this c71c114

[ukutaht]

I prefer the original solution of omitting it. It's quite likely we will move off the Jekyll-based static site in the future. And in that case, if we forget about robots.txt, we might end up in a place where we disallow all indexing for plausible.io which would be bad.

I know omitting adds more code, but it feels safer to me.

[ruslandoga]

What about copying different robots.txt into the container image based on some ARG? So that EE would get

# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-agent: *
# Disallow: /

and CE would get

User-agent: *
Disallow: /

[ruslandoga]

Or having it configured with app env and served by a custom plug.

# in endpoint.ex
plug :serve_robots_txt

on_full_build do
# ...
else
  def robots_txt do
    # or app env lookup
    """
    User-agent: *
    Disallow: /
    """
  end
end

def serve_robots_txt(conn, _opts) do
  case conn.path_info do
    ["robotx.txt"] -> conn |> send_resp(200, robots_txt()) |> halt()
    _other -> conn
  end
end

[ukutaht]

I think omitting, the way you've done, is good enough for now.

[ruslandoga]

Ok! I've added a note about the Cloud's robots.txt in 11409f3