Config file: support for multiple config files in a project

[stsewd]

Currently, the path of the config file can't be changed, it always is .readthedocs.yaml and should be at the root of the repo.

But there are users with monorepos that have several projects and their docs in one repo, so using a config file isn't something they can do, they will need to use the settings from the UI, which limit the options they can set and makes them hard to track.

By being able to change the path of the config file per-project, those users would be able to have a better experience. So, the question is how to expose this option, I have two suggestions:

• Using an env var to define the path: something like RTD_CONFIG=subproject/readthedocs.yaml
  • This doesn't require adding a new option in the UI or a new field in our DB, but since we don't show the value of env vars after they are created, it's hard to know what path we are using for a given project.
• Creating a new field project.config_path.
  • This is more explicit, if the field is empty (default) then we use our current logic to discover the config file, otherwise we use that path.

So, I'm more inclined to option B, do you have another idea of how to implement it?

[astrojuanlu]

I don't understand option B: where would project.config_path be?

[stsewd]

I don't understand option B: where would project.config_path be?

That's an option on the project admin UI.

[astrojuanlu]

Gotcha, I vote for B as well

[humitos]

I think this is a good case to come back to "project level configs Vs. version level configs" discussion and try to solve the problem in a more general way. Make a decision, plan the path going forward, and create the basic structure to keep adding "project level configs" in a clean way in the future.

We have discussed this in #6046, #4221, #5478, #8678 (there are more issues) and we also have some cards in our Trello.

NOTE that the field proposed here will be a project-level config that will affect all the versions for this project

I think this new field proposed is probably the first step towards the general solution, but I'd like to see a plan involving this thinking a little more. For example:

• add the field .config_file_path that points to the "all the version-level configs"
• set project-level settings somehow
  • a). on the config file, we can add a new key .project_level_config_path (which points to another YAML file with all the project-level configs)
  • b). add another field to the Project object in the db: .global_config_path, pointing to the new YAML file with all the project-level configs
• the configs from the new .yaml overrides the settings in the db and save them

I'm not saying this solution is good, but I think we need to think something among these lines to solve this problem and start building the new features on top of the decision we make here.

[stsewd]

Not sure if that solves the problem, or if I understand your solution. If the option is defined on a file on the repo, the user can't have different values for the same repo on the same commit/branch (which is the problem on a monorepo).

[humitos]

To have different configs/settings for the same branch you need to have two different readthedocs.yaml (e.g. readthedocs.a.yaml (e.g. builds documentation in docs/a) and readthedocs.b.yaml (e.g. builds documentation in docs/b) in that branch. So, I'm saying the config file path can be defined in the Project object adding a new file as you suggested.

Besides, I'm saying that we need another config file path for "project-level configs" where the user can set up as readthedocs.a.global.yaml (e.g. this one enables PR builder) or readthedocs.b.global.yaml (e.g. this one does not). This another config file can be defined in the readthedocs.a.yaml (option a) from my previous comment) or in another Project object field (option b) from my previous comment)

I'm not proposing a final solution. However, I'm saying that we need to figure out how to do this in a clean way.

[agjohnson]

That is a great point @humitos 💯

To back up just a bit: to support monorepos we can solve this purely at the Sphinx level, configured via env vars, and that does at least work. We could still decide this isn't a good fit for our application and solve this outside, via Sphinx extension. I would be okay if we ended here, because this isn't a widely requested feature at the end of the day.

Okay, so now if we do want to support this natively, I came to the same conclusion as Manuel.

We can tell users to use a singular python requirements file in a monorepo, but users probably would still want per-project configuration of:

• Search tweaks
• Build formats?
• Redirects (big oof here)

When we get to redirects, we need per-project and per-version configuration. So we need two files to support this. But once we also factor in monorepo support, we're talking about multiple sets of these configuration files -- as noted above.

We have a few potential options:

• A filename postfix like described above. State is stored at something like Project.configuration_postfix. Functionally similar to a Project.configuration_path, but this needs to affect both a .readthedocs.{postfix}.yaml and a .readthedocs-global.{postfix}.yaml
• Use configuration file namespaces/overrides, and still just one set of .readthedocs.yaml and .readthedocs-global.yaml
• Dynamic configuration or configuration file templating

---

Namespacing isn't as awful as it sounds. Consider something like this:

version: 2

formats: all

sphinx:
  configuration: docs/conf.py

overrides:
  "api-docs":
    sphinx:
      configuration: docs/api/conf.py
    search: {}
  "site-docs":
    sphinx:
      configuration: docs/site/conf.py
  "dev-docs":
    sphinx:
      configuration: docs/dev/conf.py

search:
  ranking:
    # Deprecated content
    api/v1.html: -1
    config-file/v1.html: -1

    # Useful content, but not something we want most users finding
    changelog.html: -6

overrides is applied last, and uses a Project.configuration_namespace to merge the overrides into the configuration object. This keeps a singular set of configuration files at least. This is maybe easier to communicate as a pattern to users than multiple sets of files. It's quite clean too.

---

Dynamic configuration is maybe a 🤯 solution, but it can also solve future configuration problems and provides a lot of flexibility to users. We're handing out foot guns with this, but you're already advanced usage if you want multiple project support.

Looking at some prior art, CircleCI has a feature for monorepo usage, called dynamic config:
https://circleci.com/docs/2.0/dynamic-config/

It's a script that generates the configuration file used in the build. It's a heavy solution, and quite weird, but could be inspiration.

But, that got me thinking, what about jinja template YAML?

version: 2

formats: all

# Basic templating can be used for variables
sphinx:
  configuration: docs/{{ readthedocs.project_slug }}/conf.py

# But we even have includes on our config
{% if readthedocs.project_slug == 'foo' %}
  {% include '.readthedocs/foo.yaml' %}
{% endif %}

---

Namespaces/overrides seems like a sweet spot for advanced usage and maintainability/support on our side.

I could see some incredibly cool things coming from template support though, but it's more of an expert level feature given you need to know jinja syntax too.

Anyways, I agree with Manuel, now is a good time to think ahead to a global configuration file. It would help to consider how we might use a global configuration file and how this might shape up that spec -- though we can simmer on an actionable plan for both redirects and global configuration.

[stsewd]

I think explicit is better, but also having two config files would be confusing (I'm talking about the idea to have one config file for global settings and another one for per-version settings).

If we have two files for global and per-version settings, we still need to choose from what branch/version we read that file, if we go with reading global options from the default version or latest, we can just use the same config file, adding the settings under the "global" key, something like:

version: 2

formats: all

sphinx:
  configuration: docs/conf.py

search:
  ranking:
    # Deprecated content
    api/v1.html: -1
    config-file/v1.html: -1

global-settings:
  redirects: ?

For the monorepo support, I still think having a setting on the admin is a good place for it instead of a file (so we don't need to guess from what file to read this setting that will tell us from what other file read the real settings), and will also give more flexibility about where to put the file, each config under each subdirectory (but we need to be explicit about relative and absolute paths).

Use configuration file namespaces/overrides, and still just one set of .readthedocs.yaml and .readthedocs-global.yaml

This is mainly to share the same config between projects, but I think most of the time there is little to share, like search the paths on each project will be different, and will also bring confusion about what overrides what (are values merged or replaced?).

[agjohnson]

If we have two files for global and per-version settings, we still need to choose from what branch/version we read that file

This is going to be a problem regardless of the number of files. There needs to be a single, canonical file + branch that is responsible for project-level settings, so this is still going to be confusing UX.

we can just use the same config file, adding the settings under the "global" key

Hrm yeah, this would be worth discussing more. Given the point above, which is less confusing: a single file with a globals key, or separate configuration file? I'm not sure there is a really strong benefit either way, so would maybe lean to a singular file.

We'd probably want something other than globals though, as this key isn't actually applied globally. That is, with globals defined on branch agj/foo, globals doesn't do anything -- they only do something on the default branch. This could be communicated perhaps just with a better configuration key naming -- on_default, when_default, something better, etc.

At that point, the maintainer's UX around project-level settings is probably similar whether they are authoring a special file or a special key in our normal file. I do like a configuration file setting for the file name more than I do a configuration postfix -- the postfix idea is much harder to communicate.

For the monorepo support, I still think having a setting on the admin is a good place for it instead of a file

Yeah, I don't think there is a way to do this without a UI setting. Even in my namespace example above, there would need to be a UI setting for the namespace to use.

This is mainly to share the same config between projects, but I think most of the time there is little to share

Yeah, it's only arguably better UX if we have a single config file. Users most likely are just going to change the Sphinx dir setting between copies.

But, if we have multiple sets of configuration files, I would do namespacing before supporting a readthedocs.{postfix}.yaml and readthedocs-globals.{postfix}.yaml. This seems much easier to communicate.

[agjohnson]

Also, noted on a call today, a configurable readthedocs.yaml path makes support harder. This was seeming like an okay option otherwise, though mostly for the use case of multiple sphinx projects. The conversation here will still be applicable to redirect and project-level configuration in our config file, but for supporting multiple projects and shared documentation sources, we're going to start with an extension instead.

[mattip]

Another data point: due to #8995, PyPy (which uses the same repo to generate https://pypy.readthedocs.io/en/latest/ * and https://rpython.readthedocs.io/en/latest/) needs to start using a config file. So we need the ability to specify multiple config files.

• An alias to https://doc.pypy.org/en/latest/

[ericholscher]

Also, noted on a call today, a configurable readthedocs.yaml path makes support harder. This was seeming like an okay option otherwise, though mostly for the use case of multiple sphinx projects. The conversation here will still be applicable to redirect and project-level configuration in our config file, but for supporting multiple projects and shared documentation sources, we're going to start with an extension instead.

This seems like the obvious path forward, similar to what we're doing with conf.py historically. I think we really need to find a way to make this configurable, and putting an option in the DB seems like the obvious place? I think it's worth any added complexity to have a good answer that's simple to understand and implement, for users with a monorepo or similar.

[stsewd]

netlify has a "base directory" option for cases like this https://docs.netlify.com/configure-builds/overview/#basic-build-settings (instead of having to specify the path to a config file, the user will specify a path to a directory).

[humitos]

@ericholscher

This seems like the obvious path forward, similar to what we're doing with conf.py historically. I think we really need to find a way to make this configurable, and putting an option in the DB seems like the obvious place?

I think conf.py in the database was a good idea originally but then it caused, and still causes, confusion to users. Now we have multiple places to specify the same configuration but one overrides the other (config file overrides db).

Also, one of the issues around "make support harder" mentioned by @agjohnson is that we (and users) will need to check multiple places to find our which is the correct configuration file read by that project in particular. By default, everybody thinks it will be .readthedocs.yaml but it may not be. We already have this confusion when having multiple config files (.readthedocs.yaml and readthedocs.yml for example).

Another "feature" that we will lose by adding more configs to the database is that deleting the project and re-importing it may not work --because configs saved into the database will be lost.

I'm not saying that adding a new config into the database is bad, but we've been avoiding this on purpose for these reasons and others. So, I think we should consider them and think more deeply if it's a good idea and if we are fine covering the extra complexity needed (on the daily basis support) before moving forward with it --even if "it's just a new field in the database" looks simple to implement.

Finally, the "namespace's idea" from @agjohnson (#8811 (comment)) doesn't look bad to me and we could not depend on an extra db field (as he suggested: Project.configuration_namespace) and use the project slug to match it (note that re-importing under the same slug would still work in that case).

[stsewd]

This issue was renamed, but I think the old title is more accurate to what we have discussed initially (support for multiple config files in one project), and I think this feature should still be considered, it's easier to share the same repo, but hard to have different requirements (well, this is possible with a custom script using build.jobs), but other settings like search can't be separated between projects. I'm opening a new issue to discuss the project level options in the config file at #9188

[ericholscher]

Just another note, with build.commands users can now run a custom build that does exactly what they want based on the READTHEDOCS_PROJECT. Not a great solution, but at least a reasonable workaround.

[CagtayFabry]

Just another note, with build.commands users can now run a custom build that does exactly what they want based on the READTHEDOCS_PROJECT. Not a great solution, but at least a reasonable workaround.

Thank you for the hint @ericholscher
could you provide a quick example (or rough list of steps) how one could use this in case of e.g. different docs in one monorepo like mentioned in the OP?

[stsewd]

could you provide a quick example (or rough list of steps) how one could use this in case of e.g. different docs in one monorepo like mentioned in the OP?

You can create a script that maps each project (using the READTHEDOCS_PROJECT env var) to a command/directory.
That script will make use of this feature https://docs.readthedocs.io/en/latest/build-customization.html#override-the-build-process.

You can also make use of this extension https://github.com/readthedocs/sphinx-multiproject.

[CagtayFabry]

great, thank you very much :)

[benjaoming]

I really like your first point @humitos, bringing in Project level vs. Version level... I'd even add Build level just to be sure 🙈

And I'm not even thinking about building new awesome features with different configurations for projects and versions, my trail of thought is from reproducible builds. I.e. if a single database field outside of Git history decides a critical build aspect, then we're moving away from reproducible builds.

I don't think that a "base directory" or "config file namespaces" fundamentally changes this.

[stsewd]

re: the implementation from #10001

This is the second idea I put on the original issue. The main downside would be: When building an old version, we will try to use the current value of the config file name, and probably those versions were using the old name.

We could solve this by introducing the same setting at the version level, if set to None, default to the value from the project, if that is None, default to our normal behavior.

But there is a workaround, changing the project-level value before building the "problematic" version, so maybe we can decide about introducing the per-version config later.

[benjaoming]

This is the second idea I put on the original issue. The main downside would be: When building an old version, we will try to use the current value of the config file name, and probably those versions were using the old name.

I'd be interested in hearing your comment to my attempt of a practical TODO for #10001 here: #10001 (comment)

[stsewd]

I'd be interested in hearing your comment to my attempt of a practical TODO for #10001 here: #10001 (comment)

I think we don't need to track the modified date or have lots of rules if we just allow users to set the config file per version explicitly, then we just have a simple check version.config_file or project.config_file or DEFAULT.

[benjaoming]

I think that a Project.rtd_conf_file_changed is very useful: It's because of the options that we'd want (potentially) to present someone who is updating a Project.rtd_conf_file with: "Do you want to overwrite the rtd_conf_file value for all existing Versions created after date YYYY-MM-DD?"

I'm not saying that the UI would be implemented in version 1 MVP... but storing that date is "free", so I'd just do that and see what happens later. It'd keep this data, it might become important. It means that we have a bit more freedom to come up with a relaxed analysis of what "reproducibility" means now and then return to the subject later.

[stsewd]

"Do you want to overwrite the rtd_conf_file value for all existing Versions created after date YYYY-MM-DD?"

Changing the value of the project level config will have effect only over versions that don't have an explicit value set (version level config has priority over project config level), the date isn't relevant in that change.

If you mean, something like "bulk" update, that's probably better done using the API, like the user can query all versions that don't have an explicit config file value set, and set it to the current value before changing the new value at the project level.

And if we want this information for analytics or similar, we already track when a project changes with our Historical models

readthedocs.org/readthedocs/projects/models.py

Line 470 in 96fe669

history = ExtraHistoricalRecords()

Isn't that easy to query, but the information is there.

[benjaoming]

@stsewd makes sense that the data can be fetched in historical records, just want to make the option explicitly available for later UI interactions.

Changing the value of the project level config will have effect only over versions that don't have an explicit value set (version level config has priority over project config level), the date isn't relevant in that change.

I would leave that up to the user. They are in charge of their own git release branches and can choose to change the config file layout here as well. Edit: But I'd also defer this entire part of the design until later, just want to make sure that we are free to implement whatever is asked for.

[ericholscher]

Changing the value of the project level config will have effect only over versions that don't have an explicit value set (version level config has priority over project config level), the date isn't relevant in that change.

I agree with @stsewd, we want this implementation to be super simple, and not over-engineered. If we find user requirements that need some complexity, we can discuss, but I really want it to be explicit.

@stsewd can you explain (or link in the above massive thread) the idea of doing it per version instead of per-project? I got sucked into another large thread around redirects in the config file, so I think refreshing the debate here makes sense.

In particular it feels like we have a few options in my mind:

• .readthedocs.yaml can define multiple projects in the same repo. This is feasible, but really messy, especially if the goal is to solve monorepo issues where different projects want control over their own docs
• Path to .readthedocs.yaml in the DB, defined by either the project or version, or both? This is how we solved this historically with conf.py, and I think the most obvious.

My primary goal here is to unblock users who are hitting this problem currently, and get a bit more data on what works. I think a path at the project level solves this, but does have downsides that we know historically around applying to all versions at once. I'm fine with Version-specific, as that's more flexible/explicit, but seems messy to maintain, but at least explicit.

To address the above points around debugging -- I think the new build pages makes this a lot more explicit with our debug view that could show an explicit .readthedocs.yaml path.

[stsewd]

@stsewd can you explain (or link in the above massive thread) the idea of doing it per version instead of per-project? I got sucked into another large thread around redirects in the config file, so I think refreshing the debate here makes sense.

I think most of the discussion about project level vs version level was mostly related to #9188, rather than having a path to the config file itself.

In particular it feels like we have a few options in my mind:

There was also the idea about namespaces that Anthony mentioned in #8811 (comment). Which is kind of similar to 1), but using different files instead of just one.

[benjaoming]

I think it's a bigger challenge to come up with a different approach than the one that's already sponsored by community, that's why I really like #10001. I can see it work, so I'm pitching in safeguards to ensure that different design choices can be left open. I don't think that's over-engineering @ericholscher. I think over-engineering is when you create complexity that's only useful in a hypothetical case. Storing a bit of extra data isn't complex. If there are things in my list of extra small TODOs to add that are in fact complex, we can trim them out 👍

[ericholscher]

Storing a bit of extra data isn't complex.

It depends :) Where do we store that data, how long do we keep it, do we show it to users, is it lies in some cases, etc?

But I don't want to get too deep into specific implementation trade offs...

Reframing the debate on our short-term goals

But I really want to focus the goal that I'm trying to achieve:

• Design goal 1: Build something that gets our users unblocked
• Design goal 2: Doesn't totally shoot us in the foot later, and gives us some options for changes
• Design goal 3: Doesn't take a ton of work/effort on our side to implement

It's goal 2 where we keep getting stuck :) I think just adding a feature-flagged project-level config solves this for now, without adding a bunch of additional work around versions, changes, updates, etc. "It's project-level, and we know how that breaks, if that works for you, here it is, just email us to enable it".

So I think the more actionable question I have is: "What downside risk is there to enable project-level rtd_yaml that we put behind a feature flag?" -- and if there is some super obvious downside, is there another approach that we can use to move forward here without getting bogged down in the various endless debates from above.

Or a @benjaoming put it:

I think it's a bigger challenge to come up with a different approach than the one that's already sponsored by community, that's why I really like #10001. I can see it work, so I'm pitching in safeguards to ensure that different design choices can be left open.

💯 -- let's do that, but not ship it to everyone, to give ourselves an option to change course if needed. Instead of engineering safeguards, let's just gate it behind email -- it's the best safeguard, and gives us information in which to further our design discussion with real use cases, instead of theoretical ideas we have.

[humitos]

This discussion went pretty deep and complex in multiple directions. I'll try to concise and expose my thoughts about the 2 different alternatives that I consider valid and doable:

New field into the database

I think the easiest way to solve this problem is by adding a new Project.config_path field into the database. This simplicity comes with some downsides like breaking the builds if the project is re-imported and not immediately knowing what's the YAML file used to build the docs (while debugging or as even as a user --we will need to show this field in the UI somewhere to avoid confusions)

Overrides at the same .readthedocs.yaml file

There is another approach here that I like more and is the "override idea" that @agjohnson explained at #8811 (comment). Summarized, it will be:

• one and only one YAML config file
• it gives it the ability to support multiple documentation set from the same repository
• it keeps the YAML file versioned per-version
• if users delete the project and re-import them under the same slug, they will keep building

build:
  os: ubuntu-22.04
  tools:
    python: "3"

sphinx:
  configuration: docs/conf.py

overrides:
  landing-with-pelican:  # project slug
    # TODO: how do I remove "build.jobs" from upper?
    build:
      commands:
      - pelican build ...
  api-documentation-sphinx:
    build:
      jobs:
        post_checkout:
          - git fetch --unshallow
    sphinx:
      configuration: docs/api/conf.py
  developer-documentation-mkdocs:
    build:
      commands:
        - mkdocs build ...

I think this implementation gives us a lot of flexibility with a small complexity in the implementation, but a clear view of what's happening and where the data is from a user perspective --plus reproducibility over time.

The downside that I'm noticing is that # TODO that I put there. How do we enforce the removal of an inhered key? This is a problem that many other software have (like Docker with docker compose files), and the solution here is to repeat the key you don't want to inherit in all the overrides that require that key. It's a small price to pay to have more flexibility and better UX, I'd say.

Note
In this example we are building 4 different projects:

• User documentation with Sphinx (regular Read the Docs build process)
• Landing pages with Pelican (build.commands)
• API documentation with Sphinx (build.jobs)
• Developer documentation with MkDocs (build.commands)

[ericholscher]

We had a call around this, and decided to move forward with the approach in #10001.

High level:

• No feature flag, just implement it on the Project model.
• A v2 would include some nice to have additions that allow DB-level Version-specific config files: yaml_file = version.yaml_file or project.yaml_file or None

Requirements:

• Ability for the user to track .readthedocs.yaml changes (beta dashboard exposes this)
• Make sure all code is updated to use the correct path

[benjaoming]

A quick update on why this issue is now closed and fixed:

The proposal from @ewdurbin in #10001 was adopted and will premiere in the next deployment. If there's any feedback, please don't hesitate to write here or open a new issue 👍

Thanks for all the thoughts, initiatives and good riddance to address this challenging new feature ❤️

[benjaoming]

Btw. it might be interesting to open up any follow-up issues to the approach, but maybe that should anticipate a bit more usage and user feedback, thoughts @ericholscher ?

[astrojuanlu]

FTR, here are the docs https://docs.readthedocs.io/en/latest/guides/setup/monorepo.html 🥳

[CagtayFabry]

I just saw this option in the web UI for the first time 👍
Thank you everyone who contributed to make this happen, awesome feature! 🚀

[benjaoming]

Thanks @CagtayFabry ❤️

(cross-posting from #10001) There's a little public example of the new feature available here:

https://github.com/readthedocs-examples/example-monorepo