Restructuring our documentation

[jchristgit]

At the moment I personally find our documentation to be hard to navigate, even as someone who is part of the team. I think we have room for improvement here.

Currently our documentation tree roughly looks like this:

├── general
│   ├── index.rst
│   └── manual-deploys.rst
├── index.rst
├── meeting_notes
│   ├── 2022-04-07.rst
│   ├── [ ... ]
│   ├── 2024-07-02.rst
│   ├── 2024-07-25.rst
│   ├── index.rst
│   └── template.rst
├── onboarding
│   ├── access.rst
│   ├── index.rst
│   ├── resources.rst
│   ├── rules.rst
│   └── tools.rst
├── postmortems
│   ├── 2020-12-11-all-services-outage.rst
│   ├── [ ... ]
│   ├── 2021-01-10-primary-kubernetes-node-outage.rst
│   └── index.rst
├── queries
│   ├── index.rst
│   ├── [ ... ]
│   └── postgres.rst
├── runbooks
│   ├── index.rst
│   └── postgresql-upgrade.rst
└── tooling
    ├── bots.rst
    └── index.rst

I find this grouping a bit confusing, because it mixes together team-internal information (like meeting notes, perhaps to a degree postmortems, queries and runbooks) with information that might be relevant to other teams (manual deployments, onboarding). I don't think the mix is inherently bad, but I think it's hard to see which parts are meant for consumption of the more casual reader.

I suggest we change our documentation structure to include different categories for reference material (like queries and meeting notes) and explanation documents like the onboarding and tooling documents, and make it a coordinated effort to restructure it and make it more useful for other teams.

I also suggest, although this might be better suited for a separate issue, that we write more documentation about the infrastructure we run, both for ourselves and for other times (possibly including how-to guides for using a service, adding PostgreSQL access, even if it's just as simple as "open an issue in the repo").

Perhaps something like this might be of interest.

I would be happy for other thoughts here.

[jchristgit]

@shtlrs would be more than happy for your input here

[shtlrs]

I agree with the fact that our current structure isn't idea nor optimal.

However, i think this is more of a X/Y problem, and that we should begin by asking ourselves this

What is it actually that we want to document ?

Once we have a response to that, we can start to remodel our current documentation skeleton, and filter out noise from the existant things that we have, and add anything that's missing.

I think some common things we'd need to document is: (based on the things you said as well):

• A diagram illustrating the current architecture that we have.
• Explaining the different deployment processes
• Explaining the different parts of our system that staff can access, and how. (This might require a proceess that won't be as trivial)
• References to different utilities we use to manage our infrastructure, and who owns them.
  etc etc.

The main problem is that docs die very fast, and it's a very difficult process to maintain them, so we shouldn't just write stuff for the sake of writing it, but write stuff to illuminate obscure parts to the outside world that we are allowed to expose, but also keep it easily identifiable and maintanable.

[jchristgit]

I agree with all you said, especially the fact that we shouldn‘t write docs for the sake of it. I think especially an explanation of the deployment processes is very important. Like, I have a PyDis project, how does that get to production?

[jchristgit]

As self-appointed Head Of Documentation I will take care of this.

[jb3]

Another thought: we actually do document more than we think.

We have 46 markdown documents in the kubernetes folder with 668 lines of markdown, our roles have 14 markdown files with 236 lines, but we don't surface these through any solution.

Should we think about surfacing these documents instead with the services that they deploy and link through to the git folders?

[jchristgit]

We can do that - with our current solution we should only have to include the myst parser and then .. include:: the files. Only caveat is e.g. headers might need to be adjusted

[jchristgit]

Joe has done it. He has done it. He has replaced our documentation, for the 5th time, and now we have a semi-decent structure.

I think there are some things to be improved though. For instance, currently the "user-specific documentation" requires users to know which component they're interacting with. For instance, information about the LDAP setup is nested in "Keycloak", whilst the user probably has no idea what a Keycloak even is, with marginally higher chance of even knowing what LDAP is. I think we still have some potential for reordering here, what do you think @jb3?

[jb3]

Definitely more room for restructing, agreed there.

I think that we can maybe have sort of a "user home" which just links into the user sections of other relevant pages (with the card style pages that we've got elsewhere).

I agree the nesting is not ideal for users so definitely better ways to surface that.

I would also like to look at surfacing better the other services we host for the benefit of the team so I'll take that away as a part of this PR.