Improving docusaurus sidebar generation (#1927)

[Mykyta-Serbynevskyi]

This PR improves automatic generation of docs sidebar via docusaurus tool.

At the moment sidebar generated via docusaurus matches sidebar from docs.airbyte.com

Things left to be done:

• Add "Career and opened positions" folder to sidebar
• Review "Project overview" sidebar folder and update it respectively to docs.airbyte.com
• Find a way how to use direct url links in sidebar for docusaurus e.g "CLI documentation" in docs.airbyte.com

[CLAassistant]

All committers have signed the CLA.

[Mykyta-Serbynevskyi]

Small update re changes introduced in this PR:

1. "Career and opened positions" folder has been added to docusaurus sidebar;
2. "Project overview" folder has been added to docusaurus sidebar;
3. In order to match sidebar structure in docs.airbyte.com docusaurus sidebar folders & sub-folders are sorted using numeric prefix, e.g:

4. Direct url links in sidebar for docusaurus are not working using markdown syntax, the following workaround is introduced instead:

[Amruta-Ranade]

The Careers and open positions folder is no longer a part of docs and shouldn't show up in the sidebar

[Mykyta-Serbynevskyi]

Hi @Amruta-Ranade

I have removed "Career-and-open-positions" folder from sidebar. Now it looks like this:

Let me know if you have any further comments or recommendations.
Thank you.

[Amruta-Ranade]

Thank you!!

[supertopher]

@Mykyta-Serbynevskyi we now have broken links when I try to build. I would guess this is from killing that folder but I didn't look super hard

[Local Search] [INFO]: Index docs-default-current written to disk
g[ERROR] Unable to build website for locale en.
[ERROR] Error: Docusaurus found broken links!

Please check the pages of your site in the list below, and make sure you don't reference any path that does not exist.
Note: it's possible to ignore broken links with the 'onBrokenLinks' Docusaurus configuration, and let the build pass.

Exhaustive list of all broken links found:

- On source page path = /project-overview/Changelog/changelog:
   -> linking to ../../02-integrations/01-sources/amplitude.md (resolved as: /02-integrations/01-sources/amplitude.md)

- On source page path = /project-overview/Changelog/connectors:
   -> linking to ../../02-integrations/01-sources/slack.md (resolved as: /02-integrations/01-sources/slack.md)
   -> linking to ../../02-integrations/01-sources/mixpanel.md (resolved as: /02-integrations/01-sources/mixpanel.md)
   -> linking to ../../02-integrations/01-sources/zoom.md (resolved as: /02-integrations/01-sources/zoom.md)
   -> linking to ../../02-integrations/01-sources/braintree.md (resolved as: /02-integrations/01-sources/braintree.md)
   -> linking to ../../02-integrations/01-sources/twilio.md (resolved as: /02-integrations/01-sources/twilio.md)
   -> linking to ../../02-integrations/01-sources/marketo.md (resolved as: /02-integrations/01-sources/marketo.md)
   -> linking to ../../02-integrations/01-sources/github.md (resolved as: /02-integrations/01-sources/github.md)
   -> linking to 03-connectors.md (resolved as: /project-overview/Changelog/03-connectors.md)
   -> linking to ../../02-integrations/01-sources/postgres.md (resolved as: /02-integrations/01-sources/postgres.md)
   -> linking to ../../02-integrations/02-destinations/postgres.md (resolved as: /02-integrations/02-destinations/postgres.md)

    at reportMessage (/Users/topher/workspace/airbyte/docusaurus/node_modules/@docusaurus/utils/lib/jsUtils.js:88:19)
    at handleBrokenLinks (/Users/topher/workspace/airbyte/docusaurus/node_modules/@docusaurus/core/lib/server/brokenLinks.js:153:35)
    at async buildLocale (/Users/topher/workspace/airbyte/docusaurus/node_modules/@docusaurus/core/lib/commands/build.js:176:5)
    at async tryToBuildLocale (/Users/topher/workspace/airbyte/docusaurus/node_modules/@docusaurus/core/lib/commands/build.js:37:20)
    at async mapAsyncSequential (/Users/topher/workspace/airbyte/docusaurus/node_modules/@docusaurus/utils/lib/jsUtils.js:36:24)
    at async build (/Users/topher/workspace/airbyte/docusaurus/node_modules/@docusaurus/core/lib/commands/build.js:72:21)

[Mykyta-Serbynevskyi]

Hi @supertopher
I could not reproduce this issue on my working PC. No broken links are found while building docusaurus sources.
Here are the steps that I used:

mkdir <dir_name>
cd <dir_name>
git clone --single-branch --branch mserbynevskyi/1927-Improve-OSS-docusaurus-tooling git@github.com:airbytehq/airbyte.git
cd airbyte/docusaurus/
yarn install && yarn build && yarn serve

I can suggest the following:

• run yarn clear cache to clear cached info
• update yarn to latest version via yarn upgrade

Let me know if it helps.

[supertopher]

@Mykyta-Serbynevskyi I cloned a fresh copy and that worked, though I found that confusing. yarn clear did work back in main airbyte repo owned by airbytehq. It certainly seems weird and quirky to need to clear the cache for that error though. Did you need to use yarn clear during development of this story? I wonder if we should add it to the docusaurus documentation for troubleshooting

[supertopher]

I don't think this is a dealbreaker, but I'd like to hear from @Amruta-Ranade.

@Mykyta-Serbynevskyi performed the work as requested: namely he built an exact copy of the OSS docs sidebar. It appears some links aren't navigable from the sidebar though (in OSS prod or in the PR). ie http://localhost:3000/operator-guides/configuring-airbyte/
This didn't work before so it's not a regression.

Furthermore I think it's okay to ask docs to address these when we see them, but again I'd like to hear from Amruta on this

[supertopher]

I scrolled and sanity checked a good number of these. Things look sane.
in visual comparison of the deployed site the navigation appeared as expected. I think this seems safe to merge once we coordinate with the docs team

nice job Mykyta

[Mykyta-Serbynevskyi]

Hi @supertopher
As for yarn clear cache question - I didn't use it during fresh & local installation of docusaurus docs.
I believe cache conficts could happen when you try build sources several times.
Thus, for end-user who tries to install docusaurus locally for the 1st time, there will not be any issues.

[grishick]

This seems to have broken connectors build
running SUB_BUILD=CONNECTORS_BASE ./gradlew clean build --stacktrace
I get:

Execution failed for task ':airbyte-integrations:bases:standard-source-test:generateSourceTestDocs'. java.io.FileNotFoundException: /Users/gregsolovyev/git/airbyte/docs/connector-development/testing-connectors/standard-source-tests.md (No such file or directory)