This repository is wired via CI/CD and scripts to listen to component dependencies and upon their release to build a new version of the documentation for them and publish it.
The repositories involved in the process are:
- /gardener/website-generator (this repository). It contains the CI/CD pipeline definition and related scripts, configuration for building the website with HUGO, including all common framework html, styles, javascript and images, as well as the scripts and build configuration for the build environment container image.
- /gardener/component is any repository in the Gardener organization contributing Docforge documentation manifest, for which /gardener/website-generator is configured to listen for releases. The repository /gardener/documentation is one such example. It contains the source content for the website, used by the builder to produce the static HTML to be served.
Currently, /gardener/documentation is configured to deliver the Gardener (core) documentation along with the other website content assets (blogs/adopters/community), but it is transitioning to a repository dedicated to only crosscutting documentation, such as tutorials and website content. The Gardener repository will be configured to contribute its own documentation upon release. All other components wil follow using the same scheme depicted on the diagram.
- /gardener/website is the home repository for the https://gradener.cloud website. It hosts the website content produced by the website-generator and is configured to have it served by GitHub Pages.
The website builds and deployments are orchestrated by Concourse CI/CD pipeline and triggered upon depending component release or upon changes in /gardener/documentation or /gardener/website-generator repositories. The build results are then pushed to /gardener/website/docs and served as a GitHub Pages site.
The build and deployment triggered by Concourse go through the stages described here. Except the first two stages, the rest are all orchestrated by the ./ci/build script.
- Container Image:
eu.gcr.io/gardener-project/gardener-website-generator
- Environment: see the Dockerfile for details. Noteworthy mentions are:
- Clone the three repositories locally:
- Symlink the
website
folder from thedocumentation
clone ascontent
in thehugo
folder inwebsite-generator
clone
website-generator/hugo/content → documentation/website
The stage is performed by a NodeJS script on the website content from website-generator/hugo/content (see the symlink step above) and goes through the following phases:
- Fetch remote pages configured in the website with front matter property
remote: URL
from other GitHub repositories. - Fetch the git commits for all pages. This is later rendered by page components showing contributors and last page update.
- Parse all files and check if any link references a remote site, which has been imported and rewrite those links to reference the local (imported) page.
The stage is performed by Hugo and goes through the following general phases:
- transform the website content (Markdown) from website-generator/hugo/content (see the symlink step above) into static HTML.
- process shortcodes in the content, if any.
- apply the appropriate layouts for the content type and configuration.
The result of the site build is output to the /docs
folder of the gardener repo clone.
The changes to the website home repo clone from previous phase are staged, commited and finally pushed to the gardener repo, where they are immediately served by GitHub Pages.
In most cases when developing site content, you will not need to run the full site build and al you need is a site preview. For the rest of the cases there are several options outlined below.
Feel free to reuse the image utilized by the CI/CD and avoid setting up the tools for the build environment. That has the advantage of keeping you up-to-date with the build setup effortlessly too. Clone the gardener-generator repo in the contianer and use the website-generator/scripts/setup script to setup the other clones as required. After you have cloned the repos in the container, you can reuse the website-generator/.ci/build script for full scale builds. Currently, there is no end-to-end script for the setup stage, so you have to come up with something yourself, mount and run it. Contributions are welcome.
In other rare case or when you cannot use Docker for some reason, see the procedure below on how to setup a build environment and run a local build.
Prerequisites:
- GoLang installed
- NodeJS/NPM installed
- a local folder setup with a Hugo and Docforge executable in it
- a GitHub token generated for the public site
Procedure:
# Fork (if you plan ot make changes here) and clone this repository locally
$ git clone https://github.com/gardener/website-generator.git
# Change to the cloned repo
cd website-generator
# Update needed submodules
git submodule update --init --recursive
# Run make (or make setup) to have all necessary repos cloned, and setup (linked) for you automatically as necessary (if they do not exist).
$ make
# It is highly recommended to supply access token before starting the build to avoid Github API rate limit restrictions.
# For basic authentication (deprecated by GitHub), use the `GIT_USER` and `GIT_PASSWORD` instead.
$ export GIT_OAUTH_TOKEN=<your-github-personal-access-token>
# Set AUTO_PUBLISH to false to instruct the build not to publish changes to the documentation and website repos. This is in the role of CI/CD builds
# and should not be taken over.
$ export AUTO_PUBLISH=false
$ make build
Normally, all steps except make build
are executed once during the initial setup. After that, the build step can then be executed numerous times.
The build results are produced in website/docs
.
The build is parameterized by means of environment variables.
The build will apply some heuristics and infer documentation
and gardener
are the names of cloned repos that are peer to website-generator
. To override their paths, use the coresponding environment variables:
GARDENER_DOCUMENTATION_PATH
sets the path to the documentation repo (default:/documentation
)GARDENER_WEBSITE_PATH
sets the path to the build output repo (default:/website
)GARDENER_GENERATOR_PATH
changes the infered location ofwebsite-generator
defaulting to the directory where the build script is executed.CONTENT
The build scripts will assume the source for building the site is available inhugo/content
(note the symlink content<==> documentation/website we setup earlier). In case you need to override its location, use theCONTENT
environment variable.
It is recommended to supply authentication credentials before starting the build to avoid Github API rate limit restrictions.
For token-based authentication (recommended), use the GIT_OAUTH_TOKEN
environment variable.
For basic authentication (deprecated by GitHub), use the GIT_USER
and GIT_PASSWORD
.
AUTO_PUBLISH
controls the post-build steps publishing the results. Normally it should not be set, unless for the CI/CD pipeline definition build step. To enable, set totrue
.
Site previews aid the process of developing site material or layouts. Similiar to builds, it is recommended to run site previews with docker, using the scripts supplied in /documentation
for that. In case you can't benefit from that for some reason, the instructions for setting it up manually are listed here.
Prerequisites:
- Cloned
/documentation
and/website-generator
repos - Symlink
/website-generator/hugo/content
<==>/documentation/website
- Hugo available on your system
The first two prerequisites are automatically ensured by running make
(or make setup
). Or you can do that manually.
Note that the
/website repo
is not necessary to perform this task.
Procedure:
cd website-generator/hugo
$ hugo serve
You can now explore the site and your changes upon save at http://localhost:1313
.
The instructions above are applicable when using Windows 10 WSL e.g. with Ubuntu. Here are a few hints on how to setup your environment accordingly.
One of the problematic integration areas in WSL are symbolic links. Despite that a good progress was made, it is still necessary to use Windows tools to create symbolic links on a Windows file system. The build and setup scripts above consider WSL environments and will automatically fallback to Windows mklink
command instead of ln
if necessary. A prerequisite to the success of this operation is to enable Windows Developer Mode (Settings > Update & Security > For developers > Use developer features).
Should you experience problems on this stage, create the symlink manually. The build script will work with existing content
symlink and not attempt to create one.
To create the content
symlink in website-generator
repo clone's hugo
folder, start CMD as Administrator and make symlink to folder (/D
)
[website-generator-repo-clone-path]>\hugo>mklink /D content <absolute-path-to-gardener-documentation-website-folder>
A successfull operation will signal output similiar to this:
symbolic link created for content <<===>> <actual-path-on-your-system>\website
Note that if you are using GitBash, it is likely to be configured to change line endings with Windows (CRLF) instead of UNIX (LF) style. You will need to change line endings in the bash scripts you plan to use. If you use VS Code look in the lower right corner and you will notice CRLF, which can be changed to LF. Notepad++ also has a Line Ending change option. Consult with your favorite tool options on how to deal with this best.
Install Docker Desktop for Windows if you plan to make use of the build image to preview changes or run the same full-scale build as the CI/CD.
A helpful article on setting up WSL to work flawlesly with Desktop Docker for Windows is available here. The list below is a TL;DR; for some key points.
- Volumes mounting
If you plan to use the image you will need to mount at least source content volume and probably a scripts volume. Docker and volume mounting has some subtleties when it comes to Windows. In short, make sure you have the
C:
drive shared in Docker Desktop and no firewall rule getting in the way, and ensure that host mount paths start with/c
and not/mnt/c
. If you setWSL=1
before running the scripts, they will try to remove the leading/mnt
. - Enable using the daemon from Docker Desktop for Windows in WSL.
- Go to Docker Desktop > Settings and check the "Expose daemon on tcp://localhost:2375 without TLS" checkbox enabled.
- Setup
.bashrc
in WSL with the following environment variableexport DOCKER_HOST=tcp://localhost:2375
- Download Hugo from GitHub.
Note: You need to download the extended version.
- Download Docforge from GitHub.
Note: In case of a problem with the executable, here is how to build your local version of Docforge.
- Create a folder there and name it appropriately.
- Extract the downloaded archive files into the folder.
- Clone the Docforge repository.
- Open a new terminal.
- In the terminal, navigate to the folder where you cloned Docforge.
- Enter
export LOCAL_BUILD=1 && ./.ci/build
.
Note: This step is only for Windows. For MAC or Linux, you should call
make build
.
- In your system, open the folder where you cloned Docforge.
- Open the "bin" folder and copy the "docforge" file there.
- Paste the file into the folder containing the Hugo executable.
- Add the binary in the PATH environment.
- Navigate to Profile -> Settings.
- Navigate to Developer settings -> Personal access tokens.
- Select "Generate new token".
- Enter your password.
- Enter a name for your token.
- Select an expiration date.
Note: While possible to create a token that never expires, it is advisable to change your tokens every couple of months.
- Check the "repo" and "admin:repo_hook" checkboxes.
- Select "Generate token".
- Copy and save your token.
Note: This is the only time you will be able to see your token. It is highly recommended to save your token somewhere on your computer.
To troubleshoot failed website production pipeline in Concourse, use fly
:
- Download fly (look for
cli:
at the bottom right of the screen at https://concourse.ci.gardener.cloud/) - Login to a target with fly:
Fly will request that you login at a URL and automatically intercept a successfull login.
$ fly -t gardener login -c https://concourse.ci.gardener.cloud/ -n gardener
- Intercept the failed container with
Example is https://concourse.ci.gardener.cloud/teams/gardener/pipelines/gardener-website-generator-master/jobs/master-head-update-job/builds/102 , where 102 is the number of the build job that you want to inspect.
$ fly -t gardener hijack -u <url-of-your-build>