Skip to content

Commit

Permalink
ci(documentation): merge index.json of doccarchives and keep CSS and …
Browse files Browse the repository at this point in the history
…JS assets for online web documentation

Signed-off-by: Pierre-Yves Lapersonne <pierreyves.lapersonne@orange.com>
  • Loading branch information
pylapp committed Nov 13, 2024
1 parent bc38100 commit b30d5db
Show file tree
Hide file tree
Showing 3 changed files with 158 additions and 42 deletions.
20 changes: 11 additions & 9 deletions docs_release/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,15 +194,15 @@ You can also [look inside this commit](https://github.com/Orange-OpenSource/ouds
## About documentation
The documentation tool in use is [Swift DocC](https://www.swift.org/documentation/docc/) ; we try as best as we can to use these conventions in our source code, and use also DocC catalogs so as to let Xcode build the doccarchives and render the documentation in the Apple fashion.
The documentation should be updated during a release ; to do that a script has been designed to update the GitHub Pages dedicated to the documentation in the OUDS iOS repository.
However, because we faced several issues with `swift package`, `xcodebuild` and `xcrun docc` commands and were not able yet to use them to generate the doccarchives and the HTML pages, a manual step must be done before. For further details, see [#95](https://github.com/Orange-OpenSource/ouds-ios/issues/95) and [#168](https://github.com/Orange-OpenSource/ouds-ios/issues/168) (in few words: not possible to manage easily multiple targets in Swift Package, _UIKit_ not supported).
The documentation tool in use is [Swift DocC](https://www.swift.org/documentation/docc/) ; we try as best as we can to use these conventions in our source code, and use also _DocC_ catalogs so as to let _Xcode_ build the _doccarchives_ and render the documentation in the Apple fashion.
The documentation should be updated during a release ; to do that a script has been designed to update the _GitHub Pages_ dedicated to the documentation in the OUDS iOS repository.
However, because we faced several issues with `swift package`, `xcodebuild` and `xcrun docc` commands and were not able yet to use them to generate the _doccarchives_ and the HTML pages, a manual step must be done for the update. For further details, see [#95](https://github.com/Orange-OpenSource/ouds-ios/issues/95) and [#168](https://github.com/Orange-OpenSource/ouds-ios/issues/168) (in few words: not possible to manage easily multiple targets in Swift Package, _UIKit_ not supported, no unified _doccarchive_ for several targets).
First, you will have to use Xcode to build the documentation (_Product > Build Documentation_) which will open the documentation viewer.
Then, **for each documentation catalog of the Swift package, i.e. for each target**, export the doccarchive **in your _Downloads_ folder**.
Today you will have to do this operation for the doccarchives *OUDS*, *OUDSComponents*, *OUDSFoundations*, *OUDSModules*, *OUDSThemesInverse*, *OUDSThemesOrange*, *OUDSTokensComponent*, *OUDSTokenSemantic* and *OUDSTokenRaw*.
First, you will have to use _Xcode_ to build the documentation (_Product > Build Documentation_) which will open the documentation viewer.
Then, **for each documentation catalog of the Swift package, i.e. for each target**, export the _doccarchive_ **in your _Downloads_ folder**.
Today you will have to do this operation for the _doccarchives_ *OUDS*, *OUDSComponents*, *OUDSFoundations*, *OUDSModules*, *OUDSThemesInverse*, *OUDSThemesOrange*, *OUDSTokensComponent*, *OUDSTokenSemantic* and *OUDSTokenRaw*.
Then, you will have to run the script `uploadWebDoc.sh` which will use these doccarchives, get their HTML content and upload the GitHub Pages branch.
Then, you will have to run the script `uploadWebDoc.sh` which will use these _doccarchives_, get their HTML content and upload the GitHub Pages branch.
```shell
# To show the help:
Expand All @@ -222,5 +222,7 @@ Keep in mind everything is stored in _/tmp_ folder with the execution timestamp,
The plugin produces a lot of files, a lot. For example for our v0.1.0, more than 6,000 files have been created for a ZIP archive of about 17 MB.
Thus, keeping all versions of the documentation is a non-sense, no one will read it and it will increase the size of the branch in our VCS tool, consuming a lot of bandwidth, and reaching limits of Git to handle large amounts of file. We would like to avoid to force developers to define a specific Git configuration to handle such massive branch.
We prefer to build ZIP documentation and Xcode doccarchives to add as artifacts of releases.
Thus, the online version of the documentation is for the last release, and each release in GitHub contains doccarchive files generated through Xcode and ZIP of HTML files picked from the *gh-pages*.
We prefer to build ZIP documentation and _Xcode_ _doccarchives_ to add as artifacts of releases.
Thus, the online version of the documentation is for the last release, and each release in GitHub contains _doccarchive_ files generated through _Xcode_ and ZIP of HTML files picked from the *gh-pages*.
You may see also the *merge-json-indexed.py* Python script: this tool is sued by the mentioned Shell script to merge easily all *index.json* files of all _doccarchives_ so as to produce one single and unique *index.json* file used mainly for the side bar menu and the highlight bar.
85 changes: 85 additions & 0 deletions docs_release/merge-json-indexes.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
#!/usr/bin/env python3
#
# Software Name: OUDS iOS
# SPDX-FileCopyrightText: Copyright (c) Orange SA
# SPDX-License-Identifier: MIT
#
# This software is distributed under the MIT license,
# the text of which is available at https://opensource.org/license/MIT/
# or see the "LICENSE" file for more details.
#
# Authors: See CONTRIBUTORS.txt
# Software description: A SwiftUI components library with code examples for Orange Unified Design System
#

import glob
import json
import os
import sys

def merge_json_files(file_list):
"""
Merges multiple JSON files into a single JSON object.
Args:
file_list (list): List of paths to the JSON files to be merged.
Returns:
dict: A dictionary containing the merged data from the JSON files.
"""
merged_data = {}

for file_name in file_list:
file_short_name = os.path.basename(file_name)
print(f"🍊 🔨 Processing file: '{file_short_name}'")

with open(file_name, "r") as f:
data = json.load(f)
for key, value in data.items():
if key in merged_data:
if isinstance(value, list):
merged_data[key].extend(value)
elif isinstance(value, dict):
merged_data[key] = merge_dicts(merged_data[key], value)
else:
merged_data[key] = value

return merged_data

def merge_dicts(dict1, dict2):
"""
Recursively merges two dictionaries.
Args:
dict1 (dict): The first dictionary to merge.
dict2 (dict): The second dictionary to merge.
Returns:
dict: The resulting dictionary from merging the two dictionaries.
"""
for key, value in dict2.items():
if key in dict1:
if isinstance(value, dict):
merge_dicts(dict1[key], value)
elif isinstance(value, list):
dict1[key].extend(value)
else:
dict1[key] = value
return dict1

# ------------ Main ------------
if __name__ == "__main__":
if len(sys.argv) != 2:
print("Usage: python3 script.py <path_to_json_directory>")
sys.exit(1)

print("🍊 👉 Processing files")
json_directory = sys.argv[1]
file_list = glob.glob(os.path.join(json_directory, '*.json')) # Will process all JSON files (suppsoed to be renamed for example, so no more details)
merged_result = merge_json_files(file_list)

output_file_path = os.path.join(json_directory, 'index.json')
with open(output_file_path, "w") as f: # The index.json file is the only expected one for the Apple documentation
json.dump(merged_result, f, indent=4)

print("🍊 👍 Completed!")
95 changes: 62 additions & 33 deletions uploadWebDoc.sh → docs_release/uploadWebDoc.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,18 +21,18 @@ set -euo pipefail
# CONTEXT:
# - `swift package` cannot manage UIKit and fails when it meets some calls to that
# - `xcrun docc` or `xcodebuild docbuild` fail to manage targets and Swift Package products
# Thus need tp rely on manualy generated doccarchive files before processing them
# Thus need to rely on manualy generated doccarchive files before processing them.
# See https://github.com/Orange-OpenSource/ouds-ios/issues/168
# See also https://github.com/Orange-OpenSource/ouds-ios/issues/95
# (╯°□°)╯︵ ┻━┻

# The folder where all the .doccarchive folders to process are
# If workspace is a Git repository you should NOT place the doccarchives inside, because they will be wiped
# The folder where all the .doccarchive folders to process are.
# If workspace is a Git repository you should NOT place the doccarchives inside, because they will be wiped.
DOCCARCHIVES_PATH="$HOME/Downloads"

# Services pages (like GitHub Pages) custom subdomain for the CNAME, don't forget to verify it in organization side for security reasons!
# For example, with GitHub pages; given the "ouds-ios" project for "Orange-OpenSource" organization,
# the custom domain "ios.unified-design-system.orange.com" will thus automatically redirect to "orange-opensource.github.io/ouds-ios"
# For example, with GitHub pages, given the "ouds-ios" project for "Orange-OpenSource" organization,
# the custom domain "ios.unified-design-system.orange.com" will thus automatically redirect to "orange-opensource.github.io/ouds-ios".
SERVICE_PAGES_DOMAIN="ios.unified-design-system.orange.com"

# Some HTML fragments to add in the HTML global page index.html
Expand All @@ -43,14 +43,14 @@ HTML_PROJECT_URL="https://github.com/Orange-OpenSource/ouds-ios"
HTML_PROJECT_COPYRIGHT="Orange SA"
HTML_PROJECT_WIKI_URL="https://github.com/Orange-OpenSource/ouds-ios/wiki"

# The name of the Git branch hosting the documentation (e.g. GitHub Pages branch defined in repository)
# We suppose all the documentation will be in this dedicated branch
# The name of the Git branch hosting the documentation (e.g. GitHub Pages branch defined in repository).
# We suppose all the documentation will be in this dedicated branch.
SERVICE_PAGES_BRANCH="gh-pages"

# Path where the documentation will be temporary
DOCS_DIRECTORY="./docs"
DOCS_DIRECTORY="../docs"

# The generated name of the ZIP containing the generated sources of documentation (for archive)
# The generated name of the ZIP containing the generated sources of documentation (for archive).
timestamp=$(date +%s)
DOCUMENTATION_ZIP_NAME="ouds-docs-$timestamp.zip"
DOCUMENTATION_ZIP_LOCATION="/tmp/$DOCUMENTATION_ZIP_NAME"
Expand Down Expand Up @@ -143,17 +143,20 @@ for arg in "$@"; do
esac
done

# Get the version of the library to add it in main page.
if [[ -z "$lib_version" ]]; then
_ "Parameter --libversion is mandatory. Exits. ($EXIT_BAD_PARAMETER)" true
exit $EXIT_BAD_PARAMETER
fi

# Upload to Git repository or not.
if [[ $use_git -eq 1 ]]; then
_ "✔️ OK, Git will be used"
else
_ "✔️ OK, Git will NOT be used"
fi

# Keep ZIP or not.
if [[ $no_zip -eq 1 ]]; then
_ "✔️ OK, no ZIP archive will be done"
else
Expand All @@ -164,7 +167,7 @@ if [[ "$use_git" -eq 0 && "$no_zip" -eq 1 ]]; then
_ "🥴 WARNING: What do you use this script for? You should at least save the doc in Git repository or in ZIP file"
fi

# Ask the user if he/she wants to go further (updating documentation updates the production website)
# Ask the user if he/she wants to go further (updating documentation updates the production website).
read -p "❓ Do you want to update the documentation? (yes/YES/Y/y): " answer
if [[ ! "$answer" =~ ^(yes|YES|Y|y)$ ]]; then
_ "👋 Bye!"
Expand All @@ -173,15 +176,16 @@ else
_ "👍 Ok, let's go!"
fi

# Check if the folder containing doccarchives exists and has doccarchives
# Check if the folder containing doccarchives exists and has doccarchive.
if [ ! -d "$DOCCARCHIVES_PATH" ]; then
_ "'$DOCCARCHIVES_PATH' does not exist, how can I get the doccarchives? Exits. ($EXIT_ERROR_BAD_PREREQUISITES)" true
exit $EXIT_ERROR_BAD_PREREQUISITES
fi

shopt -s nullglob # Allow glob patterns to return nothing if no match
shopt -s nullglob # Allow glob patterns to return nothing if no match.
doccarchives=("$DOCCARCHIVES_PATH"/*.doccarchive)

# Check of there are things to process.
if [ ${#doccarchives[@]} -eq 0 ]; then
_ "There is no doccarchive in '$DOCCARCHIVES_PATH. Exits. ($EXIT_ERROR_BAD_PREREQUISITES)" true
exit $EXIT_ERROR_BAD_PREREQUISITES
Expand All @@ -193,12 +197,12 @@ start_time=$(date +%s)
# --------------------------------

if [[ $use_git -eq 1 ]]; then
if [ -d ".git" ]; then
if [ -d "../.git" ]; then

# Xcode keeps files and dislikes updates of local branches...
_ "🚨 You should close Xcode or any software working on this workspace, before going further, just in case of..."
_ "Press any key to continue..."
read -n 1 -s # Don't care of the input, just want the user be ready in the end
read -n 1 -s # Don't care of the input, just want the user be ready in the end.

_ "✅ This is a Git repository. Please ensure the credentials you need are ready (SSH, HTTPS, GPG, etc.)"
current_branch=$(git rev-parse --abbrev-ref HEAD)
Expand All @@ -217,30 +221,37 @@ _ "👍 Documentation folder created at '$DOCS_DIRECTORY'!"
# Step 2 - For each doccarchive, copy the assets
# ----------------------------------------------

_ "👉 Generating docs..."
_ "👉 Preparing docs..."

# To store the docarchive folder names for later
# To store the docarchive folder names for later.
declare -a doccarchive_names

# Process all .doccarchive folders in the target path
# Process all .doccarchive folders in the target path.
for doccarchiveDir in "${doccarchives[@]}"; do
_ "👉 Generating docs for $doccarchiveDir..."
_ "👉 Processing docs for $doccarchiveDir..."
if [ -d "$doccarchiveDir" ]; then

# Get name of the doccarchive folder and save it
# Get name of the doccarchive folder and save it.
base_name=$(basename "$doccarchiveDir" .doccarchive)
doccarchive_names+=("$base_name")

# Prepare folders
# TODO: Refactor this part by copying only the needed files.
# WARNING: This is pure tikering (╯°□°)╯︵ ┻━┻
# We are pretty sure some files may be not that much useful and can not be copied.
# This solution works, but a refactoring should be done.

# Prepare folders.
mkdir -p "$DOCS_DIRECTORY/data/documentation"
mkdir -p "$DOCS_DIRECTORY/documentation"
mkdir -p "$DOCS_DIRECTORY/js"
mkdir -p "$DOCS_DIRECTORY/css"
mkdir -p "$DOCS_DIRECTORY/index"

# Things to copy are stored at three levels (╯°□°)╯︵ ┻━┻
# WARNING: We rely too much on how files are generated, it is tinkering...
# Copy assets for the browser and pages (dumb implementation copy everything)
cp -r "$doccarchiveDir/data/documentation/" "$DOCS_DIRECTORY/data/documentation/"

cp -r "$doccarchiveDir/documentation/" "$DOCS_DIRECTORY/documentation/"
cp -r "$doccarchiveDir/js/" "$DOCS_DIRECTORY/js/"
cp -r "$doccarchiveDir/css/" "$DOCS_DIRECTORY/css/"

index_file="$doccarchiveDir/index/index.json"
if [ -f "$index_file" ]; then
Expand All @@ -254,9 +265,13 @@ for doccarchiveDir in "${doccarchives[@]}"; do
_ "The .doccarchive folder '$doccarchiveDir' cannot be processed. Exits. ($EXIT_CANNOT_PROCESS)" true
exit $EXIT_CANNOT_PROCESS
fi
_ "👍 Docs generated for $doccarchiveDir!"
_ "👍 Docs processed for $doccarchiveDir!"
done

_ "👉 All index.json files processed, now time to merge them"
python3 merge-json-indexes.py "$DOCS_DIRECTORY/index"
_ "👍 Merge done!"

# Step 3 - Add CNAME file for GitHub Pages
# ----------------------------------------

Expand All @@ -269,7 +284,7 @@ fi
# Step 4 - Update global HTML file
# --------------------------------

# This is only to build a global index.html file referencing all other targets index.html files
# This is only to build a global index.html file referencing all other targets index.html files.

_ "👉 Updating index.html..."

Expand Down Expand Up @@ -301,15 +316,17 @@ if [[ $use_git -eq 1 ]]; then

_ "🔨 Saving things"
cp -r "$DOCS_DIRECTORY" "$DOCUMENTATION_HTML_LOCATION"

clean_repo
rm -rf "$DOCS_DIRECTORY"

_ "🔨 Checkout service pages branch, align with remote"

# Check if the local branch exists
# Check if the local branch exists.
if git show-ref --verify --quiet refs/heads/"$SERVICE_PAGES_BRANCH"; then
_ "🔨 Checking out local branch '$SERVICE_PAGES_BRANCH'"
git checkout "$SERVICE_PAGES_BRANCH"
git reset --hard origin/$SERVICE_PAGES_BRANCH # Ensure to be aligned with remote version
git reset --hard origin/$SERVICE_PAGES_BRANCH # Ensure to be aligned with remote version.
else
_ "🔨 Local branch '$SERVICE_PAGES_BRANCH' does not exist. Checking out from remote."
git fetch origin
Expand All @@ -318,19 +335,29 @@ if [[ $use_git -eq 1 ]]; then

_ "🔨 Applying changes"

# Ensure we have only updated files
rm -rf "$DOCS_DIRECTORY/data/documentation/*"
rm -rf "$DOCS_DIRECTORY/documentation/*"
rm -f "$DOCS_DIRECTORY/index/*"
# Ensure we have only updated files.
mkdir -p "$DOCS_DIRECTORY/css"
mkdir -p "$DOCS_DIRECTORY/data/documentation"
mkdir -p "$DOCS_DIRECTORY/documentation"
mkdir -p "$DOCS_DIRECTORY/img"
mkdir -p "$DOCS_DIRECTORY/index"

mkdir -p "$DOCS_DIRECTORY/js"

# The HTML shards to update, hoping we won't loose some (╯°□°)╯︵ ┻━┻
cp -r "$DOCUMENTATION_HTML_LOCATION/css/"* "$DOCS_DIRECTORY/css/"
cp -r "$DOCUMENTATION_HTML_LOCATION/data/documentation/"* "$DOCS_DIRECTORY/data/documentation/"
cp -r "$DOCUMENTATION_HTML_LOCATION/documentation/"* "$DOCS_DIRECTORY/documentation/"
cp -r "$DOCUMENTATION_HTML_LOCATION/img/"* "$DOCS_DIRECTORY/img/"
cp -r "$DOCUMENTATION_HTML_LOCATION/index/"* "$DOCS_DIRECTORY/index/"
cp -r "$DOCUMENTATION_HTML_LOCATION/js/"* "$DOCS_DIRECTORY/js/"

cp -r "$DOCUMENTATION_HTML_LOCATION/CNAME" "$DOCS_DIRECTORY/"
cp -r "$DOCUMENTATION_HTML_LOCATION/developer-og-twitter.jpg" "$DOCS_DIRECTORY/"
cp -r "$DOCUMENTATION_HTML_LOCATION/developer-og.jpg" "$DOCS_DIRECTORY/"
cp -r "$DOCUMENTATION_HTML_LOCATION/favicon.ico" "$DOCS_DIRECTORY/"
cp -r "$DOCUMENTATION_HTML_LOCATION/favicon.svg" "$DOCS_DIRECTORY/"
cp -r "$DOCUMENTATION_HTML_LOCATION/index.html" "$DOCS_DIRECTORY/"
cp -r "$DOCUMENTATION_HTML_LOCATION/nmetadata.json" "$DOCS_DIRECTORY/"

_ "🔨 Adding things (~ $files_count files)"
git add "$DOCS_DIRECTORY"
Expand All @@ -350,7 +377,7 @@ fi
# Step 6 - Compress ZIP (if relevant)
# -----------------------------------

# ZIP action must be done before reseting the Git workspace (otherwise everything will be wiped out)
# ZIP action must be done before reseting the Git workspace (otherwise everything will be wiped out).
if [[ $no_zip -eq 0 ]]; then
_ "👉 Zipping documentation folder"
zip -r "$DOCUMENTATION_ZIP_LOCATION" "$DOCS_DIRECTORY"/*
Expand All @@ -365,7 +392,9 @@ if [[ $use_git -eq 1 ]]; then
commit_hash=`git rev-parse HEAD`
_ "🔨 Going back to previous Git branch"
clean_repo
git fetch origin
git checkout "$current_branch"
git reset --hard "origin/$current_branch"
_ "👍 Pushed with commit '$commit_hash'"
fi

Expand Down

0 comments on commit b30d5db

Please sign in to comment.