Refetch content for OpenAI assistant

.github/workflows/ai-assistant-fetch-content.yml

@@ -0,0 +1,74 @@
+name: Fetch content for OpenAI assistant
+
+on:
+  workflow_dispatch:
+    inputs:
+      purge-all-content:
+        type: boolean
+        description: Purge all content
+        default: false
+
+env:
+  git-user: github-actions[bot]
+  git-email: 41898282+github-actions[bot]@users.noreply.github.com
+
+jobs:
+  update-content:
+    name: Fetch content for OpenAI assistant
+    runs-on: ubuntu-22.04
+    env:
+      PURGE_ALL_CONTENT: ${{inputs.purge-all-content}}
+      BRANCH_NAME: fetch-ai-content-${{ github.run_id }}-${{ github.run_attempt }}
+    steps:
+      - name: "Checkout code"
+        uses: actions/checkout@v4
+        with:
+          token: ${{ secrets.RELEASE_GH_DEV_ACCOUNT_PAT }}
+
+      - name: "Set up Go"
+        uses: actions/setup-go@v5
+        with:
+          go-version-file: 'go.mod'
+          # When the files to be extracted are already present,
+          # tar extraction in Golangci Lint fails with the "File exists"
+          # errors. These files appear to be present because of
+          # cache in setup-go, on disabling the cache we are no more seeing
+          # such error. Cache is to be enabled once the fix is available for
+          # this issue:
+          # https://github.com/golangci/golangci-lint-action/issues/807
+          cache: false
+
+      - name: Setup Node
+        uses: volta-cli/action@v4
+
+      - name: Set git 'user.name' and 'user.email'
+        run: |
+          git config user.name "${{ env.git-user }}"
+          git config user.email ${{ env.git-email }}
+
+      - name: Create a branch branch
+        run: |
+          git checkout -b ${{ env.BRANCH_NAME }}
+
+      - name: Fetch content
+        run: |
+          cd ./hack/assistant-setup
+          npm install
+          npm run fetch-content
+
+      - name: Push changes to remote
+        run: |
+          git add .
+          git commit -m "Refetch content for OpenAI assistant"
+          git push --set-upstream origin ${{ env.BRANCH_NAME }}
+
+      - name: Create a pull request
+        env:
+          GITHUB_TOKEN: ${{ secrets.RELEASE_GH_DEV_ACCOUNT_PAT }}
+        run: |
+          gh pr create \
+            --title "Refetch content for OpenAI assistant"  \
+             --head "${{ env.BRANCH_NAME }}" \
+             --base main \
+             --label automation \
+             --body "This Pull Request is created by automation to update the content for OpenAI assistant."

.github/workflows/ai-assistant-reconfigure.yml

@@ -0,0 +1,40 @@
+name: Reconfigure OpenAI assistant
+
+on:
+  push:
+    tags:
+      - '*'
+    branches:
+      - main
+
+env:
+  OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+
+jobs:
+  reconfigure-ai-assistant:
+    name: Reconfigure OpenAI assistant
+    runs-on: ubuntu-22.04
+    steps:
+      - name: Set environment variables
+        run: |
+          if [[ $GITHUB_REF == refs/tags/* ]]; then
+            # tag push          
+            echo "ASSISTANT_ENV=prod" >> $GITHUB_ENV
+          elif [[ $GITHUB_REF == refs/heads/* ]]; then
+            # branch push
+            echo "ASSISTANT_ENV=dev" >> $GITHUB_ENV
+          fi
+
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Setup Node
+        uses: volta-cli/action@v4
+
+      - name: Reconfigure assistant
+        run: |
+          cd ./hack/assistant-setup
+          npm install
+          npm start

.github/workflows/release.yml

@@ -12,7 +12,7 @@ permissions:
   contents: write
 
 jobs:
-  trigger-relase:
+  trigger-release:
     name: Trigger release
     runs-on: ubuntu-latest
     steps:

README.md

@@ -58,12 +58,27 @@ This repository shows Botkube Cloud plugins.
 
 ## Release
 
+### Fetch content for OpenAI assistant
+
+The AI plugin uses Botkube content (website, docs, blog posts, etc.). To refresh it, follow the steps:
+
+1. Navigate to the [` Fetch content for OpenAI assistant` GitHub Actions workflow](https://github.com/kubeshop/botkube-cloud-plugins/actions/workflows/ai-assistant-fetch-content.yml).
+1. Optionally check the "Purge all content" checkbox.
+1. Trigger the pipeline.
+
+
+The content is shared between dev and prod AI plugin, and is synchronized during the [plugin relese](#release-plugins-for-botkube-agent).
+
+### Release plugins for Botkube Agent
+
 The latest plugins from `main` are released automatically to the bucket `botkube-cloud-plugins-latest`.
-To release a new version of the plugins:
+To release a new production version of the plugins:
 
 1. Navigate to the [`Trigger release` GitHub Actions workflow](https://github.com/kubeshop/botkube-cloud-plugins/actions/workflows/release.yml).
 1. Provide the target version in the `version` input. Do not forget about the `v` prefix.
 
    For example, if you want to release version `1.2.3`, you should provide `version: v1.2.3`.
 
 1. Trigger the pipeline.
+
+For both latest and production plugins, the OpenAI assistant for AI plugin is reconfigured automatically concurrently to the plugins build.

hack/assistant-setup/README.md

@@ -32,17 +32,28 @@ To use your own assistant, modify the `assistantID` variable in the `index.ts` f
 
 ## Development
 
-## Refetch content for file search
-
-> **NOTE:** The process uses [Jina.AI Reader API](https://github.com/jina-ai/reader) and usually takes 10-15 minutes. All files will be removed before the process starts.
+### Refetch content for file search
 
+The process uses [Jina.AI Reader API](https://github.com/jina-ai/reader) and usually takes a few minutes minutes.
 To scrap the content from the latest Botkube website and Botkube Docs, run the following command:
 
 ```sh
 npm run fetch-content
 ```
 
-## Format code
+By default, before refetching content starts:
+
+- downloaded files starting with [package.json](package.json)`https://botkube.io/blog/*` and `https://botkube.io/learn/*` prefixes are kept and won't be updated,
+- all other content is removed.
+
+To refetch all content, run the following command:
+
+```sh
+export PURGE_ALL_CONTENT=true # default: false
+npm run fetch-content
+```
+
+### Format code
 
 To format code, run:
 

hack/assistant-setup/content-fetcher/main.go

@@ -7,6 +7,7 @@ import (
 	"net/http"
 	"net/url"
 	"os"
+	"path/filepath"
 	"regexp"
 	"strings"
 	"time"
@@ -27,6 +28,7 @@ const (
 	maxRetries          = 5
 	retryInterval       = 1 * time.Second
 	httpCliTimeout      = 1 * time.Minute
+	purgeAllContentEnv  = "PURGE_ALL_CONTENT"
 )
 
 var excludedDocsPagesRegex = regexp.MustCompile(`^https:\/\/docs\.botkube\.io\/(?:\d+\.\d+|next)\/.*`)
@@ -44,9 +46,17 @@ func main() {
 		},
 	}
 
-	log.Infof("Removing old %q directory...", contentDir)
-	err := os.RemoveAll(contentDir)
-	loggerx.ExitOnError(err, "while removing old directory")
+	shouldPurgeAllContent := os.Getenv(purgeAllContentEnv) == "true"
+
+	if shouldPurgeAllContent {
+		log.Infof("Purging all content as the %s env var is set. Removing old %q directory...", purgeAllContentEnv, contentDir)
+		err := os.RemoveAll(contentDir)
+		loggerx.ExitOnError(err, "while purging old content")
+	} else {
+		log.Infof("Removing some of the content as the %s env var is not set", purgeAllContentEnv)
+		err := removeFrequentlyUpdatedContent(log)
+		loggerx.ExitOnError(err, "while removing old content")
+	}
 
 	log.Info("Fetching Botkube sitemap...")
 	marketingPages, err := fetcher.getURLsToDownloadFromSitemap(marketingSitemapURL)
@@ -60,7 +70,7 @@ func main() {
 	pagesToFetch := fetcher.preparePageList(docsPages, marketingPages)
 	log.Infof("Found %d pages to fetch", len(pagesToFetch))
 
-	log.Infof("Creating %q directory...", contentDir)
+	log.Infof("Creating %q directory if doesn't exist...", contentDir)
 	err = os.MkdirAll(contentDir, os.ModePerm)
 	loggerx.ExitOnError(err, "while creating directory")
 
@@ -71,10 +81,19 @@ func main() {
 			errs = multierror.Append(errs, err)
 			continue
 		}
-		log.WithFields(logrus.Fields{
+
+		fetchLogger := log.WithFields(logrus.Fields{
 			"url":      page,
 			"filePath": filePath,
-		}).Infof("Fetching and saving page %d of %d...", i+1, len(pagesToFetch))
+			"progress": fmt.Sprintf("%d/%d", i+1, len(pagesToFetch)),
+		})
+
+		if _, err := os.Stat(filePath); err == nil {
+			fetchLogger.Info("Skipping as the file already exists")
+			continue
+		}
+
+		fetchLogger.Infof("Downloading page...")
 
 		err = retry.Do(
 			func() error {
@@ -97,6 +116,37 @@ func main() {
 	log.Infof("Saved %d docs pages", len(pagesToFetch))
 }
 
+var notFrequentlyUpdatedContentPrefixes = []string{
+	"botkube.io__blog__",
+	"botkube.io__learn__",
+}
+
+func removeFrequentlyUpdatedContent(log logrus.FieldLogger) error {
+	return filepath.WalkDir(contentDir, func(path string, d os.DirEntry, err error) error {
+		if err != nil {
+			return err
+		}
+
+		if d.IsDir() {
+			return nil
+		}
+
+		for _, prefix := range notFrequentlyUpdatedContentPrefixes {
+			if strings.HasPrefix(d.Name(), prefix) {
+				log.WithField("path", path).Debug("Skipping removing not frequently updated content")
+				return nil
+			}
+		}
+
+		err = os.Remove(path)
+		if err != nil {
+			return err
+		}
+
+		return nil
+	})
+}
+
 type contentFetcher struct {
 	log     logrus.FieldLogger
 	httpCli *http.Client

hack/assistant-setup/content/botkube.io.md

@@ -3,14 +3,12 @@ Title: Collaborative Kubernetes Troubleshooting Platform | Botkube
 URL Source: https://botkube.io/
 
 Markdown Content:
-[![Image 1: Kusk Kubernetes](https://assets-global.website-files.com/633705de6adaa38599d8e258/6338148fa3f8a509639804fa_botkube-logo.svg)](https://botkube.io/)
-
 A virtual SRE, powered by AI
 ----------------------------
 
-### Powered by
+Botkube, the AI-Powered Kubernetes Troubleshooting Platform
 
-[![Image 2: Monitoring Kubernetes notifications in chat platforms](https://assets-global.website-files.com/633705de6adaa38599d8e258/642da9080827c967a39b0043_automation_new.gif)](https://botkube.io/solutions/enabling-developers)
+[![Image 1: Monitoring Kubernetes notifications in chat platforms](https://assets-global.website-files.com/633705de6adaa38599d8e258/642da9080827c967a39b0043_automation_new.gif)](https://botkube.io/solutions/enabling-developers)
 
 Empowering DevOps
 -----------------
@@ -19,15 +17,16 @@ Error to resolution in one thread: Centralize alerting and troubleshooting in on
 
 Botkube integrates essential notifications, with context, and troubleshooting tools directly into your chat platform. Powered by artificial intelligence and automation, it allows teams to tackle common issues without wasting time on repetitive tasks.
 
-[Developer Self-Service ----------------------](https://botkube.io/solutions/enabling-developers)
+Developer Self-Service
+----------------------
 
 Developers aren’t Kubernetes experts, and they don’t have to be!
 
 Botkube empowers developers to self-service and solve Kubernetes problems from the comfort of their existing chat platforms without extensive K8s knowledge. Our AI assistant provides actionable insights and recommendations directly to developers, enabling quick problem resolution without heavy reliance on support teams.
 
-[![Image 3: Monitoring Kubernetes notifications in chat platforms](https://assets-global.website-files.com/633705de6adaa38599d8e258/654d07eb1993ada26a1f17b1_Enabling_Developers.gif)](https://botkube.io/solutions/enabling-developers)
+[![Image 2: Monitoring Kubernetes notifications in chat platforms](https://assets-global.website-files.com/633705de6adaa38599d8e258/654d07eb1993ada26a1f17b1_Enabling_Developers.gif)](https://botkube.io/solutions/enabling-developers)
 
-![Image 4: Monitoring Kubernetes notifications in chat platforms](https://assets-global.website-files.com/633705de6adaa38599d8e258/654d0856e99da8d8bd98bf27_flux-diff-1.gif)
+![Image 3: Monitoring Kubernetes notifications in chat platforms](https://assets-global.website-files.com/633705de6adaa38599d8e258/654d0856e99da8d8bd98bf27_flux-diff-1.gif)
 
 Improving Reliability
 ---------------------
@@ -36,9 +35,6 @@ Botkube gives you reliable Kubernetes monitoring in minutes, not days.
 
 With powerful filtering, Botkube gets the right notifications to the right teams. Lighten their workload by automating responses to common problems. Accelerate mean time to recovery by receiving alerts, troubleshooting, and resolving issues all in the same place — the chat platform where you already spend much of your time.
 
-See Botkube in Action
----------------------
-
 Why our users love Botkube: Case Study
 --------------------------------------
 
@@ -58,7 +54,7 @@ Join Maria, Developer Advocate for Botkube, in an insightful workshop that lever
 
 Speakers:
 
-![Image 5: Mateusz Stostok, Botkube Backend Engineer](https://assets-global.website-files.com/634fabb21508d6c9db9bc46f/6408ed63e5b48fed17e54625_SE6Pjp9PW9TaOwePHJXRaxaLQgYdT2HX_5PYASmvIx8.jpeg)
+![Image 4: Mateusz Stostok, Botkube Backend Engineer](https://assets-global.website-files.com/634fabb21508d6c9db9bc46f/6408ed63e5b48fed17e54625_SE6Pjp9PW9TaOwePHJXRaxaLQgYdT2HX_5PYASmvIx8.jpeg)
 
 Maria Ashby
 
@@ -69,41 +65,22 @@ Botkube
 Latest Blog Post
 ----------------
 
-User Quotes
------------
-
-![Image 6: automating tests staging cluster](https://assets-global.website-files.com/633705de6adaa38599d8e258/6387ccf6ff66597d5f414815_botkube-quote-sign.svg)
-
-The advantages of BotKube are its versatility and efficiency in managing and monitoring Kubernetes clusters. It offers seamless integration with various messaging platforms and provides real-time alerts and notifications. Its appeal is enhanced by Its user-friendly interface, and extensive customization options
-
-![Image 7: automating tests staging cluster](https://assets-global.website-files.com/633705de6adaa38599d8e258/6387ccf6ff66597d5f414815_botkube-quote-sign.svg)
-
-The advantages of BotKube are its versatility and efficiency in managing and monitoring Kubernetes clusters. It offers seamless integration with various messaging platforms and provides real-time alerts and notifications. Its appeal is enhanced by Its user-friendly interface, and extensive customization options
-
-![Image 8: three blue circles on a black background](https://assets-global.website-files.com/633705de6adaa38599d8e258/6387cc6cd11dba9de0d3578f_botkube.gif)
-
-![Image 9: automating tests staging cluster](https://assets-global.website-files.com/633705de6adaa38599d8e258/6387ccf6ff66597d5f414815_botkube-quote-sign.svg)
-
-For me, monitoring health and performance of the infrastructure in realtime was done with the help of Botkube. Botkube provided me security features such as unauthorized access attempts.
-
-![Image 10: three blue circles on a black background](https://assets-global.website-files.com/633705de6adaa38599d8e258/6387cc6cd11dba9de0d3578f_botkube.gif)
-
-![Image 11: a pair of white speech bubbles on a black background](https://assets-global.website-files.com/633705de6adaa38599d8e258/64dbb86f74cb6622225312c2_quote-icon.svg)
+![Image 5](https://assets-global.website-files.com/633705de6adaa38599d8e258/64dbb86f74cb6622225312c2_quote-icon.svg)
 
 For me, monitoring health and performance of the infrastructure in realtime was done with the help of Botkube. Botkube provided me security features such as unauthorized access attempts.
 
-![Image 12: a pair of white speech bubbles on a black background](https://assets-global.website-files.com/633705de6adaa38599d8e258/64dbb86f74cb6622225312c2_quote-icon.svg)
+![Image 6](https://assets-global.website-files.com/633705de6adaa38599d8e258/64dbb86f74cb6622225312c2_quote-icon.svg)
 
 The advantages of BotKube are its versatility and efficiency in managing and monitoring Kubernetes clusters. It offers seamless integration with various messaging platforms and provides real-time alerts and notifications. Its appeal is enhanced by Its user-friendly interface, and extensive customization options
 
-![Image 13: a pair of white speech bubbles on a black background](https://assets-global.website-files.com/633705de6adaa38599d8e258/64dbb86f74cb6622225312c2_quote-icon.svg)
+![Image 7](https://assets-global.website-files.com/633705de6adaa38599d8e258/64dbb86f74cb6622225312c2_quote-icon.svg)
 
 “Perfect! When i have the repo i will write here, thank you very much for the support! I really like this project!”
 
 Stay in the Loop
 ----------------
 
-![Image 14: Kusk Kubernetes](https://assets-global.website-files.com/633705de6adaa38599d8e258/636d3117b8612105c60e0bd9_botkube-front-right.svg)
+![Image 8: Kusk Kubernetes ](https://assets-global.website-files.com/633705de6adaa38599d8e258/636d3117b8612105c60e0bd9_botkube-front-right.svg)
 
 Join the Botkube Community in one of these channels