Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentloader CRD and add audio handler #771

Merged
merged 3 commits into from
Feb 28, 2024

Conversation

nkwangleiGIT
Copy link
Contributor

  1. Add documentloader CRD to handle uploaded files from user(read file from s3)
  2. Add audio handler using whisper, so we can transcribe audio to txt and then chat with audio content

if !ok || len(files) == 0 {
return args, errors.New("empty file list")
}
if err := cli.Get(ctx, types.NamespacedName{Namespace: p.RefNamespace(), Name: p.Ref.Name}, instance); err != nil {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should split resource get into func (p *Executor) Init, like

func (k *Knowledgebase) Init(ctx context.Context, cli client.Client, _ map[string]any) error {
instance := &v1alpha1.KnowledgeBase{}
if err := cli.Get(ctx, types.NamespacedName{Namespace: k.RefNamespace(), Name: k.Ref.Name}, instance); err != nil {
return fmt.Errorf("can't find the knowledgebase in cluster: %w", err)
}
k.Instance = instance
return nil
}

And add func (p *Executor) Ready() like

func (k *Knowledgebase) Ready() (isReady bool, msg string) {
return k.Instance.Status.IsReadyOrGetReadyMessage()
}
method for app health check.

Since the state update has not been implemented yet, we can implement this along with the state update in one pr.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's add it to make documentloader always ready, and check if we need to add some logic later.

for _, file := range files {
ossInfo := &arcadiav1alpha1.OSS{Bucket: p.RefNamespace()}
ossInfo.Object = filepath.Join("upload", file)
klog.Infoln("handling file", ossInfo.Object)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should use klog.FromContext(ctx).xxx to replace klog, We add a requestid to each request in the logs to make debugging easier, in

func addRequestIDToLog(c *gin.Context, id string) {
ctx := c.Request.Context()
logger := klog.FromContext(ctx)
newLogger := logger.WithValues("requestID", id)
newLogger.Info("new request")
c.Request = c.Request.WithContext(klog.NewContext(ctx, newLogger))
c.Next()

But it looks like this is an issue elsewhere as well, and we can leave that for a subsequent pr along with an update.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, klog it more easy to use

)

const (
Group = "arcadia.kubeagi.k8s.com.cn"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to double check that since we are using group name arcadia.kubeagi.k8s.com.cn, should we move these files to api/base?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think about this before, but documentloader is a Node that should run with app runtime, so place it to app-node.
api/base seems just resources that does not have any Node run logic, so don't use it although they have the same group. Let's keep it for now?

@nkwangleiGIT nkwangleiGIT merged commit 327c540 into kubeagi:main Feb 28, 2024
3 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants