Enhancement: Add user feedback for responses

[azaylamba]

Issue #151
Description of changes: Added basic feedback mechanism for responses generated by the chatbot.
As part of this change, we have stored the user feedback in S3 bucket.

We are storing the feedback in the following format:
{
"feedbackId": "",
"sessionId": "",
"userId": "",
"key": "",
"prompt": "",
"completion": "",
"model": "",
"feedback": "1/0",
"createdAt": ""
}

This feedback can be used by AWS Athena/Glue etc to do analysis and also to train the model if required. We are keeping one object per feedback in S3 'STANDARD_IA' class.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[azaylamba]

@massi-ang could you please have a look?

[bigadsoleiman]

Hi @azaylamba thanks for your submission. I really like this feature as it opens up to offline analyses and more importantly fine tuning of models where possible and I think this should be the end goal.

Though, I think this should be stored in a form of dataset that then can be used for fine-tuning. See Bedrock's prepare for finetuning docs which requires dataset in the following format:

{"prompt": "<prompt text>", "completion": "<generated text>"}

Now, in this case, we can bring on more fields such as

{
   "prompt": "<prompt text>", 
   "completion": "<generated text>", 
   "model" ..., 
   "feedback": 0/1,
   "timestamp":  ..., 
   ...any other data of interest
}

and stored in S3 maybe with a Infrequent Access storage class.

The advantages would be:

• Cheaper option
• Admins or systems can leverage Athena to query this feedbacks
• Glue can be used to re-format this feedbacks for model-specific finetuning
• Given the above we could then leverage QuickSight and embed rich dashboard in the chatbot without coding custom charts

You can even store 1 object per feedback, Athena, Glue/EMR will do the heavy-lifting to merge them when needed.

[azaylamba]

@bigadsoleiman Thanks for your input. I will try the approach you suggested.

[azaylamba]

@bigadsoleiman I have modified the code to store the user feedback in S3 bucket as per the suggested format. Please have a look.

[bigadsoleiman]

Thanks!! going in right direction.

One thing I would change is the value of the feedback, I would avoid strings such as thumbsUp/thumbsDown as it's easier in a context of EMR, Athena to deal with numbers especially when it comes to aggregations so I'd rather use 0/1

Unless there is a specific reason I'm missing for using thumbsUp/thumbsDown, let me know.

[bigadsoleiman]

Let's handle storage class in CDK through lifecycle rules instead.

something along these line should ensure that objects are transitioned to the right class and it's easier to handle more complex rules in future

    const userFeedbackBucket = new s3.Bucket(this, 'UserFeedbackBucket', {
       ...
      lifecycleRules: [{
        transitions: [{
          storageClass: s3.StorageClass.INFREQUENT_ACCESS,
          transitionAfter: cdk.Duration.days(0), // Transition immediately after creation
        }],
      }],
    });
  }

[azaylamba]

The minimum allowed value of transitionAfter days is 30. So we cannot use transitionAfter: cdk.Duration.days(0),
The recommended approach to directly store the objects in STANDARD_IA is setting the storage class at object level.

[bigadsoleiman]

let's set the lifecycle rules here for simplicity and having them handled in one place (instead of object creation)

    const userFeedbackBucket = new s3.Bucket(this, 'UserFeedbackBucket', {
       ...
      lifecycleRules: [{
        transitions: [{
          storageClass: s3.StorageClass.INFREQUENT_ACCESS,
          transitionAfter: cdk.Duration.days(0), // Transition immediately after creation
        }],
      }],
    });
  }

[bigadsoleiman]

Why CORS are enabled here? We are not dealing with objects directly from web clients

[bigadsoleiman]

Don't think we need transfer acceleration here right?

[azaylamba]

@bigadsoleiman Addressed the review comments related to cors, transfer acceleration and feedback type.

[azaylamba]

thanks for your inputs @bigadsoleiman