Programming exercises: Add Athena Feedback Suggestions

[pal03377]

Feature Template Spec Version 0.1

Quick links:

• Feature proposal (this document): Programming exercises: Add Athena Feedback Suggestions #7094
• Client: Development: Add client support for programming exercise feedback suggestions #7099
• Server: Programming exercises: Add server support for feedback suggestions #7136
• Client + Server merged: Programming exercises: Add support for feedback suggestions (both other PRs combined) #7138

Context

We have revived feedback suggestions by Athena to text exercises (#6861). We want to extend this to programming exercises as well.

Problem

Athena can generate feedback suggestions for programming exercises, but Artemis cannot interact with Athena and show them.

Motivation

Athena could already help a lot of tutors save time by intelligently suggesting feedback for text exercises with the Athena-CoFee system. We want to explore new ways to achieve the same for programming exercises. We will attempt this with the ThemisML system and a module connected to the APIs for GPT3.5 and GPT4.

---

Requirements Engineering

Existing (Problematic) Solution / System

There are no feedback suggestions available for programming exercises.

Proposed System

We show feedback suggestions for programming exercises and offer to accept or reject them. Because it would be annoying to click around in files to find all suggestions, the default will be to NOT accept a feedback suggestion. Still, you will receive a warning if you missed a suggestion and neither accepted nor rejected it.

Requirements

1. FR: Provide Feedback Suggestions Athena can deliver feedback suggestions to the LMS when a tutor starts grading a submission. Feedback suggestions are generated based on the capabilities of the selected assessment module and analysis parameters.

2. FR: Review Suggestions Once Athena has finished processing all incoming data and can provide feedback suggestions, Artemis can display a UI for tutors to view these feedback suggestions for programming exercises. The UI should be accessible only to authorized tutors who are grading the exercise.

3. FR: Accept Feedback Suggestions When a tutor is content with a given feedback suggestion, they can accept it from within the LMS.

4. FR: Reject Feedback Suggestions When a tutor does not want to apply a feedback suggestion, they can reject it from within the LMS.

5. NFR: Usability: Feedback Accessibility Tutors should be able to effortlessly view and interpret the feedback suggestions provided by Athena.

6. NFR: Performance: Immediate Grading Tutors should be able to start grading a submission immediately, even if Athena has not yet provided feedback suggestions. In this case, suggestions from Athena should load asynchronously, allowing grading to proceed without delay.

7. NFR: Security: Data Leakage Prevention Confidential data, such as student submissions or feedback, should not be leaked outside Artemis and Athena. Appropriate data protection measures must be in place.

---

Analysis

Analysis Object Model

Tutors will be able to get Feedback Suggestions for Assessments on Submissions in Programming Exercises:

Dynamic Behavior

---

System Design

Subsystem Decomposition

Here are the full changes made from the text + programming suggestions integration with Athena:

Server:

Client:

New endpoints:

• api/athena/exercises/{exerciseId}/solution-repository: Returns the solution repository of the exercise as a ZIP file for Athena to download and process for suggestions
• api/athena/exercises/{exerciseId}/template-repository: Returns the template repository of the exercise as a ZIP file for Athena to download and process for suggestions
• api/athena/exercises/{exerciseId}/tests-repository: Returns the tests repository of the exercise as a ZIP file for Athena to download and process for suggestions
• api/athena/exercises/{exerciseId}/submissions/{submissionId}/repository: Returns the submission repository as a ZIP file for Athena to download and process for suggestions

Persistent Data Management

• Already approved: Mark feedback suggestions coming from Athena in the DB by prefixing their text with FeedbackSuggestion:
• Distinguish between feedback suggestions that are accepted as-is and modified ones: The prefix is either FeedbackSuggestion:accepted: or FeedbackSuggestion:adapted:
• Already approved: Add a new column feedback_suggestions_enabled to the exercises table to mark exercises where automatic feedback suggestions from Athena are enabled. When the migration happens, this is true for all text exercises with assessment_type = SEMI_AUTOMATIC.
• Choose the Athena module based on a Spring config entry for the corresponding exercise type
• Store submissions, assessments, and feedback as before.

Access Control / Security Aspects

Only tutors will be able to view feedback suggestions. The feedback-suggestions endpoint will be only available to users with the role TA or higher.

All new endpoints returning repositories have to be authenticated using the shared Athena secret. They should only work for exercises where feedback suggestions are enabled.

All access control systems set in #6861 stay as they are.

Change Checklist

• Change Athena<*>Service-s from Development: Re-implement integration of feedback suggestion service Athena #6861 to accept general exercises, submissions, and feedback instead of "only" text exercise content (e.g., TextSubmission objects)
• Let the /feedback-suggestions endpoint from Development: Re-implement integration of feedback suggestion service Athena #6861 return instances of the de.tum.in.www1.artemis.service.dto.athena.TextFeedbackDTO (renamed to FeedbackDTO to be more general). The client will interpret the indices from the feedback reference instead of the server converting it to TextBlockRefs.
• Extend the client-side AthenaService to allow programming exercises by changing the return type of getFeedbackSuggestions.
• Extend the client-side Feedback model with a Feedback.isFeedbackSuggestion helper method to show them inline.
• Extend the client-side CodeEditorTutorAssessmentContainerComponent to load and show feedback suggestions.
• Add a boolean feedbackSuggestions for exercises to determine whether feedback suggestions are enabled for them
• Add the endpoints for repositories and authenticate them
• Schedule Athena for more/all exercise types
• Add a checkmark for enabling feedback suggestions in exercise settings
  • Programming exercises (new)
  • Text exercises (change in logic)
  • Only show the option on Athena-enabled servers
• Add a setting for the choice of the Athena module
• Add multiple tests
  • server
  • client

---

UI / UX

• A tutor starts a new assessment of a manually graded programming exercise. Artemis chooses a submission for them, using Athena.
• The tutor gets the submission. They can immediately start grading. Feedback suggestions are displayed in the UI as soon as they are available. Suggestions can be accepted or rejected by clicking a button on the feedback cards showing the suggestions. The feedback suggestion cards are visually distinguishable from normal feedback in that they have slightly transparent text and a purple background. This indicates they are "only" suggestions and are not applied by default.
• All files that include feedback suggestions are marked with a light bulb in the file tree.
• The tutor is done and submits the assessment. If no feedback suggestions are neither accepted or rejected, Artemis shows a popup warning the tutor about this. If the tutor is okay with this, the assessment is submitted. Otherwise, grading continues with the same submission.

Feedback suggestions are only loaded for submissions without an existing assessment, i.e., the "Save" button was never pressed. The accept/reject state for the feedback suggestions is temporary initially: If a TA reloads or cancels the assessment, they get the feedback suggestions again. This has two advantages:

1. When TAs change, the feedback suggestions are kept
2. More importantly, feedback suggestions don't need to be stored in Artemis before they are actually accepted

The downsides of this behavior are negligible because usually, a TA will assess a whole submission in a single go and not go back anyway.

Feedback Suggestions in the programming assessment editor

Suggestions are clearly marked as such; it is also clear that the feedback is not automatically given without the TA's consent.

Feedback Suggestion on hover

The slight text transparency goes away.

Unreferenced feedback suggestion

They follow a similar visual style.

Unreferenced feedback suggestion on hover

Accepted feedback suggestion

Accepted feedback suggestions look like normal feedback, with an additional "Accepted Suggestion" badge only visible to tutors.

Accepted unreferenced feedback suggestion

Tooltips