Artemis: Interactive Learning with Individual Feedback

Artemis brings interactive learning to life with instant, individual feedback on programming exercises, quizzes, modeling tasks, and more. Offering customization for instructors and real-time collaboration for students, this platform bridges creativity and education. Embrace a new era of engaging, adaptive learning and artificial intelligence support with Artemis, where innovation meets inclusivity. Find out more on https://artemisapp.github.io

Main goals

1. User experience: Provide an intuitive and engaging interface that enhances the learning experience for both students and instructors. Ensure ease of use across diverse learning scenarios, fostering seamless interaction and accessibility.

2. Scalable infrastructure: Build a robust platform capable of supporting large-scale courses with thousands of participants simultaneously. Ensure high availability, performance, and adaptability as the platform grows in functionality and user base.

3. Constructive alignment: Align learning goals, activities, and assessments through well-integrated features such as the exam mode that support instructional design principles. Enable the seamless implementation of interactive learning methodologies, ensuring that every feature contributes meaningfully to learning goals.

4. Learning analytics: Leverage data to provide actionable insights into student performance and engagement. Offer detailed feedback to both learners and instructors, enhancing personalized learning experiences and supporting data-driven decisions for educational improvement while ensuring data privacy and security.

Main features and modules

1. Exercises with automatic and manual assessment. Instructors have many configuration options.
   • Team exercises: Instructors can configure team exercises with real time collaboration and dedicated tutors per team.
   • Multiple submissions: Students can participate in exercises multiple times. Instructors can configure the number of submissions and the submission deadline.
   • Practice mode: Students can practice exercises as often as they want.
2. Programming exercises with version control, automatic individual feedback (and assessment) based on test cases and static code analysis (executed using continuous integration) for any programming language.
   • Instant feedback: Students receive immediate and individual feedback on submissions. Instructors can customize feedback messages easily, hide feedback during the working time (e.g., with hidden tests)
   • Interactive: Instructors integrate interactive instructions based on tasks and UML diagrams directly into the dynamic problem statements. They can define hints for difficult exercise parts.
   • Independent: Instructors can customize programming exercises to support any programming language. To simplify the setup, Artemis includes sophisticated templates for the most common languages (e.g., Java, Python, C, Haskell, Kotlin, VHDL, Assembler, Swift, Ocaml, ...)
   • Local and online: Students can participate locally in their preferred IDE or in the online code editor (without any setup)
   • Manual reviews are possible directly in the online code editor or in Orion
   • Policies: Instructors can define submission policies (e.g., penalties after 10 attempts) to prevent that students try out all possibilities without thinking.
   • Grading: Instructors have many options to configure grading, analyze the results based on tests and static code analysis categories and re-evaluate the results
   • Secure: Test cases and student code run in Docker environments on build agents. Test frameworks such as Ares simplify the creation of structural and dynamic tests and prevent that students can cheat.
3. Build agents Integrated code lifecycle elevates the programming exercise experience and seamlessly integrates version control and continuous integration, creating a streamlined, cohesive environment. Unlike external systems, this integrated approach minimizes maintenance overhead and is tailored to the specific needs of a learning platform, offering unparalleled flexibility, performance, and scalability. It simplifies the initial setup for administrators and ensures a smoother, integrated workflow for students and instructors alike. Build agents are responsible for executing the build, test, and static code analysis jobs for programming exercises. They easily integrate into the Artemis infrastructure.
4. Quiz exercises with multiple choice, drag and drop, and short answer questions
   • Modeling quizzes: Instructors can easily create drag and drop quizzes based on UML models
   • Different modes: Quizzes support a live mode (rated) during lectures, a practice mode for students to repeat the quiz as often as they want, and an exam mode (see below)
5. Modeling exercises based on the easy-to-use online modeling editor Apollon with semi-automatic assessment using machine learning concepts
   • Multiple diagram types: Artemis supports 7 UML diagram types (e.g. class diagrams) and 4 additional diagrams (e.g. flow charts)
   • Easy to use: Students can create models easily using drag and drop, they can explain the models using additional text.
   • Integrated feedback: Reviews can provide feedback and points directly next to the model elements.
6. Text exercises with manual, semi-automatic assessment based on supervised machine learning and natural language processing (NLP) using Athena
   • Integrated feedback: Reviews can provide feedback and points directly next to the text segments.
   • Language detection: Artemis detects the language of the submission and shows the word and character count.
7. File upload exercises allow full flexibility to instructors. Students can create any kind of file (e.g. PDF, PNG) and submit it to Artemis when they have completed their work. Artemis allows instructors and tutors to download the files and assess them manually based on structured grading criteria (see below in the section Assessment).
8. Exam mode: Instructors can create online exams with exercise variants, integrated plagiarism checks, test runs and student reviews. You can find more information on Exam mode student features and on Exam mode instructor features.
9. Assessment: Artemis uses double-blind grading and structured grading criteria to improve consistency and fairness.
   • Training process: It integrates an assessment training process (based on example submissions and example assessments defined by the instructor), has a grading leader board, and allows students to rate the assessments. Students can complain or ask for more feedback.
   • Grading: Instructors can configure grade keys for courses and exams to automatically calculate grades and display them to students. Grades can be easily exported as csv files to upload them into university systems (such as Campus online). They can define bonus configurations for final exams to improve student grades according to their grades from a midterm exam or course exercises.
10. Communication: Instructors can post announcements. Students can ask questions, post comments, and react to other posts in channels or private chats. Tutors can filter unanswered questions.
    • Notifications: Artemis supports customizable web and email notifications. Users can enable and disable different notification types.
    • Mobile apps: Users can use mobile apps on iOS and Android for an improved user experience.
11. Lectures: Instructors can upload lecture slides, divide lectures into units, integrate video streams, lecture recordings, and exercises into lectures, and define competencies.
12. Atlas: Atlas is the competency-based management system that supports instructors and students with adaptive learning and learning analytics.
    • Adaptive learning: Artemis allows instructors and students to define and track competencies. Students can monitor their progress towards these goals, while instructors can provide tailored feedback. This approach integrates lectures and exercises under overarching learning objectives.
    • Learning analytics: Artemis integrated different statistics for students to compare themselves to the course average. It allows instructors to evaluate the average student performance based on exercises and competencies.
    • Learning paths: Based on the competency model and students' individual progress, Artemis creates learning paths that guide students through the course content.
13. Iris: Artemis integrates Iris, a LLM based virtual assistant that supports students and instructors with common questions and tasks.
    • Questions: Iris supports students with answering questions about exercises, lectures, and the learning performance instantly.
    • Pro-active assistance: Iris can pro-actively communicate with the students, help them with the next steps in their learning experience and motivate them to continue.
14. Athena: Artemis integrates Athena, a machine learning-based tool that supports instructors with the assessment of text, modeling and programming exercises. Athena offers different modules including automatic feedback suggestions based on generate AI.
15. Tutorial groups: Artemis supports the management of tutorial groups of a course. This includes planning the sessions, assigning responsible tutors, registering students and tracking the attendance.
16. Plagiarism checks: Artemis integrates plagiarism checks for programming exercises (based on JPlag), text exercises, and modeling exercises in courses and exams. It allows notifying students about identified plagiarism. Students can review and comment on the allegation.
17. LTI: Artemis supports the Learning Tools Interoperability (LTI) standard to integrate Artemis into other learning management systems (LMS) such as Moodle and edX.

Additional features and aspects

1. Integrated markdown editor: Markdown is used to format text content across the platform using an integrated markdown editor.
2. Scalable: Artemis scales to multiple courses with thousands of students simultaneously using it. In fact, the largest course had 2,400 students. Administrators can easily scale Artemis with additional build agents in the continuous integration environment.
3. High user satisfaction: Artemis is easy to use, provides guided tutorials. Developers focus on usability, user experience, and performance.
4. Customizable: It supports multiple instructors, editors, and tutors per course and allows instructors to customize many course settings including user management,
5. Open-source: Free to use with a large community and many active maintainers.

Roadmap

The Artemis development team prioritizes the following areas in the future. We welcome feature requests from students, tutors, instructors, and administrators. We are happy to discuss any suggestions for improvements.

• Short term: Further improve the communication features with mobile apps for iOS and Android
• Short term: Add the possibility to use Iris for questions on all exercise types and lectures (partly done)
• Short term: Provide GenAI based automatic feedback to modeling, text and programming exercise with Athena
• Short term: Improve the LTI integration with Moodle
• Medium term: Improve the REST API of the server application
• Medium term: Integrate an online IDE (e.g. Eclipse Theia) into Artemis for enhanced user experience
• Medium term: Add more learning analytics features while preserving data privacy
• Medium term: Improve the user experience, usability and navigation
• Medium term: Add automatic generation of hints for programming exercises
• Medium term: Add GenAI support for reviewing exercises for instructors
• Medium term: Add GenAI support for learning analytics (partly done)
• Long term: Explore the possibilities of microservices, Kubernetes based deployment, and micro frontends
• Long term: Allow students to take notes on lecture slides and support the automatic updates of lecture slides
• Long term: Develop an exchange platform for exercises

Contributing

We welcome contributions from both members of our organization and external contributors. To maintain transparency and trust:

• Members: Must use their full real names and upload a professional and authentic profile picture. Members can directly create branches and PRs in the repository.
• External Contributors: Must adhere to our identity guidelines, using real names and authentic profile pictures. Contributions will only be considered if these guidelines are followed.

We adhere to best practices as recommended by GitHub's Open Source Guides and their Acceptable Use Policies. Thank you for helping us create a respectful and professional environment for everyone involved.

We follow a pull request contribution model. For detailed guidelines, please refer to our CONTRIBUTING.md. Once your pull request is ready to merge, notify the responsible feature maintainer on Slack:

Maintainers

The following members of the project management team are responsible for specific feature areas in Artemis. Contact them if you have questions or if you want to develop new features in this area.

Goal	Responsible maintainer
User experience	Ramona Beinstingel (@rabeatwork)
Scalable infrastructure	Matthias Linhuber (@mtze)
Constructive alignment	Stephan Krusche (@krusche)
Learning analytics	Maximilian Anzinger (@maximiliananzinger)

Feature / Module	Responsible maintainer
Exercises	Stephan Krusche (@krusche)
Programming exercises	Stephan Krusche (@krusche)
Build agents	Robert Jandow (@robertjndw)
Quiz exercises	Timor Morrien (@Hialus)
Modeling exercises	Felix Dietrich (@FelixTJDietrich)
Text exercises	Felix Dietrich (@FelixTJDietrich)
File upload exercises	Elisabeth Friesinger (@easy-lisi)
Exam mode	Stephan Krusche (@krusche)
Assessment	Maximilian Sölch (@maximiliansoelch)
Communication	Stephan Krusche (@krusche)
Lectures	Patrick Bassner (@bassner)
Atlas	Maximilian Anzinger (@maximiliananzinger)
Iris	Patrick Bassner (@bassner)
Athena	Maximilian Sölch (@maximiliansoelch)
Tutorial Groups	Ramona Beinstingel (@rabeatwork)
Plagiarism checks	Markus Paulsen (@MarkusPaulsen)
LTI	Maximilian Anzinger (@maximiliananzinger)

Setup and guidelines

Development setup, coding, and design guidelines

• How to set up your local development environment
• Server coding and design guidelines
• Client coding and design guidelines
• Code Review Guidelines
• Performance Guidelines

Documentation

The Artemis documentation is available here. You can find a guide on how to write documentation.

Server setup

Setting up Artemis in your development environment or a demo production environment is really easy following the instructions on https://docs.artemis.cit.tum.de/dev/setup. When you want to support programming exercises, we recommend using the Integrated Code Lifecycle.

Artemis can also be set up in conjunction with external tools for version control and continuous integration:

1. LocalVC and Jenkins
2. GitLab and Jenkins (deprecated)
3. GitLab and GitLab CI (experimental and deprecated)

Artemis uses these external tools for user management and the configuration of programming exercises.

Administration setup

If needed, you can configure self service user registration.

Building for production

To build and optimize the Artemis application for production, run:

./gradlew -Pprod -Pwar clean bootWar

This will create a Artemis-.war file in the folder build/libs. The build command compiles the TypeScript into JavaScript files, concatenates and minifies the created files (including HTML and CSS files). It will also modify index.html so it references these new files. To ensure everything worked, run the following command to start the application on your local computer:

java -jar build/libs/*.war --spring.profiles.active=dev,localci,localvc,artemis,scheduling,buildagent,core,local

(You might need to copy a yml file into the folder build/libs before, also see development setup)

Then navigate to http://localhost:8080 in your browser.

Refer to Using JHipster in production for more details.

The following command can automate the deployment to a server. The example shows the deployment to the main Artemis test server (which runs a virtual machine):

./artemis-server-cli deploy username@artemis-test0.artemis.in.tum.de -w build/libs/Artemis-7.8.1.war

Architecture

The following diagram shows the top level design of Artemis which is decomposed into an application client (running as Angular web app in the browser) and an application server (based on Spring Boot). For programming exercises, the application server connects to a version control system (VCS) and a continuous integration system (CIS). Authentication is handled by an external user management system (UMS).

While Artemis includes generic adapters to these three external systems with a defined protocol that can be instantiated to connect to any VCS, CIS, or UMS, it also provides 3 concrete implementations for these adapters to connect to.

Server architecture

The following simplified UML component diagram exemplary shows more details of the Artemis application server architecture and its REST interfaces to the application client.

Deployment

The following UML deployment diagram shows a typical deployment of Artemis application server and application client. Student, Instructor, and Teaching Assistant (TA) computers are all equipped equally with the Artemis application client being displayed in the browser.

The Continuous Integration Server typically delegates the build jobs to local build agents within the university infrastructure or to remote build agents, e.g., hosted in the Amazon Cloud (AWS).

Data model

The Artemis application server uses the following (simplified) data model in the MySQL database (notice that the actual data model is more complex by now). It supports multiple courses with multiple exercises. Each student in the participating student group can participate in the exercise by clicking the Start Exercise button. Then a repository and a build plan for the student (User) will be created and configured. The initialization state helps to track the progress of this complex operation and allows recovering from errors. A student can submit multiple solutions by committing and pushing the source code changes to a given example code into the version control system or using the user interface. The continuous integration server automatically tests each submission and notifies the Artemis application server when a new result exists. In addition, teaching assistants can assess student solutions and "manually" create results.

Please note that the actual database model is more complex. The UML class diagram above omits some details for readability (e.g., lectures, student questions, exercise details, static code analysis, quiz questions, exam sessions, submission subclasses, etc.)

Artemis Community

There is a growing community of university instructors who are using Artemis.

Communication

We communicate using GitHub issues and pull requests. Additionally, you can join us on Slack to ask questions and get support. If you are interested, please send an email to Stephan Krusche.

Universities / Schools with Artemis in Use

The following universities are actively using Artemis or are currently evaluating Artemis.

• Technical University of Munich
  https://artemis.cit.tum.de
  Main contact person: Stephan Krusche

• LFU Innsbruck, Uni Salzburg, JKU Linz, AAU Klagenfurt, TU Wien
  https://artemis.codeability.uibk.ac.at
  codeAbility project
  Main contact person: Michael Breu

• University of Stuttgart
  https://artemis.sqa.ddnss.org
  Main contact person: Steffen Becker

• Universität Passau
  https://artemis.fim.uni-passau.de (only accessible via the university network/VPN)
  Main contact person: Benedikt Fein

• Karlsruhe Institute of Technology
  https://artemis.praktomat.cs.kit.edu
  Main contact person: Dominik Fuchß

• Hochschule München
  https://artemis.cs.hm.edu
  Main contact person: Michael Eggers

• Technische Universität Dresden
  Main contact person: Andreas Domanowski

• Hochschule Heilbronn
  Main contact person: Jörg Winckler

• Maria-Theresia-Gymnasium München
  Main contact person: Valentin Herrmann

Interested universities

• HU Berlin
  Main contact person: Lars Grunske

• Westsächsische Hochschule Zwickau
  Main contact person: Heiko Baum

• Technische Universität Chemnitz
  Main contact person: Danny Kowerko

• Universität zu Köln
  Main contact person: Andreas Vogelsang

• Technische Universität Dortmund
  Main contact person: Falk Howar

• Universität Bielefeld
  Main contact person: Daniel Merkle

• Universität Ulm
  Main contact person: Matthias Tichy

• Imperial College London
  Main contact person: Robert Chatley

• University of South Australia
  Main contact person: Srecko Joksimovic