📝 Note: This is the code repository of our paper "MedGAME: Storytelling Gamification Empowered by Large Language Models for Medical Education".

---

- Before diving into our code configuration, we would like to first present a silhouette of our MedGame...

What kind of educational application does MedGame aim to deliver?

_{An interactive storytelling game for medical education that can be automatically designed, planned and generated with LLM-driven workflow.}

Case 1

Case 2

_{(Note: Sometimes GitHub may fail to render the MedGame demo GIFs above. If this happens, you can download them directly from ./Assets/GIFS.)}

---

- Next, we introduce the building blocks of a Clinical Story: ACT and Scene.

What is an ACT in Clinical Story?

_{An ACT serves as a container for multiple Scenes, representing a major phase in the patient's clinical journey.}

ACT Structure - Example 1

ACT Structure - Example 2

What is a Scene in Clinical Story?

_{A Scene is a focused clinical encounter within an ACT, containing narrative segments and various interactive elements.}

Scene Structure - Example 1

Scene Structure - Example 2

3 Types of Questions (Puzzle Nodes) within a Scene

_{Each Scene contains interactive puzzle nodes that test student knowledge through different question formats.}

Single Choice

Batch Question (Multiple Selection)

Interactive Question

---

- Now let's dive into the technical details of our MedGame framework...

Repository Structure

MedGame/
├── README.md                         # This file
├── requirements.txt                  # Python dependencies
│
├── MedGame_Pipeline/                 # Core MedGame framework
│   ├── config.py                     # Configuration (API keys placeholder)
│   ├── prompts.py                    # LLM prompts for narrative design
│   ├── pydantic_models.py            # Clinical story tree schema
│   ├── generation_factory.py         # Phase 1: Medical Narrative Designer
│   ├── movie_builder.py              # Phase 2: Story Director (Visual)
│   ├── audio_video_pipeline.py       # Phase 2: Multimodal Execution (Audio)
│   └── infinitetalk_runner.py        # Phase 2: Multimodal Execution (Video)
│
├── MedGAME_UI/                       # Interactive Game Player Application
│   ├── BackEnd/                      # FastAPI backend server
│   │   ├── app/
│   │   │   ├── api/                  # API endpoints (cases, game, media, player)
│   │   │   ├── models/               # Data models (media, player, story)
│   │   │   ├── services/             # Business logic services
│   │   │   ├── config.py             # Backend configuration
│   │   │   └── main.py               # FastAPI application entry
│   │   ├── requirements.txt          # Backend dependencies
│   │   └── run.py                    # Backend launcher
│   ├── Frontend/                     # React + TypeScript frontend
│   │   ├── src/
│   │   │   ├── components/           # UI components (game, interaction, layout)
│   │   │   ├── pages/                # Page components (HomePage, GamePage)
│   │   │   ├── stores/               # State management (gameStore)
│   │   │   ├── types/                # TypeScript type definitions
│   │   │   └── api/                  # API client
│   │   ├── vite.config.ts            # Vite build configuration
│   │   └── tailwind.config.js        # Tailwind CSS configuration
│   ├── Data_Samples/                 # Sample game data for demo
│   ├── run.sh                        # Start frontend & backend
│   └── stop.sh                       # Stop all services
│
├── Shells/                           # Training & evaluation scripts
│   ├── Training/
│   │   ├── MedicalNarrativeGeneration/   # Phase 1 training
│   │   └── StoryDirection/               # Phase 2 training
│   ├── Evaluation/
│   │   ├── MedicalNarrativeGeneration/   # Phase 1 evaluation
│   │   └── StoryDirection/               # Phase 2 evaluation
│   ├── Evaluation_0shot_StoryDirection/  # Zero-shot evaluation
│   └── Evaluation_2shot_MedicalNarrativeGeneration/  # Two-shot evaluation
│
└── Assets/
    ├── GIFS/                         # Demo GIFs for README
    ├── images/                       # Static images (Banner, etc.)
    └── SFT_data/                     # Training/evaluation datasets

MedGame Pipeline

The MedGame pipeline follows a two-phase generative workflow as described in Section 4 of the paper (see Figure 1):

• Phase 1: Script Synthesis (Steps 1-2) — The Medical Narrative Designer transforms raw patient summaries into a structured Game Script.
• Phase 2: Orchestration and Execution (Steps 3-6) — The Story Director decomposes the script into multimodal generation primitives and coordinates task execution.

---

Medical Narrative Designer

Corresponds to Steps 1-2 in Figure 1 · Implementation: generation_factory.py

The Medical Narrative Designer $f_{\text{des}}$ transforms unstructured clinical records into structured, interactive pedagogical narratives:

$$\mathcal{T} = f_{\text{des}}(S_{\text{pat}} \mid \Phi_{\text{des}})$$

where $\mathcal{T}$ is the hierarchical Clinical Storyline and $S_{\text{pat}}$ is the raw patient summary. The synthesis is guided by a multi-faceted prompt $\Phi_{\text{des}}$ that incorporates:

Component	Description
Few-shot exemplars $\mathcal{E}$	Clinical examples for in-context learning
Pydantic schema	Structural constraints enforcing output validity
Medical rubrics $\mathcal{R}$	Standards for clinical reasoning and pedagogical design
Narrative elements $\{\mathcal{P}, \mathcal{L}\}$	Character and scene selection constraints

---

Story Director

Corresponds to Steps 3-4 in Figure 1 · Implementation: movie_builder.py

The Story Director $f_{\text{dir}}$ serves as the orchestration layer that transforms the Clinical Storyline into an executable multimodal generation plan:

$$\mathcal{G} = f_{\text{dir}}(\mathcal{T} \mid \Phi_{\text{dir}})$$

where $\mathcal{G}=(V,E)$ is a directed acyclic graph (DAG) of generation primitives. Each node $v \in V$ represents a self-contained multimodal generation task. The prompt $\Phi_{\text{dir}}$ incorporates narrative elements and toolset API specifications.

Key concepts:

• Initial frame synthesis: Establishes canonical visual reference by fixing identity and scene layout
• Dependency edges $(u \rightarrow v) \in E$: Encode asset-level dependencies between tasks
• Asset propagation: Promotes long-horizon visual coherence by reusing upstream assets

---

Multimodal Task Execution

Corresponds to Steps 5-6 in Figure 1 · Implementation: audio_video_pipeline.py, infinitetalk_runner.py

The system materializes interactive assets by executing generation primitives following the DAG:

$$\mathbf{a}_{v_i} = \mathcal{M}(v_i \mid \{\mathbf{a}_u\}_{(u,v_i) \in E})$$

where $\mathcal{M}$ is the multimodal toolset and $\{\mathbf{a}_u\}$ denotes inherited assets from upstream tasks. This dependency-aware execution preserves visual identity and narrative continuity across outputs.

Installation

Environment Setup

We use two conda environments:

• medgame: For the MedGame pipeline
• medgame_llm: For LLM training and inference

# Create MedGame pipeline environment
conda create -n medgame python=3.10
conda activate medgame
pip install -r requirements.txt

# Create LLM training environment
conda create -n medgame_llm python=3.10
conda activate medgame_llm
pip install ms-swift[all]
pip install unsloth

Configuration

Before running the pipeline, configure your API keys in MedGame_Pipeline/config.py:

# Replace with your actual API keys
LLM_API_BASE = "YOUR_API_BASE_URL"
LLM_API_KEY = "YOUR_API_KEY"

MedGAME UI (Game Player Application)

For setting up the interactive game player UI (Frontend + Backend), please refer to MedGAME_UI/README.md.

Usage

Running the MedGame Pipeline

from MedGame_Pipeline.generation_factory import Generation_Factory

# Initialize factory
factory = Generation_Factory()

# Run Phase 1: Script Synthesis (Steps 1-2)
factory.pipeline_with_scene_and_character_constraints(
    input_text=patient_summary,
    output_dir="./output",
    model_version="3",
    reasoning_effort="low"
)

Training Models

# Train Medical Narrative Generation model
conda activate medgame_llm
cd Shells/Training/MedicalNarrativeGeneration
bash train_qwen3_8b_medical_narrative_generation.sh

# Train Story Direction model
cd Shells/Training/StoryDirection
bash train_qwen3_8b_story_direction.sh

Evaluation

# Evaluate Medical Narrative Generation
cd Shells/Evaluation/MedicalNarrativeGeneration
bash eval_qwen3_8b_medical_narrative_generation.sh

# Evaluate Story Direction
cd Shells/Evaluation/StoryDirection
bash eval_qwen3_8b_story_direction_sft.sh

Citation

If you find this repo useful, please consider giving us a star and citing our paper:

@misc{wu2026medgamestorytellinggamificationempowered,
      title={MedGame: Storytelling Gamification Empowered by Large Language Models for Medical Education}, 
      author={Qian Wu and Xinrong Zhou and Zizhan Ma and Kai Chen and Zheyao Gao and Xun Lin and Hongqiu Wu and Longfei Gou and Yixiao Liu and Ann Sin Nga Lau and Qi Dou},
      year={2026},
      eprint={2607.21570},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.21570}, 
}

Contact

If you have any question / want to collaborate with us, please email to qianwu@link.cuhk.edu.hk. Thanks