Help Us Shape the Next Generation of Doctor-AI Interfaces

This study evaluates whether AI-generated decision support alongside ultrasound images improves gynecologists' classification performance on ovarian tumor cases. You will classify real, retrospective patient cases into one of 10 diagnostic subtypes, sometimes on your own and sometimes with the help of different forms of AI assistance, such as a top-1 prediction, a full probability distribution, a certainty score, a saliency heatmap, a conformal prediction set, or retrieved similar images. Your input will directly inform which AI-support designs help clinicians and which only get in the way.

Study Structure

Your work is organised into blocks of 128 cases, each taking around one hour to complete. You do not need to work through the blocks in one sitting: log in, complete a block, and return to continue whenever it suits you.

The study runs in two parts. In Part 1, you classify 896 cases across 7 blocks, with different conditions shown inside the blocks. Once all participants have completed Part 1, we identify the best-performing AI-assistance conditions and combine them into two new conditions for Part 2, where you classify a further 256 cases across 2 blocks. Part 2 opens only once every participant has finished Part 1, so there may be a pause between the two parts. You will receive an email as soon as Part 2 is ready to begin.

Why Participate?

Shape clinical AI tools. Your classifications feed directly into a pre-registered analysis that will determine which AI-assistance designs are worth deploying in real gynecological ultrasound workflows.
Get acknowledged. Participants who complete the study will be acknowledged in the resulting research publication.
Receive a personalized performance report. After completing the study, you get a report of your own classification accuracy and how it compares with other participating gynecologists, offering a useful reflection on your diagnostic performance across a large, varied case set.
Flexible participation. Cases are grouped into blocks of 128, each taking around one hour, so you can contribute in the time you have rather than in one long sitting.

Your Personalized Report

After you and the other participants completed the evaluations, you receive a report summarizing your classification accuracy, how it compares with the other participants, and how your confidence ratings tracked your actual performance. The report will show your diagnostic accuracy per class as well as your binary classification performance.

Example of a personalized classification performance report

Who Can Participate?

We are looking for around 60 gynecologists, both junior and senior, who meet the following criteria:

Basic training in gynaecological ultrasound
At least one year of clinical experience working with gynaecological ultrasound imaging.
Sufficient English proficiency to read study instructions and interface labels without assistance.

We welcome both junior and senior doctors, as comparing performance across experience levels is part of what makes the study valuable. We use the EFSUMB (European Federation of Societies for Ultrasound in Medicine and Biology) competence levels as a guide: junior corresponds to EFSUMB Level 1 (able to perform and report standard, routine gynaecological ultrasound examinations independently, but would refer more complex or unusual findings to a more experienced colleague), and senior corresponds to EFSUMB Level 2 or 3 (able to independently assess complex or atypical findings, and/or hold a supervisory, training, or expert role). If you trained outside this framework, just use whichever of the two descriptions above best matches your own practice.

How to Participate

Enroll via the study's REDCap survey. When you finish, you are taken to registration with your one-time key already filled in — set a password there.
Log in with that key and password and read the short study introduction, which explains the task and interface in detail.
Classify cases at your own pace, one hour-long block at a time.
Receive your personalized report once you have completed your evaluations.