Using GenAI to develop multiple-choice questions

In collaboration with Prof. Annemiek Snoeckx (Faculty of Medicine and Health Sciences, UAntwerp)

1. Multiple-choice questions in assessment and education

Examinations constitute an important instrument for evaluating students’ learning process and the knowledge and skills they have acquired. Depending on the intended learning outcomes, a range of assessment formats may be employed, including oral or written examinations  (ECHO teaching tip, 2026). For assessing factual knowledge, conceptual understanding, and application, multiple-choice questions (MCQs) are particularly suitable. Consequently, they are widely used in both formative and summative assessment contexts.
MCQs can take various forms depending on their purpose. For instance, questions requiring justification or identifying the best or worst answer can assess understanding and critical thinking, whereas reordering or matching items are more suited to evaluating relationships and processes (Sabbe, 2013; NBME, 2024).

The use of MCQs offers several advantages. They enable the efficient assessment of large cohorts within a limited timeframe, while ensuring objectivity and broad coverage of the curriculum. Moreover, well-designed distractors can help differentiate between partial and complete understanding by revealing common misconceptions or gaps in knowledge. The digitalisation of assessment further enhances these efficiencies.

The quality of MCQs depends heavily on their construction. Each item consists of a stem, the correct answer, and several distractors. The stem should present the problem clearly, concisely, and unambiguously. Negative phrasing and irrelevant information are best avoided. The answer options should be grammatically consistent with the stem, plausible, substantively distinct, and of comparable length. In most cases, three answer options strike an effective balance between readability and reliability (Sabbe, 2013; NBME, 2024).

Developing high-quality MCQs is a time-intensive process, particularly when combined with the wide range of other academic responsibilities faced by teaching staff. As a result, exam questions are often reused over multiple years. While understandable, this practice carries certain risks: knowledge evolves, curricula change, and items may become outdated. Continuous development of new, up-to-date questions therefore remains essential. Emerging technologies, such as generative artificial intelligence (GenAI), offer potential support in this process (Artsi et al., 2024; UNESCO, 2023).

2. Artificial intelligence and large language models

Recent advances in GenAI have led to significant developments in natural language processing, particularly through the emergence of large language models (LLMs). These models, including systems such as ChatGPT, GPT-4, and Claude, are trained on vast corpora of text. They are capable of generating new content, answering questions, and structuring information in ways that often appear both human-like and accurate. The application of GenAI is attracting growing interest in knowledge-intensive domains such as education. It has the potential to support learning, improve access to knowledge, and reduce workload, thereby creating more space for personalised guidance. At the same time, these developments prompt critical reflection on how best to integrate human expertise with AI capabilities. In this sense, GenAI represents not only a technological innovation but also a driver of educational transformation (Hang et al., 2023).

This teaching tip explores how teaching staff can use LLMs to develop MCQs, outlines key considerations, and provides practical guidance for implementation.

3. What can LLMs offer for MCQ-based assessment?

LLMs can support educators in several ways (Artsi et al., 2024; UNESCO, 2023):

More efficient item development 

One of the primary advantages of GenAI is speed. Whereas manually constructing MCQs is time-consuming, LLMs can rapidly generate multiple draft or sample items. Reduced time investment allows for the development of larger item pools, facilitating more frequent renewal of question banks and helping to ensure that assessment materials remain current.

Beyond generating new items, LLMs can also be applied to existing MCQ sets to:

• Revise and restructure large item banks.
• Adapt existing questions to different levels of difficulty.
• Assemble multiple equivalent versions of an examination with comparable levels of difficulty.

Quality assurance 

Evaluating the quality of MCQs requires both time and expertise, and ensuring consistency and comparability across items remains a significant challenge. Conducting such evaluations efficiently is particularly demanding when working with large item banks. Assessing MCQs in terms of item writing flaws (IWFs) is especially complex in extensive question sets. IWFs refer to errors or weaknesses in item construction, such as ambiguous wording, negatively phrased stems, or inconsistently structured response options. LLMs may, in the future, play a valuable role in systematically screening and standardising items, thereby supporting the validity and reliability of examinations. In addition, LLMs can assist in correcting grammatical and linguistic inaccuracies, reducing the need for manual post-editing.

Greater variation and enhanced learning experiences 

GenAI facilitates the generation of varied items tailored to different student groups or levels. This enables educators to design assessments that better align with diverse learner needs. For example, first-year students may be presented with more foundational knowledge questions, whereas master’s students may engage with complex case-based or interpretative tasks. In addition, GenAI opens up opportunities for more formative assessment by enabling the rapid generation of personalised practice questions with immediate feedback.

Enhanced feedback for students 

Research indicates that MCQs are most effective for learning when students receive not only the correct answer but also explanations for why that answer is correct and why the alternatives are incorrect. In practice, however, such detailed feedback is rarely provided due to the time required to develop it. LLMs can support this process by automatically generating rationales for each answer option (Ch’en et al., 2025).

4. Challenges and considerations

Despite their potential, the use of LLMs also raises important considerations for educators (Law et al., 2025; Rincon et al., 2025; Ahmed et al., 2025).

General considerations

• Limited currency: Many models do not have access to real-time information. More recent developments may therefore be absent unless using models with updated access.
• Reliability of the output: GenAI can sometimes produce inaccurate or unsubstantiated responses that nevertheless appear convincing, often referred to as 'hallucinations'. Critical review by the educator remains essential.
• Referencing and transparency: LLMs rarely provide verifiable sources. This is particularly problematic in educational and assessment contexts, where accuracy and traceability are crucial.
• Bias and ethics: Training data may contain biases, which can be reflected in model outputs.
• Privacy and confidentiality: Sensitive or identifiable information about students or colleagues should not be shared.
• Over-reliance: While GenAI can reduce workload, it should not replace the educator’s professional judgement and pedagogical decision-making.
• Implementation challenges: The responsible and practical integration of GenAI into teaching remains an evolving area, with unresolved questions regarding logistics, cost, and staff development.

Considerations specific to MCQs

• Scientific evidence: Research on GenAI-generated MCQs is still emerging. Existing studies are limited in number, often conducted in English, and typically situated in controlled environments rather than authentic educational contexts.
• Difficulty level: Evidence suggests that AI-generated items tend to be less challenging than those developed by experienced educators, limiting their effectiveness in assessing higher-order cognitive skills such as application and analysis.
• Discriminatory power: Relatedly, such items may be less effective at distinguishing between high- and low-performing students.
• Quality of distractors: The plausibility of incorrect options remains a known limitation. Implausible distractors reduce the validity of assessment.
• Rationales and feedback: Although LLMs can generate explanations, these are often superficial or insufficiently nuanced, limiting their current pedagogical value.
• Role of the educator: LLMs are best used to generate templates or initial drafts. Comprehensive review, refinement, and validation by the educator remain indispensable.

5. Getting started with LLMs for MCQs

The importance of effective prompting 

When using LLMs to develop MCQs, the quality of the prompt is critical. A prompt refers to the instruction provided to the model; the clearer, more specific, and better structured the prompt, the higher the quality of the output. This process, often referred to as prompt engineering, is an emerging field that examines how models can be guided to produce relevant, usable, and reliable outputs (Kiyak et al., 2024; Ch’en et al., 2025; Artsi et al., 2024).

Core principles for effective prompt use

6. Step-by-step guide for developing MCQs with LLMs

Based on the principles above, teaching staff can follow a step-by-step approach. This helps to harness the benefits of GenAI while simultaneously keeping known challenges (such as the quality of distractors or overly simple questions) under control (Kiyak et al., 2024; Bhowmick et al., 2023).

6. What does the future hold?

The use of LLMs for developing MCQs is still in its infancy, and there is currently very little scientific research on this topic. Moreover, the existing literature is often based on English-language use (which is not easily implementable within the context of Dutch higher education), and research has been conducted in specific research settings, without a clear translation to everyday practice.

At the same time, models are developing at a rapid pace, with new developments following one another in quick succession. New generations of GenAI models (for example, OpenAI's reasoning models or DeepSeek-R1) can already reason better and build up their answers step by step.

In addition, there are first systems specifically designed to support educators in assessment development, such as Questgen, QuizRise, and others (Hang et al., 2023).

The future therefore looks promising, with considerable confidence that AI will offer opportunities to support educators in developing, revising, and evaluating MCQs. For teaching staff, this means that experimentation is worthwhile, but that critical evaluation and human validation remain indispensable.

7. Conclusion

LLMs can accelerate and enrich the development of MCQs, but their output should serve only as a starting point. Human oversight remains essential to ensure quality and didactic value. The technology is developing at a rapid pace, so experimentation and critical learning are crucial. Do not wait until GenAI is perfect, but begin using it today: experiment, try things out, evaluate critically, and progressively discover how it can support you.

​Lees deze tip in het Nederlands​