In 1992, the Mathematics Department at Tulane University (New Orleans, USA), took a bold decision of moving away from the traditional pen-paper based descriptive tests to pen-paper tests with only multiple-choice questions (MCQs). During that year, I was teaching Calculus to the 1st year undergraduate students, who are referred to as Freshmen. My first reaction was to label this decision ridiculous. And, I had three strong reasons to back my decision:
It was impossible to test a student's ability and knowledge in Calculus using MCQs as most problems required multi-step solutions or graphing skills.
The MCQ-based test will be easy and taken casually by students. The academic rigor will be compromised.
The average score of the students on the MCQ-based test will increase significantly. Some students who deserve an "F" may end up passing the course.
All my apprehensions were satisfactorily addressed during the training program that was conducted for our department by the assessment experts from the Education Department. Over the next 5 years, I happily used MCQ-based tests and discovered that all my hypothesis were false.
Till today, I have been involved in creating 100,000+ assessment items for some of the most recognized and reputed education and publishing companies globally. I have interacted with several researchers in assessment and learned the finer nuances of MCQ-based assessments. I have successfully trained more than 100 people to write high-quality assessment items. And, in this article, I am going to share a step-wise guide to create amazingly effective MCQs.
First Things First
On 17th November 2020, The Indian Express published a story about a survey conducted by Azim Premji University. It stated, "More than 90% of teachers felt that no meaningful assessment of children's learning was possible during online classes." The survey was very accurate, but 90% of teachers were absolutely wrong. Let me explain.
More than 90% of teachers have never received any training to design and develop effective online assessments. Such teachers obviously believe that online assessment can never be meaningful. Based on the above findings of the survey, you can choose two options:
Option A: Reject the adoption of online assessment
Option B: Train the teachers to conduct online assessment
Those who choose option A will be making a big mistake and going against the well-established research and time-tested practices of online assessments. Those who choose option B will read this article to the very end and try to implement this guide in their classrooms and schools.
If you are not sure which option is correct, let me point out that the National Education Policy 2020, strongly recommends online assessment. In fact, it speaks about adaptive testing to be used by every school. Adaptive testing is not possible without MCQ-based tests.
Question Bank: A collection of questions and clones
Every school and every teacher must begin to think about creating a database of questions, referred to as a question bank. This is important because good MCQ questions are like rare gems and they must be protected, preserved, and reused as often as possible. In fact, the quality of the question bank is often the real differentiator between the quality of education that a school or a teacher imparts.
In 1999, I was a consultant to eGurucool, one of India's first initiative in eLearning. It was there that I came in contact with Dr. V. Natarjan, who had written several books and research papers on assessment. Together with Dr. Natarajan, we initiated the mammoth exercise of creating a question bank for every subject and every class of the school education curriculum. I learned a lot from Dr. Natarajan whose passion for assessment was unmatched. He introduced me to the basic terminology used for assessment banks.
An item is a question. Each item consists of a stem with 4 or 5 choices. There can be one or more correct choices. The incorrect choices are referred to as distractors. Each choice has to have a rationale that explains the scenario in which the respondent (student) will select that choice. The correct choices are based on the possible solutions (answers) of an item. The distractors are based on the most common errors or popular myths associated with the item. A clone of an item is created by a minor modification to the stem or the choices of the item. Each item can be used to create 5 - 20 clones depending on the type of item or the concept on which it is based.
We will now use the above terminology for the rest of this article.
Classifying Items with Depth of Knowledge Levels
When we created items at eGurucool, Bloom's taxonomy was the preferred way to classify items. Since the last 10 years, items are being tagged globally, using Norman Web's four Depth of Knowledge (DoK) Levels as shown in the image below.
DoK Levels are different from Bloom's taxonomy. In fact, DoK levels are not a taxonomy and these levels were developed for the sole purpose of applying Bloom's taxonomy in a standardized manner across different systems of assessments. The relationship between the two classifications can be understood from the following image.
Understanding these DoK levels is important if you want to create tests that assess the students' level of thinking.
For the last 20 years, we have been watching Kaun Banega Crorepati. We are all pretty familiar with the format of the show. What is the DoK level of the questions asked on KBC?
All the questions on KBC are DoK 1 level. The difference between the question worth Rs 1000 and Rs 10000000 is the level of difficulty. And, this reveals one more important aspect of creating items.
For each DoK level, we can create items of different levels of difficulty.
DoK 2 items ask the students to:
explain concepts, ideas, or procedures, or
apply these concepts, ideas, or procedures to new contexts
DoK 3 level items focus on strategic thinking. To frame these items, we must understand and use strategic thinking ourselves. Otherwise, it will become very challenging to create such items. In a nutshell, these items require students to think of a strategy to get the correct response. This strategy may involve deep analysis of the problem, multiple concepts or ideas, and a combination of different procedures or processes.
DoK 4 items require extended thinking. These items would involve creating a unique solution to the problem by evaluating multiple facets in a logical manner. MCQs based on DoK 4 levels are rarely created because the time frame required to answer these questions extends beyond the time frame available for MCQ tests.
As the DoK level increases, more time is required to frame an item. In my experience, these time frames are:
DoK 1 item: 1 hour
DoK 2 item: 3 hours
DoK 3 item: 6 hours
So, be prepared to invest time to build a question bank. If you rush through this process, you will end up with trash. But there is good news too. Each MCQ item that you create can be cloned to create multiple items. So, the return on your time is not that bad after all.
Creating an Item
Now, it's time to dive into the item creation process. So, here are the steps.
Identify the Learning Outcome: Be as concise as you can. One item SHOULD assess only one learning outcome. From the response of the student, you should be able to conclude whether or not the student has achieved the learning outcome. And it the student has not attained the learning outcome you should be able to identify the learning gaps that need to be addressed.
Choose a DoK level: Choose the DoK level for which you want to create the item. On a standardized test, a typical distribution of DoK levels is 20% (DoK 1), 60% (DoK 2), 20% (DoK 3). But, you may want to reduce the percent of DoK 2 items in favor of DoK 3 items if you want to increase the cognitive challenge of the tests.
Create a stem: The stem should be complete, clear, comprehensible, and concise. Use appropriate vocabulary, short sentences, and avoid any redundant information. The idea is not to confuse or trick the student but to provide all the information necessary to get the correct response. The stem may even include a visual if it helps the student understand the problem.
Write the correct solution(s)/answer(s): Though we prefer one correct answer, it is ok to have multiple correct answers to an MCQ. Most learning management systems allow you to create tests with such questions. In fact for DoK 3 questions, you may wish to have more than one correct option to see if students can think differently about the same problem.
Identify common errors or misconceptions: This is the key to writing valid distractors. Every concept can be misinterpreted or misunderstood. These are the misconceptions you want to capture. Also, every procedure or recall can be erroneously reproduced.
Create correct options: Properly phrase the answers/solutions to create correct options. The phrasing has to be done in a manner that students can easily understand.
Create distractors: Use the misconceptions and common errors to create the distractions. Make sure to use only ONE misconception or error to frame one distractor. If the student chooses this distractor as a response, you must be able to identify the EXACT misconception or the error that needs to be addressed.
Write down the rationale for each distractor: This is an important step that is often neglected by item writers (teachers). Most assessment items have to go through a post-creation review by an expert. When the reviewer is evaluating the item, she should know the reason why you chose a particular distractor. In the absence of this rationale, the time taken by the reviewer will double or triple.
Write down the feedback for each option: For each option, correct and incorrect, you must provide students constructive feedback. In case of correct answer, let the student know that she has achieved the learning outcome. In case of incorrect answer, your feedback must include the gap you have identified as well as the corrective action the student has to take to eliminate that gap.
Assign a difficulty level: The difficulty levels of an item can be Easy (1), Moderate (2), or Challenging (3). This is pretty tough initially because for you every question would be much easier than for your students. Also, the difficulty level of an item is never fixed nor uniform. It may vary depending on the current knowledge or skills of the students. An item that is easy for some students may be challenging for others. So, the best way is to think of your entire class as a group. How many of these students do you expect to get the correct answer for this item? If this estimate is 50% or more, assign the difficulty level as 1. If this estimate is 15% or less assign the difficulty level as 3.
Training and Practice
Writing assessment items is not an "obvious" task. It requires training as well as practice. In my experience, it is very difficult initially for teachers to make accurate and valid assessments for DoK 2 and DoK 3 levels as they have hardly encountered such problems themselves. In fact, after several days of training, most teachers are not able to create good DoK 3 items. But, the good thing is that your ability and skills increase with practice if you incorporate self-evaluation, peer-evaluation, and expert-evaluation as a means to collect feedback on the items you create.
In 20011, I headed a project in which we had to create 8000 MCQs for a company based in Dublin, Ireland. This was a challenging project and we had to recruit a team of 40 people to meet the deadlines. We spent close to 3 months training this team. Here are the three observations I made from this exercise:
Younger teachers were easier to train as compared to more experienced teachers. This could be because it is more difficult to unlearn rather than learn.
Only 20% of the items that were created made it through the final selection. So, please do not assume that every item you create will be a valid item.
Only teachers with a deep understanding of their subject could create DoK 2 and DoK 3 level items.
If you plan to enable the teachers of your school to write MCQs, do not underestimate the process or the training requirements. You will need patience and perseverance to succeed. There is no short cut.
Commentaires