COVER STORY

Is there something that we dread more than preparing for an exam, or sitting for it? Perhaps not. Unless, you consider the wait before the results transpire. Frolicking aside, we are stressed beyond anything, wondering if we would ace the paper or stand out among all our mates; more so if the questions were, in our mind, ‘tough’ or ‘out of syllabus’. As much as we all like to blame the system, there’s one examination board that is currently working hard to eliminate all possibilities of human error and to provide a result sheet that is essentially fair to all those awaiting it. The Aga Khan University Education Board (AKU-EB) has a solution - its e-marking system, a unique feature it offered to put on full display for the stakeholders earlier this month after having concluded its April/May 2016 Examinations across Pakistan.

Now, in its phase of assessment, Dr. Shehzad Jeeva  (Director, AKU-EB) along with his team including Dr. Naveed Yousuf (Associate Director, Assessment), Ms. Aamna Pasha (Assistant Director, Teachers Development), and Hanif Shariff and Aqeel Farooq (Associate Director and Manager, Operations, respectively) conducted a briefing session on the whole process while also elaborating on the e-marking systems.

Behind the scenes

The idiom ‘more than meets the eye’ rings true when it comes to the setup they have at their facility. And apparently everything that they do is not without purpose. Take this meeting, for instance. How else could they ensure these principals have the confidence that the system is safe and convey the same to the parents and students they have a one-on-one interaction with, if not by inviting them to see it’s not all about scanners and screens?

A model board to create a model student body

A year down the road of having introduced the CCTVs at the exam centres, they have transparency and credibility of the qualification to their credit. But, if one is to improve performances and continue excelling in the academic sector, then we have to take into account and  consider the fact that AKU-EB has trained 850 middle-school teachers, and reviewed and revised nine syllabi in the last five months. Since all examinations that are being conducted by the AKU-EB have to be reliable, valid and fair (based on best assessment practices), the process is pretty much rigorous, divided into three segments: examination paper development, e-marking and compilation of results.

To begin with, examination development is important because therein the people in charge can assert whether theirs is in alignment with the syllabi and its objectives, with exam specifications, with required cognitive and appropriate difficulty level, and fair to all. Based on a national curriculum, the syllabi include not just clear student learning objectives but also examination specifications (or ‘blueprints’).  What they’ve done is to direct groups of five experts to review the questions available to them. Two of them are content experts; the remaining three, specialists teaching the same level (HSC or SSC), though not necessarily the subject specialists, in a position to go through the structure, clarity of language and stimuli being used in the question.

Moving on, we have the e-marking system to maintain anonymity and transparency throughout the process. When it introduced the system in 2007, AKU-EB became the first ever board in Pakistan to endorse e-marking. Students should get what they deserve, right? Neither less nor more. It’s easy enough for on-lookers to comment how computer-assisted marking will not just be a timely analysis but also protect individual candidate against marked error(s).  However, a lot of human effort is still needed.

The cover sheet of the examination answer booklet contains all the information regarding the student - student’s name, school name, candidate ID and exam centre ID. There’s a numeric code or barcode, too. Once the exams are administered, the operation staff, who additionally do not have access to scoring, cuts off the edges of the booklets off into single sheets within a closed room before giving it to the graders. Assuming one does not have the power to decode the barcode (if it’s not hidden on the screenshot presented for e-marking), it goes unstated that they cannot be matched again; in other words, there’s no way to identify and know which Multiple Choice questions/Constructed Response Questions/Extended Response Questions has been answered by which student (gender, school or region). And then sheets are randomly passe    d through scanners and then compiled and locked. Bottom line’s complete anonymity is maintained from the minute the booklet is received till the results are announced to minimise biasness.

The last stage, the compilation of results, is what the esteemed guests have really gathered for - the topic took centre stage in the Q/A discussion which ensued after the presentation by Dr. Naveed Yousuf ended.  Yes, what I am trying to say here is it warrants a separate section.

Content validity and fairness of marking schemes

When a copy of marking scheme is revised and approved for a given paper, two senior markers (or ‘seeders’ as they are referred to here) upon request select a small population of students who are representatives of both genders, of performance levels and of different regions across Pakistan. They then sit down to independently mark these scripts. The report generated confirms whether the marking scheme is indeed clear enough. If it was the case, the scores for each question and each student given by the two markers would be the same. If there happens to be a discrepancy, on the other hand, it will be discussed. Was the marking scheme not clear enough? A very unique response is common with problem solving questions and because students are taught differently in different regions; was this a reason which is why they were unsure how to mark it? Did the senior markers and subject experts agree that such a response should get credit? The marking scheme is revised accordingly, made more clear and more structured (and thus more reliable, valid, and fair), and e-marking starts.

The subject specialists and other seniors have an initial discussion with the e-markers to guide and lead them with respect to the marking scheme as well as checking hints so that they are at the same level of understanding. Yet, no matter what you do, human error is involved. There might be a slightly lenient marker just as there might be a slightly strict marker. The Board tries to neutralise this effect: it distributes questions instead of students among the e-markers believing no student ought to be at a slight (dis)advantage than the other when it comes to educational assessment. Each and every script is scored by as many markers as there are number of questions on the paper.

A senior marker (who is most likely a seeder or a very senior marker who has been available with AKU-EB as a senior teacher) is paired with five other markers to simultaneously review their scoring trends as well as the time taken by each marker for a single question. This senior marker would naturally become familiar with the average scoring pattern of the students and if they observe any deviation (even if by chance of fatigue), they could pull the plug then and there only, re-discuss and/or redefine before proceeding further.

To put it into perspective, the scores are siphoned through three levels and then published: the questions are ‘marked’, ‘reviewed’ and ‘verified’.

On maintaining quality

“That’s a lot of work,” you must be thinking. But if e-marking is completed, the process itself is far from over. Students of AKU-EB (oh, and those who are not, be informed), your scores will now be ready to be analysed. Internationally recognised indicators are used to calculate the reliability indices of the paper, the standard error measurements, the mean scores, the standard deviations for each question in the paper even if that figure happens to be 4,000 administered in April/May alone, for instance.

 “Treat the patient, not the lab”

To challenge the teachers themselves with the exact questions as set in the exams is a rarity; to see the latter admit they are “not too sure what’s the correct answer” and the Board giving those students due credit is rarer still. This is not to suggest that the AKU-EB in any way undermining the content expertise of these highly regarded teachers who, nevertheless, get to be the final decision makers.

All this work and dedication to treat the patient’s pain and not what the lab reports say pays off when they are able to truly discriminate between the students who know well and who don’t. Having completed a very rigorous post-examination analysis for all questions in the paper, the supervisors compile the results (‘aggregate scores’ or ‘total scores’), which is checked to evaluate the number of A-graders, B-graders and so on, the overall aim of this whole exercise.

“A good question is half of knowledge”

Coming to the Q/A session I briefly touched upon earlier, a natural query that cropped up was regarding the criteria of a senior marker so involved in the set-up. Years of teaching, years of being affiliated with AKU-EB as a marker and, later, as a senior marker, performance in the organisation, whether they are distinguished enough, whether they have attended the training, and whether they have been a part of the development of AKU-EB’s syllabi sums up the holistic approach to senior marker selection. If one looks closely, the criteria locate ones who know the syllabi, about the AKU-EB and are well aware of the examination paper.

In the faintest of attempts to capture the big fish, to find a flaw in this system, a question that coerced Dr. Shehzad Jeeva to address the small assembly before him was concerning a marking scheme for P2. Without dismissing the idea outright, he chose to argue it with actual facts. According to him, the major reason one doesn’t see any marking schemes or solved papers for P2 is because AKU-EB came into being only because the schools at the time wanted to eradicate rote learning - and, of course, the quality education they were already providing. While they do share certain aspects during the teacher training for good response and bad response, publishing it would become a hindrance to conceptual learning they’re so eager to promote! (And he’s right; we may be learners all our lives, each with a unique point of view, but at times like these there seems to be no harm in going through solved answers. Pretty soon, it becomes a habit to memorise and we prefer writing it precisely as it is than present our own idea.)

“We in Pakistan are the highest producers of A and A*. Should we be proud or should we be worried about it? People say we’ve cracked the code. I say no, we’ve not cracked the code...I’ve personally seen lots of students who’re attending tuition centres and each one of them carries a marksheet booklet in their hands. Where’s the education? Where’s the imagination? It’s a number game....”

So, most of these students securing As in A-levels may not really be competent. They are just ‘test-wise’; they know how to get good scores. If you compare it to the local boards, most of them know how to win the sympathy of the examiner, leaving a comment like “tabiyat sahi nahi hai” on their answer copy. If, however, a student is scoring 98, 99, or 100 percent marks in an AKU-EB examination, that kid rightly deserves a pat twice on the back because it’s difficult to impress multiple markers.

Seeing is believing

The visit ended with a walk to the room where the markers sat busy working with the e-marking system - well, technically, there was a lunch and signing off in the comments book, but I’m sure we can omit that here. The curious visitors were handed a pass and divided into groups as they stepped across the threshold into a hush-hush environment.

There are eight options in the software - marked, remarked, parked, rescan, reviewed, unmarked, escalate and verified - for examiners to mark Secondary School Certificate and Higher Secondary School Certificate. One marker can pursue the answer key from the software only to check, verify and save the responses as marked. They can park the question if they need more time to assess it; if this confusion entails from some conceptual misunderstanding, they can leave comments and escalate the question to the respective senior marker; if it’s due to an unclear picture, clicking rescan would send it to the operations department.

The following terms are often misunderstood and misused. They are constantly referred to in the article; hence, the need for clarity.

Reliability: Also called ‘accuracy’ in layman’s terms, a test is said to be more reliable if there are less errors.

Validity: To be valid, a test must provide consistently accurate measurements, assessing what it’s supposed to assess. Therefore, a test must be reliable.

Fairness: The examination and the scores administered need to be impartial across the student population within the greater student body.