Final Exam Survival -
Designing, Preparing for, and Taking It.

I have no home; I make awareness my dwelling.

I have no means; I make understanding my means.

I have no eyes; I make the flash of lightning my eyes.

I have no strategy; I make “unshadowed by thought” my strategy.

I have no enemy; I make carelessness my enemy.

(A few related verses from the samurai creed of an unnamed samurai;
Japan c.a. 13^th century)

(photo credit)

I have been teaching SE350 the operating systems course at the University of Waterloo this term. This Monday I concluded the lecture and now the students enter the final stage towards the final exam. After I finished designing the exam yesterday, I thought about some valuable lessons that I made during the process. These may be useful for other instructors in getting their exam right.

In addition my student quality indicators still are in the red for some of the students (more about this measure in a future post; but yes guys you were profiled big time) and it appears that they do not have the right methodology to prepare for the exam. Having designed this exam and being a student for the better part of the decade also enables me to talk about some useful lessons.

The Profs View - Exam Design Tactics

My original hiring committee told me that my job is teaching an operating systems course. In a bit over 35 lectures I explained the students anything from basic computer architecture to file system design. I also motivated them a bit about the field of embedded systems. My assignment is a success, if the overwhelming majority of students walk out of the course with a reasonable understanding of operating system concepts. So what does this have to do with the exam: The exam is one of the means to make sure only those walk out of the door with high grades that do actually understand these concepts.

Understanding happens at several levels and a certain spread in preparation. Here is a rough scale where I want to see certain student groups.

Grade	Description
0-5	At the very low end you have one or two students, who never showed up during the lecture and are stupid enough to take the exam. They will have little to no understanding of operating systems.
5-35	Then there will be a few who went to the lecture but did not make the effort to properly prepare for the exam at all. They also failed the related course project. Although they showed effort, the acquired skills will be of little use in practice.
35-60	Ending up in this range can have a diversity of reasons. Lack of concentration, panic etc. Someone who falls in this range should have a little understanding of the concepts, went to the tutorials, and followed the lecture. However, the level of understanding is not enough to really release them into the public. I would be seriously worried if those guys ever write system software.
60-80	Students in this range know the concepts. They followed the lecture, went to the tutorials and can solve simple problems by applying the algorithms of the lecture. Most of the students in this range also did well during their course project and reinforced the stuff learned in the lecture by applying it on the project. They usually also considered additional materials like the course text for the exam preparation. I would be relatively confident to release those guys into the public. In practice they will make the appropriate decisions with some aid based on what they have learned in the course.
80-100	The elite. These guys know stuff and can solve complex problems easily. They probably had close to no life and just focused on the exam preparation. I would be very confident to release those guys into the field because they know their stuff. The fact that they put so much preparation into it should show up in the marks as well as being better than the rest of the crowd.

First of all, I will not have any multiple-choice questions. Personally, I regard these tests as one of the more brain-dead approaches to evaluate students (long philosophical discussion skipped here).

My exam is designed in a way that I try to catch the 1-35 percentiles with dedicated questions. I have two questions in my exam that will just ask you to define and distinguish some very basic concepts. No one who at least went to the lecture should get less than one 1 point on the exam.

The 35 onwards percentiles are covered in composite problems. These problems usually start with explaining some concepts that go a little beyond definitions. The main part is usually an algorithm or explaining the internal workings of some exemplified system. Just mastering that will get them in the 60 – 80 range. To make it easier all individual composite problems are dedicated to particular individual problem areas.

To reward the elite, I usually throw in an “ice-berg” question at the end of the composite problem that one could only answer if they really know the concept well.

Since some of the concepts are less important from my perspective, those lesser important topics will have a lower weighting than the more important topics. The less important topics will also not be covered in-depth.

One important factor to consider is the marking of the exam. Avoid open-ended questions that require the students to make heavy assumptions about a problem. This will result in quite a diversity of answers that no TA really wants to mark. Avoid scattering questions randomly across the problem areas. In my case TAs were assigned to the lab and I dedicated one particular TA to the tutorial. That way you will have TAs marking the exam that had little to no exposure to the materials. You do not want to have each one of them to mark more than one particular problem area.

Another thought is to not include sophisticated coding questions. The time pressure and nature of the exam will cause most of them to make fugitive errors that may quickly deteriorate their scores. Real code you can edit, debug, compile, and massage over time. You cannot do that on paper. I would only ask them about code they have seen before in the book, tutorials, lecture slides, or likely written themselves.

The Exam-Takers View – How the hell do I prepare for that exam?

(funny, but even these guys won’t help you with that)

It is quite funny but that thought crossed my mind in a Karate sparring session today. As it turns out a possible approach to effectively prepare “mentally” for the exam is analogous to generating and maintaining muscle memory of certain techniques and applying them in practice.

Kata – Memorizing and Maintaining the Building Blocks

Kata in martial arts is often seen as the central building block in many martial arts. In Karate (one of the things I do besides Tai Chi and Kendo) it is a sequence of basic moves, each of which has a specific application and is exercised against an imaginary opponent. The point of learning and practicing Kata is to memorize the moves and their application. This is actually quite similar to learning operating system concepts. In the lecture several core concepts, requirements, and algorithms were described that should be “known” for the final exam.

When I was an undergraduate student I managed to pass my Bachelors degree with a perfect score. This was possible with an effective system of learning techniques and knowing how to work the system of electives in a way that I always had the best marks included in the final score. The system I used was coined by a Swiss guy called Gregor Staub. He wrote a book (in German) called “Mega Memory” in which he described the following idea among a many other techniques for specific memorization tasks.

The key principle to this technique is repetition. According to him you need to at least 6 repetitions to reinforce a learned concept. These repetitions need to occur in certain time intervals. The chance to forget a concept that you have just learned by tomorrow is very high if you do not revisit it again within 20 minutes.

His proposal was to distil the concepts of lectures down into small note cards (a.k.a. flash cards). These cards only contain the essential information that you need to reinforce the concept. For example I distilled my 755 slides of the lecture down to 84 (4 7/8" x 3 1/2") cards. Each of those cards carries a concept title on the back and a sequential card number.

Those cards are a very nice way to subdivide information. You can easily dissect concepts from the lecture. Once you have your deck, you need to reinforce the concepts.

Day 1:

To illustrate that approach, assume you take the first 20 cards out of your deck:

1. Memorize all those cards until you fluently can reproduce the concepts by just looking at the topic headline of the card.
2. Cards that you know perfectly well put on a separate stack. Let’s call this “know stack”.
3. Repeat 1 and 2 until you know all cards perfectly well and they are now on the “know stack”.
4. Now wait 20 minutes.
5. Do 1-3 over again.
6. Enough for today

Day 2:

1. Revisit all cards from yesterday again. This should yield a “know-yesterday-” and a “forgot stack”. Ideally, you should end up somewhere like 17-know and 3-forgot concepts.
2. Now take 20 more cards from your deck and put them on the “forgot” stack.
3. Repeat procedure from day 1 with the “forgot” stack.

Day 3-6 and onwards:

Now the same thing happens over again. You revisit each of those days separately and place the “known” concepts on respective “know-day-x” stacks. All forgotten and new concepts get placed on the “forgot” stack. On day six, you should end up with 6 separate stacks that you know quite well. The oldest of those stacks was successfully revisited 6 times and sits in long-term memory. You can safely put that old stack away in a separate box. You do not need to revisit it again until 3 months from now to make it permanent.

Over the years I made some amendments to the original technique. Instead of learning by titles you can use the back sides to formulate specific questions about the cards content. That speeds up the reinforcing process quite significantly. For very big exams, for example my preparation for the IBM-certified Specialist Software Quality exam, I ended up with close to 400 concept cards that contained 3 terms of software quality techniques and quality management. To speed up the reinforcing process, I bespoke MP3s with the these concept card questions and the related answers that I listened to after getting out of bed in the morning and when I got back to bed. This one also helped to speed up the reinforcing process quite significantly.

This technique is a very powerful tool to memorize concepts. It should be said that the information placed on the note-cards is very subjective. Depending on how well you already know a particular subject will dictate how many cards you dedicate to it. That said learning from someone else’s cards is not a good idea. The entire process of distilling the lecture materials down is already a good learning experience. Do not underestimate the overhead of this distillation process. It took me 2 working days to distil 755 slides down to 84 cards… and I’m the guy who taught the course.

This process of memorization will put you into the 60-80-crowd; however, to do really well you need to gear up a bit.

Kumite – Reinforcing by Application

Kumite generally refers to sparring in Karate. In contrast to Kata you train with a real individual to develop a particular skill or practice for competition. In martial arts just knowing the individual moves and Kata will not necessarily make you a good competitor.

So what has this to do with exam preparations? Well now that you know the “Kata” you need to know how to apply it in practice.

Practical problems can be found in the course book and were presented in the tutorials. Create a small booklet or folder of these problem descriptions on the front and their solutions on the back. Solve these problems on a separate sheet of paper and then compare your solution…

• Every time you have some free time, pick a problem at random and solve it.
• Every time you are etching to play your favourite MMORPG, pick a problem at random and solve it instead.
• Every time you feel like goofing off on Facebook or Youtube, pick a problem at random and solve it instead.
• Every time you feel like talking things over with your friend on the cell, pick a problem at random and solve it instead.
• First thing in the morning … pick a problem and solve it.
• Last thing in the evening … pick a problem and solve it.
• I think you got the idea … pick a problem and solve it!

If you really dedicate some time to this you will master the art of operating systems, which will certainly bring you closer from the 60-80s to the 100. This step will cost a lot of effort but the fruits are rewarding. In martial arts this is also the sweaty part.

If you followed that through there should be no trouble in taking the exam. Some basic exam taking rules are printed on the current revision of the exam itself; don’t worry you will see it soon enough.

References

• http://en.wikipedia.org/wiki/Family_Guy
  Always funny always cheers me up when I am stressed out… btw. pick a problem and solve it instead of goofing off on the net!!!
• http://en.wikipedia.org/wiki/Kumite
• http://en.wikipedia.org/wiki/Kata
• http://www.megamemory.ch/
  Gegor Staub’s website (in German)