Final Exam Survival -
Designing, Preparing for, and Taking It.

I have no home; I make awareness my dwelling.

I have no means; I make understanding my means.

I have no eyes; I make the flash of lightning my eyes.

I have no strategy; I make “unshadowed by thought” my strategy.

I have no enemy; I make carelessness my enemy.

(A few related verses from the samurai creed of an unnamed samurai;
Japan c.a. 13^th century)

(photo credit)

I have been teaching SE350 the operating systems course at the University of Waterloo this term. This Monday I concluded the lecture and now the students enter the final stage towards the final exam. After I finished designing the exam yesterday, I thought about some valuable lessons that I made during the process. These may be useful for other instructors in getting their exam right.

In addition my student quality indicators still are in the red for some of the students (more about this measure in a future post; but yes guys you were profiled big time) and it appears that they do not have the right methodology to prepare for the exam. Having designed this exam and being a student for the better part of the decade also enables me to talk about some useful lessons.

The Profs View - Exam Design Tactics

My original hiring committee told me that my job is teaching an operating systems course. In a bit over 35 lectures I explained the students anything from basic computer architecture to file system design. I also motivated them a bit about the field of embedded systems. My assignment is a success, if the overwhelming majority of students walk out of the course with a reasonable understanding of operating system concepts. So what does this have to do with the exam: The exam is one of the means to make sure only those walk out of the door with high grades that do actually understand these concepts.

Understanding happens at several levels and a certain spread in preparation. Here is a rough scale where I want to see certain student groups.

Grade	Description
0-5	At the very low end you have one or two students, who never showed up during the lecture and are stupid enough to take the exam. They will have little to no understanding of operating systems.
5-35	Then there will be a few who went to the lecture but did not make the effort to properly prepare for the exam at all. They also failed the related course project. Although they showed effort, the acquired skills will be of little use in practice.
35-60	Ending up in this range can have a diversity of reasons. Lack of concentration, panic etc. Someone who falls in this range should have a little understanding of the concepts, went to the tutorials, and followed the lecture. However, the level of understanding is not enough to really release them into the public. I would be seriously worried if those guys ever write system software.
60-80	Students in this range know the concepts. They followed the lecture, went to the tutorials and can solve simple problems by applying the algorithms of the lecture. Most of the students in this range also did well during their course project and reinforced the stuff learned in the lecture by applying it on the project. They usually also considered additional materials like the course text for the exam preparation. I would be relatively confident to release those guys into the public. In practice they will make the appropriate decisions with some aid based on what they have learned in the course.
80-100	The elite. These guys know stuff and can solve complex problems easily. They probably had close to no life and just focused on the exam preparation. I would be very confident to release those guys into the field because they know their stuff. The fact that they put so much preparation into it should show up in the marks as well as being better than the rest of the crowd.

First of all, I will not have any multiple-choice questions. Personally, I regard these tests as one of the more brain-dead approaches to evaluate students (long philosophical discussion skipped here).

My exam is designed in a way that I try to catch the 1-35 percentiles with dedicated questions. I have two questions in my exam that will just ask you to define and distinguish some very basic concepts. No one who at least went to the lecture should get less than one 1 point on the exam.

The 35 onwards percentiles are covered in composite problems. These problems usually start with explaining some concepts that go a little beyond definitions. The main part is usually an algorithm or explaining the internal workings of some exemplified system. Just mastering that will get them in the 60 – 80 range. To make it easier all individual composite problems are dedicated to particular individual problem areas.

To reward the elite, I usually throw in an “ice-berg” question at the end of the composite problem that one could only answer if they really know the concept well.

Since some of the concepts are less important from my perspective, those lesser important topics will have a lower weighting than the more important topics. The less important topics will also not be covered in-depth.

One important factor to consider is the marking of the exam. Avoid open-ended questions that require the students to make heavy assumptions about a problem. This will result in quite a diversity of answers that no TA really wants to mark. Avoid scattering questions randomly across the problem areas. In my case TAs were assigned to the lab and I dedicated one particular TA to the tutorial. That way you will have TAs marking the exam that had little to no exposure to the materials. You do not want to have each one of them to mark more than one particular problem area.

Another thought is to not include sophisticated coding questions. The time pressure and nature of the exam will cause most of them to make fugitive errors that may quickly deteriorate their scores. Real code you can edit, debug, compile, and massage over time. You cannot do that on paper. I would only ask them about code they have seen before in the book, tutorials, lecture slides, or likely written themselves.

The Exam-Takers View – How the hell do I prepare for that exam?

(funny, but even these guys won’t help you with that)

It is quite funny but that thought crossed my mind in a Karate sparring session today. As it turns out a possible approach to effectively prepare “mentally” for the exam is analogous to generating and maintaining muscle memory of certain techniques and applying them in practice.

Kata – Memorizing and Maintaining the Building Blocks

Kata in martial arts is often seen as the central building block in many martial arts. In Karate (one of the things I do besides Tai Chi and Kendo) it is a sequence of basic moves, each of which has a specific application and is exercised against an imaginary opponent. The point of learning and practicing Kata is to memorize the moves and their application. This is actually quite similar to learning operating system concepts. In the lecture several core concepts, requirements, and algorithms were described that should be “known” for the final exam.

When I was an undergraduate student I managed to pass my Bachelors degree with a perfect score. This was possible with an effective system of learning techniques and knowing how to work the system of electives in a way that I always had the best marks included in the final score. The system I used was coined by a Swiss guy called Gregor Staub. He wrote a book (in German) called “Mega Memory” in which he described the following idea among a many other techniques for specific memorization tasks.

The key principle to this technique is repetition. According to him you need to at least 6 repetitions to reinforce a learned concept. These repetitions need to occur in certain time intervals. The chance to forget a concept that you have just learned by tomorrow is very high if you do not revisit it again within 20 minutes.

His proposal was to distil the concepts of lectures down into small note cards (a.k.a. flash cards). These cards only contain the essential information that you need to reinforce the concept. For example I distilled my 755 slides of the lecture down to 84 (4 7/8" x 3 1/2") cards. Each of those cards carries a concept title on the back and a sequential card number.

Those cards are a very nice way to subdivide information. You can easily dissect concepts from the lecture. Once you have your deck, you need to reinforce the concepts.

Day 1:

To illustrate that approach, assume you take the first 20 cards out of your deck:

1. Memorize all those cards until you fluently can reproduce the concepts by just looking at the topic headline of the card.
2. Cards that you know perfectly well put on a separate stack. Let’s call this “know stack”.
3. Repeat 1 and 2 until you know all cards perfectly well and they are now on the “know stack”.
4. Now wait 20 minutes.
5. Do 1-3 over again.
6. Enough for today

Day 2:

1. Revisit all cards from yesterday again. This should yield a “know-yesterday-” and a “forgot stack”. Ideally, you should end up somewhere like 17-know and 3-forgot concepts.
2. Now take 20 more cards from your deck and put them on the “forgot” stack.
3. Repeat procedure from day 1 with the “forgot” stack.

Day 3-6 and onwards:

Now the same thing happens over again. You revisit each of those days separately and place the “known” concepts on respective “know-day-x” stacks. All forgotten and new concepts get placed on the “forgot” stack. On day six, you should end up with 6 separate stacks that you know quite well. The oldest of those stacks was successfully revisited 6 times and sits in long-term memory. You can safely put that old stack away in a separate box. You do not need to revisit it again until 3 months from now to make it permanent.

Over the years I made some amendments to the original technique. Instead of learning by titles you can use the back sides to formulate specific questions about the cards content. That speeds up the reinforcing process quite significantly. For very big exams, for example my preparation for the IBM-certified Specialist Software Quality exam, I ended up with close to 400 concept cards that contained 3 terms of software quality techniques and quality management. To speed up the reinforcing process, I bespoke MP3s with the these concept card questions and the related answers that I listened to after getting out of bed in the morning and when I got back to bed. This one also helped to speed up the reinforcing process quite significantly.

This technique is a very powerful tool to memorize concepts. It should be said that the information placed on the note-cards is very subjective. Depending on how well you already know a particular subject will dictate how many cards you dedicate to it. That said learning from someone else’s cards is not a good idea. The entire process of distilling the lecture materials down is already a good learning experience. Do not underestimate the overhead of this distillation process. It took me 2 working days to distil 755 slides down to 84 cards… and I’m the guy who taught the course.

This process of memorization will put you into the 60-80-crowd; however, to do really well you need to gear up a bit.

Kumite – Reinforcing by Application

Kumite generally refers to sparring in Karate. In contrast to Kata you train with a real individual to develop a particular skill or practice for competition. In martial arts just knowing the individual moves and Kata will not necessarily make you a good competitor.

So what has this to do with exam preparations? Well now that you know the “Kata” you need to know how to apply it in practice.

Practical problems can be found in the course book and were presented in the tutorials. Create a small booklet or folder of these problem descriptions on the front and their solutions on the back. Solve these problems on a separate sheet of paper and then compare your solution…

• Every time you have some free time, pick a problem at random and solve it.
• Every time you are etching to play your favourite MMORPG, pick a problem at random and solve it instead.
• Every time you feel like goofing off on Facebook or Youtube, pick a problem at random and solve it instead.
• Every time you feel like talking things over with your friend on the cell, pick a problem at random and solve it instead.
• First thing in the morning … pick a problem and solve it.
• Last thing in the evening … pick a problem and solve it.
• I think you got the idea … pick a problem and solve it!

If you really dedicate some time to this you will master the art of operating systems, which will certainly bring you closer from the 60-80s to the 100. This step will cost a lot of effort but the fruits are rewarding. In martial arts this is also the sweaty part.

If you followed that through there should be no trouble in taking the exam. Some basic exam taking rules are printed on the current revision of the exam itself; don’t worry you will see it soon enough.

References

• http://en.wikipedia.org/wiki/Family_Guy
  Always funny always cheers me up when I am stressed out… btw. pick a problem and solve it instead of goofing off on the net!!!
• http://en.wikipedia.org/wiki/Kumite
• http://en.wikipedia.org/wiki/Kata
• http://www.megamemory.ch/
  Gegor Staub’s website (in German)

Planning for the First University-Level Teaching Position

“She made herself a wall and told the kids to overcome it. Students who do not hit that wall are inferior and weak to failure. Nowadays the kids do not take instructors seriously anymore because the instructors do not try to build that wall. […] They will be quick to forget me but they will never forget Akutsu-Sensei.”
(Namiki-Sensei about Akutsu-Sensei, じょおうのきょうしつ - The Queens Classroom EP11)

Fortune smiled at me this term. A key professor took a parental leave from the ECE department and they were looking for a new instructor. This course is relatively tough to teach. I suspect because none of the faculty was willing to jump in and previous sessional instructors were also unavailable, they actually considered getting in some foreign aid. Having assisted this course before, I was taken into consideration for the interview process.

The interview process was straightforward. It was a series of interviews focusing on the overall capabilities of an instructor. First they want to see you in person and assess that you are not a moron. Self presentation-, listening-, and observation skills get you through that. The next thing they focus on is teaching quality assurance. In my case the interview was targeted at a specific course. You should know what your course will be about and have a detailed plan how you would organize the lecture. That includes not only the content of the lecture but also managerial considerations as to where to allocate TAs and how many TAs you get / are expected to work with. Finally someone will be invited to assess your background knowledge about specific topics of the course. This one could eventually be your biggest thread. You should now your stuff well. Some interviewers might drop you like a hot dish: “How can this guy dare to teach , if he doesn’t even know ”. Other interviewers might be leaner and still think you are smart enough to prepare well for your assignment.

If you pass this stage they will ask you to perform a demo lecture about a sub-section of the course. Mine went well and I was notified that I got accepted.

So now you got the job and your schedule for the next term will be shot. I expect to have little time for anything else than teaching this term. Trust me, you should not do this for money as you will end up putting far more hours in it than “expected”. If you are really short on cash consider Teaching Assistantships. If you screw up as instructor, you not only have an angry mob of undergrads after you, but you also risk your reputation in the department. Since academic communities are usually small, next time you apply for a faculty or instructor position that will be considered. An angry mob of undergraduates cannot really hurt you as long as you obey university policies and master some martial arts skills ;), but they can be really annoying. To have your peace of mind and be able to look back on an effective term those issues should, however, be avoided in the first place.

The first thing I worried about is, what resources and additional man power I would be getting. The next thing is to get in touch who-ever was instructing and or supporting the course before. The more old material you can get, the less stuff you have to prepare next term. In my case I was very lucky to connect with the previous instructor and lab instructor of the course. Most of the materials were obtained, slides, past exams, quizzes, and lab materials.

Engineering the Wall

•••••••••••••••▼The Wall▼•••••••••••••••

┬┴┬┴┬┴┬┴┬┴┬┴┬┴┬┴┬┴┬┴┬

┴┬┴┬┴┬┴┬┴┬┴┬┴┬┴┬┴┬┴┬┴

┬┴┬┴┬┴┬┴┬┴┬┴┬┴┬┴┬┴┬┴┬

┴┬┴┬┴┬┴┬┴┬┴┬┴┬┴┬┴┬┴┬┴
(by some unknown very geeky ASCII artist)

Since the students have to metaphorically overcome your wall, you need to engineer it in the first place. The design depends on your resources (strength), the students’ experiences (height) and their cultural background (slope & approach) and your objectives.
Most of the findings in this section are a team effort and arose from discussion and high commitment from lab instructor, previous course instructors and TAs.

Thoughts on Wall Strength

I was in the lucky opportunity to get three excellent TAs and an experienced lab instructor. We have three lab sessions of three hours each every other week and weekly tutorials. With the lab you can get the students a deep dive on operating systems and acquiring practical development skills.

Thoughts on Wall Height

Traditionally, and in our case, we let them develop a simple embedded operating system from scratch. Given that they only have less than four months to complete the assignment is (expectedly) challenging. They are software engineers and come with some background in embedded assembly programming and C/C++ programming. Since they are software engineers, I also have no reservations exposing them to the full breath and depth of the operating systems subject.

Learning operating systems is not a mentally challenging task. The algorithms are relatively simple (scheduling, I/O interaction, concurrency) and easy to grasp. Most of the technical material is closely tied to the advances in the hardware community and ends up a lot to learn but is not really mentally challenging.

What is, however, quite challenging is putting the learned material to use in the lab. Therefore, I the course was traditionally worked to make the students make focus a large part of their effort on the project. Given the short lab timeframe, only the basics are covered in the lab.

Passing the lab will not let the students pass the final exam and just learning for the final exam will hardly make them pass the course. Lately the department enforced a minimum 50% weight on the final exam. If it was according to me, I would put much more weight on the project because these skills will certainly pay most of the dividends in their later career. In the past, instructors tried to combine these two areas, to have them evaluate code samples in quizzes/exams as well, which generally dragged the exam average down.

Thoughts on Wall Slope and Approach

As I mentioned before designing this part is the most challenging and depends on your objectives, the environment, and also the cultural background of your students.

Let’s start with the cultural background (or better university culture). Going to a traditional German university for my undergraduate and graduates study (well I fought at the European Council to have my Diploma recognized as MSc…long story), you come from a very different end of university culture than you see in North America. Nowadays the system is changing in Germany, but when I was doing my degree, the five year study were divided into two parts. A section of four basic semesters (i.e., there were only two terms per year) and a section of four advanced semesters. At my time the tuition was bogus, I paid like 70 EUR per term, and the study was largely government financed. As a result the student had no economic stake in pursuing the study and “the system” had great interest in letting unfit students fail early. Therefore the students would usually be sent through hell and high waters during the first four terms to extinct anyone who is not fit or does not commit to their studies. Operating systems would potentially fall into this primary study. After these four terms all course marks would be merged to establish a mid-examination mark. During my years it was not unusual to barely pass this with about 4 out of 5 where 1 is best (that would account for about~55% - 60%, on a grading scheme that is usually tougher than Canada). After that your program grades are reset and you pursue advanced studies that are usually comprised of several electives. During the advanced studies, exams are usually carried out orally, which have their own flavour but are, from my perspective, easier. To have an idea about the practical implications: my course of studies started with about 50 students. After the mid-examination we were about 12, who barely made it. As of today, I know about 6 who eventually graduated.

Many of them now work as research engineers in German automotive companies, pursued PhDs in Germany (actually all of those guys finished already… time to hurry up for me), or entered the middle or higher management of software companies like SAP.

The courses at my school usually consist of a lecture and occasional labs or tutorials. The courses spanned sometimes multiple terms before you had an exam and were back loaded. That means there was only one single final exam, which you would only be allowed to write after you finished all terms of the course. In my case, I had to wait one year to be able to write my math exam. Exams usually spanned multiple hours. This math exam for instance took five hours. Labs, tutorials and assignments were usually not marked; there were a few exceptions for practical courses like Embedded Systems, which usually occurred at the advanced level. All that counted was the final exam. Procrastination or trying to learn everything 2 weeks before the exam was in almost all cases fatal. Most lecturers were also following the traditional Latin meaning of “study” = “studere” in German: “nach etwas streben, sich um etwas bemühen”: To pursue something yourself; to make an effort yourself. Therefore, it was not usual to ask for problems in the exam that were not covered in detail in the lecture or tutorial. Lecture and tutorial were used to “illustrate” the material not necessarily to cover it in enough depth for the exam. The student was expected to read the course text, optional materials and develop the necessary skills to survive the exam. Only then they’d be valued as an Engineer, or otherwise extinct and thrown out of the program early. This property actually provided you with several freedoms that are hard to recognize first. Only a few instructors cared if you attend the lecture, tutorials, or labs, as long as you survive the exam. This could actually buy you plenty of free time, if you were well organized and skilled. Unfortunately, thanks to EU, austerity, and the introduction of student fees this original tough system has lost much of its traditional flavour over the past 5 years.

Then you come to North America, specifically Waterloo, and see a very different picture for undergraduate studies.

First, students here are actually heavily financially invested into their studies and the university runs as business. Therefore, letting them fail early seems to be big taboo here. The university does not seem to want to disgruntle students. Also university evaluations are carried out in a very different manner. If you had a few very bright people graduating from your program in Germany, you have many people graduating in Canada, with a few being very bright and starting off businesses. The few very bright individuals in both instances would dominate the news and be the selling point of your university. On the other hand, you still have a large mass of students who do not end up becoming CEOs or top-managers in Canada. Out of those, it is quite surprising how many people in Canada actually end up doing things that are totally unrelated to their initial study. It is in so far a surprise to me, because they not only spend a lot of time on it, but they also blew a lot of cash on it. Actually, most of this money comes from student loans, enslaving them for years to debt (another “business model” from the Canadian government).

Second, this is something that I would call North-American culture that has also been analyzed in the (controversial) research community.
These are some ideas that apply to the emergent mass and not necessarily individuals, however, may become emergent for an entire course.
Canadians seem to have the highest level of individualism (IDV), a low level of uncertainty avoidance (UAE), a low long term outlook (LTO) and a low power distance (PDI) compared to other regions.

As a result of LTO and UAE, studies tend to be more applied here and focus on the short-term “market” instead of fundamental research. Also hard-core corporate research labs are found elsewhere than in Canada. Thus, the study is much more applied here and incorporates more labs than in Germany. Therefore, the students also prefer a higher level of practical guidance, instead of leaving them to themselves and punishing them hard if they get it wrong. They also seem to work more ad-hoc here.

That may make them very prone to procrastination (i.e. ignoring the course/lab). They will eventually focus their attention where the pressure/fun comes from (other courses, evening activities …). Other areas may not have this problem. An instructor of mine in Germany, was teaching university-level math courses 20 years ago in Japan… he had trouble setting the standards high enough not to bore the students. Unfortunately, their system also lost much of its flavour over the past 20 years.

Talking to people about this issue there are a couple options to mitigate procrastination and promote continuous participation...

One option is a front-loaded course where most of the efforts are spent on a mid-term examination. You communicate to them that they should focus their attention on the mid-term and that will get burned there regardless (say scores in the high 60s) and an easier final (say scores expected in the high 80s). On the lab side, you also force them to focus their efforts in getting the project design right very early. This was done in the past by having them deliver a comprehensive software design document about half-way through the project. This approach seems to be the favoured option for many instructors here; well some are leaner on the mid-terms and still burn the students on the finals. However, for the actual lesson this does not provide optimal value. At the time the students write the software design document they have little idea what they are going to implement. It was usually communicated to them during the marking, how good or bad their design was and how the real thing should look like. Another issue is, because many instructors follow this model in other courses as well, the students will experience load surges during the term. Here is a good one, try to get hold of a UW undergrad during mid-term periodJ.

Another option is an even workload. Instead of making them write a mid-term, you let them do multiple short quizzes. That way you enforce continuous participation and give them a bit more time during mid-terms. Because the main lab-take-home is acquiring the development skills, we skip the SDD and focus on code deliverables right from the start that are spread out in multiple milestones. Finally, they will include the written design and the lessons learned in a final report. This poses more workload on the TAs and lab instructors in terms of marking, but almost entirely avoids procrastination on the student side and over the long run provides them with more insight into the course material. The risk of course failures is therefore reduced because they are implicitly prepared for the final exam and final course deliverable if they follow through. If they do not, they will eventually feel the pain of low marks right from the start and need to react and presumably catch up with the workload.

We (my lab instructor and me) favour and will use the even workload option, even when this means that I need to share some of the TA workload myself. I want to provide these guys with a high-quality course and ensure that they actually acquire skills in the lab and remember some of the course materials.

Summary

Teaching a course is not going to be a walk in the park. It takes plenty of effort and a good team of TAs and lab instructors. If you teach a course, consider the crowd, their background, their skills, and the established university culture. Be sure to work for the students and not against them. If you are from a different background, noticing such deviations is sometimes hard.

Also thanks to the previous instructors and the current lab instructor for in-depth discussions and input on the subject.

We will see how well that plays out in my case. As Clausewitz once said: “No plan survives the first battle!” … we will see if they tear my wall down J

References

• Willmanns & Hehl, “Praxis und Paradoxa des Innovationsmanagements” (In German).
  This is a good book for innovation and research management in Germany. After reading it, you notice the imprint that you’re educational and cultural background makes on you and how to deal with it.
• Hofstede, Cultural Dimensions.
  He quantitatively tries to analyze different cultures. The measures should not be stereotyped on individuals but give a rough idea what to expect, when you travel to a different background. His research is however quite controversial.
• じょおうのきょうしつ - The Queens Classroom.
  This is a TV series that I came across on a trip to Japan. The teacher has a great interest to prepare the kids for the tough and terribly competitive world in Japan, where bullying and forced subordination are a constant. The methods applied are very traditional and hardly represent what’s actually happening today. It is also exposing how traditional values are now ridiculed over there. Such draconian methods would obviously fail in adult education, specifically in Canada. It’s however a very curious TV series.
• Pink Floyd – The Wall.
  A song of the late 70s. It deals with the abuse by teachers and the situation of schools in Britain at the time. The wall in this case is however used as a metaphor for isolation and not effort (as in the citation before).
• Various management adult education books.