I originally bought Supermemo to study for the Oracle DBA exams. Since then I was delayed in taking the exams, but continued using Supermemo in order to stay ready. Recently I took two of the exams, my first exams since starting to use Supermemo. It seems only fair that I report the results. So here's what happened...
[My Study]
[The 90%
Promise]
[Practice]
[Test Day]
[Lessons
Learned]
My Study
I've administered Oracle databases for several years now as part of my job. But it's hard to reflect that adequately on a resume, since my title was usually "Senior Engineer" or "Senior Analyst" or even "S.I. V"--bland little titles for a jack-of-all-trades. So I decided to add an Oracle Certification to my resume to validate my past experience.
Being a busy professional, I bought some "Cliffs Notes" style study manuals. There are several series like that, and they're all decent, so I won't name the specific one. The books I used were decent, but not best-of-breed: they were full of typos and had a handful of errors as well. But they were adequate for exam preparation.
The first test was SQL and PL/SQL, and I took it before I'd even heard of Supermemo. I've been using ANSI SQL for years on all sorts of databases, so this one was a snap. I skimmed the book for a week or so, scheduled the test, took and passed it.
The second test I scheduled was "Oracle Backup and Recovery". Since I'm being brutally honest, I have to admit I failed it--by one question. I committed the sin I'd warned my students about before hundreds of tests: I'd let my guard down because the first test was such a snap. So I'd prepared for a week or so, and never addressed one glaring gap in my knowledge of Oracle-specific tools. As a result, I blew a half-dozen questions or so, any one of which would have been enough for a passing grade.
Humbled and chastened, I began an internet search for a tool to aid my reviews. As I said on my main Supermemo page, Supermemo was not the software I'd envisioned; I wanted an adaptive, self-scoring quiz which would simulate the actual Oracle exams. But Supermemo was the best I could find, and it ran on my Palm Pilot, so I bought it and started in.
Progress was slow, for a simple reason: I'm a busy professional. Work claimed a lot of my time. And in my spare time, I'd started preparing for a Solaris sysadmin certification, and a Java 2 certification, to validate more of my work experience which is hard to document. All in all, it probably took me a month or so of spare-time study to review the Backup and Recovery book, and commit its contents to Supermemo. After Backup and Recovery, there are three more Oracle DBA exams, and each book took maybe a month to go through, in the same way.
But that month was stretched out over most of a year, as I studied in my spare time. Which is why Supermemo really saved my bacon! Normally, coming back to my study after a gap of six weeks or more would find my mind a complete blank! I would have had to start over from the beginning. But during those gaps I'd continued doing my daily quizzes with Supermemo, so the information was still fresh in my mind when I dug in for more.
Now, after about a year of preparation--on again, off again--I read that Oracle is going to retire the Oracle 8 DBA exams. So I have to finish them before the end of March, 2002. That lit a fire under me, and I took stock. Of the four pending exams, I've been ready for two of them for months now, so I scheduled them for Wednesday, Feb. 6. Remembering my embarrassing experience last year, I decided to actually calculate my odds of success.
The 90% Promise and Margins o' Safety
By default, Supermemo promises a retention of about 90% if used faithfully. On the Windows version this is adjustable, either up or down, but on the Palm Pilot it is not. Since I'm a Palm user, I'm stuck with the 90% retention rate. But it's important to remember that 90% means 90% of the material committed into Supermemo. If you study only half of the material, and commit it all in Supermemo, then you will remember about 90% of that 50%, which is only 45% of the total.
Why am I pointing that out? Because I studied for the exams using the "Cliffs Notes" type of study materials: professional summaries of the most important information. They omitted less important topics in the interest of time. I then committed most, but not all, of that material to Supermemo. So how should I interpret the 90% promise? I would not expect to know 90% of the total information, but rather 90% of what I covered, which is less than the entire book--and the book of course represents less than the total test information.
Anyway, being a Mathematician I couldn't rest until I'd estimated in advance my margin of safety. If the study notes covered 95% of the material, and I committed 95% of that, then I committed about 90.25% of the total to Supermemo. If I then remembered 90% of that, I could expect to know a little over 81% of the total going into the test.
That means I was approaching a 60-question test, in which I stood an 81% chance of answering any given question correctly, and a 19% chance of answering incorrectly. Mathematicians call that a "Binomial experiment": there are exactly two outcomes, with known probabilities for each one. To make a long story short, it was easy to calculate that I should expect about 49 out of 60 questions right, and could be 95% certain that I would get between 43 and 55 right.
Passing for one exam--Oracle Backup and Recovery--was 42 right, or a score of 70%. I could be 95% certain of passing that one, even on my worst day. Passing on the other exam--Oracle Performance Tuning--was 48 out of 61, or a score of about 78.7% (where do they come up with these numbers?).
The interval 43-55 was not very comforting in the latter case, so I looked at the normal approximation to the binomial distribution, and calculated the probability of getting a passing score for both exams. For Backup and Recovery: about 98.7%. For Performance Tuning: about 69.5%. Not the best odds in the world on that latter test, so even with Supermemo caution was needed.
I should point out, though, that my analysis was overly pessimistic. Tests are not truly random: the test-makers always rig them to make sure that the most important topics are not accidentally left out. So if the test-makers, and the authors of my study notes, both agree on what's "important", then my chances were considerably better than the above calculations suggest. And of course they had better agree, since otherwise the study-quide--makers would soon be out of business. Anyway, I stuck with my pessimistic assessment, since that's the kind of guy I am.
Practice Makes Perfect
The review materials included practice exams--even a computer-based test which simulated the actual exam. First I took the practice tests written by the authors of the study notes I'd used. Time after time, I scored about 90%. So I guess Dr. Wozniak is right: his algorithm does indeed guarantee 90% retention of the committed material.
The trouble is, those practice tests covered the exact same material as the notes I was studying from. The question is, how adequate were the notes? So I looked at practice tests from other publishers. There was one problem: the tests I'm taking are due to be retired in a couple of months. The only books I could find all pertain to newer versions of the test. Still, I gave them a shot.
Using those practice tests, I consistently scored 80-90% correct on Backup and Recovery. That's right in the ballpark I'd expected: I was expecting scores between 72% and 92%, by my first pessimistic analysis. So apparently my calculations were about right--and also, apparently, Backup and Recovery hasn't changed much in the newest version of Oracle.
Performance Tuning, on the other hand, was a tougher test. My scores ranged from about 80% down to about 60%. But hopefully that was because new versions or Oracle work differently than Oracle 8! So after each practice test, I double-checked each question I got wrong. If it involved a new feature, then I disqualified the question and recalculated my score. When I did that, my scores were again pretty consistently between 80% and 90% correct.
Of course, I also discovered gaps in my knowledge in this way. When that happened, I added flashcards to supermemo so I would know the material next time. (At least, I'd stand a 90% chance of knowing the material!)
This is a good time to point out that I was really relying on Supermemo to pull me through. I had lent my Performance Tuning book to a friend, so there was no way I could use it to "cram" for the test. Whatever I could remember, thanks to Supermemo, was all I had to go on.
Test Day!
I scheduled the two tests back-to-back. Backup and Recovery was first. I expected to pass it--it was almost 99% certain I would--so I scheduled it first. If I passed, the boost to my morale would help on the next test. And if I'd taken Performance Tuning first and failed, the resulting depression could easily make me blow the other test.
I am, generally speaking, a great test taker. I burn through them fast and steady, never looking back, and score very high. But I usually take tests in the context of a University course--where I'm guaranteed that my class notes give 100% coverage of the test material, and where of course I crammed before each test. This time I'd done no cramming, and was not guaranteed 100% coverage of the material. So I had to fight my natural tendencies, and force myself to take the test seriously; if I breezed through it, I could easily fail.
The testing software helped a little. You can check off questions which you weren't sure about, and review them later. So whenever I wasn't 100% certain of the answer, I checked the question for review. That took discipline--I'm not only a good test taker, but also a very good guesser. And a horrible second guesser! So I usually do not schedule questions for review, unless I truly have no clue of the answer.
After completing all 60 (or 61) questions, I looked at the summary page to assess my condition. I counted the unchecked questions, and multiplied that number by .9 to see if it was a passing score. It wasn't. I estimated the number of questions remaining to make up a passing score, and divided by .9, to estimate how many more I would need to be pretty sure of passing. For Backup and Recovery that was 12 questions, and for Performance Tuning it was about 16. (I don't remember exactly, and they collected my scratch paper after the test.)
With that target number of questions in mind, I went back through the marked questions. (I never looked at the unmarked questions again.) On many of them I was pretty sure I had the right answer, but I carefully examined the other alternatives to satisfy myself that I knew why none of them was the answer. If I could do that, I unchecked the question and made a tally mark on my scratch paper. If I could not, but I could narrow it down to two options, then I left it checked off but put a tally in a separate "50/50" column on my scratch paper. I went this way through all of the marked questions.
When I was finished, I counted the questions which I had unchecked. Then I counted half of the questions which I'd tallied in the "50/50" column. On the backup and recovery exam, I was positive about enough questions to be sure of a passing grade, ignoring any "50/50" questions. Holding my breath, I clicked "DONE".
On the performance tuning exam, I was not certain of enough questions to pass: I was only certain of 42 questions. Eighteen other questions were marked in my "50/50" column. So I estimated that I had a score of 51, and was 95% sure my score was between 47 and 55. I calculated that using the binomial distribution again, assuming the 42 were correct and giving the last 18 each a 50/50 chance. Interestingly, that was very nearly the same confidence interval I had predicted before even entering the exam room. But it was close--way too close. Since I'd given each question my best shot, I took several deep breaths, held it in, and clicked "DONE".
The computer scored my tests, and printed out the results. In Backup and Recovery I scored 54 out of 60: exactly 90% correct. In Performance Tuning I scored 54 out of 61: about 88.5% correct. Whoo hoo! Piotr had delivered on his 90% promise!
Lessons Learned
My original analysis was obviously reasonable. I was 95% certain of a score below 55, and scored 54 on both tests. It's also obvious that my analysis was somewhat pessimistic: scoring right at the top of the range suggests that either my notes gave better than 95% coverage, or else they agreed very well with Oracle's ideas about which questions are the most important. I'm quite sure the latter is true, or as I said the publishers of the study notes would have gone out of business.
Apparently the "90% Promise" is indeed accurate. Using Supermemo, I did score consistently near 90% on practice tests by the publishers of my notes, and that translated into a score of 90% on the actual test because the notes gave appropriate coverage of the test material. Since I'm still using Supermemo, even after the test is over, I'm confident that I could take the test over again, at any time in the future, and score about 90%.
On the other hand, as I've said elsewhere, Supermemo is not magic. If I had relaxed my discipline during the Performance Tuning test, I would almost certainly have failed it. You might not use the elaborate statistical methods that I did, but that was really just a trick to make myself focus. Like me, you will definitely need to make sure you pay attention to the test in order to claim the "90% Promise" of Supermemo. You'll have to wrack your brains, and skip hard questions, and come back to them, just like you always did. But with Supermemo and an honest effort, you really can expect the answers to be there when you need them.
