Whoops, Tim already wrote a blog post with this same name. I'm using it anyway.
I was the problem czar for the Harvard-MIT November Tournament, which happened on Sunday, November 7, 2010. As problem czar, I was responsible for making sure that enough problems got written on time for the contest to run. That in itself was a huge task, made no easier by the relatively small amount of help that the February problem czars gave me. I personally wrote the majority of the test, with help from Travis in some key areas (he wrote many of the geometry problems), and from Jacob in not as key areas (he kept giving me problems about Bayesian inference...).
66 problems later, I get the (somewhat delayed) response from testsolvers: "Too Hard."
What is too hard?
Honestly I am probably too biased to comment, but I don't believe that it's possible for something to be too hard on its own. Being too hard can cause other issues, such as a lack of distinguishing power, but being too easy can also cause that. And I am fully aware of the issues that come from a test that lacks distinguishing power.
So when I see a complaint that something is too hard, what do I think? I feel that it is extremely more likely that some aspect of the undertaker is off rather than that the task is actually too hard. And it seems like others are the opposite -- they would much rather blame the test than the test takers, especially when they are the test taker.
Actually, I do that too. Lots of my blog posts last year were about things that I disagreed with in tests. Was it really the tests that were at fault or was it me? I think it was some of both. I don't think it's deniable that the HMMT calculus test was unable to distinguish between the top 4 competitors (hint: they tied), but at the same time it wasn't like the test gave them absolutely zero opportunity to distinguish themselves (the tie was at 28, not 50).
Regardless, I feel like there is this idea that if lots of people get two or fewer problems then the test is too hard. So, people say, make there be a few problems that everyone will get. But why? Now I've reduced my contest from 10 problems to 7. That doesn't seem useful at all.
I've rarely seen someone look at a test and say ``This is too easy,'' but many people will look at a test and say ``This is too hard.''
Wednesday, November 24, 2010
Subscribe to:
Post Comments (Atom)
"and from Jacob in not as key areas (he kept giving me problems about Bayesian inference...)"
ReplyDeleteDid you include any of them? =P
Let's assume we're actually trying to administer a contest for the purpose of ranking the competitors in some unambiguous way. I agree with you that having fluff problems at the beginning is useless, but that doesn't mean that the comments from test solvers are wrong or biased, just that they are irrelevant. What you actually want to do is get people to test solve under actual time conditions, take their scores, and see if the exam distinguishes between people properly, based on your own assessment of the test solvers. Their comments might be valuable as well, but only if they are problem specific.
Incidentally, how many problems did you have to scrap in the name of difficulty?
This comment has been removed by the author.
ReplyDeletethis might be irrelevant but i just wanted to ask you since ur never on gmail anymore :P. Do you ever feel smart? or do you only look at what you dont know? I still dont think im smart by any means for being a sophomore in multivar but there are those times when i talk with kids from other schools and its just mind blowing for them.
ReplyDelete