During the weekend I helped out at my wife’s booth at the local Oktoberfest (which was oddly low on beer and sausage, but full of distillers and wings.) I was happy to do it, but I was feeling that we weren’t getting quite the traffic at this festival as we need. As proof of this I offer the fact I could have snuck away from the booth on all three days with no ill effect.
On the first night, I discovered a local used bookstore hiding in the oldest bank building in that town (it hasn’t been the oldest bank in town for a half-century), and because the festival flagged a good bit on Sunday during the Steelers game, I was able to leave my wife to tend the store and walk over there to scan their shelves. I did make one purchase, event though I’m pretty sure I have a copy in my garage. I found a copy of the New Dictionary of Cultural Literacy. I bought it because it could go to Seton if I already have it, and because it reminded me of one of my most successful schemes.
This was at a time prior to the introduction of A-sets and middle school sets, so we were trying to fill the easiest sets available with questions. Overall, our questions were missing the mark in difficulty and there were significant numbers of rewrites required for the questions. We were hitting the marks for circuit veteran teams, but if you looked at the numbers for events for teams that played once a year or once a semester, the scores seemed uncomfortably low.
Now this wasn’t a problem with all the questions going to Invitational Series sets, it was a significant minority of the questions. They were taking up time, and they if we could focus the writers to stick to easier questions (or more accurately more commonly learned question clues) we could fix this pretty easily by bending the average difficulty down.
Writers always have the habit of writing to enrich their own knowledge, and to gravitate toward the things that they find fascinating and novel. Both of these work great if you are trying to write something for your own level of competition, and get very troubling when you are writing for competitions where the players don’t have your current level of experience. At the time, most of our writers were college players or recent graduates, so they were still writing questions both to make money, and to learn things. A lot of them ended up writing questions which reflected things that they wanted to know as a player, and not things that would be answerable by a player five to ten years less experienced. The writer is fascinated with something they hadn’t seen before, meaning those who see that clue presented to them are even less likely to have seen it before.
In this situation the writer’s incentive is to write things closer to their level of experience. We needed a way to incentivize the writers to write for the lower levels of competition. And it needed to be something that reduced the cycle of rewrites, which cost editor time, and delayed writer payment. One of our methods of incentivizing the writers at this point was to run contests to fill needs in particular subjects. We had done a couple of these over that summer to fill needs in science and fine arts, and we were about to push for literature. But this was right at the time of the great literature difficulty expansion in quiz bowl, and I was worried we’d end up with a lot of lit which was not truly acceptable for high school, but would be seen as reasonable expansion for college questions. I proposed an alternative, based on my copy of The Dictionary of Cultural Literacy that I had purchased a couple months earlier.
I had mentioned the Dictionary of Cultural Literacy in The Five Foot Bookshelf, and it’s an extension of a book length essay Cultural Literacy by E. D. Hirsch. One of the thrusts of the essay was that there exists parallel to basic literacy (knowing how to read the printed word and what it means) a sense of cultural literacy, knowing the symbolic meaning of words as they are chosen for communication. In order for the general population to know what someone is saying there needs to be some core knowledge of the symbolism to extract the full meaning from communication.
The problem that I was trying to solve with the contest was a related one. I needed our writers to write things that would not necessarily be answered by everyone, but that would seem reasonable by everyone playing on them. The best way to do that would be to get writers to write about subjects that weren’t necessarily quiz bowl staples, weren’t necessarily curriculum staples, but were subjects encountered by students with an interest in learning, and a curiosity about the world. The things that get referenced in another piece of writing, conversation, film or music. The dictionary is full of that sort of material, and I used it to set up my competition.
I know that some of the conclusions drawn in the Cultural Literacy essay are now controversial, and I think I even realized that at the time. Because of that, I didn’t use the dictionary as the sole source. (I salted a couple dozen entries that came from the Dictionary of Global Culture or were shared between them.) But the list was chosen as a representative sample of what was taught in a variety of curricula, and used by writers as references in their work.
I pulled the titles of about 500 entries, and compiled them in a list that roughly corresponded to the percentages of the high school distribution of quiz bowl questions, within limits. The Technology section was already out of date, I didn’t need any Sports or Pop Culture questions at that point, and I didn’t want to touch current events via this method.
At the beginning of the contest, I offered up the list to all the writers, and gave them the rules. You can write any question you want for the next 30 days, but if you write a question for a high school packet that mentions any of these subjects in a clue or the answer, you get a point. Most points gets a gift card, second place gets a smaller one. If you think there’s a case to be made for this question to get a point without mentioning the particular clue, write a note on the question.
Some particulars of this contest’s design:
Giving them targets to build from rather than subjects that need to be filled was key to this. At the time, we had a perpetual set of deadlined “needs lists” that had to be filled, but it was incumbent on the writer to come up with the idea to fit the needs. Here we were giving them the seed to inspire them, and not worrying about where exactly it needed to fit.
It needed to be a big enough canon of answers that someone couldn’t gain from cramming it, I had no expectation that the writers would keep the list quiet. If there had been 20 answers on that list, someone could have worked the system a little.
It needed to be a big enough list of targets that someone couldn’t simply decide to step through the list and hope to complete it. A small enough list that you think you can complete it, and you start at the beginning. If it’s a little larger, you may think you’re smarter than everyone else, and you’ll start at the bottom. But if it’s so big you can’t ever hope to complete it in time? You triage, you look through the list, and pick and choose the low hanging fruit. That’s exactly what I wanted because each writer’s low-hanging fruit is different.
I wanted the prize to be small enough that they wouldn’t get into an all-consuming fight, but large enough that everyone would consider the competition. You make the prize big for this, and the favorite is going to be the writer who writes the most questions normally, and people get disappointed when they finish out of the money. If you make it too small, people who don’t write the most questions normally don’t consider the prize at all. Because my goal was to get the most out of the most writers, I went for something significant at the time, but not life altering. $75 total on two gift cards.
I also wanted to make it a big enough target list that the writers who were inclined to bend the rules did so in the most interesting ways. If I had to look through all the questions they provided to score them, I wanted some measure of entertainment. And they delivered. John Paul Jones was on the list, and two separate writers theorized they could write about Led Zeppelin. I also included Smith, Johnson, and Adams as targets figuring they’d interpret the rest of the answer line however they wanted to, and automatic repeat checking would figure it out for me.
It was a success. We got over 2000 questions that month that hit one of the targets, doubling production of the previous month. That got us ahead of the production crunches that usually marked the next couple of months. It also prompted an update to our website. About a year after, if a writer logged into the website, and chose a category that had needs, they could ask for suggested topics which could fill the category.
This was an experiment that worked for us, because it incentivized writers to act in a certain way, by leveraging the value of constraint. By giving them a wide-ranging list of inspirations, we got them past the difficult stage of question ideation. By choosing the list, we constrained their process to things that would be more likely to be useful to the compilation process.
So as I’m sitting here with the copy of the book by my side, and you’re wondering why I’m writing about this… consider that there’s lots of people out there who need lots of questions over the next few years. And it’s in everyone’s interest to find proper incentives to produce questions which match the expectations of teams across the spectrum of experience and practice. This is a way to guide your writers, whether for a packet, a tournament set, or larger.
We had the first practice at Seton. I was initially worried that I might not be welcome there but that problem subsided immediately. They had advertised and recruited well, and we were facing a situation where I had to put a second hub of buzzers out. There were seven kids signed up. I had prepared by taking and writing an simulated game of the program, enough to have three teams competing, but we only used one third of it in the practice. I then gave them a novice level packet which I had specifically chosen because it had at least one answer match with the TV show. You always want to reinforce the idea that everything repeats, and doing it in the first practice justifies the idea of conducting practice.
Finally, the kittens have settled in. They have been confined to Catie’s room until their vet appointment next week. With Catie away for long stretches this past week, Dana and I have taken turns napping in her room, making sure the little hellions have supervision and acclimating them to living with humans, and knowing how hard to bite a finger without hurting anyone. They’re adorable, and they know it, which is always dangerous. They’ve been introduced to the wonders of The Wet, the activity of the cat tree and the redfly that glows along the wall. They’ve already marked me up, with a claw scratch across my wrist, and it will soon be time to introduce them to the other ladies of the house. And then we will see if this will all work out.