Week 130: Making the unknown unknown known
Holes and how to shrink them.
[A set of variations on a theme, taking the information gained from viewing episodes and charting the questions, and applying them to improve the team.]
On holes
Holes are a colloquial term for the things your team doesn't know. And by that, I mean the team doesn't know that. Holes in quiz bowl are communally owned, but are removed and reduced by the actions of the individual.
Where your team's holes lie is one of the most important things to recognize. Once you know the holes in your team, you can begin to fill in the gaps in your team's knowledge, and prioritize the holes by size (the sheer number of clues and answers), and repeatability (how often they come up in questions).
In order to figure out where the holes are in your team, and to be able to prioritize them, you first need to examine the content of matches. When we asked you to chart rounds, you got this raw data. That data can then be grouped into sets of information and you can evaluate the size of each group. Before the first practice, take this list and place it in order, largest count to smallest.
Entering the next practice you can have a list of types of information, and as you go through practice, you can check off the ones where your team is able to answer the questions which correspond to those groups, and which of those questions they can't answer correctly. Split your marks up so you have data from each round of general practice. You can then look at your list after each practice, and check down the list for the next big area that your team doesn't know. That is your team's next biggest hole to fill.
Comparing that data to your practice, you can see which of the clusters of information both appear frequently and which appear consistently. Something that hits only once in a round, but never absent, and is from a much smaller set of answers is more valuable to us, because that smaller set of answers may be teachable in a few minutes. Consistency in the past is reliability for the future.
This is an iterative procedure that depends on practice to give you new data to work from. It will not necessarily be rapid improvement once you get past the first week, but it will be improvement always oriented in the correct direction.
At a certain point the list flattens out, and you end up with a large number of possible next steps to examine, which are not distinguished from each other. Fortunately, there’s a steep peak to the list which can keep us occupied well past the first week.
When you start in competition you don't know where your holes are, because you don't know anything about the types of questions that can be asked. To use the terminology I always attribute to Donald Rumsfeld: this is an unknown unknown. To contrast: a known unknown is something you can identify as a problem, an unknown unknown is something you can’t see from your currently available information.
Before we can work on improving the team by attacking what they don’t know, you have to learn what among the things they don’t know will actually be important. A coach for quiz bowl has three steps in their process: learning what’s going to be appearing in the rounds their team has to play, learning what from that set of things their team doesn’t currently know, and providing practice and study material so their team can overcome the deficit. Only the last of these is moving from a known unknown to knowledge. The first two are managing an unknown unknown, and extracting some amount of known unknowns from it.
The unknown unknown consists of a set of holes, and you don't know the size of the hole, and you don't know how often things pass through the hole. But with enough data collected from packets, you can begin to evaluate the size of the holes, and begin prioritizing those holes. When time is of the essence, you want to tackle the biggest possible improvements that still fit into your schedule.
The first book was written as a study of offense in quiz bowl. That is taking what is presented to you and extracting the maximum value from each question that is presented to you. This book is a study of defense; because, in television quiz bowl there is so much to defend. Any tossup question is contested by multiple teams, and your team's job is to prevent points from going to the other team by answering the question before them.
A hole in your team's knowledge is undefended territory. The goal of any team should be to have fewer, smaller holes in their knowledge than their opponent. In the circuit, there can be large holes in your knowledge which is unconvertible by your opponent, because they don’t know it either. But in televised quiz bowl the writer’s goal is to have every question in play, so there will be much less safe territory.
We don't know coming into the practice what categories the writers are filling the rounds with, or what sources they are using, or what their vision is for a packet that is good for competition. We don't get that data, unless they tell us specifically through the format (announcing categories with each question), or through the care they use in writing a particular question. Our only evidence to their vision is in the questions and answers.
The other thing that can be done with the grouping of data from your charting is to make some evaluation of what the question writers deem important. Had we the time, we would be able to evaluate what the writer believes is important from the content of their questions. But we don't have that time so we can estimate it best by looking at the types of answers that they use in their questions. This gives us some idea of their tendencies in their writing style. “Do they write questions where the answers are geography?” We can also make an estimate of the techniques they use to create their questions. “Do they write questions that quote statistics, or publicly quote sources? Do they write questions that refer to other things but are fundamentally a question of vocabulary?”
[Away from the holes for now. An aside from my reading through twitter mentions of quiz bowl this week.]
Two tweets from a twitter thread about science questions asked why do the clues in so many forms of competition seem to focus away from the science.
While I find it interesting as an explanation, I am holding to an observation that I’m putting in the book: For most quizzing, science questions are approximately vocabulary. Vocabulary clues are canonically shorter than either uniquely identifying or differentiating scientific clues. If your goal is to simply produce uniquely identifying clues in the most efficient way, going for vocabulary is a reasonable way to go. And since specialized knowledge to uniquely identify and differentiate scientific answers is confined to a subset of writers and players, it is folly to expect scientific clues when vocabulary accomplishes the task the writer set out to do, make questions answerable by a larger fraction of their audience. And once that is the prevailing trend in writing, it affects the perception of the clues. They are perceived as acceptably easy because they are repeated, and then they become easy because the repetition of clues begins to resemble spaced repetition.
Science questions and clues are a special case of the route tree model I mentioned in week 43, and assuming that stays in the book, I will probably use it to explain science questions in short formats.