Divergent Thinking Tests: Why They Measure Creative Potential, Not Creativity

Few tools in psychology have stirred as much enthusiasm — and as much suspicion — as divergent thinking tests. Early on, many people apparently believed they could replace IQ tests. Then critics showed that their predictive power was moderate at best, and the pendulum swung to the other extreme: these tests, and creativity tests in general, were questioned and largely dismissed.

In 2012, Mark A. Runco and Selcuk Acar of the University of Georgia published a review in the Creativity Research Journal to bring some order to that debate. The paper appeared in a special issue on indicators of creative talent, and its question is a precise one: how reliable and how valid are divergent thinking tests, and what do they actually indicate?

---

Many Answers to One Question

Divergent thinking tests ask open-ended questions that allow numerous responses. For the authors, this way of thinking is a good metaphor for the kind of cognition that should lead to original ideas, and it is easiest to understand by contrast with convergent thinking, which typically leads to conventional, “correct” ideas and solutions. The distinction — explained in detail in our article on divergent vs. convergent thinking — goes back to J. P. Guilford, who tied what he called divergent production to creative potential.

Potential is, in fact, the paper’s central idea. Divergent thinking often leads to originality, the central feature of creativity, but it is not the same thing as creative thinking: someone can do well on a divergent thinking test and never actually perform creatively. That is why Runco and Acar treat these tests as indicators — that is, predictors. A good indicator is reliable and accurate, but every prediction carries some uncertainty, and the potential it estimates may or may not be fulfilled.

---

How It Is Measured: Four Ways to Count Ideas

In 1981, Hocevar identified four types of creativity tests: divergent thinking tests, attitude and interest inventories, personality inventories and biographical inventories. Divergent thinking tests dominated creativity assessment for decades, and that created a problem: they were sometimes treated as tests of creativity, a view the authors consider untenable.

Most of these tests now look at four indexes:

• Fluency: how many ideas a person produces.
• Originality: how novel, or statistically infrequent, an idea is within a group.
• Flexibility: how many different conceptual categories the ideas cover.
• Elaboration: the least common index; it shows when someone follows one associative pathway for some distance.

Too often only fluency is used, and the authors advise against it: the other indexes carry information of their own, and fluency is less closely tied to creativity than originality and flexibility are. The classic batteries include Guilford’s, Wallach and Kogan’s, and the Torrance Tests of Creative Thinking, which remain enormously popular (on their validity, see our study of the Torrance tests).

---

Are They Reliable?

Reliability tells us whether a test gives stable, consistent results, and it is the starting point for any assessment: an unreliable test cannot be valid. When scoring requires judges — deciding whether an idea is original, for instance — their agreement has to be checked. The figures Runco and Acar gather are reassuring: Meeker reported interrater reliability well above 0.90, Urban and Jellen reported 0.90, and Wallach and Kogan 0.92, on a scale where 1 would be perfect agreement.

Another kind of reliability is internal consistency: whether the different tasks in a test point in the same direction. In one review by Cropley, these values ranged from 0.42 to 0.97, though most were above 0.70. There was one warning sign. Hocevar’s analyses showed that originality scores became unreliable once fluency was controlled, and the conclusion was that the sheer number of ideas “contaminated” originality. Runco and Albert (1985) qualified that: for some tests and some samples, especially at higher levels of ability, originality stays reliable even after fluency is taken out.

---

Do They Measure What They Claim to Measure?

Validity asks whether a test really measures what it is meant to measure, and it is the opposite of bias. A test that leans heavily on language, for example, will favor highly verbal examinees regardless of their creative ability. The authors also point to biases linked to socioeconomic status and experience.

A sensitive point is the link with intelligence. According to threshold theory, some level of general intelligence is necessary but not sufficient for creative work; at moderate and high levels, the evidence confirms that the two separate. That supports the discriminant validity of these tests — in other words, they measure something other than general intelligence. Context matters too: Wallach and Kogan found that the separation was low with test-like directions and quite clear when the task was presented as a game.

Predictive validity — whether a test anticipates future creative achievement — depends on choosing the right criterion. Kogan and Pankove found poor predictions for fifth graders and only slightly better ones for tenth graders, while other studies reported much better results with the same tests. Runco, Plucker and Lim argued that many earlier criteria depended on opportunities and extracognitive factors rather than on ideation. With a scale that asks only about generating ideas, correlations with divergent thinking tests were reassuring, and the lack of any significant link with grade point average added further evidence of discriminant validity.

---

How to Score the Ideas

More significant than the new tests, the authors note, are the alternative ways of scoring them. Torrance’s four indexes are highly intercorrelated and laborious to use, so Wallach and Kogan proposed using only two: fluency and uniqueness. Others added quality. Cropley, for example, suggested giving 4 points to ideas produced by less than 1% of the population, 3 to those produced by 1–2%, 2 for 3–6%, 1 for 7–15% and 0 for the rest.

Every system has a cost. Dividing originality by fluency removes the contamination but hides productivity: someone who gave two ideas, one of them unique, would score the same as someone who gave ten, five of them unique. Simply adding the indexes together is, for the authors, the least defensible option. Among the alternatives that worked best are weighting fluency by how rare each idea is, and asking judges to rate each person’s whole pool of ideas instead of one idea at a time — which also saves a great deal of time.

---

Generating Is Not Enough: The Role of Evaluation

On its own, Runco and Acar explain, divergent thinking would lead to wild ideas: some might be creative, but many would be unrealistic, useless or irrelevant. That is why judgment and evaluation are part of the creative process. When Khandwalla asked people to think aloud while working on open-ended problems, the most frequent path combined generating ideas and evaluating them: far from working against each other, the two reinforced one another.

Poor evaluation has costs of its own. It can make someone abandon a promising line of thought too soon, a problem known as premature closure, and the studies reviewed show that people make more errors when judging highly original ideas, whose merit they tend to underestimate. The good news is that this skill can be trained: when Runco and Basadur trained a group of managers specifically in evaluating ideas, their correct choices increased and their incorrect choices decreased.

Children evaluate, too. In one study, fourth to sixth graders were more accurate when judging how popular ideas were than when judging how creative they were, and their accuracy was unrelated to intelligence. Another study, with third to fifth graders, found no sign of the fourth-grade slump that Torrance and others had described: in that sample, divergent thinking actually peaked in fourth grade.

---

A Good Indicator, Not a Verdict

Runco and Acar close with the image of the pendulum. After the early enthusiasm and the backlash, more careful psychometric studies showed that the validity of these tests is as good as what is found throughout psychological assessment, with some coefficients above 0.6. The foundation of that conclusion is simple: divergent thinking tests are not tests of creativity but estimates of the potential for creative problem solving. As the authors put it, “DT is not synonymous with creativity” (p. 72).

With that premise accepted, research is already looking further. Vartanian and colleagues found that people with higher divergent thinking ability are faster at judging whether two concepts are related, a speed that is unrelated to IQ. And Runco and colleagues found that fluency is linked to several genes involved in dopamine reception, while originality is not.

For researchers, the paper is a guide to choosing criteria and scoring systems with care. For educators, it is a reminder that a high score signals potential, not guaranteed creativity, and that assessing students’ creativity means looking beyond the number of ideas. For professionals, it shows that generating options is only part of the work: those options also have to be evaluated well. That is why the creativity course devotes Week 6 to recognizing divergent and convergent thinking, and Week 8 to developing divergent thinking.

---