Win-Win Case-in-Point Study
How Aggregated Figures Can Conceal True System Performance
People are asking for better schools with no clear idea how to improve education, or even how to define improvement in education.
Deming, Dr. W.E. The New Economics, 3rd ed. (p. 6)
THE AIM for this brief post is to call your attention to a recent article I came across about declining academic achievement scores in reading and math among US students, as indicated in the National Assessment of Educational Progress’ benchmark scoring in their flagship publication, The Nation’s Report Card, as it directly relates to the case John Dues makes in the second chapter of his book, Win-Win, about how our prevailing ways of thinking about figures can mislead us into drawing the wrong conclusions. (See my review of Win-Win, here.)
From the article:
Dues’ opens Chapter 2, Transformation from Mythology to the New Philosophy, by recounting a similar academic achievement report, A Nation at Risk, published forty years ago by the National Commission on Excellence in Education, and a subsequent report by a team of engineers that slightly up-ended its conclusions less than a decade later, attributing fault to an obscure statistical phenomena known as Simpson’s Paradox.
Dues explains:
… and continues with this surprising discovery:
In sum: Under the prevailing theory of management used almost everywhere, figures like test scores can be misconstrued as a true proxy for a system’s performance when, in reality, they’re more representative of normal, common-causes of variation, in this case the make-up of the student cohorts taking the test over time.
Rx? As Dues concludes:
The philosophical foundation Dues recommends is, of course, Deming’s System of Profound Knowledge, which can help dispel the management myths he goes on to enumerate in the chapter. Two domains of this theory stand out: Appreciation for a System, in this case all the interactions that occur behind and around the students who write the tests, and Theory of Variation, which helps to understand the differences in the cohorts who write the tests over time, and under what conditions.
Reflection Questions
Consider the article and opening of Chapter 2 of Dues’ book and the built-in problems that come from the way we’ve been conditioned to think about figures, especially those that have been aggregated and possibly confounded by Simpson’s Paradox. What other examples have you observed in your work where data presented one way leads to conclusions that are contradicted when analyzed from another? What were the results? Did they help or hamper improvement efforts? What else was attempted? How could knowledge of the four domains of Deming’s System of Profound Knowledge help overcome making the mistakes?
Last question: Have you picked up a copy of Dues’ book, yet? Get it on Amazon or via the publisher.