Enumerative Versus Analytic Studies

Determining a Rational Basis For Action

The distinction between enumerative and analytic studies is extremely important in the design and analysis of either complete counts or samples. In both types of study the ultimate aim is to provide a rational basis for action. A problem exists, and something is to be done about it. In the enumerative problem something is to be done to some portion of the contents of the bowl, regardless of the reasons why that portion is so large or small. In the analytic problem, on the other hand, something is to be done to regulate and predict the results of the cause system that has produced the universe (city, market, lot of industrial product, crop of wheat) in the past and will continue to produce it in the future.

- Dr. W.E. Deming. Some Theory of Sampling. (p. 247)

Use of data requires knowledge about the different sources of uncertainty. Measurement is a process. Is the system of measurement stable or unstable?

Use of data requires also understanding of the distinction between enumerative studies and analytic problems. An enumerative study produces information about a frame. Theory of sampling and design of experiments are enumerative studies. Our Census is an enumerative study. Another example is a shipload of iron ore. Buyer and seller need to know how much iron is on board.

The interpretation of results of a test or experiment is something else. It is prediction that a specific change in a process or procedure will be a wise choice, or that no change will be better. Either way the choice is prediction. This is known as an analytic problem, or a problem of inference, prediction. Tests of significance, t-test, chi-square, are useless as inference — i.e., useless for aid in prediction.

- Dr. W.E. Deming. The New Economics, 3rd ed. (p. 68)

IN his work as a statistician, Dr. Deming had long-observed a problem in how statistical theory and techniques were being stretched beyond their intended purpose to resolve problems for which statistics could not account, specifically the mis-application of enumerative statistical methods to analytic questions of prediction. As a consequence, Deming perceived that management were being mislead into making decisions for action on an irrational basis, resulting in losses in quality. It was therefore imperative in his opinion that management learn to distinguish the two types of application through proper instruction by a “master” and not “hacks” ! Accordingly we’ll take a brief look at each beginning with a definition by two of Deming’s contemporaries, Dr. Donald J. Wheeler and David S. Chambers, in their excellent text Understanding Statistical Process Control:

An Enumerative Study proceeds by investigation of the material in a Frame. A Frame is a list of physical units of some kind, identifiable by serial number, any or all of which may be selected and investigated.

An enumerative study is one in which action will be taken on the material in the Frame to be studied… The aim of a statistical study in an Enumerative problem is descriptive — how many or how much? The aim is not to find out why there are so many, or why there is so much, but merely to quantify the material in the Frame.

An Analytical Study is one in which action will be taken on the process or cause-system that produced the Frame studied, with the intent being to improve practice in the future. Here interest centers in future product, not in the material studied.

(p. 329)

Ergo: We want to use enumerative studies to answer questions in the present about the estimated composition of a population of items through random sampling, eg. a census, content of iron in a shipment of iron ore, app feature users who are university educated between 40-45 years, etc., and analytic studies to predict the behaviour of a process into the future using past performance data to understand when and whether to intervene, eg. a Process Behaviour Chart analysis of the daily number of defects produced, tests performed per hour, daily throughput of work items, complaint resolution times, or quantity of daily reported cases older than one week over the past 30d:

Recalling Lessons from the Red Bead Experiment

To further clarify our understanding, we can revisit the Red Bead Experiment using the lenses of enumerative and analytical studies. Specifically, we might ask the question of whether the results of each willing worker’s drawing of fifty beads was in fact random and what rational basis for action the supervisor had in reacting to each worker’s performance with various coarse management contrivances and inducements.

Recall the procedure of the experiment calls for each worker to draw a lot of 50 white beads from a receptacle containing 4,000 beads in an 80/20 distribution of white to red beads using a paddle with fifty indentations. Over successive drawings we find that each worker’s lot contains wide ranges of red beads from 4 or 5 all the way to the mid-to-high 20s. This is an enumerative study that answers present questions about counts and quantities, but has no predictive power to tell us why these quantities exist.

Using the lens of an analytic study, we could answer predictive questions about the future performance of any worker using the apparatus by plotting their daily counts of red beads in a Process Behaviour Chart and analyzing it for stability and predictability, ie. all data points falling within the control limits.

Doing so, we may learn that there’s no rational basis for taking action against the worker for their results as they have little control over the distribution of red beads in each drawn lot, but we could observe changes in counts over time by working on the components of the system, including the mixture of beads in the receptacle and the ability for each worker to introduce improvements to the process.

Dr. Deming would drive home the differences between enumerative and analytic studies by asking participants what they would predict the average number of red beads to settle down to over time - a trap!

The answer that comes forth spontaneously from the audience is that it must be 10 because 10 is 20 per cent of 50, the size of the lot. Wrong.

We have no basis for such a statement. As a matter of fact, the cumulated average for paddle No. 2 over many experiments in the past has settled down to 9.4 red beads per lot of 50. Paddle No. 1, used for 30 years, shows an average of 11.3.

The paddle is an important piece of information about the process. Would the reader have thought so prior to these figures? …

If we were to form lots by use of random numbers, then the cumulative average, the statistical limit of x-bar, would be 10. The reason is random numbers pay no attention to color, nor to size, nor to any other physical characteristic of beads, paddle, or employee. Statistical theory (of probability) as taught in the books for theory of sampling and theory of distribution applies in the use of random numbers, but not in experiences of life. Once statistical control is established, then a distribution exists, and is stable.

- Dr. W.E. Deming. Out of the Crisis. (pp. 351-352)

Thus, we can now appreciate how the mis-application of statistical theory can mislead us into believing things that we have no rational basis for assuming and cause us to treat problems of prediction as though they were certainties of counts and quantities. A grievous error…

Reflection Questions

Consider the distinction between enumerative and analytic studies you see around you each day and how they can become confounded. What inferences do you see made about data using quantitative analyses to predict the future?

As an example to get you started, consider how COVID19 metrics are collected, reported, and used as basis for deciding on different interventions. What type of study is required to know the number of COVID19 cases in a population? What would be required to conduct such a study? What type of study is required to evaluate the effectiveness of interventions? What would be required to conduct this type of study?

Extra Credit Reading

Deming, Dr. W.E. On the Distinction Between Enumerative and Analytic Surveys (Journal of The American Statistical Association, Vol. 48, 1953, pp. 244-248)

Deming, Dr. W.E. On Probability as a Basis For Action (The American Statistician, Vol 29, No. 4, 1975) (pp. 146-148)

Deming, Dr. W.E. Out of the Crisis. (pp. 131-132)

Deming, Dr. W.E. Some Theory of Sampling (Chapter 7)

Deming, Dr. W.E. The New Economics, 3rd ed. (pp. 68-69)

Latzko, William J. and Saunders, David M. Four Days with Dr. Deming. (pp. 191-192)

Neave, Dr. Henry R. The Deming Dimension (p. 272)

Wheeler, Dr. Donald J. and Chambers, David. Understanding Statistical Process Control (pp. 329-330)