Economic Observations

Oskar Morgenstern was one of the founders of mathematical game theory, as co-author with John von Neumann in 1944 of Theory of Games and Economic Behavior. He was also the author in 1950 (revised in 1963) of a book that should be of substantial interest to philosophers of the social sciences. The book was called On Accuracy of Economic Observations, and it called into question the unreasonable level of confidence that economists had in economic statistics in the post-war period. Here is how Morgenstern described his goals in the first edition:

This study is the preliminary result of a prolonged concern with the properties of economic data. Most recently the problem of accuracy arose again in connection with an investigation into the structure of the American economy by means of input-output tables. These are arrangements of the sales and purchases of industries which show the interconnectedness of the entire economy. They are attempts to provide information Quesnay intended to give in the ancient "Tableau Economique"; they were first used by W. W. Leontief. (v)

He puts the central message of the book in these words in the preface to the second edition:

This book addresses itself to both the general reader and the professional economist. The former will see that decisions made in business and in public service are based on data that are known with much less certainty than generally assumed by the public or the government. The second will discover that even most widely accepted figures frequently have error components of unexpected magnitude, and consequently cast doubt on many currently accepted analyses in economics. (vii)

Morgenstern's analysis in this book is a very sober assessment of the challenges that confront the task of assembling national or regional economic statistics -- employment, production, prices, trade, and so on. In Part Two of the book he has separate chapters on trade, prices, mining, agriculture, employment, national income, and growth.
His perspective is a scientific one; he is fully supportive of the need to treat economic reality on the basis of empirical data. But he insists on the crucial importance of carefully reviewing the instruments through which our economic observations are made, and assessing the cumulative range of error that is created by those instruments. In other words, he constructs an applied epistemology to accompany economic observation: a detailed accounting of the scope and limits of accuracy in these observations.
Our problems begin, Morgenstern believes, with the fact that economic statistics are generally reported without any indication of error estimates. "Yet it ought to be clear a priori that most economic statistics should not be stated in the manner in which they are commonly reported, pretending an accuracy that may be completely out of reach and for the most part is not demanded.... It will be seen later that national income and consumers' spending power probably cannot be known now in part without an error of +/- 10 to 15 percent" (8).
Morgenstern specifically discusses ten major sources of error in the estimation of economic statistics:

Lack of designed experiments
Hiding of information, lies
The training of observers
Errors from questionnaires
Mass observations
Lack of definition and classification
Errors of instruments
The factor of time
Observations of unique phenomena
Interdependence and stability of errors

Morgenstern illustrates the effects of these kinds of factors on the degree of accuracy attainable in several areas of economic observation. Particularly interesting for contemporary concerns is his analysis of employment statistics (chapter XIII). Here he points out that there are conceptual issues that compound the effects of difficulties of measurement; for example, how do we define an "involuntary status of employment"? How do we handle "frictional" unemployment? But he notes as well that there are substantial political pressures on the unemployment numbers (216). Morgenstern then digs into the methodologies involved in the four major sources of employment statistics in the 1950s: Bureau of Labor Statistics, Bureau of the Census, Bureau of Employment Security, and the US Department of Agriculture. (It must be born in mind that this analysis describes the methods used in the 1950s, not contemporary methods.) Here is a description of the method used by the Census Bureau:

The data are based on a geographically stratified random sample of about 35,000 households. These households are randomly chosen from sampling cells which have themselves been chosen according to a multi-stage cluster sampling procedure involving the use of probabilities proportional to size, from over 300 geographical areas. The sample is taken in the middle of each month to maximize comparability with other monthly series. (219)

Each of the four bureaus mentioned here conduct their observations using somewhat different methods; and each sometimes arrive at significantly different estimates. For example, for the year 1950 the Census estimate of non-agricultural employment was 52.3%; BLS found 45.2%; and BES found 34.3%. "The three counts of non-agricultural employment differ widely by four to eighteen million people [from 1946 to 1961]" (224). Each is a reasonable and justifiable estimate; but there are broad differences among them.
Here is an interesting hypothetical graph that Morgenstern introduces to demonstrate how sensitive some economic judgments are to variations within the error range of a given variable. The solid graph plots national income against time as observed by an official national data time series. The lightly shaded area represents the error range around this variable. And the dashed line represents a hypothetical graph of the movements of national income that is entirely consistent with the empirical information associated with the national estimate -- but leads to diametrically different conclusions about the timing of the business cycle.

None of this should be taken to imply that we shouldn't pay attention to economic statistics, or that there are not rigorous methods for attempting to measure these kinds of variables. Rather, Morgenstern's point is that all scientific observation involves complex instruments; and it is incumbent upon the researcher to carefully assess the sources and magnitude of possible errors in his or her observations. Further, economists and policy makers need to exercise appropriate humility in making use of economic statistics to support this policy reform or that; they need to recognize the possibility that the estimates and time series of statistical variables upon which they rely are significantly skewed and inaccurate.
I wonder whether contemporary econometricians would seriously disagree with Morgenstern's skepticism about claims of several decimal points of precision for variables like national income, growth, or employment. Is there a current-generation refutation of Morgenstern's concerns?
(G. William Skinner wrote a highly interesting article in 1987 about the problems inherent in population, tax, and production data for historical Sichuan; "Sichuan's Population in the Nineteenth Century: Lessons from Disaggregated Data" (link). His analysis includes some of the same sources of error and bias that Morgenstern identifies.)
(Ian Hacking's The Emergence of Probability: A Philosophical Study of Early Ideas about Probability, Induction and Statistical Inference is a very interesting treatment of the history of the development of the mathematics of social statistics.)