The ecological data from 1983-1984 were aggregated on a county level, and supplemented with county data from a nationwide mortality survey in 1973-1975 as well as select demographic information from the Population Atlas of China.

The size and scope of the China Study are impressive. As a result, some dietary advocates have aggressively promoted the China Study as "proof" that vegan diets are optimal or best. However, a closer look at the study reveals important limitations that impact the reliability, usefulness, and interpretation of the study results. Many dietary advocates are quick to cite the China Study without discussing the limitations inherent in such a study.

• Level of aggregation of the study data yields, at most, 65 observations (data points) for analysis. The data in the China Study are aggregated at the county level. The result is that for most health/disease/dietary factors to be analysed, there are 65 (or less) observations (data points). This is important for two reasons:
  • The study is often described as authoritative and reliable--characteristics that are usually associated with "large" data sets. When one learns there are only 65 observations (and hundreds of variables), it suddenly seems far less authoritative. Note that the term "large" is relative; for simple analysis of a very few variables, 65 data points may be adequate, but for sophisticated models involving several variables, hundreds (or even thousands) of data points may be appropriate.

  • The limit of 65 observations places limits on the number of variables that can be analyzed simultaneously (via multivariate--that is, multiple-variable--techniques).

    Side note: (statistical, can skip if you wish) Even a simple technique like some of the regression methods based on splines may be seriously limited on a data set of only 65 points. Spline methods are becoming increasingly important, because unlike traditional regression techniques they do not assume the functional form of a relationship between variables. (That is, they do not assume ahead of time what particular mathematical relationship may exist between one variable and another.)

  Such limits are quite frustrating on a data set that includes hundreds of variables. Campbell [1989] appears to acknowledge this (p. 2):

  Although uniquely comprehensive... it is not yet clear how satisfactory the analysis of multiple factor effects will be upon disease risk, given the limited number of survey counties (65). More complete evaluations of the virtually unlimited interactions between different disease causes may have to await the addition of still more dietary and lifestyle studies of disease mortality rates.

• Limits on the use of geographical correlations, the primary data of the China Study monograph (Junshi et al. [1990]). The China Study monograph [Junshi et al. 1990] provides geographic correlations which are of limited direct use. Although it is possible to develop statistical models in which the dependent variable is a correlation, models constructed using the underlying variables from the relevant correlation may be far more meaningful and useful. Peto, writing in Junshi et al. [1990, pp. 76-77] notes:

  Although geographic variation in particular disease rates can provide clear evidence that, in areas where those rates are high, that disease is largely avoidable, they may provide frustratingly unclear evidence as to exactly how it can be avoided...

  An even more striking example of the limitations of geographic correlations is that oesophageal cancer in China has no clear geographic association with smoking, and has a significantly (P < 0.01) negative association with daily alcohol intake.

  Peto and Doll [1981], as cited by Peto in Junshi et al. [1990], also remind us that attempts to separate causality from confounding factors in geographical data via the technique of multiple regression are often unsuccessful (or, my comment--misleading, which is even worse).

• The China Study report lists only 6 statistically significant correlations between meat-eating and disease mortality. Further, 4 of the correlations are negative, which indicates that the mortality rate for that disease decreased as meat consumption increased. The two diseases that had positive correlations with meat consumption are schistosomiasis, a parasite, and pneumoconiosis and dust disease.

  Thus, the direct evidence of the study is hardly the condemnation of meat consumption that veg*n dietary advocates may claim it to be. It should be noted here that correlation is a measure only of linear relationships, and other analytical methods may yield different results. Despite the possibility of the existence of more complicated statistical relationships, it seems quite odd, given the interpretations of the study made by veg*n dietary advocates, that meat intake generally did not correlate with disease mortality. (See table 5033, pp. 634-635 of Junshi et al. [1990].)

• Ecological studies (like the China Study) generate hypotheses, they do not prove them. Campbell and Junshi [1994] concisely state this limitation (p. 1155S):

  First, this study is ecological and includes 6,500 individuals residing in 130 villages. Thus according to widely held assumptions, any inferences concerning cause-and-effect relationships should be considered to be hypothetical only, with validation to be provided only by intervention or prospective analytic studies on individuals.

  Thus we note that the China Study requires backup clinical studies before making inferences or drawing conclusions. The main hypothesis of the China Study is whether diets that are predominantly plant foods reduce chronic diseases. However, some veg*n advocates go far beyond the main hypothesis of the study, and claim it proves that veg*n diets are "better" than all omnivore diets. Further, such claims may be made without supporting clinical studies, and without regard for the actual range of diets included in the study. (The latter point is discussed later herein.)

• The ecological fallacy, and its impact on ecological inference. Freedman [1999, p. 1] provides a brief overview of the ecological fallacy (boldface emphasis below is mine):

  In 19th century Europe, suicide rates were higher in countries that were more heavily Protestant, the inference being that suicide was promoted by the social conditions of Protestantism (Durkheim 1897). A contemporary example is provided by Carroll (1975): death rates from breast cancer are higher in countries where fat is a larger component of the diet, the idea being that fat intake causes breast cancer. These are "ecological inferences," that is, inferences about individual behavior drawn from data about groups...

  The ecological fallacy consists in thinking that relationships observed for groups necessarily hold for individuals: if countries with more Protestants have higher suicide rates, then Protestants must be more likely to commit suicide; if countries with more fat in the diet have higher rates of breast cancer, then women who eat fatty foods must be more likely to get breast cancer. These inferences may be correct, but are only weakly supported by the aggregate data.

  ...However, it is all too easy to draw incorrect conclusions from aggregate data.... For example, recent studies of individual-level data cast serious doubt on the link between breast cancer and fat intake (Holmes et al. 1999).

  Eliminating group-level confounding has no necessary relation to individual-level confounding. Greenland and Robins [1994a] provide an interesting and insightful statistical critique of ecologic studies, furnishing examples that demonstrate how ecologic and individual-level studies are subject to different forms of confounding and errors. In conclusion, they state that [Greenland and Robins 1994, p. 749], "...conditions that guarantee no confounding in an ecologic study are logically independent of the conditions that guarantee no confounding in an individual-level study..." That, of course, sharply limits the relevance and reliability of efforts to use ecological data to make inferences about individuals.

  They also discuss how the techniques commonly used for "adjusting" ecological data for covariates (other variables) might not reflect the true underlying conditions due to non-additivity, non-linear relationships, and other factors (i.e., the common practice of a linear adjustment may be inappropriate and/or inadequate for many covariates). Further, they point out that having a large number of regions in an analysis does not guarantee a randomization of the underlying relationship (if any) between covariates and regions.

  Averaging at the aggregate level prevents assessment of individual-level relationships. Commenting on Greenland and Robins [1994a], Piantadosi [1994, p. 763] makes the cogent remarks:

  It is impossible to study ecologic analyses very carefully and come away with much confidence in them. Averaging is a process that necessitates some loss of information... When analyzing such aggregated data, we not only lose all ability to extend inferences reliably to less aggregated data but we even lose the ability to estimate the direction and magnitude of bias...

  I encourage epidemiologists to understand the deficiencies of group-level analyses and limit them to the role of hypothesis generation.

  Readers with some knowledge of statistics will find the above papers, plus the related papers of Greenland and Robins [1994b] and Cohen [1994] to be of interest, and to provide a good introduction to the statistical limitations (and problems) that are relevant when one tries to make ecological inferences. This underscores the earlier point (above) that the China Study merely generates hypotheses, it does not prove them.

• Results of a study done in one culture or society do not necessarily apply to other cultures or societies. (They potentially can, but they also might not.) Campbell and Junshi [1994] make this point (p. 1155S):

  A second general comment on comparing residents of rural China with residents of highly industrialized societies is that it may not be appropriate to extrapolate diet-disease relationships across cultures.

  Campbell and Junshi [1994] then go on to briefly discuss the linkage between diet and health, and how the incidence of chronic disease changes when groups change diets. In that regard, the discussion in the previous section of chronic health problems among the Aborigines of Australia, after adopting high-carbohydrate diets, is relevant.

  Wenxun et al. [1990] provide actual examples of differences in Chinese and Western disease patterns (p. 1033):

  An examination of the correlations among CVD (cardiovascular disease) mortality rates, erythrocyte [red blood cell] fatty acids, blood lipid indices, and selected environmental factors indicates that some relationships reported for Western populations were absent in China whereas other relationships were observed. For example, there was no correlation between county means [averages] of plasma cholesterol and county mortality rates for any of three CVD types. Thus, factors other than total dietary lipid or plasma cholesterol may be important in explaining the geographic distribution of CVD mortality rates within China.

  One probably will not find a consensus on the reasons why extrapolation of study results from one culture to another is unreliable. However, a number of probable factors include: differences in income and lifestyle, differences in level of industrialization causing differences in exercise levels and exposure to chemicals, minor but significant differences in genetics, differences in food preparation practices, and so on.

• Lack of actual income data for the survey participants is a serious flaw. It makes adjustment of the data for the effect of income less reliable. Ideally, one would like to adjust data for income in statistical analyses, as it is presumably a good proxy variable for the degree of Westernization. (Many of the degenerative diseases common in Western societies are in effect, diseases of affluence, hence adjusting the data for actual income may be important in statistical analysis.) However, the China Study has no direct measures of the income of the participants. Instead, the study includes demographic data that could be utilized to yield estimates of county average, per-capita income. However, such an estimate is a poor substitute for actual income data on the study participants. Also, per-capita income may be skewed by the presence of a few, very high-income individuals living in an otherwise (very) poor county.

• Attempts to use the China Study to prove that all omnivore diets are bad is yet another logical fallacy. Ultimately, attempts to claim that the China Study "proves" all omnivore/faunivore diets are bad fail as yet another logical fallacy. Basically, none of the county diets in the China Study were vegan diets, and none were evolutionary diets (and, by the way, none were the SAD/SWD diet). Most were high carbohydrate, grain-centered diets (though one county reported high consumption of both meat and dairy--reminder: dairy was never a part of humanity's evolutionary diet). Campbell, writing in Junshi et al. [1990], reports (p. 63):

  The national mean [average] percentage energy intake obtained from animal foods was observed to be 5.7%, with a range of 0.1-59.4%.

  Thus we observe that extrapolation to strict vegan or evolutionary diets (or even the SAD diet) go beyond the range of the China Study data, and hence such projections are less reliable statistically. Also, as none of the China Study diets were evolutionary diets, and the meat consumed came from domesticated rather than wild animals, the results from such (Chinese) diets cannot be extrapolated to evolutionary diets (i.e., yet another logical fallacy).

• Recent research in Australia found that a high-starch Chinese diet did not reduce risk of colon cancer. The paper of Muir et al. [1998] describes a randomized, crossover dietary intervention study in which 12 people (of European descent) in Australia followed for 3 weeks each a high-starch diet similar to the diet of low-income comunities in China, and later the SAD (standard Australian diet, in this context). The Chinese-style diet lowered serum cholsterol and fecal pH. However, for all other fecal markers tested, the results from the Chinese diet were worse than for the SAD. Muir et al. conclude [1998, p. 372]:

  These results suggest that consumption of high-starch diet alone is insufficient to reduce the risk of developing colon cancer.

  As the above reflects results of only one study, the usual cautions on interpretation apply.

• Collaborative analysis of 5 large studies finds no difference in death rates from cancer, vegetarian vs. non-vegetarian. A recent collaborative analysis of 8,300 deaths among 76,000 people in 5 prospective studies [Key et al. 1998] compared death rates of vegetarians and non-vegetarians for a number of diseases. The studies involved included large proportions of vegetarians. Key et al. [1998] found that vegetarians were less likely to die of ischemic heart disease than non-vegetarians, but there were no differences in the death rates for a number of types of cancer: stomach, large bowel (colon), lung, breast, and prostate.

  Note: In the five studies analyzed by Key et al. [1998], four of the studies defined vegetarians as people who did not eat any meat or fish, with non-vegetarians being all other people (i.e., this would include those who did eat dairy and eggs in their diets). The other study in the analysis classed as vegetarians those who claimed to be vegetarians; a followup on that study 5 years later found that only 66% of the self-identified vegetarians ate meat or fish less than one time per month. (This suggests some mis-classification error in that study or, as the followup was 5 years later, it may reflect people who abandoned the veg*n diet.)

GO TO NEXT PART OF ARTICLE

(Instinct vs. Intelligence in Diet: Where is the Line?)

Return to beginning of article

SEE REFERENCE LIST

SEE TABLE OF CONTENTS FOR:
PART 1 PART 2 PART 3 PART 4 PART 5 PART 6 PART 7 PART 8 PART 9

---

GO TO PART 1 - Brief Overview: What is the Relevance of Comparative Anatomical and Physiological "Proofs"?

GO TO PART 2 - Looking at Ape Diets: Myths, Realities, and Rationalizations

GO TO PART 3 - The Fossil-Record Evidence about Human Diet

GO TO PART 4 - Intelligence, Evolution of the Human Brain, and Diet

GO TO PART 5 - Limitations on Comparative Dietary Proofs

GO TO PART 6 - What Comparative Anatomy Does and Doesn't Tell Us about Human Diet

GO TO PART 7 - Insights about Human Nutrition & Digestion from Comparative Physiology

GO TO PART 8 - Further Issues in the Debate over Omnivorous vs. Vegetarian Diets

GO TO PART 9 - Conclusions: The End, or The Beginning of a New Approach to Your Diet?

Back to Research-Based Appraisals of Alternative Diet Lore