Does This Analysis Make My Tropics Look Big?

Guest Post by Willis Eschenbach

There is a new paper in Nature magazine that claims that the tropics are expanding. This would be worrisome because it could push the dry zones further north and south, moving the Saharan aridity into Southern Europe. The paper is called “Recent Northern Hemisphere tropical expansion primarily driven by black carbon and tropospheric ozone”, by Robert Allen et al. (paywalled here , supplementary information here  , hereinafter A2012). Their abstract says:

Observational analyses have shown the width of the tropical belt increasing in recent decades as the world has warmed. This expansion is important because it is associated with shifts in large-scale atmospheric circulation and major climate zones. Although recent studies have attributed tropical expansion in the Southern Hemisphere to ozone depletion the drivers of Northern Hemisphere expansion are not well known and the expansion has not so far been reproduced by climate models. Here we use a climate model with detailed aerosol physics to show that increases in heterogeneous warming agents—including black carbon aerosols and tropospheric ozone—are noticeably better than greenhouse gases at driving expansion, and can account for the observed summertime maximum in tropical expansion.

Setting aside the question of their use of a “climate model with detailed aerosol physics“, they use several metrics to measure the width of the tropics—the location of the jet stream (JET), the mean meridional circulation (MMC), the minimum precipitation (PMIN), the cloud cover minimum (CMIN), and the precipitation-evaporation (P-E) balance. Figure 1 shows their observations and model results for how much the tropics have expanded, in degrees of latitude per decade.

FIGURE 1. ORIGINAL CAPTION FROM A2012: Figure 2 | Observed and modelled 1979–1999 Northern Hemisphere tropical expansion based on five metrics. a, Annual mean poleward displacement of each metric, as well as the combined ALL metric. … CMIP3 models are grouped into nine that included time-varying black carbon and ozone (red); three that included time-varying ozone only (green); and six that included neither time-varying black carbon nor ozone (blue). Boxes show the mean response within each group (centre line) and its 2σ uncertainty. Observations are in black. In the case of one observational data set, trend uncertainty (whiskers) is estimated as the 95% confidence level according to a standard t-test.

I note in passing that the error bars of the observations are very wide. In fact, they barely establish the change as being different from zero, and in a couple cases are not statistically significant.

Now, several people have asked me recently how I can analyze a paper so quickly. There are some indications that set off alarms, or that tell me where to look. In this case, the wide error bars set off the alarms. I also didn’t like that instead of giving the claimed expansion per decade, they reported the total expansion over the 28 years of the study … that’s a second red flag, as it visually exaggerates their results. Finally, the following paragraph in A2012 told me where to look:

We quantify tropical width using a variety of metrics^5,11: (1) the latitude of the tropospheric zonal wind maxima (JET); (2) the latitude where the Mean Meridional Circulation (MMC) at 500 hPa becomes zero on the poleward side of the subtropical maximum; (3) the latitude where precipitation minus evaporation (P-E) becomes zero on the poleward side of the subtropical minimum; (4) the latitude of the subtropical precipitation minimum (PMIN); and (5) the latitude of the subtropical cloud cover minimum over oceans (CMIN). To obtain an overall measure of tropical expansion, we also average the trends of all five metrics into a combined metric called ‘ALL’. Expansion figures quoted in the text will be based on ALL unless otherwise specified.

What told me where to look? Well, the sloppy citation. Note that they have not given citations for each of the 5 claims. Instead, they have put no less than seven citations at the head of the list of the five groups of observations and model results. That, to me, is a huge red flag. It means that there is no way to find out the source of each of the five individual observational results in A2012. So I went to look at the citations. They are as follows:

5. Zhou, Y. P., Xu, K.-M., Sud, Y. C. & Betts, A. K. Recent trends of the tropical hydrological cycle inferred from Global Precipitation Climatology Project and International Satellite Cloud Climatology Project data. J. Geophys. Res. 116, D09101 (2011).

6. Bender, F., Ramanathan, V. & Tselioudis, G. Changes in extratropical storm track cloudiness 1983–2008: observational support for a poleward shift. Clim. Dyn. http://dx.doi.org/10.1007/s00382-011-1065-6 (2011).

7. Son, S.-W., Tandon, L. M., Polvani, L. M. & Waugh, D. W. Ozone hole and Southern Hemisphere climate change. Geophys. Res. Lett. 36, L15705 (2009).

8. Polvani, L. M., Waugh, D. W., Correa, G. J. P. & Son, S.-W. Stratospheric ozone depletion: the main driver of twentieth-century atmospheric circulation changes in the Southern Hemisphere. J. Clim. 24, 795–812 (2011).

9. Son,S.-W. et al. Impact of stratospheric ozone on Southern Hemisphere circulation change: a multimodel assessment. J. Geophys. Res. 115, D00M07 (2010).

10. Kang, S. M., Polvani, L. M., Fyfe, J. C.& Sigmond, M. Impact of polar ozone depletion on subtropical precipitation. Science 332, 951–954 (2011).

11. Johanson, C. M. & Fu, Q. Hadley cell widening: model simulations versus observations. J. Clim. 22, 2713–2725 (2009).

For no particular reason other than that it was available and first in the list, I decided to look at the Zhou paper, “Recent trends of the tropical hydrological cycle inferred from Global Precipitation Climatology Project and International Satellite Cloud Climatology Project data”. Also, that was a citation that refers to the minimum precipitation (PMIN) for both hemispheres, as used in A2012. Figure 2 shows results from the Zhou paper:

Figure 2. ORIGINAL CAPTION FROM ZHOU: Figure 4. Time‐latitude cross sections of zonal mean seasonal precipitation and the corresponding linear trend with latitude. Solid orange lines mark the 2.4 mm d−1 precipitation threshold which is used as the boundaries of subtropical dry band. The boundary at the high and low latitude of the dry band is used as a proxy of the boundary of Hadley cell and ITCZ, respectively. Solid black lines indicate latitude with minimum precipitation. Dashed red lines mark the Hadley cell boundary determined by the 250 Wm−2 threshold using HIRS OLR data.

Now, the black line in these four frames show the minimum precipitation, so that must be where they got the PMIN data. So I went to look at what the Zhou paper says about the trend in the minimum precipitation PMIN. That’s shown in their Figure 5:

Figure 3. ORIGINAL CAPTION FROM ZHOU: Figure 5. Linear trends of the latitude of minimum precipitation, ITCZ, and Hadley cell boundaries inferred from GPCP for each season and the year marked on the horizontal axis for (a) the Northern Hemisphere and (b) the Southern Hemisphere. … Leftmost, middle, and rightmost bars in each group are for minimum precipitation, Hadley cell, and ITCZ boundary, respectively. For quantities significant at the 90% level, bars are shaded green, blue, and orange, respectively.

Now, let me stop here and discuss these results. I’m interested in the “Year” category for minimum precipitation (green), since that’s what they used in the A2012 paper. Note first that the minimum precipitation results that they are using are not even significant at the 90% level, which is very weak. But it’s worse than that. This paper shows one and only one result that is significant at the 90% level out of a total of six “YEAR” results.

This brings up a very important and routinely overlooked problem with this kind of analysis. While we know that one of these six “YEAR” results appears to be (weakly) significant at the 90% level, they’ve looked at six different categories to find this one result. What is often ignored is that the real question is not whether that one result is significant at the 90% level. The real question is, what are the odds of finding one 90% significant result purely by chance when you are looking at six different datasets?

The answer to this is calculated by taking the significance level to the sixth power, namely 0.9⁶, which is 0.53 … and that means that the odds of finding a single result significant at the 90% level in six datasets are about fifty/fifty.

And that, in turn, means that their results are as meaningless as flipping a coin to determine whether the tropics are expanding on an annual basis. None of their results are significant.

It also means that the data from the Zhou paper which are being used in the A2012 paper are useless.

Finally, I couldn’t reproduce either the average value, or the error bars on that average, in the A2012 “ALL” data. Here are the “ALL” values from my Figure 1 (the A2012 Figure 2):

Item, Value, Error

JET, 0.45, 1.09

P-E, 0.75, 0.29

MMC, 0.24, 0.08

PMIN, 0.17, 0.51

CMIN, 0.33, 0.06

ALL, 0.33, 0.12

When I average the five values, I get 0.39, compared to their 0.33 … and the problem is even greater with the error bars. The error of an average is the square root of the sum of the squares of the errors, divided by the number of data points N. This calculates out to an error of 0.25 … but they get 0.12.

Does this mean that the tropics are not expanding? Well, no. It tells us nothing at all about whether the tropics are expanding. But what it does mean is that their results are not at all solid. They are based at least in part on meaningless data, and they haven’t even done the arithmetic correctly. And for me, that’s enough to discard the paper entirely.

w.

PS: I suppose it is possible that they simply ignored the results from the Zhou paper and used the results from another of their citations for the minimum precipitation PMIN … but that just exemplifies the problems with their sloppy citations. In addition, it brings up the specter of data shopping, where you look at several papers and just use the one that finds significant results. And that in turn brings up the problem I discussed above, where you find one significant result in looking at several datasets.