April 8, 2004

 

To:       Nature Publishing Group

 

Re:      Second Revision of 2004-01-14277

 

Enclosed please find a second revision of 2004-01-14277, a comment on a paper by Mann, Bradley and Hughes (MBH98), published in Nature in 1998, revised in accordance with referee comments enclosed in an email of March 9, 2004 from Rosalind Cotter and then reduced to 800 words in accordance with an email of March 26, 2004 from Rosalind Cotter. We have also included supplementary information for reviewing purposes.

 

We once again express our appreciation for the consideration extended to us by Nature.

 

Sincerely,

 

Stephen McIntyre

Ross McKitrick

 

April 8, 2004

To:       Nature Editors and Reviewers

From:    Stephen McIntyre and Ross McKitrick

Re:             Manuscript 2004-01-14277

 

Reply to Referees and MBH

 

This cover letter describes our second revision. Our first revision addressed the reviews of the referees and MBH, and in doing so increased the length of the submission. We were requested to meet a length limit of 800 words, with a supporting SI document for reviewing purposes. This request is complied with in this submission. All comments made in the reply to referees in our first revision are re-stated here; all changes discussed here are relative to the original submission. Accordingly, consideration of the reply here does not require any use of our first revision.

 

Our submission analyzes in detail the effect of “key” indicators on the temperature reconstruction carried out in Mann et al (1998) (MBH98), with particular attention to principal component analysis of their North American tree ring network. In respect to perhaps the most contentious issue, we categorically confirm that our early 15th century calculations included the NOAMER PC1 and PC2.

 

Our re-submission is accompanied by supplementary information in two Parts: (1) back-up discussions for statements made in the text, since space limitations generally foreclosed much methodological details; (2) computer scripts used to generate both figures and statistics in the text and Supplementary Information Part 1.

 

We will discuss the comments of the referees, the rebuttal by Professor Mann and colleagues (MBH), then the principal changes to the manuscript from the original submission.

 

Referee #1.

 

1.       We certainly share the unease of referee #1 in respect to “standardisation based on a small segment of the series to the whole series.” Indeed the influence of this procedure in MBH98 is one of our principal concerns. We have applied the referee’s phrase in our text to help make it clear that this is also our concern. We note that Mann et al. did not dispute our description of their procedure insofar as this step is concerned (although they disagree with our conclusions as to the influence of the procedure). Our studies indicate that the principal impact of their procedure results from the de-centering of some series by removing  the mean calculated over the small segment.

 

2.       We agree that the results are very sensitive to the presence/absence of early data and that the selection criteria need to be carefully examined. We hope that recognition of this sensitivity will be an outcome of our paper. We note that Mann et al. do not dispute our analysis of the quality issues pertaining to the Stahle/SWM data and now assert that this series does not have a material impact (a conclusion with which we agree). Similarly, they do not dispute our observation that the Twisted Tree series does not affect the early 15th century reconstruction. We have moved some footnote comments regarding the Gaspé series to the main text and amplified these comments.

 

We have added RE goodness-of-fit statistics as requested, but like referee #1, we do not believe that RE statistics dispose of the matter. RE statistics are useful, but improvements of fit are not justified if they are attained by use of ad hoc procedures or retaining data known on other grounds to be unsuitable (through not having a linear relationship to temperature). Other statistics like R2 are also useful for assessing the validity of a model. We have added comments in the text (and particularly the SI) to deal with this issue.

 

We agree with referee #1 that there is an upward trend in both reconstructions towards the end of the series in the former Figure 4c (Figure 1e) and we do not argue that this feature is an artifact of the PC procedure. Our concern is with the relative levels of early 15th and 20th century temperature index values. We argue that the PC procedure in MBH98 depresses early 15th century values of their index and that this problematic methodological step materially affects the relative levels of early 15th and 20th century temperature index values.

 

Referee #2

We agree with referee #2 that examination of software may be of interest to readers. We have included scripts to produce all figures and statistics in our SI Part 2.

 

1.       We confirm that in our original submission all our early 15th century calculations included two NOAMER PCs. This is proven in the software which we have included (the lookup table controlling the number of PCs used for each region for each calculation sub-interval is shown at the end of the script.) We agree with referee #2 that MBH’s reply does not take this into account. Their “simulation” - in which the two NOAMER PCs were deleted - is an inaccurate characterization of our procedure and, to that extent, their criticisms of the behaviour of their “simulation” is irrelevant.

 

2.       This point would seem to be directed at Mann et al. The reason for the similarity in the shape of the early 15th century portions of the temperature indexes in the two cases (the simulation “MM04c” – excluding the NOAMER PCs, and our calculation in Figure 1e (formerly Figure 4c) – using a conventional algorithm) is that, in both cases, the Graybill high-latitude sites (see explanation in text) do not dominate the NH temperature index.

 

3.       We have carried out calculations of RE and R2 statistics for various reconstructions and, as noted in our comments for referee #1, have added sentences on this issue and a discussion in the SI.

 

4.       This comment is directed to MBH.

 

Reply to MBH

We respond here to what we believe to be the most significant points.

 

1.       MBH argued that many of the criticisms of the quality of the MBH98 data base presented in MM03 and of the Stahle/SWM series in MM04, even if correct, do not affect early 15th century calculations. We agree that the Stahle/SWM and Twisted Tree series do not affect early 15th century values, even though they were previously cited by Mann et al. (2003) as “key indicators” for this period. We have exported the discussion of these series to SI, leaving only a sentence of reference. We note that MBH did not offer a defense of the quality of the early portion of the Stahle-SWM series or offer a defense for the obsolete version of the Twisted Tree series that they used. We agree that many quality defects in the underlying MBH98 dataset (discussed in MM03) do not affect early 15th century values and did not present these discussions in this resubmission. We focused in this submission on the truly “key indicators” – the North American PC1 and Gaspé series - rather than on all possible data quality issues. We note that Mann et al. will be publishing a Corrigendum in Nature acknowledging many of the other defects.

 

2.       In their Appendix, MBH objected to our sole focus on the Sheep Mountain series, pointing out that other series were also highly weighted. So we have broadened the point by analyzing all 16 series with weights of at least 25% of that given to Sheep Mountain, the most heavily weighted series. (See SI for a listing).  We noticed that 15 of the 16 sites are high-altitude sites due to a single researcher, Donald Graybill. We identified a presentation of 12 of these sites in Graybill and Idso (1993) (which was also discussed in Mann et al. (1999).) Graybill and Idso specifically stated that the 20th century growth in these sites were not accounted for by local or regional temperature and hypothesized that these trees  (selected for cambial dieback) contained signals of direct 20th century CO2 fertilization. Indeed Mann et al. (1999) explicitly acknowledged this one year later in their GRL paper and introduced a nonlinear transformation to try and adjust for CO2 fertilization effects. This analysis helped us understand the unpublished, so-called “censored” version of the NOAMER PC calculations at the MBH98 FTP site. We found that 19 of the 20 sites so “censored” were Graybill sites and that all of the above 16 sites were so censored. We refer in the revised text to some findings regarding these sites, notably that when these sites are removed from the MBH roster, their PC1 loses its hockey stick shape and is virtually identical to our PC1. This shows that the unconventional MBH98 algorithm yields results that differ from a conventional PC calculation by overweighting series widely regarded as not having a linear relationship to temperature, which MBH98 stated to be a prerequisite of their method. We believe that our results are significantly enhanced through responding to this objection and that this analysis sheds a great deal of light both on the critical NOAMER PC1 and on how much significance can be placed on RE and other verification statistics. We have added a discussion of this analysis in the Comment and the SI.

 

3.       MBH argue that their results are validated by RE statistics. We have carried out calculations of RE and R2 statistics on both stepwise models and the model using only the AD1400 roster and have reported these in the re-submission and SI. Since we show that the proxies contributing to “improved” RE values do not meet the linearity assumptions of the MBH98 model, the RE statistics and any improvement in it are spurious. We have also reported on the R2 statistic (which is low) on the basis that a good model should meet all statistical tests, not merely an RE test. The extremely low R2 values are further evidence of the spuriousness of the apparent RE fits.

 

4.       MBH attempt to rebut our argument about the influence of their PC methodology by recalculating their index in a linear regression with all proxies available in AD1400 uniformly weighted – thus using the 70 North American tree ring series previously incorporated into the NOAMER PCs as individual proxies in the regression. This experiment does not test the robustness of their method. The 20 Graybill-Idso proxies go from being dominant members of their NOAMER PC1, which is only one of 22 indicators in the AD1400 regression roster (but which strongly depresses the early 15th century index because of its unphysical 2-6 σ negative values), to constituting 20 out of 95 series in a regression equation, and thereby still dominate the behaviour of the model. To test the degree to which their results depend on use of an unconventional PC algorithm, they should simply replace it with a conventional PC algorithm, which is precisely what we did. The conclusion of such a test is that their results do depend on use of use of data with pronounced 20th century nonlinearity, overweighted by their unconventional PC algorithm. In any case, even if they can obtain results similar to their original results using a wholly different method, that does not establish the validity of the methods used in MBH98. Introduction of a new methodology would require full presentation and justification.

 

5.       Even now, MBH may argue that they had good reasons to pre-scale their proxies to the 1902-1980 mean and standard deviation, and then to detrended standard deviations, and so forth, in order to emphasize the contribution of the Graybill-Idso series, although it is hard to contemplate what those reasons would be. However, they did not provide such reasons in MBH98, nor did they disclose their use of an unconventional PC algorithm, nor did they alert their readers that their unconventional PC algorithm was so crucial to the final results or that these results depended so heavily on overweighting a very unique set of tree ring sites with cambial dieback which many authors (MBH included) have elsewhere flagged as problematic as temperature proxies. At a minimum, our results show the sensitivities involved in these problematic assumptions - information which we believe to be essential to a balanced assessment of the results asserted in MBH98.

 

6.       MBH state that our analysis of the effect of the extrapolation of the early portion of the Gaspé series is a mere “technicality”. This is hardly correct. Our interest in this series was drawn by the fact that the series as used did not coincide with the archived version at WDCP and that it was the only one of the over 400 series used by MBH98 in which there was an extrapolation of the early part of the series. We also noticed that this unique extrapolation was not disclosed in MBH98. It was hard to avoid noticing that this extrapolation permitted insertion of this series into the AD1400 roster, and, once there, it strongly depressed the early 15th century values (which otherwise seemed to want to go up.) When we carried out before and after calculations, we found a dramatic impact from this one series (see SI), which seems inconsistent to us with robustness requirements. Further examination showed that the early decades of the series consisted of only one tree. Tree-ring site “chronologies” are normally based on averages of multiple overlapping tree ring series and single-tree chronologies are not generally regarded as meeting minimum signal standards. Also the 20th century portion of this series is highly non-linear and has almost no correlation to local temperature, and the site is located in a forested area nowhere near the northern treeline (of which it is supposed to be a member). In response to MBH’s criticism on this point, we have moved a brief discussion of this proxy from a footnote to the main text and amplified it somewhat, as well as presenting additional material in the SI, in order to demonstrate substantive reasons for excluding the Gaspé extrapolation. (Notwithstanding our general concerns, we retained the series in calculations after 1450 where it has at least 3 trees and its impact is attenuated.) Because of the highly unique, ad hoc and undisclosed nature of the extrapolation and the substantial impact of the extrapolation, generally accepted methods would place the onus on MBH to justify the extrapolation and not on us to justify its exclusion. Such justification would require a demonstration of the impact of the extrapolation, why there should be an exception to permit the extrapolation, consideration of whether the pre-1450 portion of the series meets minimum signal requirements and whether the tree ring width series itself meets MBH98 requirements for being a temperature proxy. None of this has been provided to date.

 

Changes to the manuscript

For space reasons, we have exported the figures showing the Sheep Mountain-Mayberry Slough site series and the red noise simulation and associated discussion to the SI. We have combined two panels of our figure on principal components and three panels of figures on northern hemisphere reconstructions into one figure and exported the balance to the SI. We have kept summary sentences.

 

We have exported the discussion of the Stahle/SWM and Twisted Tree series to the SI, again keeping only a summary sentence. Our points here were not seriously contested by MBH and do not have a material impact on contentious early 15th century results.

 

As noted in our discussion of Professor Mann’s response, we have added a discussion of the high altitude sites that dominate the NOAMER PC1 and a discussion of the “censored” PC1 calculation at the MBH98 FTP site.

 

We have included RE statistics, as requested (with additional statistical data in the SI). Again, we note that the largest improvement in RE fit comes from using series computed with questionable methodology, resulting in the over-weighting of series that do not meet the required linearity assumptions of MBH98 methodology.

 

We wish to express our thanks to the referees and editors of Nature, and to Professors Mann, Bradley and Hughes, for considering our work.

 

Sincerely

 

Stephen McIntyre and Ross McKitrick