Chronology of Correspondence with Professor Mann (original Oct. 28, 2003; updated Jan. 14, 2004, postscript June 2004)
Following is the actual correspondence with Prof. Mann, together with related materials, showing that (1) pcproxy was a file used by Mann et al. long prior to my request; (2) no request was ever made for an Excel file or any other special collation; (3) the data used in MM03 was obtained from a directory at Mann's FTP site <ftp://holocene.evsc.virginia.edu/pub/sdr/pcproxy.txt> (which was deleted without notice between Oct. 29, 2003 and Nov, 8, 2003); (4) Mann was directly contacted and asked to confirm the validity of the data, when problems were identified. (5) at Mann's FTP site, not all files are indexed and unindexed files can be moved into the pub/ directory without changing their time stamp.
1. On August 31, 2001, the figure below, referring to the file pcproxy.txt, was located at Scott Rutherford's web page, dated long before the April 2003 inquiry to Mann This figure was archived by the Wayback Machine from Scott Rutherford's website. (See <http://web.archive.org/web/*/http://deschutes.gso.uri.edu/~rutherfo/climaterecon.html>.)
2. On July 30, 2002, the website ftp://holocene.evsc.virginia.edu appears to have started up, since all directories are date-stamped to that date. Many files within the website are dated May 22, 2000 and may have been transferred.
3. On August 8, 2002, the files pcproxy.txt and pcproxy.mat were saved at ftp://holocene.evsc.virginia.edu/pub/sdr. The file pcproxy.mat has a dateline of August 8, 2002 as well. (These files were deleted by Professor Mann shortly after publication of MM03, but I specifically verified the dates before they were deleted.)
4. On April 8, 2003, I sent the following request for the FTP location of MBH98 to Mann directly. No request was made for an Excel spreadsheet or other special format.
Dear Dr. Mann, I have been studying MBH98 and 99. I located datasets for the 13 series used in 99 at ftp://eclogite.geo.umass.edu/pub/mann/ONLINE-PREPRINTS/Millennium/DATA/PROXIES/ (the convenience of the ftp: location being excellent) and was intereseted in locating similar information on the 112 proxies referred to in MBH98, as well as listing (the listing at http://www.ngdc.noaa.gov/paleo/ei/data_supp.html is for 390 datasets, and I gather/presume that many of these listed datasets have been condensed into PCs, as mentioned in the paper itself. Thank you for your attention. Yours truly, Stephen McIntyre, Toronto, Canada
At the time and up to at least January 2004, Mann's website page for MBH98 <http://www.people.virginia.edu/~mem6u/mbh98.html> contained hyperlinks for data to his old FTP site at UMass and to the NOAA FTP site, but NOT to the FTP site at ftp://holocene.evsc.virginia.edu.
5. On April 9, 2003, Mann didn't seem to know where the FTP site was (!) and referred the matter to Scott Rutherford. (As an aside, no fewer than 10 articles were archived at Mann's FTP site in March and April 2003.)
Dear Mr. McIntyre, These data are available on an anonymous ftp site we have set up. I've forgotten the exact location, but I've asked my Colleague Dr. Scott Rutherford if he can provide you with that information. best regards, Mike Mann
6. On April 9, 2003, after the referral from Prof. Mann, the request for an FTP location was sent to Scott Rutherford.
Dear Dr. Rutherford, Prof. Mann indicated that you would be able to locate the ftp for the MBH proxies and I would appreciate it if you could forward me the information. Thank you for your attention. Yours truly, Steve McIntyre
7. On Apr. 11, 2003, Rutherford stated that the data were not located in one FTP site:.
Steve, The proxies aren't actually all in one ftp site (at least not to my knowledge). I can get them together if you give me a few days. Do you want the raw 300+ proxies or the 112 that were used in the MBH98 reconstruction? Scott
8. On April 11, 2003, I replied to Rutherford further showing that no request was being made for data to be organized in spreadsheet.
[Scott]
9. On April 22, 2003, I sent a reminder to Rutherford.
[Scott] Any
luck with this?
10. On April 22, 2003, Rutherford apologized for the delay.
Steve, I was just going to send you an e-mail. Honest. I haven't forgotten about you. Bear with me a little longer. Your request is just about at the top of the stack. Regards, Scott
11. On April 23, 2003, Rutherford sent the following email to me. (This email was recovered from an archive after the first version of this chronology was posted in November 2003.).
Steve, OK, I think I have it all straight now. You can get the data via anonymous ftp at holocene.evsc.virginia.edu/pub/sdr. There is a binary matlab file called pcproxy.mat and a plain text file called pcproxy.txt. The data run from 1400 to 1997 in rows and each proxy is in a column. Attached to this e-mail is a file called backto1820.list. This is a plain text file that lists the proxies in order (I hope) from left to right in the data file. (Lines beginning with -9.9 are flags for the directory where the data file is stored, not much help to you, but they may help you understand what each proxy set is.) You will also want to get http://www.ngdc.noaa.gov/paleo/ei/data_supp.html to determine what the proxies are. Note that this is the network that uses the PCs of the dense tree ring networks. I hope this helps. Let me know if you have questions. Regards, Scott
The file pcproxy.txt (and pcproxy.mat) were deleted from the ftp site within a few days of publication of McIntyre and McKitrick (2003) in late October 2003, but is here. The file enclosed by Rutherford listing proxies is shown here, with an identical version of this file is at the Mann FTP site here. As noted above, the files pcproxy.txt and pcproxy.mat were dated August 8, 2002 at the ftp site. Note that the directory locations referred to as being "not much help to you" are the directory locations in the directory ftp://holocene.evsc.virginia.edu/pub/MBH98/, now said to be the correct location of the data. This hardly suggests that the MBH98 directory was then in the public domain.
12. On Sept. 9, 2003, I sent the following question to Prof. Mann raising questions about his temperature principal component calculations and received no reply.
Dear Prof. Mann,
I have tried
diligently to reconstuct your termperature principal components as described in
MBH98, but without success and would appreciate some assistance.
I downloaded
hadcrut2.dat from CRU (July 2003 edition), truncated the data to 1902-1995 and
further truncated it to the 1082 cells at gridpoints.loc and arranged as 1082
time-series with 1128 monthly readings. This step was successful as I could
match your map of cell locations. I standardized each series to mean 0 and sd 1
for the period 1902-95. In MBH98,
you say that you carried out “conventional” PCA, but there is so much
missing data that conventional PCA failed when I tried. In particular, 4 cells
had no values at all and I don’t see why they were included in your selection.
Most PCA algorithms balk at missing data or exclude it. How did you deal
with the extensive missing data?
I downloaded the
EOFs, PCs and eigenvector loadings from ftp://eclogite.geo.umass.edu/pub/mann/MANNETAL98/EIGENVECTORS/ .
I spliced the EOFs into a 16x1082 matrix and the PCs (pc01.out, etc.)
into a 92x16 matrix. I made a diagonal of the first 16 values in column 2 of “tpca-eigenvals.out",
which look like eigenvalues, and carried out an expansion.
I then deducted the grid-box values generated from this expansion from
the Jones data as above; calculated variance for each year across available
cells and made a sum, comparing this to the variance similarly calculated in the
standardized Jones data. I
obtained very low/much lower explained variance from this than you got.
I also tried some experiments and it also doesn’t seem to me that the
first 16 EOFs maximize explained variance, as they should.
I would appreciate any assistance or clarification which you could give.
Regards, Steve
McIntyre
13. On Sep. 10, 2003, I raised questions to Rutherford about potential collation errors in the file pcproxy.txt .
Dear Scott, In the file pcproxy.txt which you sent me a while ago, a disporoportionate number of series start in the years *99 or *49, for example, series 73-83, 86-92 and 106-107. Is this intentional? Is this also the case in the underlying data? Thanks, Steve
14. On. Sep. 11, 2003, Rutherford answered as follows. Rutherford's reply suggests that he was not the one who collated pcproxy.txt - that it was "before his time".
Steve, These would be the principal components (PCs) of dense tree ring networks. As trees drop out of the network the PCs are recalculated every 50 or 100 years. So I think what is going on is that once the tree network is too sparse, Mike stopped calculating the PCs at the last step where there was a decent network. (This was before my time, but I'm pretty sure that's story.) Scott
15. On Sept. 25, I sent the following inquiry to Prof. Mann on pcproxy.txt, seeking, among other questions, specific verification that pcproxy.txt was the correct file.
Dear Prof Mann
Here is the pcproxy.txt file sent to me last April by Scott Rutherford at your
direction. It contains some missing data after 1971. Your 1998 paper does
not describe how missing data in this period is treated and I wanted to verify
that it is the correct file. How did you handle missing data in this period? In
earlier periods, it looks like you changed the roster of proxies in each of the
periods described in the Supplementary Information using only proxies available
throughout the entire period. I have obtained quite close replication of the
rpc1 in the 20th century by calculating coefficients for the proxies and then
calculating the rpc's using the minimization procedures described in MBH98 and
the selection of PCs in the Supplementary Information. The reconstruction
is less close in earlier periods. I also don't understand the reasoning
for reducing the roster of eigenvectors in earlier periods. The
description in MBH98 was necessarily very terse and is still very terse in the
Supplementary Information; is there any more detailed description of the
reconstruction methodology to help me resolve this? Thank you for your
attention.
Yours truly,
Steve McIntyre,
Toronto, Canada
16. On Sep. 25, Prof. Mann failed to provide such verification and stated that he would not respond to further inquiries.
Dear Mr. McIntyre,
A few of the series terminate prior to the nominal 1980 termination date of the
calibration period (the earliest such instance, as you note, is 1971). In such
cases, the data were continued to the 1980 boundary by persistence of the final
available value. These details in fact, were provided in the supplementary
information that accompanied the Nature article. That information is
available here (see first paragraph):
ftp://eclogite.geo.umass.edu/pub/mann/ONLINE-PREPRINTS/MultiProxy/data-supp.html
and here:
http://www.ngdc.noaa.gov/paleo/ei/data_supp.html
The results,
incidentally, are insensitive to this step; essentially the same reconstruction
is achieved if a calibration period terminating in 1970 (prior to the
termination of any of the proxy series) was used instead.
Owing to numerous demands on my time, I will not be able to respond to further
inquiries.
Other researchers have successfully implemented our methodology based on the
information provided in our articles [see e.g. Zorita, E., F. Gonzalez-Rouco,
and S. Legutke, Testing the Mann et al. (1998) approach to paleoclimate
reconstructions in the context of a 1000-yr control simulation with the ECHO-G
Coupled Climate Model, J. Climate, 16, 1378-1390, 2003.]. I trust,
therefore, that you will find (as in this case) that all necessary details are
provided in the papers we have published or the supplementary information links
provided by those papers.
Best of luck with your work.
Sincerely,
Michael E. Mann
17. On Oct. 27, 2003, MM03 was released.
18. On Oct. 29, 2003, the statements by Professor Mann referred to above were released through David Appell: Reference was made to the files pcproxy.txt.
"The
McIntye and McKitrick ("M&M") Energy and Environment paper
criticizing the work of Mann, Bradley, and Hughes is wrong, says Mann.
In short, here's what happened: M&M asked an associate of Mann to
supply them with the Mann et. al. proxy data in an Excel spreadsheet, even
though the raw data is available
here. [ftp://holocene.evsc.virginia.edu/pub/MBH98/] An error was made in preparing this Excel file, in which the early
series were successively overprinted by later and later series, and this is the
data M&M used. ....
Mann adds:
.... The authors had access to the full data, which has been available on a public ftp site for nearly two years. When they noticed, as described in their paper, some signs of problems with the Excel spreadsheet version of the data, one might think that they would have bothered to check the data available on our public ftp site."
19. On Oct. 29, 2003, we determined that pcproxy.txt and pcproxy.mat were both still at ftp://holocene.evsc.virginia.edu/pub/sdr and downloaded copies of these files as well as the contents of ftp://holocene.evsc.virginia.edu/pub/MBH98. The data in ftp://holocene.evsc.virginia.edu/pub/sdr/pcproxy.txt can be exactly traced to ftp://holocene.evsc.virginia.edu/pub/MBH98/ for all 81 non-principal component series (#1-68, #100-112). The collation errors identified in MM03, which are associated with splicing of different versions of PC series, affect 21 PC series (#72-92), but do not affect 10 other series. In our own calculations, we obviously noticed the problems with the MBH98 principal component series and carried out fresh calculations for series #68-80 and #83-99 using the maximal available period. We used MBH98 data only for series #81-83, where no listing of original sites was available (see revised proxy4.txt) Thus, of the various Audit Issues listed at the time, only Issues 2 and 3 are affected by the difference between the two data versions at the Mann FTP site.
20. On November 8, 2003, we re-visited the FTP site and determined that both pcproxy.txt and pcproxy.mat had been deleted from Mann's FTP site.
21. On November 12, 2003, the directory at FTP/UMass containing MBH98 data was deleted.
22. In mid-November, 2003, John Daly wrote to the webmaster at UMass asking that the directory be restored, which it was a few days later.
23. In mid-November 2003, the graphic referred to above was identified at the home page of Scott Rutherford. This website was deleted shortly thereafter.
Postscript June 2004: We were provided the following URL link in April 2004 containing a draft version of the SI attached to the July 1, 2004 Corrigendum <ftp://holocene.evsc.virginia.edu/pub/sdr/temp/nature> dated December 10, 2003. If you google up-directory to <ftp://holocene.evsc.virginia.edu/pub/sdr/temp> you will find that the sub-directory is not indexed. If you knew that the directory was there and had the correct URL, you could presumably have located the files, but without the exact call, you could not have found it. We checked these directories in November and December 2003 and did not locate these files. In July 2004, this directory was moved to the location <ftp://holocene.evsc.virginia.edu/pub/MANNETAL98>, where it still had a date stamp of December 10, 2003. However, it was first indexed only in July 2004. Thus, the date stamp on an indexed file does not indicate the first date at which the file was retrievable by index. Some other dates:
Dec. 9, 2003 - Rutherford posts at FTP/temp/nature a README outlining directory structure; the datalist1400,... and data1400,... files showing the makeup of the stepwise calculations in PROXY directory; the figure data in FIGURE directory; temperature data set, temperature PCs, temperature EOFs in INSTRUMENTAL directory; , .
Jan. 27, 2004 - Rutherford posts up README in FIGURE directory; eigenvalues, temperature gridcell locations and gridcell standard deviations in INSTRUMENTAL directory; .
Jan. 30, 2004 - Rutherford posts up add-delete file.
Feb. 3, 2004 - README file in PROXY directory
April 1-2, 2004 - README and textfile in METHODS directory
May 6, 2004 - README in INSTRUMENTAL directory;