From: Ben Santer <santer1@llnl.

gov>
To: "'Philip D. Jones'" <p.jones@uea.ac.uk>
Subject: [Fwd: Re: [Fwd: Typo in equation 12 Santer.]]
Date: Thu, 30 Oct 2008 20:14:41 -0700
Reply-to: santer1@llnl.gov

<x-flowed>
Dear Phil,

I thought you'd be interested in my reply to Gavin (see forwarded email).

Cheers,

Ben
----------------------------------------------------------------------------
Benjamin D. Santer
Program for Climate Model Diagnosis and Intercomparison
Lawrence Livermore National Laboratory
P.O. Box 808, Mail Stop L-103
Livermore, CA 94550, U.S.A.
Tel: (925) 422-3840
FAX: (925) 422-7675
email: santer1@llnl.gov
----------------------------------------------------------------------------

</x-flowed>
X-Account-Key: account1
Return-Path: <santer1@llnl.gov>
Received: from mail-2.llnl.gov ([unix socket])
by mail-2.llnl.gov (Cyrus v2.2.12) with LMTPA;
Thu, 30 Oct 2008 20:10:53 -0700
Received: from nspiron-1.llnl.gov (nspiron-1.llnl.gov [128.115.41.81])
by mail-2.llnl.gov (8.13.1/8.12.3/LLNL evision: 1.7 $) with ESMTP id
m9V3Arh7024023;
Thu, 30 Oct 2008 20:10:53 -0700
X-Attachments: None
X-IronPort-AV: E=McAfee;i="5300,2777,5419"; a="30418306"
X-IronPort-AV: E=Sophos;i="4.33,519,1220252400";
d="scan'208";a="30418306"
Received: from dione.llnl.gov (HELO [128.115.57.29]) ([128.115.57.29])
by nspiron-1.llnl.gov with ESMTP; 30 Oct 2008 20:10:53 -0700
Message-ID: <490A773D.20807@llnl.gov>
Date: Thu, 30 Oct 2008 20:10:53 -0700
From: Ben Santer <santer1@llnl.gov>
Reply-To: santer1@llnl.gov
Organization: LLNL
User-Agent: Thunderbird 1.5.0.12 (X11/20070529)
MIME-Version: 1.0
To: Gavin Schmidt <gschmidt@giss.nasa.gov>
CC: Karl Taylor <taylor13@llnl.gov>
Subject: Re: [Fwd: Typo in equation 12 Santer.]
References: <1224543811.19301.2452.camel@isotope.giss.nasa.gov>
In-Reply-To: <1224543811.19301.2452.camel@isotope.giss.nasa.gov>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

<x-flowed>
Dear Gavin,
There is no typo in equation 12. The first term under the square root in
equation 12 is a standard estimate of the variance of a sample mean
(see, e.g., "Statistical Analysis in Climate Research", Zwiers and
Storch, their equation 5.24, page 86). The second term under the square
root sign is a very different beast - an estimate of the variance of the
observed trend. As we point out, our d1* test is very similar to a
standard Student's t-test of differences in means (which involves, in
its denominator, the square root of two pooled sample variances).

In testing the statistical significance of differences between the model

average trend and a single observed trend, Douglass et al. were wrong to
use sigma_SE as the sole measure of trend uncertainty in their
statistical test. Their test assumes that the model trend is uncertain,
but that the observed trend is perfectly-known. The observed trend is
not a "mean" quantity; it is NOT perfectly-known. Douglass et al. made a
demonstrably false assumption.

Bottom line: sigma_SE is a standard estimate of the uncertainty in a

sample mean - which is why we use it to characterize uncertainty in the
estimate of the model average trend in equation 12. It is NOT
appropriate to use sigma_SE as the basis for a statistical test between
two uncertain quantities (see our comments in our point #3, immediately
before equation 12). The uncertainty in the estimates of both modeled
AND observed trend needs to be explicitly incorporated in the design of
any statistical test comparing modeled and observed trends. Douglass et
al. incorrectly ignored uncertainties in observed trends.

Our Figure 6A is not a statistical test. It does not show the standard
errors in the observed trends at discrete pressure levels (which would
have made for a very messy Figure, given that we show results from 7
different observational datasets). Had we attempted to show the observed
standard errors in Figure 6A, I suspect that standard errors from the
RICH, IUK, RAOBCORE-v1.3, and RAOBCORE 1.4 datasets would have
overlapped with the multi-model average trend at most pressure levels. I
can easily produce such a Figure if necessary.

With best regards,

Ben

Gavin Schmidt wrote:

> Ben, Just thought I'd check with you first. I don't think there is a
> problem - but I think the question is really alluding to is our comment
> about Douglass et al 'being wrong' in using sigma_SE - since if we use
> it in the denominator in the d1* test, it can't be wrong, see?
>
> My response would be that we are testing a number of different things
> here: d1* tests whether the ensemble mean is consistent with the obs
> (given their uncertainty). Whereas our figure 6 and the error bars shown
> there are testing whether the real world obs are consistent with a
> distribution defined from the model ensemble members.
>
> gavin
>
> -----Forwarded Message-----
>
>> From: lucia liljegren <lucia@rankexploits.com>
>> To: gschmidt@giss.nasa.gov
>> Subject: Typo in equation 12 Santer.
>> Date: 20 Oct 2008 15:46:51 -0500
>>
>> Hi Gavin,
>>
>> Someone commenting at ClimateAudit is suggesting that equation 12
>> contains a typo. They are under the impression the 1/nm does not
>> belong in the circled term. Rather than going back and forth with "is
>> not a typo", "is so a typo", I figured I'd just ask you. Is there a
>> typo in equaltion 12 below.
>>
>> ----
>>
>
>>
>>
>>
>> BTW: I think Santer is pretty good paper.
>>
>> Thanks, Lucia
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>

--
----------------------------------------------------------------------------
Benjamin D. Santer
Program for Climate Model Diagnosis and Intercomparison
Lawrence Livermore National Laboratory
P.O. Box 808, Mail Stop L-103
Livermore, CA 94550, U.S.A.
Tel: (925) 422-3840
FAX: (925) 422-7675
email: santer1@llnl.gov
----------------------------------------------------------------------------

</x-flowed>