←
Chapter page 19 / 3812 Confidence intervals with bootstrapping
→
English

12  Confidence intervals with bootstrapping

In this chapter, we expand on the familiar idea of using a sample proportion to estimate a population proportion. That is, we create what is called a confidence interval, which is a range of plausible values where we may find the true population value. The process for creating a confidence interval is based on understanding how a statistic (here the sample proportion) varies around the parameter (here the population proportion) when many different statistics are calculated from many different samples.

If we could, we would measure the variability of the statistics by repeatedly taking sample data from the population and compute the sample proportion. Then we could do it again. And again. And so on until we have a good sense of the variability of our original estimate.

When the variability across the samples is large, we would assume that the original statistic is possibly far from the true population parameter of interest (and the interval estimate will be wide). When the variability across the samples is small, we expect the sample statistic to be close to the true parameter of interest (and the interval estimate will be narrow).

The ideal world where sampling data is free or extremely cheap is almost never the case, and taking repeated samples from a population is usually impossible. So, instead of using a “resample from the population” approach, bootstrapping uses a “resample from the sample” approach. In this chapter we discuss in detail the bootstrapping process.

As seen in Chapter 11, randomization is a statistical technique suitable for evaluating whether a difference in sample proportions is due to chance.

Randomization tests are best suited for modeling experiments where the treatment (explanatory variable) has been randomly assigned to the observational units and there is an attempt to answer a simple yes/no research question.

For example, consider the following research questions that can be well assessed with a randomization test:

  • Does this vaccine make it less likely that a person will get malaria?
  • Does drinking caffeine affect how quickly a person can tap their finger?
  • Can we predict whether candidate A will win the upcoming election?

In this chapter, however, we are instead interested in a different approach to understanding population parameters. Instead, of testing a claim, the goal now is to estimate the unknown value of a population parameter.

For example,

  • How much less likely am I to get malaria if I get the vaccine?
  • How much faster (or slower) can a person tap their finger, on average, if they drink caffeine first?
  • What proportion of the vote will go to candidate A?

Here, we explore the situation where the focus is on a single proportion, and we introduce a new simulation method: bootstrapping.

Bootstrapping is best suited for modeling studies where the data have been generated through random sampling from a population. As with randomization tests, our goal with bootstrapping is to understand variability of a statistic. Unlike randomization tests (which modeled how the statistic would change if the treatment had been allocated differently), the bootstrap will model how a statistic varies from one sample to another taken from the population. This will provide information about how different the statistic is from the parameter of interest.

Quantifying the variability of a statistic from sample to sample is a hard problem. Fortunately, sometimes the mathematical theory for how a statistic varies (across different samples) is well-known; this is the case for the sample proportion as seen in Chapter 13.

However, some statistics do not have simple theory for how they vary, and bootstrapping provides a computational approach for providing interval estimates for almost any population parameter. In this chapter we will focus on bootstrapping to estimate a single proportion, and we will revisit bootstrapping in Chapter 19 through Chapter 21, so you’ll get plenty of practice as well as exposure to bootstrapping in many different datasettings.

Our goal with bootstrapping will be to produce an interval estimate (a range of plausible values) for the population parameter.

12.1 Medical consultant case study

People providing an organ for donation sometimes seek the help of a special medical consultant. These consultants assist the patient in all aspects of the surgery, with the goal of reducing the possibility of complications during the medical procedure and recovery. Patients might choose a consultant based in part on the historical complication rate of the consultant’s clients.

12.1.1 Observed data

One consultant tried to attract patients by noting the average complication rate for liver donor surgeries in the US is about 10%, but her clients have had only 3 complications in the 62 liver donor surgeries she has facilitated. She claims this is strong evidence that her work meaningfully contributes to reducing complications (and therefore she should be hired!).

We will let \(p\) represent the true complication rate for liver donors working with this consultant. (The “true” complication rate will be referred to as the parameter.) We estimate \(p\) using the data, and label the estimate \(\hat{p}.\)

The sample proportion for the complication rate is 3 complications divided by the 62 surgeries the consultant has worked on: \(\hat{p} = 3/62 = 0.048.\)

Is it possible to assess the consultant’s claim (that the reduction in complications is due to her work) using the data?

No. The claim is that there is a causal connection, but the data are observational, so we must be on the lookout for confounding variables. For example, maybe patients who can afford a medical consultant can afford better medical care, which can also lead to a lower complication rate. While it is not possible to assess the causal claim, it is still possible to understand the consultant’s true rate of complications.

Parameter.

A parameter is the “true” value of interest.

We typically estimate the parameter using a point estimate from a sample of data. The point estimate is also known as the statistic.

For example, we estimate the probability \(p\) of a complication for a client of the medical consultant by examining the past complications rates of her clients:

\[\hat{p} = 3 / 62 = 0.048~\text{is used to estimate}~p\]

12.1.2 Variability of the statistic

In the medical consultant case study, the parameter is \(p,\) the true probability of a complication for a client of the medical consultant. There is no reason to believe that \(p\) is exactly \(\hat{p} = 3/62,\) but there is also no reason to believe that \(p\) is particularly far from \(\hat{p} = 3/62.\) By sampling with replacement from the dataset (a process called bootstrapping), the variability of the possible \(\hat{p}\) values can be approximated.

Most of the inferential procedures covered in this text are grounded in quantifying how one dataset would differ from another when they are both taken from the same population. It does not make sense to take repeated samples from the same population because if you have the means to take more samples, a larger sample size will benefit you more than separately evaluating two sample of the exact same size. Instead, we measure how the samples behave under an estimate of the population.

Figure 12.1 shows how the unknown original population can be estimated by using the sample to approximate the proportion of successes and failures (in our case, the proportion of complications and no complications for the medical consultant).

A small sample of 3 red marbles and 4 white marbles is taken from a large population with predominately unknown individual values. The sample is then replicated infinitely many times to create a proxy population where the values are known to be 3/7 red and 4/7 white.
Figure 12.1: The unknown population is estimated using the observed sample data. Note that we can use the sample to create an estimated or bootstrapped population from which to sample. The observed data include three red and four white marbles, so the estimated population contains 3/7 red marbles and 4/7 white marbles.

By taking repeated samples from the estimated population, the variability from sample to sample can be observed. In Figure 12.2 the repeated bootstrap samples are obviously different both from each other and from the original population. Recall that the bootstrap samples were taken from the same (estimated) population, and so the differences are due entirely to natural variability in the sampling procedure.

The same unknown large population is given with a sample of 3 red and 4 white marbles. After the proxy population is created (infinite replicates of the sample), new resamples of size 7 can be taken from the proxy population. Three resamples of size 7 are shown: Resample 1 has 2/7 red; Resample 2 has 4/7 red; and Resample k has 5/7 red.
Figure 12.2: Bootstrap sampling provides a measure of the sample to sample variability. Note that we are taking samples from the estimated population that was created from the observed data.

By summarizing each of the bootstrap samples (here, using the sample proportion), we see, directly, the variability of the sample proportion, \(\hat{p},\) from sample to sample. The distribution of \(\hat{p}_{boot}\) for the example scenario is shown in Figure 12.3, and the full bootstrap distribution for the medical consultant data is shown in Figure 12.6.

The same unknown large population with a sample of 3 red and 4 white marbles; the proxy population which is infinite with 3/7 red marbles; and the k Resamples of size 7 are shown. From each of the resamples the bootstrapped proportion of red is calculated (shown as 2/7, 4/7, and 5/7). Many many resamples are taken and summarized in a dotplot of the bootstrapped proportions. The proportions range from 0/7 to 7/7 in a bell shape with the majority of bootstrapped proportions falling between 1/7 and 6/7.
Figure 12.3: The bootstrapped proportion is estimated for each bootstrap sample. The resulting bootstrap distribution (dotplot) provides a measure for how the proportions vary from sample to sample

It turns out that in practice, it is very difficult for computers to work with an infinite population (with the same proportional breakdown as in the sample). However, there is a physical and computational method which produces an equivalent bootstrap distribution of the sample proportion in a computationally efficient manner.

Consider the observed data to be a bag of marbles 3 of which are success (red) and 4 of which are failures (white). By drawing the marbles out of the bag with replacement, we depict the exact same sampling process as was done with the infinitely large estimated population.

Shown is the unknown large population with a sample of 3 red and 4 white marbles. Without creating the infinitely large proxy population, resamples are taken from the original sample (by sampling with replacement from the sample). Three resamples of size 7 are shown: Resample 1 has 2/7 red; Resample 2 has 4/7 red; and Resample k has 5/7 red.
Figure 12.4: Taking repeated resamples from the sample data is the same process as creating an infinitely large estimate of the population. It is computationally more feasible to take resamples directly from the sample. Note that the resampling is now done with replacement (that is, the original sample does not ever change) so that the original sample and estimated hypothetical population are equivalent.
Top image includes the steps of (1) a large unknown population, (2) observed sample of size 7 (with 3 red and 4 white), (3) creation of an infinitely large proxy population, and (4) three resamples. (5) Many resamples are considered with a dotplot of bootstrapped proportions. The bottom image follows the same process without the infinitely large proxy population. That is, in the bottom image a (1) single sample is taken from the original population and (2) the three resamples are taken directly from the observed data (using sampling with replacement). (3) Again, many resamples are considered with a dotplot of bootstrapped proportions.
Figure 12.5: A comparison of the process of sampling from the estimate infinite population and resampling with replacement from the original sample. Note that the dotplot of bootstrapped proportions is the same because the process by which the statistics were estimated is equivalent.

If we apply the bootstrap sampling process to the medical consultant example, we consider each client to be one of the marbles in the bag. There will be 59 white marbles (no complication) and 3 red marbles (complication). If we choose 62 marbles out of the bag (one at a time with replacement) and compute the proportion of simulated patients with complications, \(\hat{p}_{boot},\) then this “bootstrap” proportion represents a single simulated proportion from the “resample from the sample” approach.

In a simulation of 62 patients, about how many would we expect to have had a complication?1

One simulation isn’t enough to get a sense of the variability from one bootstrap proportion to another bootstrap proportion, so we repeat the simulation 10,000 times using a computer.

Figure 12.6 shows the distribution from the 10,000 bootstrap simulations. The bootstrapped proportions vary from about zero to 11.3%. The variability in the bootstrapped proportions leads us to believe that the true probability of complication (the parameter, \(p\)) is likely to fall somewhere between 0% and 11.3%, as these numbers capture 95% of the bootstrap resampled values.

The range of values for the true proportion is called a bootstrap percentile confidence interval, and we will see it again throughout the next few sections and chapters.

Histogram of 10,000 bootstrapped proportions. The bootstrap 2.5 percentile proportion is zero and the 97.5 percentile is 0.113.
Figure 12.6: The original medical consultant data is bootstrapped 10,000 times. Each simulation creates a sample from the original data where the probability of a complication is \(\hat{p} = 3/62.\) The bootstrap 2.5 percentile proportion is 0 and the 97.5 percentile is 0.113. The result is: we are confident that, in the population, the true probability of a complication is between 0% and 11.3%.

The original claim was that the consultant’s true rate of complication was under the national rate of 10%. Does the interval estimate of 0% to 11.3% for the true probability of complication indicate that the surgical consultant has a lower rate of complications than the national average? Explain.

No. Because the interval overlaps 10%, it might be that the consultant’s work is associated with a lower risk of complications, or it might be that the consultant’s work is associated with a higher risk (i.e., greater than 10%) of complications! Additionally, as previously mentioned, because this is an observational study, even if an association can be measured, there is no evidence that the consultant’s work is the cause of the complication rate (being higher or lower).

12.2 Tappers and listeners case study

Here’s a game you can try with your friends or family: pick a simple, well-known song, tap that tune on your desk, and see if the other person can guess the song. In this simple game, you are the tapper, and the other person is the listener.

12.2.1 Observed data

A Stanford University graduate student named Elizabeth Newton conducted an experiment using the tapper-listener game.2 In her study, she recruited 120 tappers and 120 listeners into the study. About 50% of the tappers expected that the listener would be able to guess the song. Newton wondered, is 50% a reasonable expectation?

In Newton’s study, only 3 out of 120 listeners (\(\hat{p} = 0.025\)) were able to guess the tune! That seems like quite a low number which leads the researcher to ask: what is the true proportion of people who can guess the tune?

12.2.2 Variability of the statistic

To answer the question, we will again use a simulation. To simulate 120 games, this time we use a bag of 120 marbles 3 are red (for those who guessed correctly) and 117 are white (for those who could not guess the song). Sampling from the bag 120 times (remembering to replace the marble back into the bag each time to keep constant the population proportion of red) produces one bootstrap sample.

For example, we can start by simulating 5 tapper-listener pairs by sampling 5 marbles from the bag of 3 red and 117 white marbles.

W W W R W
Wrong Wrong Wrong Correct Wrong

After selecting 120 marbles, we counted 2 red for \(\hat{p}_{boot1} = 0.0167.\) As we did with the randomization technique, seeing what would happen with one simulation isn’t enough. In order to understand how far the observed proportion of 0.025 might be from the true parameter, we should generate more simulations. Here we have repeated the entire simulation ten times:

\[0.0417 \quad 0.025 \quad 0.025 \quad 0.0083 \quad 0.05 \quad 0.0333 \quad 0.025 \quad 0 \quad 0.0083 \quad 0\]

As before, we’ll run a total of 10,000 simulations using a computer. As seen in Figure 12.7, the range of 95% of the resampled values of \(\hat{p}_{boot}\) is 0.000 to 0.0583. That is, we expect that between 0% and 5.83% of people are truly able to guess the tapper’s tune.

Histogram of 10,000 bootstrapped proportions. The bootstrap 2.5 percentile proportion is zero and the 97.5 percentile is 0.0583.
Figure 12.7: The original listener-tapper data is bootstrapped 10,000 times. Each simulation creates a sample where the probability of being correct is \(\hat{p} = 3/120.\) The 2.5 percentile proportion is 0 and the 97.5 percentile is 0.0583. The result is that we are confident that, in the population, the true percent of people who can guess correctly is between 0% and 5.83%.

Do the data provide convincing evidence against the claim that 50% of listeners can guess the tapper’s tune?3

12.3 Confidence intervals

A point estimate provides a single plausible value for a parameter. However, a point estimate is rarely perfect; usually there is some error in the estimate. In addition to supplying a point estimate of a parameter, a next logical step would be to provide a plausible range of values for the parameter.

12.3.1 Plausible range of values for the population parameter

A plausible range of values for the population parameter is called a confidence interval. Using only a single point estimate is like fishing in a murky lake with a spear, and using a confidence interval is like fishing with a net. We can throw a spear where we saw a fish, but we will probably miss. On the other hand, if we toss a net in that area, we have a good chance of catching the fish.

If we report a point estimate, we probably will not hit the exact population parameter. On the other hand, if we report a range of plausible values – a confidence interval – we have a good shot at capturing the parameter.

If we want to be very certain we capture the population parameter, should we use a wider interval (e.g., 99%) or a smaller interval (e.g., 80%)?4

12.3.2 Bootstrap confidence interval

As we saw above, a bootstrap sample is a sample of the original sample. In the case of the medical complications data, we proceed as follows:

  • Randomly sample one observation from the 62 patients (replace the marble back into the bag so as to keep the population constant).
  • Randomly sample a second observation from the 62 patients. Because we sample with replacement (i.e., we do not actually remove the marbles from the bag), there is a 1-in-62 chance that the second observation will be the same one sampled in the first step!
  • Keep going one sampled observation at a time …
  • Randomly sample the 62nd observation from the 62 patients.

Bootstrap sampling is often called sampling with replacement.

A bootstrap sample behaves similarly to how an actual sample from a population would behave, and we compute the point estimate of interest (here, compute \(\hat{p}_{boot}\)).

Based on theory that is beyond this text, we know that the bootstrap proportions \(\hat{p}_{boot}\) vary around \(\hat{p}\) similarly to how different sample proportions (i.e., values of \(\hat{p}\)) vary around the true parameter \(p.\) Therefore, an interval estimate for \(p\) can be produced using the \(\hat{p}_{boot}\) values themselves.

95% bootstrap percentile confidence interval for a parameter \(p.\)

The 95% bootstrap confidence interval for the parameter \(p\) can be obtained directly using the ordered \(\hat{p}_{boot}\) values.

Consider the sorted \(\hat{p}_{boot}\) values. Call the 2.5% bootstrapped proportion value “lower”, and call the 97.5% bootstrapped proportion value “upper”.

The 95% confidence interval is given by: (lower, upper)

In Section 16.1 we will discuss different percentages for the confidence interval (e.g., 90% confidence interval or 99% confidence interval).

Section Section 16.1 also provides a longer discussion on what “95% confidence” actually means.

12.4 Chapter review

12.4.1 Summary

Figure 12.8 provides a visual summary of creating bootstrap confidence intervals.

Full depiction of the bootstrapping process. Cartoon of the population and potential bootstrap samples is shown, each with a different bootstrap proportion. A dot plot of the bootstrapped proportions shows how they are naturally variable.
Figure 12.8: We will use sampling with replacement to measure the variability of the statistic of interest (here the proportion). Sampling with replacement is a computational tool which is equivalent to using the sample as a way of estimating an infinitely large population from which to sample.

We can summarize the bootstrap process as follows:

  • Frame the research question in terms of a parameter to estimate. Confidence Intervals are appropriate for research questions that aim to estimate a number from the population (called a parameter).
  • Collect data with an observational study or experiment. If a research question can be formed as a query about the parameter, we can collect data to calculate a statistic which is the best guess we have for the value of the parameter. However, we know that the statistic won’t be exactly equal to the parameter due to natural variability.
  • Model the randomness by using the data values as a proxy for the population. In order to assess how far the statistic might be from the parameter, we take repeated resamples from the dataset to measure the variability in bootstrapped statistics. The variability of the bootstrapped statistics around the observed statistic (a quantity which can be measured through computational technique) should be approximately the same as the variability of many observed sample statistics around the parameter (a quantity which is very difficult to measure because in real life we only get exactly one sample).
  • Create the interval. After choosing a particular confidence level, use the variability of the bootstrapped statistics to create an interval estimate which will hope to capture the true parameter. While the interval estimate associated with the particular sample at hand may or may not capture the parameter, the researcher knows that over their lifetime, the confidence level will determine the percentage of their research confidence intervals that do capture the true parameter.
  • Form a conclusion. Using the confidence interval from the analysis, report on the interval estimate for the parameter of interest. Also, be sure to write the conclusion in plain language so casual readers can understand the results.

Table 12.1 is another look at the bootstrap process summary.

Table 12.1: Summary of bootstrapping as an inferential statistical method.
Question Answer
What does it do? Resamples (with replacement) from the observed data to mimic the sampling variability found by collecting data from a population
What is the random process described? Random sampling from a population
What other random processes can be approximated? Can also be used to describe random allocation in an experiment
What is it best for? Confidence intervals (can also be used for bootstrap hypothesis testing for one proportion as well)
What physical object represents the simulation process? Pulling marbles from a bag with replacement

12.4.2 Terms

The terms introduced in this chapter are presented in Table 12.2. If you’re not sure what some of these terms mean, we recommend you go back in the text and review their definitions. You should be able to easily spot them as bolded text.

Table 12.2: Terms introduced in this chapter.
bootstrap percentile confidence interval confidence interval sampling with replacement
bootstrap sample parameter statistic
bootstrapping point estimate

12.5 Exercises

Answers to odd-numbered exercises can be found in Appendix A.12.

  1. Outside YouTube videos. Let’s say that you want to estimate the proportion of YouTube videos which take place outside (define “outside” to be if any part of the video takes place outdoors). You take a random sample of 128 YouTube videos5 and determine that 37 of them take place outside. You’d like to estimate the proportion of all YouTube videos which take place outside, so you decide to create a bootstrap interval from the original sample of 128 videos.

    1. Describe in words the relevant statistic and parameter for this problem. If you know the numerical value for either one, provide it. If you don’t know the numerical value, explain why the value is unknown.

    2. What notation is used to describe, respectively, the statistic and the parameter?

    3. If using software to bootstrap the original dataset, what is the statistic calculated on each bootstrap sample?

    4. When creating a bootstrap sampling distribution (histogram) of the bootstrapped sample proportions, where should the center of the histogram lie?

    5. The histogram provides a bootstrap sampling distribution for the sample proportion (with 1000 bootstrap repetitions). Using the histogram, estimate a 90% confidence interval for the proportion of YouTube videos which take place outdoors.

    6. Interpret the confidence interval in context of the data.

  1. Chronic illness. In 2012 the Pew Research Foundation reported that “45% of US adults report that they live with one or more chronic conditions.” However, this value was based on a sample, so it may not be a perfect estimate for the population parameter of interest on its own. The study was based on a sample of 3014 adults. Below is a distribution of 1000 bootstrapped sample proportions from the Pew dataset. (Pew Research Center 2013) Using the distribution of 1,000 bootstrapped proportions, approximate a 92% confidence interval for the true proportion of US adults who live with one or more chronic conditions and interpret it.

  1. Social media users and news, bootstrapping. A poll conducted in 2022 found that 50% of U.S. adults get news from social media sometimes or often. However, the value was based on a sample, so it may not be a perfect estimate for the population parameter of interest on its own. The study was based on a sample of 12,147 adults. Below is a distribution of 1,000 bootstrapped sample proportions from the Pew dataset. (Pew Research Center 2022) Using the distribution of 1,000 bootstrapped proportions, approximate a 98% confidence interval for the true proportion of US adult social media users (in 2022) who get at least some of their news from Twitter. Interpret the interval in the context of the problem.

  1. Bootstrap distributions of \(\hat{p}\), I. Each of the following four distributions was created using a different dataset. Each dataset was based on \(n=23\) observations. The original datasets had the following proportions of successes: \[\hat{p} = 0.13 \ \ \hat{p} = 0.22 \ \ \hat{p} = 0.30 \ \ \hat{p} = 0.43.\] Match each histogram with the original data proportion of success.

  1. Bootstrap distributions of \(\hat{p}\), II. Each of the following four distributions was created using a different dataset. Each dataset was based on \(n=23\) observations.

    Consider each of the following values for the true popluation \(p\) (proportion of success). Datasets A, B, C, D were bootstrapped 1000 times, with bootstrap proportions as given in the histograms provided. For each parameter value, list the datasets which could plausibly have come from that population. (Hint: there may be more than one dataset for each parameter value.)

    1. \(p = 0.05\)

    2. \(p = 0.25\)

    3. \(p = 0.45\)

    4. \(p = 0.55\)

    5. \(p = 0.75\)

  1. Bootstrap distributions of \(\hat{p}\), III. Each of the following four distributions was created using a different dataset. Each dataset had the same proportion of successes \((\hat{p} = 0.4)\) but a different sample size. The four datasets were given by \(n = 10, 100, 500\), and \(1000\).

    Consider each of the following values for the true popluation \(p\) (proportion of success). Datasets A, B, C, D were bootstrapped 1000 times, with bootstrap proportions as given in the histograms provided. For each parameter value, list the datasets which could plausibly have come from that population. (Hint: there may be more than one dataset for each parameter value.)

    1. \(p = 0.05\)

    2. \(p = 0.25\)

    3. \(p = 0.45\)

    4. \(p = 0.55\)

    5. \(p = 0.75\)

  1. Cyberbullying rates. Teens were surveyed about cyberbullying, and 54% to 64% reported experiencing cyberbullying (95% confidence interval). Answer the following questions based on this interval. (Pew Research Center 2018)

    1. A newspaper claims that a majority of teens have experienced cyberbullying. Is this claim supported by the confidence interval? Explain your reasoning.

    2. A researcher conjectured that 70% of teens have experienced cyberbullying. Is this claim supported by the confidence interval? Explain your reasoning.

    3. Without actually calculating the interval, determine if the claim of the researcher from part (b) would be supported based on a 90% confidence interval?

  1. Waiting at an ER. A 95% confidence interval for the mean waiting time at an emergency room (ER) of (128 minutes, 147 minutes). Answer the following questions based on this interval.

    1. A local newspaper claims that the average waiting time at this ER exceeds 3 hours. Is this claim supported by the confidence interval? Explain your reasoning.

    2. The Dean of Medicine at this hospital claims the average wait time is 2.2 hours. Is this claim supported by the confidence interval? Explain your reasoning.

    3. Without actually calculating the interval, determine if the claim of the Dean from part (b) would be supported based on a 99% confidence interval?

  1. About 4.8% of the patients (3 on average) in the simulation will have a complication, as this is what was seen in the sample. We will, however, see a little variation from one simulation to the next.↩︎

  2. This case study is described in Made to Stick by Chip and Dan Heath. Little known fact: the teaching principles behind many OpenIntro resources are based on Made to Stick.↩︎

  3. Because 50% is not in the interval estimate for the true parameter, we can say that there is convincing evidence against the hypothesis that 50% of listeners can guess the tune. Moreover, 50% is a substantial distance from the largest resample statistic, suggesting that there is very convincing evidence against this hypothesis.↩︎

  4. If we want to be more certain we will capture the fish, we might use a wider net. Likewise, we use a wider confidence interval if we want to be more certain that we capture the parameter.↩︎

  5. There are many choices for implementing a random selection of YouTube videos, but it isn’t clear how “random” they are.↩︎

中文

12  基于自助法的置信区间

在本章中,我们将进一步探讨用样本比例来估计总体比例这一熟悉的概念。也就是说,我们将构建所谓的 置信区间,它是一个合理取值的范围,真实的总体值可能位于其中。构建置信区间的方法基于理解当从许多不同样本计算出许多不同统计量时,统计量(此处为样本比例)如何围绕参数(此处为总体比例) 变化 。

如果可以的话,我们会通过反复从总体中抽取样本数据并计算样本比例来衡量统计量的变异性。然后我们可以再做一次。再来一次。如此反复,直到我们对最初估计值的变异性有了充分的了解。

当样本之间的变异性较大时,我们会认为最初的统计量可能远离所关注的真实总体参数(且区间估计会较宽)。当样本之间的变异性较小时,我们预期样本统计量接近所关注的真实参数(且区间估计会较窄)。

抽样数据免费或极其便宜的理想世界几乎从未出现过，而且从总体中重复抽样通常是不可能的。因此，自助法（bootstrapping）不是采用“从总体中重抽样”的方法，而是采用“从样本中重抽样”的方法。在本章中，我们将详细讨论自助法的过程。

如 第 11所示，随机化是一种统计技术，适用于评估样本比例之间的差异是否由偶然因素造成。

随机化检验最适合用于对实验进行建模，其中处理（解释变量）被随机分配给观测单元，并且试图回答一个简单的是/否研究问题。

例如，考虑以下可以用随机化检验很好地评估的研究问题：

  • 这种疫苗能降低一个人感染疟疾的可能性吗？
  • 摄入咖啡因会影响一个人敲击手指的速度吗？
  • 我们能预测候选人A是否会赢得即将到来的选举吗？

然而，在本章中，我们感兴趣的是理解总体参数的另一种方法。现在的目标不是检验某个断言，而是估计总体参数的未知值。

例如：

  • 如果我接种了疫苗，我感染疟疾的可能性会降低多少？
  • 如果先摄入咖啡因，一个人平均敲击手指的速度会快多少（或慢多少）？
  • 候选人A将获得多少比例的选票？

在这里，我们探讨关注单一比例的情形，并介绍一种新的模拟方法： 自助法.

自助法最适合用于对通过从总体中随机抽样生成的数据进行建模的研究。与随机化检验一样，我们使用自助法的目标是理解统计量的变异性。与随机化检验（模拟如果处理以不同方式分配，统计量会如何变化）不同，自助法将模拟统计量在从总体中抽取的不同样本之间如何变化。这将提供关于统计量与所关注的参数有多大差异的信息。

量化一个统计量在不同样本之间的变异性是一个困难的问题。幸运的是，有时关于统计量如何（在不同样本之间）变化的数学理论是众所周知的；样本比例就是这种情况，如 第13章.

中所示。 第 19 然而，有些统计量并没有关于其如何变化的简单理论，而自助法（bootstrapping）为几乎任何总体参数提供区间估计提供了一种计算方法。在本章中，我们将专注于使用自助法来估计单个比例，并且我们将在 第 21至

中重新讨论自助法，因此你将获得大量练习，并在许多不同的数据设置中接触自助法。

12.1 我们使用自助法的目标是为总体参数生成一个区间估计（一个合理取值的范围）。

医疗顾问案例研究

12.1.1 观测数据

观测数据

一位顾问试图通过以下说法来吸引患者：美国肝供体手术的平均并发症率约为10%，但在她经手的62例肝供体手术中，她的客户只出现了3例并发症。她声称这是强有力的证据，表明她的工作对减少并发症有实质性贡献（因此应该雇用她！）。 \(p\) 我们将用 参数参数 \(p\) 。）我们使用数据来估计 \(\hat{p}.\)

，并将该估计值记为 \(\hat{p} = 3/62 = 0.048.\)

并发症率的样本比例是3例并发症除以该顾问经手的62例手术：

不。该论断声称存在因果关系，但数据是观察性的，因此我们必须警惕混杂变量。例如，也许请得起医疗顾问的病人也能负担更好的医疗护理，这同样可能导致更低的并发症发生率。虽然无法评估这一因果论断，但仍然可以了解该顾问真实的并发症发生率。

参数。

一个 参数 是所关心的“真实”值。

我们通常使用来自数据样本的点估计来估计参数。点估计也被称为 统计量.

例如，我们通过考察该医疗顾问过去客户的并发症发生率来估计其客户发生并发症的概率 \(p\) 。

\[\hat{p} = 3 / 62 = 0.048~\text{is used to estimate}~p\]

12.1.2 统计量的变异性

在医疗顾问案例研究中，参数是 \(p,\) ，即该医疗顾问的客户发生并发症的真实概率。我们没有理由认为 \(p\) 恰好等于 \(\hat{p} = 3/62,\) ，但也没有理由认为 \(p\) 与 \(\hat{p} = 3/62.\) 相差特别大。通过从数据集中进行有放回抽样（这一过程称为自助法），可以近似得到可能的 \(\hat{p}\) 值的变异性。

本文中涵盖的大多数推断方法都基于量化来自同一总体的两个数据集之间的差异。从同一总体中重复抽样是没有意义的，因为如果你有条件抽取更多样本，更大的样本量会比分别评估两个完全相同大小的样本更有益。相反，我们测量样本在总体估计下的表现。

图 12.1 展示了如何通过使用样本来近似成功与失败的比例（在我们的例子中，即医疗顾问出现并发症和无并发症的比例）来估计未知的原始总体。

A small sample of 3 red marbles and 4 white marbles is taken from a large population with predominately unknown individual values. The sample is then replicated infinitely many times to create a proxy population where the values are known to be 3/7 red and 4/7 white.
图 12.1：使用观测样本数据来估计未知总体。请注意，我们可以利用样本创建一个用于抽样的估计总体（即自助法总体）。观测数据包含三颗红色弹珠和四颗白色弹珠，因此估计总体包含 3/7 的红色弹珠和 4/7 的白色弹珠。

通过从估计总体中重复抽样，可以观察到样本之间的变异性。在 图 12.2 中，重复的自助法样本显然彼此不同，也与原始总体不同。请记住，这些自助法样本取自同一个（估计的）总体，因此差异完全来自抽样过程中的自然变异性。

The same unknown large population is given with a sample of 3 red and 4 white marbles. After the proxy population is created (infinite replicates of the sample), new resamples of size 7 can be taken from the proxy population. Three resamples of size 7 are shown: Resample 1 has 2/7 red; Resample 2 has 4/7 red; and Resample k has 5/7 red.
图 12.2：自助法抽样提供了样本间变异性的度量。请注意，我们是从由观测数据创建的估计总体中抽样的。

通过对每个自助法样本进行汇总（这里使用样本比例），我们可以直接看到样本比例 \(\hat{p},\) 在样本之间的变异性。示例情景中 \(\hat{p}_{boot}\) 的分布显示在 图 12.3中，医疗顾问数据的完整自助法分布显示在 图 12.6.

The same unknown large population with a sample of 3 red and 4 white marbles; the proxy population which is infinite with 3/7 red marbles; and the k Resamples of size 7 are shown. From each of the resamples the bootstrapped proportion of red is calculated (shown as 2/7, 4/7, and 5/7). Many many resamples are taken and summarized in a dotplot of the bootstrapped proportions. The proportions range from 0/7 to 7/7 in a bell shape with the majority of bootstrapped proportions falling between 1/7 and 6/7.
中。

图 12.3：为每个自助法样本估计自助法比例。所得的自助法分布（点图）提供了比例在样本之间变化程度的度量

事实证明，在实践中，计算机很难处理无限总体（具有与样本相同的比例构成）。然而，有一种物理和计算方法，能够以计算高效的方式产生等价的样本比例自助法分布。 将观测数据视为一袋弹珠，其中 3 颗是成功（红色），4 颗是失败（白色）。通过有放回地从袋中取出弹珠，我们描绘了与使用无限大的估计总体时完全相同的抽样 过程

Shown is the unknown large population with a sample of 3 red and 4 white marbles. Without creating the infinitely large proxy population, resamples are taken from the original sample (by sampling with replacement from the sample). Three resamples of size 7 are shown: Resample 1 has 2/7 red; Resample 2 has 4/7 red; and Resample k has 5/7 red.
。
Top image includes the steps of (1) a large unknown population, (2) observed sample of size 7 (with 3 red and 4 white), (3) creation of an infinitely large proxy population, and (4) three resamples. (5) Many resamples are considered with a dotplot of bootstrapped proportions. The bottom image follows the same process without the infinitely large proxy population. That is, in the bottom image a (1) single sample is taken from the original population and (2) the three resamples are taken directly from the observed data (using sampling with replacement). (3) Again, many resamples are considered with a dotplot of bootstrapped proportions.
图12.5：从估计的无限总体中抽样与从原始样本中有放回地重抽样这两个过程的比较。注意，自助法比例的点图是相同的，因为估计统计量的过程是等价的。

如果我们将自助抽样过程应用于医疗顾问的例子，我们把每位客户视为袋中的一颗弹珠。袋中有59颗白色弹珠（无并发症）和3颗红色弹珠（有并发症）。如果我们从袋中抽取62颗弹珠（一次一颗，有放回），并计算模拟患者中出现并发症的比例， \(\hat{p}_{boot},\) 那么这个“自助”比例就代表了“从样本中重抽样”方法得到的一个模拟比例。

在一次62名患者的模拟中，我们预计大约会有多少人出现并发症？1

一次模拟不足以了解一个自助比例与另一个自助比例之间的变异性，因此我们使用计算机将模拟重复10,000次。

图 12.6 展示了10,000次自助模拟的分布。自助比例的变化范围大约从0到11.3%。自助比例的变异性使我们相信，并发症的真实概率（参数， \(p\)）很可能落在0%到11.3%之间的某个位置，因为这些数值涵盖了95%的自助重抽样值。

真实比例的取值范围称为 自助百分位置信区间，在接下来的几节和几章中我们还会再次见到它。

Histogram of 10,000 bootstrapped proportions. The bootstrap 2.5 percentile proportion is zero and the 97.5 percentile is 0.113.
图12.6：原始医疗顾问数据经过10,000次自助抽样。每次模拟都从原始数据中抽取一个样本，其中出现并发症的概率为 \(\hat{p} = 3/62.\) 自助2.5百分位比例为0，97.5百分位为0.113。结果是：我们有信心认为，在总体中，出现并发症的真实概率介于0%和11.3%之间。

最初的断言是该顾问的真实并发症发生率低于全国10%的水平。对于并发症真实概率0%到11.3%的区间估计，是否表明这位外科顾问的并发症发生率低于全国平均水平？请解释。

不能。因为该区间与10%重叠，所以有可能该顾问的工作与较低的并发症风险相关，也有可能与较高的并发症风险（即大于10%）相关！此外，如前所述，由于这是一项观察性研究，即使能够测量出关联，也没有证据表明该顾问的工作是并发症发生率（无论较高还是较低）的原因。

12.2 敲击者与听者案例研究

这里有一个你可以和朋友或家人一起玩的游戏：选一首简单而广为人知的歌曲，在桌上敲出曲调，看看对方能否猜出这首歌。在这个简单的游戏中，你是敲击者，对方是听者。

12.2.1 观测数据

斯坦福大学研究生伊丽莎白·牛顿（Elizabeth Newton）进行了一项使用敲击者-听者游戏的实验。2 在她的研究中，她招募了120名敲击者和120名听者参与研究。大约50%的敲击者预期听者能够猜出歌曲。牛顿想知道，50%是一个合理的预期吗？

在牛顿的研究中，120名听者中只有3人（\(\hat{p} = 0.025\)）能够猜出曲调！这个数字似乎相当低，这让研究者提出问题：能猜出曲调的人的真实比例是多少？

12.2.2 统计量的变异性

为了回答这个问题，我们将再次使用模拟。为了模拟120局游戏，这次我们使用一个装有120颗弹珠的袋子，其中3颗是红色的（代表猜对的人），117颗是白色的（代表没能猜出歌曲的人）。从袋中抽样120次（记住每次都要把弹珠放回袋中，以保持红色弹珠的总体比例不变）就产生了一个自助样本。

例如，我们可以先通过从装有3颗红色和117颗白色弹珠的袋子中抽取5颗弹珠，来模拟5对敲击者-听者。

W W W R W
错误 错误 错误 正确 错误

选出120颗弹珠后，我们数到2颗红色，对应于 \(\hat{p}_{boot1} = 0.0167.\) 正如我们在随机化技术中所做的那样，仅观察一次模拟的结果是不够的。为了了解观察到的比例0.025可能离真实参数有多远，我们应该生成更多次模拟。这里我们将整个模拟重复了十次：

\[0.0417 \quad 0.025 \quad 0.025 \quad 0.0083 \quad 0.05 \quad 0.0333 \quad 0.025 \quad 0 \quad 0.0083 \quad 0\]

和之前一样，我们将使用计算机总共运行10,000次模拟。如 图 12.7所示，95%的重抽样 \(\hat{p}_{boot}\) 值范围为0.000到0.0583。也就是说，我们预计真正能够猜出敲击者曲调的人的比例在0%到5.83%之间。

Histogram of 10,000 bootstrapped proportions. The bootstrap 2.5 percentile proportion is zero and the 97.5 percentile is 0.0583.
图12.7：原始的听者-敲击者数据被自助法重抽样了10,000次。每次模拟都会创建一个猜对概率为 \(\hat{p} = 3/120.\) 的样本。2.5百分位比例为0，97.5百分位为0.0583。结果是我们可以确信，在总体中，能够猜对的人的真实比例介于0%和5.83%之间。

这些数据是否提供了令人信服的证据，反驳50%的听者能猜出敲击者曲调的说法？3

12.3 置信区间

点估计为参数提供了一个单一的可能值。然而，点估计很少是完美的；通常估计中会存在一些误差。除了给出参数的点估计之外，下一个合乎逻辑的步骤是为该参数提供一个可能的 数值范围 。

12.3.1 总体参数的可能数值范围

总体参数的一个合理取值范围被称为 置信区间。仅使用单点估计就像在浑浊的湖里用鱼叉捕鱼，而使用置信区间则像用渔网捕鱼。我们可以在看到鱼的地方投出鱼叉，但很可能会错过。另一方面，如果我们在那个区域撒网，就很有机会捕到那条鱼。

如果我们报告点估计，我们很可能无法恰好命中总体参数。另一方面，如果我们报告一个合理取值范围——即置信区间——我们就有很大机会捕捉到该参数。

如果我们想非常有把握地捕捉到总体参数，应该使用更宽的区间（例如99%）还是更小的区间（例如80%）？4

12.3.2 自助法置信区间

如上所述， 自助样本 是原始样本的一个样本。以医疗并发症数据为例，我们按以下步骤进行：

  • 从62名患者中随机抽取一个观测值（把弹珠放回袋中，以保持总体不变）。
  • 从62名患者中随机抽取第二个观测值。由于我们是有放回抽样（即我们实际上并不把弹珠从袋中取出），第二个观测值有1/62的概率与第一步抽取的观测值相同！
  • 一次一个观测值地继续抽取……
  • 从62名患者中随机抽取第62个观测值。

自助抽样通常被称为 有放回抽样.

自助样本的表现类似于来自总体的真实样本，我们据此计算感兴趣的点估计（此处计算 \(\hat{p}_{boot}\)).

基于超出本文范围的理论，我们知道自助法比例 \(\hat{p}_{boot}\) 围绕 \(\hat{p}\) 的变化方式，类似于不同样本比例（即 \(\hat{p}\)的值）围绕真实参数 \(p.\) 的变化方式。因此，可以使用 \(p\) 值本身来生成 \(\hat{p}_{boot}\) 的区间估计。

参数的95%自助百分位置信区间 \(p.\)

参数 \(p\) 的95%自助置信区间可以直接使用排序后的 \(\hat{p}_{boot}\) 值得到。

考虑排序后的 \(\hat{p}_{boot}\) 值。将2.5%的自助比例值称为“下限”，将97.5%的自助比例值称为“上限”。

95%置信区间为：(下限, 上限)

在 第 16.1 节 我们将讨论置信区间的不同百分比（例如，90%置信区间或99%置信区间）。

第 第 16.1 节 还对“95% 置信度”的实际含义进行了更详细的讨论。

12.4 本章复习

12.4.1 小结

图 12.8 提供了创建自助法置信区间的可视化总结。

Full depiction of the bootstrapping process. Cartoon of the population and potential bootstrap samples is shown, each with a different bootstrap proportion. A dot plot of the bootstrapped proportions shows how they are naturally variable.
图 12.8：我们将使用有放回抽样来衡量目标统计量（此处为比例）的变异性。有放回抽样是一种计算工具，它等价于将样本作为对从中抽样的无限大总体的估计方式。

我们可以将自助法过程总结如下：

  • 将研究问题表述为对一个待估计的参数。 置信区间适用于旨在从总体中估计某个数值（称为参数）的研究问题。
  • 通过观察性研究或实验收集数据。 如果研究问题可以表述为关于参数的查询，我们就可以收集数据来计算一个统计量，它是我们对参数值的最佳猜测。然而，我们知道由于自然变异性，统计量不会恰好等于参数。
  • 通过将数据值作为总体的替代来对随机性进行建模。 为了评估统计量与参数之间可能相差多远，我们从数据集中反复重抽样，以衡量自助统计量的变异性。自助统计量围绕观测统计量的变异性（一个可以通过计算技术测量的量）应近似等于许多观测样本统计量围绕参数的变异性（一个很难测量的量，因为在现实生活中我们只能得到恰好一个样本）。
  • 创建区间。 在选定特定的置信水平后，利用自助统计量的变异性创建一个区间估计，希望能捕捉到真实参数。虽然与手头特定样本相关的区间估计可能捕捉到参数，也可能没有捕捉到，但研究者知道，在其一生中，置信水平将决定其研究置信区间中能够捕捉到真实参数的百分比。
  • 形成结论。 使用分析得到的置信区间，报告目标参数的区间估计。此外，务必用通俗易懂的语言撰写结论，以便普通读者能够理解结果。

表 12.1 是对自助法（bootstrap）过程总结的另一种视角。

表 12.1：自助法作为一种推断统计方法的总结。
问题 答案
它的作用是什么？ 从观测数据中进行有放回的重抽样，以模拟从总体中收集数据时出现的抽样变异性
所描述的随机过程是什么？ 从总体中随机抽样
还可以近似哪些其他随机过程？ 也可用于描述实验中的随机分配
它最适合用于什么？ 置信区间（也可用于单一比例的自助法假设检验）
什么实物可以代表这一模拟过程？ 有放回地从袋中抽取弹珠

12.4.2 术语

本章中介绍的术语列于 表 12.2。如果您不确定其中一些术语的含义，我们建议您回到正文中复习它们的定义。您应该能够很容易地发现它们是 粗体文本.

表 12.2：本章介绍的术语。
自助百分位置信区间 置信区间 有放回抽样
自助样本 参数 统计量
自助法 点估计

12.5 练习

奇数编号习题的答案见 附录 A.12.

  1. YouTube 之外的户外视频。 假设你想估计在户外拍摄的 YouTube 视频的比例（将“户外”定义为视频中任何部分在室外拍摄）。你随机抽取了 128 个 YouTube 视频5 并确定其中 37 个在户外拍摄。你想估计所有在户外拍摄的 YouTube 视频的比例，因此决定基于这 128 个视频的原始样本创建一个自助置信区间。

    1. 用文字描述这个问题中相关的统计量和参数。如果你知道其中任何一个的数值，请给出。如果不知道数值，请解释为什么该值是未知的。

    2. 分别用什么符号来表示统计量和参数？

    3. 如果使用软件对原始数据集进行自助法（bootstrap）抽样，每个自助样本上计算的统计量是什么？

    4. 在创建自助样本比例的自助抽样分布（直方图）时，直方图的中心应该位于何处？

    5. 该直方图给出了样本比例的自助抽样分布（1000次自助重复）。利用该直方图，估计在户外拍摄的YouTube视频比例的90%置信区间。

    6. 结合数据背景解释该置信区间。

  1. 慢性疾病。 2012年，皮尤研究基金会报告称“45%的美国成年人报告他们患有一种或多种慢性疾病”。然而，该数值基于一个样本，因此就其本身而言，可能并不是对所关注总体参数的完美估计。该研究基于3014名成年人的样本。以下是基于皮尤数据集的1000个自助样本比例的分布。 (皮尤研究中心 2013) 利用1000个自助比例的分布，近似求出患有一种或多种慢性疾病的美国成年人真实比例的92%置信区间，并对其进行解释。

  1. 社交媒体用户与新闻，自助法。 2022年开展的一项民意调查发现，50%的美国成年人有时或经常从社交媒体获取新闻。然而，该数值基于一个样本，因此就其本身而言，可能并不是对所关注总体参数的完美估计。该研究基于12147名成年人的样本。以下是基于皮尤数据集的1000个自助样本比例的分布。 (皮尤研究中心 2022) 利用1000个自助比例的分布，近似求出（2022年）至少从Twitter获取部分新闻的美国成年社交媒体用户真实比例的98%置信区间，并结合问题背景解释该区间。

  1. 的自助抽样（Bootstrap）分布 \(\hat{p}\), I. 以下四个分布分别由不同的数据集创建。每个数据集基于 \(n=23\) 个观测值。原始数据集的成功比例如下： \[\hat{p} = 0.13 \ \ \hat{p} = 0.22 \ \ \hat{p} = 0.30 \ \ \hat{p} = 0.43.\] 将每个直方图与原始数据的成功比例相匹配。

  1. 的自助抽样（Bootstrap）分布 \(\hat{p}\)，II。 以下四个分布分别由不同的数据集创建。每个数据集基于 \(n=23\) 个观测值。

    针对真实总体 \(p\) （成功比例）的以下每个取值进行考虑。数据集 A、B、C、D 各进行了 1000 次自助抽样，自助抽样比例如所给直方图所示。对于每个参数值，列出可能来自该总体的数据集。（提示：每个参数值可能对应多个数据集。）

    1. \(p = 0.05\)

    2. \(p = 0.25\)

    3. \(p = 0.45\)

    4. \(p = 0.55\)

    5. \(p = 0.75\)

  1. 的自助抽样（Bootstrap）分布 \(\hat{p}\)，III。 以下四个分布分别由不同的数据集创建。每个数据集具有相同的成功比例 \((\hat{p} = 0.4)\) ，但样本量不同。这四个数据集分别为 \(n = 10, 100, 500\)和 \(1000\).

    针对真实总体 \(p\) （成功比例）的以下每个取值进行考虑。数据集 A、B、C、D 各进行了 1000 次自助抽样，自助抽样比例如所给直方图所示。对于每个参数值，列出可能来自该总体的数据集。（提示：每个参数值可能对应多个数据集。）

    1. \(p = 0.05\)

    2. \(p = 0.25\)

    3. \(p = 0.45\)

    4. \(p = 0.55\)

    5. \(p = 0.75\)

  1. 网络欺凌率。 一项针对青少年的网络欺凌调查显示，54% 到 64% 的受访者报告曾遭受网络欺凌（95% 置信区间）。请根据该区间回答以下问题。 (皮尤研究中心（Pew Research Center）2018 年)

    1. 一家报纸声称大多数青少年都经历过网络欺凌。这一说法是否得到该置信区间的支持？请解释你的理由。

    2. 一位研究人员推测 70% 的青少年经历过网络欺凌。这一说法是否得到该置信区间的支持？请解释你的理由。

    3. 在不实际计算区间的情况下，判断 (b) 部分中研究人员的说法是否会得到 90% 置信区间的支持？

  1. 在急诊室等待。 急诊室 (ER) 平均等待时间的 95% 置信区间为 (128 分钟，147 分钟)。请根据该区间回答以下问题。

    1. 当地一家报纸声称该急诊室的平均等待时间超过 3 小时。这一说法是否得到该置信区间的支持？请解释你的理由。

    2. 该医院的医学院院长声称平均等待时间为 2.2 小时。这一说法是否得到该置信区间的支持？请解释你的理由。

    3. 在不实际计算区间的情况下，判断 (b) 部分中院长的说法是否会得到 99% 置信区间的支持？

  1. 在模拟中，大约4.8%的患者（平均3人）会出现并发症，因为这是样本中所观察到的情况。不过，每次模拟之间会看到一些细微的变化。↩︎

  2. 本案例研究描述于 《让创意更有黏性》（Made to Stick） 一书，作者为奇普·希思（Chip Heath）和丹·希思（Dan Heath）。一个鲜为人知的事实是：许多OpenIntro资源背后的教学原则正是基于 《让创意更有黏性》（Made to Stick）.↩︎

  3. 由于50%并不在真实参数的区间估计之内，我们可以说，有令人信服的证据反对“50%的听众能猜出曲调”这一假设。此外，50%与最大重抽样统计量之间有相当大的距离，这表明有 非常 令人信服的证据反对这一假设。↩︎

  4. 如果我们想更有把握捕到鱼，可以使用更宽的渔网。同样，如果我们想更有把握捕获参数，就使用更宽的置信区间。↩︎

  5. 实现YouTube视频随机选择的方法有很多种，但并不清楚它们有多“随机”。↩︎