5.1 Scatterplots for paired data

A scatterplot provides a case-by-case view of data for two numerical variables. In Figure 1.2, a scatterplot was used to examine the homeownership rate against the percentage of housing units that are in multi-unit structures (e.g., apartments) in the county dataset. Another scatterplot is shown in Figure 5.1, comparing the total income of a borrower total_income and the amount they borrowed loan_amount for the loan50 dataset. In any scatterplot, each point represents a single case. Since there are 50 cases in loan50, there are 50 points in Figure 5.1.

Looking at Figure 5.1, we see that there are many borrowers with income below $100,000 on the left side of the graph, while there are a handful of borrowers with income above $250,000.

5.2 Dot plots and the mean

Sometimes we are interested in the distribution of a single variable. In these cases, a dot plot provides the most basic of displays. A dot plot is a one-variable scatterplot; an example using the interest rate of 50 loans is shown in Figure 5.3.

The mean, often called the average is a common way to measure the center of a distribution of data. To compute the mean interest rate, we add up all the interest rates and divide by the number of observations.

The sample mean is often labeled \(\bar{x}.\) The letter \(x\) is being used as a generic placeholder for the variable of interest and the bar over the \(x\) communicates we are looking at the average interest rate, which for these 50 loans is 11.57%. It’s useful to think of the mean as the balancing point of the distribution, and it’s shown as a triangle in Figure 5.3.

The loan50 dataset represents a sample from a larger population of loans made through Lending Club. We could compute a mean for the entire population in the same way as the sample mean. However, the population mean has a special label: \(\mu.\) The symbol \(\mu\) is the Greek letter mu and represents the average of all observations in the population. Sometimes a subscript, such as \(_x,\) is used to represent which variable the population mean refers to, e.g., \(\mu_x.\) Oftentimes it is too expensive to measure the population mean precisely, so we often estimate \(\mu\) using the sample mean, \(\bar{x}.\)

The mean is useful because it allows us to rescale or standardize a metric into something more easily interpretable and comparable. Suppose we would like to understand if a new drug is more effective at treating asthma attacks than the standard drug. A trial of 1,500 adults is set up, where 500 receive the new drug, and 1000 receive a standard drug in the control group. Results of this trial are summarized in Table 5.1.

Comparing the raw counts of 200 to 300 asthma attacks would make it appear that the new drug is better, but this is an artifact of the imbalanced group sizes.

Instead, we should look at the average number of asthma attacks per patient in each group:

• New drug: \(200 / 500 = 0.4\) asthma attacks per patient
• Standard drug: \(300 / 1000 = 0.3\) asthma attacks per patient

The standard drug has a lower average number of asthma attacks per patient than the average in the treatment group.

5.3 Histograms and shape

Dot plots show the exact value for each observation. They are useful for small datasets but can become hard to read with larger samples. Rather than showing the value of each observation, we prefer to think of the value as belonging to a bin. For example, in the loan50 dataset, we created a table of counts for the number of loans with interest rates between 5.0% and 7.5%, then the number of loans with rates between 7.5% and 10.0%, and so on. Observations that fall on the boundary of a bin (e.g., 10.00%) are allocated to the lower bin. The tabulation is shown in Table 5.2, and the binned counts are plotted as bars in Figure 5.4 into what is called a histogram. Note that the histogram resembles a more heavily binned version of the stacked dot plot shown in Figure 5.3.

Histograms provide a view of the data density. Higher bars represent where the data are relatively more common. For instance, there are many more loans with rates between 5% and 10% than loans with rates between 20% and 25% in the dataset. The bars make it easy to see how the density of the data changes relative to the interest rate.

Histograms are especially convenient for understanding the shape of the data distribution. Figure 5.4 suggests that most loans have rates under 15%, while only a handful of loans have rates above 20%. When the distribution of a variable trails off to the right in this way and has a longer right tail, the shape is said to be right skewed.⁵

Figure 5.5 shows a density plot which is a smoothed out histogram. The technical details for how to draw density plots (precisely how to smooth out the histogram) are beyond the scope of this text, but you will note that the shape, scale, and spread of the observations are displayed similarly in a histogram as in a density plot.

Variables with the reverse characteristic – a long, thinner tail to the left – are said to be left skewed. We also say that such a distribution has a long left tail. Variables that show roughly equal trailing off in both directions are called symmetric.

In addition to looking at whether a distribution is skewed or symmetric, histograms can be used to identify modes. A mode is represented by a prominent peak in the distribution. There is only one prominent peak in the histogram of interest_rate.

A definition of mode sometimes taught in math classes is the value with the most occurrences in the dataset. However, for many real-world datasets, it is common to have no observations with the same value in a dataset, making this definition impractical in data analysis.

Figure 5.6 shows histograms that have one, two, or three prominent peaks. Such distributions are called unimodal, bimodal, and multimodal, respectively. Any distribution with more than two prominent peaks is called multimodal. Notice that there was one prominent peak in the unimodal distribution with a second less prominent peak that was not counted since it only differs from its neighboring bins by a few observations.

Looking for modes isn’t about finding a clear and correct answer about the number of modes in a distribution, which is why prominent is not rigorously defined in this book. The most important part of this examination is to better understand your data.

5.4 Variance and standard deviation

The mean was introduced as a method to describe the center of a variable, and variability in the data is also important. Here, we introduce two measures of variability: the variance and the standard deviation. Both of these are very useful in data analysis, even though their formulas are a bit tedious to calculate by hand. The standard deviation is the easier of the two to comprehend, as it roughly describes how far away the typical observation is from the mean.

We call the distance of an observation from its mean its deviation. Below are the deviations for the \(1^{st},\) \(2^{nd},\) \(3^{rd},\) and \(50^{th}\) observations in the interest_rate variable:

\[ \begin{aligned} x_1 - \bar{x} &= 10.9 - 11.57 = -0.67 \\ x_2 - \bar{x} &= 9.92 - 11.57 = -1.65 \\ x_3 - \bar{x} &= 26.3 - 11.57 = 14.73 \\ &\vdots \\ x_{50} - \bar{x} &= 6.08 - 11.57 = -5.49 \\ \end{aligned} \]

If we square these deviations and then take an average, the result is equal to the sample variance, denoted by \(s^2\):

\[ s^2 = \frac{(-0.67)^2 + (-1.65)^2 + (14.73)^2 + \cdots + (-5.49)^2}{50 - 1} = \frac{0.45 + 2.72 + \cdots + 30.14}{49} = 25.52 \]

We divide by \(n - 1,\) rather than dividing by \(n,\) when computing a sample’s variance. There’s some mathematical nuance here, but the end result is that doing this makes this statistic slightly more reliable and useful.

Notice that squaring the deviations does two things. First, it makes large values relatively much larger. Second, it gets rid of any negative signs.

The standard deviation is defined as the square root of the variance:

\[s = \sqrt{25.52} = 5.05\]

While often omitted, a subscript of \(_x\) may be added to the variance and standard deviation, i.e., \(s_x^2\) and \(s_x^{},\) if it is useful as a reminder that these are the variance and standard deviation of the observations represented by \(x_1,\) \(x_2,\) …, \(x_n.\)

Like the mean, the population values for variance and standard deviation have special symbols: \(\sigma^2\) for the variance and \(\sigma\) for the standard deviation.

In practice, the variance and standard deviation are sometimes used as a means to an end, where the “end” is being able to accurately estimate the uncertainty associated with a sample statistic. For example, in Chapter 13 the standard deviation is used in calculations that help us understand how much a sample mean varies from one sample to the next.

5.5 Box plots, quartiles, and the median

A box plot summarizes a dataset using five statistics while also identifying unusual observations. Figure 5.8 provides a dot plot and a box plot of the interest_rate variable from the loan50 dataset.⁹

The dark line inside the box represents the median, which splits the data in half. 50% of the data fall below this value and 50% fall above it. Since in the loan50 dataset there are 50 observations (an even number), the median is defined as the average of the two observations closest to the \(50^{th}\) percentile. Table 5.3 shows all interest rates, arranged in ascending order. We can see that the \(25^{th}\) and the \(26^{th}\) values are both 9.93, which corresponds to the thick line in Figure 5.8 (b).

When there are an odd number of observations, there will be exactly one observation that splits the data into two halves, and in such a case that observation is the median (no average needed).

The second step in building a box plot is drawing a rectangle to represent the middle 50% of the data. The length of the box is called the interquartile range, or IQR for short. It, like the standard deviation, is a measure of variability in data. The more variable the data, the larger the standard deviation and IQR tend to be. The two boundaries of the box are called the first quartile (the \(25^{th}\) percentile, i.e., 25% of the data fall below this value) and the third quartile (the \(75^{th}\) percentile, i.e., 75% of the data fall below this value), and these are often labeled \(Q_1\) and \(Q_3,\) respectively.

Extending out from the box, the whiskers attempt to capture the data outside of the box. The whiskers of a box plot reach to the minimum and the maximum values in the data, unless there are points that are considered unusually high or unusually low, which are identified as potential outliers by the box plot. These are labeled with a dot on the box plot. The purpose of labeling the outlying points – instead of extending the whiskers to the minimum and maximum observed values – is to help identify any observations that appear to be unusually distant from the rest of the data. There are a variety of formulas for determining whether a particular data point is considered an outlier, and different statistical software use different formulas. A commonly used formula is that any observation beyond \(1.5\times IQR\) away from the first or the third quartile is considered an outlier. In a sense, the box is like the body of the box plot and the whiskers are like its arms trying to reach the rest of the data, up to the outliers.

5.6 Robust statistics

How are the sample statistics of the interest_rate dataset affected by the observation, 26.3%? What would have happened if this loan had instead been only 15%? What would happen to these summary statistics if the observation at 26.3% had been even larger, say 35%? The three conjectured scenarios are plotted alongside the original data in Figure 5.9, and sample statistics are computed under each scenario in Table 5.4.

The median and IQR are called robust statistics because extreme observations have little effect on their values: moving the most extreme value generally has little influence on these statistics. On the other hand, the mean and standard deviation are more heavily influenced by changes in extreme observations, which can be important in some situations.

You might not be surprised that the answer to the question “which is better, the mean or the median?” is: it depends. The two statistics measure different things, and so their use is dependent on the context in the analysis. Consider the following scenarios:

• Is it better to measure the average profit per customer or the median profit per customer?
  • If concern is about the overall profit margin of the company, the mean is a better measure to assess what is happening across the company. The company could have a positive median profit per customer and still be unprofitable.
  • If concern is around understanding the profit per typical customer, possibly to understand the growth headroom for the company’s profit, the median profit per customer would tell you more about individual customer profits.
• If you operate an app and want to know how long it takes for the app to open on your customers’ phones, do you want the mean amount of time or the median amount of time?
  • The mean leads to an understanding of the overall amount of time being wasted in opening the app.
  • The median tells you about the typical user experience.
  • However, if the app takes less than 5 milliseconds to launch for 50% of the users but more than 10 seconds to launch for 10% of the users, the median doesn’t give the information you need. In that scenario, you might want an upper percentile, like the 95th percentile.

Regardless of the choice of centrality statistic (either mean or median), for most analyses, it is important to consider more than just the centrality. Other statistics like upper and lower percentiles, IQR, or standard deviation provide information about the variability of the observations. And visualizing the data through a graphical representation will typically provide a wealth of information necessary for understanding the full data picture associated with the research question.

5.7 Transforming data

When data are very strongly skewed, we sometimes transform them, so they are easier to model. Figure 5.10 (a) and Figure 5.10 (b) show right-skewed distributions: distribution of the percentage of unemployed people and the distribution of the population in all counties in the United States. The distribution of population is more strongly skewed than the distribution of percentage unemployed, hence the log transformation results in a much bigger change in the shape of the distribution.

There are some standard transformations that may be useful for strongly right skewed data where much of the data is positive but clustered near zero. A transformation is a rescaling of the data using a function. For instance, a plot of the logarithm (base 10) of unemployment rates and county populations results in the new histograms in Figure 5.10 (b). The transformed data are symmetric, and any potential outliers appear much less extreme than in the original dataset. By reigning in the outliers and extreme skew, transformations often make it easier to build statistical models for the data.

Transformations can also be applied to one or both variables in a scatterplot. A scatterplot of the population change from 2010 to 2017 against the population in 2010 is shown in Figure 5.11 (a). It’s difficult to decipher any interesting patterns because the population variable is so strongly skewed. However, if we apply a log\(_{10}\) transformation to the population variable, as shown in Figure 5.11 (b), a positive association between the variables is revealed. In fact, we may be interested in fitting a trend line to the data when we explore methods around fitting regression lines in Chapter 7.

Transformations other than the logarithm can be useful, too. For instance, the square root \((\sqrt{\text{original observation}})\) and inverse \(\bigg ( \frac{1}{\text{original observation}} \bigg )\) are commonly used by data scientists. Common goals in transforming data are to see the data structure differently, reduce skew, assist in modeling, or straighten a nonlinear relationship in a scatterplot.

5.8 Mapping data

The county dataset offers many numerical variables that we could plot using dot plots, scatterplots, or box plots, but they can miss the true nature of the data as geographic. When we encounter geographic data, we should create an intensity map, where colors are used to show higher and lower values of a variable. Figure 5.12 shows intensity maps for poverty rate in percent (poverty), unemployment rate in percent (unemployment_rate), homeownership rate in percent (homeownership), and median household income in $1000s (median_hh_income). The color key indicates which colors correspond to which values. The intensity maps are not generally very helpful for getting precise values in any given county, but they are very helpful for seeing geographic trends and generating interesting research questions or hypotheses.

5.9 Chapter review

5.10 Exercises

Answers to odd-numbered exercises can be found in Appendix A.5.