The Box-Percentile Plot

Warren W. Esty, Jeffrey D. Banfield and Frank E. Harrell

Abstract

A variant of the boxplot is proposed in which the sides contain the information of a percentile plot (which is equivalent to the empirical cumulative distribution function). Unlike boxplots, there is no question about how long to draw the whiskers, nor is there loss of information due to grouping. Side-by-side comparisons of distributions are especially effective. In spite of including more detail, the impact on statistically-untrained readers remains similar to that of traditional boxplots.

Keywords: Boxplot, Distribution, Graphical Data Analysis, Tree Invasions.

1 Introduction.

Consider the question of how to graphically communicate the information in several related sets of continuously-distributed data. This article introduces the box-percentile plot, a modified version of the boxplot that has two distinct advantages over previous versions. One is that there are no questions about how it should be drawn – the plot configuration is justified by the properties of empirical distributions and does not require arbitrary choices about box configuration or whisker length. The other is that it includes details which are useful to statistically-trained readers without weakening the impact on statistically-untrained readers – it is not necessary to choose between showing general behavior or detail (Tukey, 1977). The box-percentile plot can be used as both an exploratory tool and a space-saving method of publishing information in non-statistical scholarly articles which otherwise might take two or more graphs to convey.

Unlike the boxplot, which uses width only to emphasize the middle 50 per cent of the data, the box-percentile plot uses width to encode information about the distribution of the data over the entire range of data values. Figure 1 plots the highest points in 50 states and the heights of 219 volcanos (the same data used by Tukey for exhibit 5 of his Chapter 2) using box-percentile plots and standard boxplots. It is clear that the box-percentile plots convey the same graphical impression as the boxplots and contain additional information about the shape of the distributions. We will look at these plots in more detail in Section 3.2.

The idea behind constructing a box-percentile plot is simple. At any height the width of the irregular “box” is proportional to the percentile of that height, up to the 50th percentile, and above the 50th percentile the width is proportional to 100 minus the percentile. Thus, the width at any given height is proportional to the percent of observations that are more extreme in that direction. As in boxplots, the median, 25th, and 75th percentiles are marked with line segments across the box. Other percentiles may be emphasized if desired.

To illustrate the effectiveness of the additional information contained in the box-percentile plot (compared to the boxplot), consider the three artificial data sets shown as histograms in Figure 2. The distributions are clearly different. The first data set (Fig. 2A) is from a standard normal distribution; the second data set (Fig. 2B) is from a uniform distribution with an extreme outlier on each end, and the third data set (Fig. 2C) comes from a tri-modal distribution. Their boxplots are indistinguishable, but Figure 2D shows the box-percentile plots allow the observer to distinguish and identify the three distributions. We have obviously constructed this data to emphasize a strength of the box-percentile plot, however, in section 3 we will use real data sets to further illustrate how the information provided by box-percentile plots can be used to solve problems that boxplots are unable to address.

1.1 Background

Tukey (1990) reminds us of the significance of impact, the distinction between prospecting and transfer of information, and the importance of recognizing the purpose to be served. Side-by-side boxplots indicate both the general magnitude of the observations in each set and permit rough comparisons between the sets. Even statistically-untrained readers can quickly grasp the basic distributional information displayed by the plot. The typical values and the general differences in the distribution of the values from set to set are easy to see. Then, only a small amount of statistical training is needed to be able to properly interpret the information that the quartiles and extreme values provide about the dispersion of the data. Thus, for communicating data, especially to scholarly but statistically-untrained readers, boxplots are often very effective and have become a standard graphical tool in many sciences.

Given the effectiveness of boxplots at communicating a few salient distributional details, the question naturally arises if boxplots can be modified to convey more information to the statistically-trained reader without confusing the picture for the uninitiated. Width has been used to encode information about sample size or confidence intervals (McGill, Tukey, and Larson). Shading has also been used to indicate confidence intervals (Benjamini). Another modification is to use width to indicate an estimated probability density. Unfortunately, the estimated probability density depends strongly on the method used to estimate it, so one set of data could yield widely varying plots (Benjamini). Variants of the boxplot without an actual box have also been proposed (Tufte, Tukey (1977), Tukey (1990)). So far, none of these variants have found their way into common usage.

If the underlying principle of the basic, unmodified, boxplot method of visual presentation is accepted, there remains the question of precisely how to construct the plot (Frigge, Hoaglin, and Iglewicz). Boxplots emphasize a few key percentiles and mark them on the vertical scale. Horizontal marks are used primarily to emphasize the location of the middle 50 per cent of the data. They group data in the middle half into quartiles and group much of the rest into whiskers. Problems with boxplot construction are that the length of the whiskers and the treatment of extreme values are somewhat arbitrarily determined and no one definition has been accepted by all users. One way is to draw whiskers out to the 10th and 90th percentiles and mark each of the more extreme observations individually (Cleveland). In other versions the whiskers extend to the most extreme values (Tukey (1977) p.40, Tufte), or you may use discretion to chose how many extreme values to indicate separately, in which case the whiskers extend out to them (Tukey (1977) p.41). Still another version (Tukey’s “schematic plot” (1977) p.44 and 47f.; Chambers, Cleveland, Kleiner, and Tukey) uses a far more complicated method to determine the lengths of the whiskers. The definitions of “steps” and “adjacent values” is this context are arbitrary. Recently Lee and Tu (1997) have developed a general graphical tool, the BLiP plot, which can produce many variations (34 options) on most standard distributional graphs, including boxplots. In the examples in their paper they use the .025 and .975 quantiles as the boundary for the whiskers in their boxplots. There is no general agreement as to what constitutes the basic boxplot. Thus when you see a boxplot you must scrutinize the accompanying text to determine how the plot has been constructed. The box-percentile plot eliminates the need to make these arbitrary choices.

Neither the boxplot nor the box-percentile plot is intended as a substitute for plots of estimated probability density functions. They do not plot densities and cannot be interpreted with the sophisticated (integral calculus) idea that area represents observations. The variable width BLiP plot (Lee and Tu, 1997) uses a simple central difference estimator of the density function to produce graphs that look similar to box-percentile plots but the encoded information and the interpretation is very different. As with all density estimates, the variable width BLiP plot requires a smoothing parameter and, depending on the choice of the parameter, can have several different representations of the same data. All distributional graphs that are based on density esitmates entail arbitrary choices, and perhaps the most appealing feature of box-percentile plots is that they do not require such choices.

There are other types of diagrams which effectively display individual data sets without grouping and without questions about how they should be defined. In particular, percentile plots and empirical cumulative distribution functions (cdf) display all the data and have straightforward definitions. For comparing several data sets, however, neither plot is suitable. If the graphs for several data sets are side-by-side the critical horizontal comparison is difficult to make, although summary lines can help (Cleveland, Fig. 3.19).

Plotting several data sets on the same percentile or empirical cdf plot may lead to problems in detection and evaluation of differences between curves (Cleveland, Fig. 3.75 and 1.5).

2 The Box-Percentile Plot

The box-percentile plot combines the virtues of boxplots (ease of interpretation, ability to compare several data sets simultaneously) with those of percentile plots (display all the data, no arbitrary choices in construction). The idea is to use the width (as in the boxplot) to emphasize the middle of the data and to continue to use width (but not in an arbitrary fashion) to give less emphasis to the more extreme data (as whiskers and outliers are given less emphasis in boxplots). Thus the box-percentile plot “boxes” are wide in the middle (like boxplot boxes), narrow away from the middle (as are whiskers) and very narrow at the extremes. Unlike boxplots, the width contains precise information about the distribution of the data. They contain all the information of percentile plots and permit an easy and accurate assessment of symmetry. Also, since the data are not grouped, grouping can never conceal significant information as it would in some examples (Tukey’s (1977) weight of nitrogen example, p.50).

2.1 Construction

Let the number of observations be n and the observed values be ordered lowest-to-highest as y_{(1)}, y_{(2)}, ..., y_{(n)}. Each y-value is plotted as a distinct point, so no information is lost. Let the desired maximum width of the box be w. If the data is a random sample, under rather general conditions it can be proved that the expected probability between the i^{th} and (i + 1)^{st} order statistic is 1/(n + 1), so in percentile plots the data-point y_{(k)} is marked at height y_{(k)} above the horizontal coordinate k/(n + 1). In box-percentile plots the data-point y_{(k)} is marked at height y_{(k)} at distance kw/(n + 1) on either side of a vertical axis of symmetry, if y_{(k)} is less than or equal to the median. If y_{(k)} is greater than the median, it is plotted at height y_{(k)} at distance (n+1−k)w/(n+1) on either side of the vertical axis of symmetry. If the data is a population rather than a sample, then division by n + 1 is inappropriate. In that case y_{(k)} could be plotted at distance (k − 1)w/(n − 1) from the axis for y_{(k)} less than or equal to the median and at distance (n − k)w/(n − 1) if y_{(k)} is above the median.

3 Examples

3.1 Simulated Data

Box-percentile plots not only facilitate comparison of a few key percentiles, as is the case with boxplots, they also permit comparison of complete distributions. However, if one wishes to ignore the additional information contained in box-percentile plots they can be used in the same manner as boxplots with no additional effort on the part of the reader. On the other hand, if one wishes to use the additional information provided by box-percentile plots then a small amount of training is needed. This subsection uses simulated data to help familiarize the reader with some of the common patterns that emerge in box-percentile plots. The following subsections will illustrate the use of box-percentile plots with real data sets.

Figures 2A, B, and C show the histograms of three simulated datasets and Figure 2D shows the corresponding box-percentile plots. There are 500 observations in each sample so the plots have settled down to a fairly stable shape.

The data set in Figure 2A is from a normal distribution with mean 0 and variance 1. The corresponding box-percentile plot in Figure 2D is typical of a normal box-percentile plot. There is a single mode (no flat vertical lines) that occurs at the median. The plot is vertically symmetric about the median and the sides of the box are convex. The data in Figure 2B is from a uniform distribution with two single outliers, one in each direction. The corresponding box-percentile plot in Figure 2D indicates there are outliers and the main body of the data is uniformly distributed.

Outliers cause a long, thin line leading from the main body of the plot to the outliers. If there are several outliers in one direction, the “arm” of the box-percentile plot may have some width but they are still easily identifiable as unusual values that may not belong to the main body of data. Figure 3 shows a normal data set with several outliers in one direction. Notice how the main body of the box-percentile plot has the typical shape of a normal distribution while the narrow arm leading to the outliers easily identifies that set of points as outliers, without an arbitrary definition of “outlier.” With regular boxplots, if some of the observations are regarded as outliers, extending the whiskers out to the extreme values may give a misleading impression of where the data lie so outliers are generally marked individually. This is not a problem with box-percentile plots. However, if one wishes to define outliers and then emphasize them, the box-percentile plot can be modified to do this.

The other feature of the box-percentile plot of the uniform data with outliers in Figure 2D is the “diamond” shape of the main body of the plot. This is typical of a uniform distribution (the percentile plot of a uniform distribution is linear). Compare the box-percentile plot of the uniform data with outliers from Figure 2D with the box-percentile plot of normal data with outliers in Figure 3. It is easy to distinguish between the distributions of the main body of the data, even in the presence of outliers.

The tri-modal box-percentile plot in Figure 2D illustrates the typical feature of a multimodal distribution, vertical lines in the outline of the box. The “valleys” between modes have few observations relative to the “peaks”, so there is little change in the percentiles in those regions which translates into flat, near vertical lines.

Figure 4 shows the histogram and box-percentile plot of a lognormal data set. This illustrates the typical pattern of a box-percentile plot for skewed data.

3.2 Volcanos

Figure 1 shows box-percentile plots and boxplots for two datasets, the highest point in each of the 50 states and the heights of 219 volcanos. In the boxplots it appears that the states data is skewed toward the higher values and there are a few outliers in the volcano data. The boxplots give no detailed information, other than location, about how these two datasets are related.

The box-percentile plots provide a far more informative view of the data. We can see from the box-percentile plot that the states data is bimodal rather than skewed. The shorter heights appear somewhat uniformly distributed between 0 and about 8000 feet. There is another group of states with heights between 13000 and 15000 feet and a single outlier (Alaska) at 21000 feet. Note that the boxplot for the states data gives the impression that there may be several states with maximum heights between 8000 and 11000 feet (the upper part of the box) while the box-percentile plot clearly shows that this is not the case.

The box-percentile plot of the volcano data suggests that the outliers shown in the boxplot are really the tail of a skewed distribution. The shape of the box-percentile plot for the volcano data indicates it may be lognormal but not as extremely skewed as the data shown in Figure 4.

3.3 Tree Invasions

Box-percentile plots provide a means to easily compare distributions and, as any good exploratory tool should, they can lead researchers to ask new questions about the data. A tree invasion is encroachment of trees into a region where they have not traditionally grown. This is a serious problem in many parts of the world due to the loss of farming and grazing land. Figure 5A shows several data sets from a study of tree invasions in the upper Madison valley of Montana (Hansen, Wyckoff, and Banfield, 1995). A sample of trees were cored (a small part of the trunk is removed so the age of the tree can be determined by counting the tree rings) at several different invasion sites to determine when invasions occurred. Invasions show up as modes in the age distribution of the trees.

One of the more noticeable features of Figure 5A is the difference between the shapes of the plots which indicate different invasion histories. The trees at site 1 have a fairly uniform age distribution indicating a constant increase in the number of trees. In contrast, site 2 had an invasion (sudden increase in trees) in the early 1900’s while site 4 had a strong invasion during the mid to late 1930’s. Sites 5 and 6 have similarly shaped distributions that start at different times. Could there be something about the morphology of the land that could cause similar patterns? Sites 4 and 7 have similiar distributions until the strong invasion at site 4 in the late 1930’s during which time site 7 had virtually no increase in trees germinating. Is there a difference between the fire or grazing histories of these two sites during that period? Also, the shape of the distributions for sites 4 and 7 after 1940 is different, the rate of germination at site 4 is smaller than that at site 7 (the width of the box is not changing as fast for site 4), could this indicate the sudden influx of trees at site 4 during the 1930’s is suppressing the germination of new trees in the years that follow?

Box-percentile plots are, of course, not the only graphical tool that should be used in the exploration and analysis of this data. They are, however, a powerful tool for an initial view of what is occuring, in terms of tree invasions, over a large geographic area. In the hands of geographers, familiar with the potential causes of tree invasions, climate data, and grazing histories of the sites, the differences in the age distributions that can be seen and compared across sites using box-percentile plots provide insights and comparisons that are simply not available with other methods.

4 Conclusion

The box-percentile plot is not an all-purpose graph, but it does everything the boxplot does, and more, without being more difficult to interpret. And, fortunately, its construction is based on principles of mathematical statistics, not on arbitrary rules. There is no question about how long to draw the whiskers or how to plot the outliers. Thus this new type of graph not only has a visual impact which provides incisive comparisons, it presents the information in a statistically justified manner.

The authors have developed a set of S functions for constructing and plotting box-percentile plots. The software is available from the authors or by sending email, containing the single line send bpplot from S, to statlib@lib.stat.cmu.edu . A listing of all software available from Statlib may be obtained by sending the message send index to the same address.

5 References

Benjamini, Y. (1988), “Opening the box of a boxplot.” The American Statis-

tician, 42.4, 257-262. Chambers, J., W. Cleveland, B. Kleiner, and P. A. Tukey (1983) Graphical Methods of Data Analysis, Belmont, CA: Wadsworth International Group.

Cleveland, W. (1985) The Elements of Graphing Data, Monterey, CA: Wadsworth

Advanced Books and Software. Frigge, M., D. Hoaglin, and B. Iglewicz (1989) “Some implementations of the

boxplot.” The American Statistician, 43.1, 50-54. Hansen, K., W. Wyckoff, and J. Banfield (1995) “Historical Grazing and For-

est Invasion in Southwestern Montana.” Journal of Forest and Conservation History, in press.

Lee, J. Jack and Tu, Z. Nora (1997) “A Versatile One-Dimensional Distribu-

tion Plot: The PLiP Plot.” The American Statistician, 51.4, 353-358. McGill, R., J.W. Tukey, and W. Larsen (1978) “Variations of boxplots.” The

American Statistician, 32.1, 12-16. Tufte, E. (1983) The Visual Display of Quantitative Information, Cheshire,

CT: Graphics Press. Tukey, J.W. (1977) Exploratory Data Analysis, Reading, MA: Addison Wes-

ley Tukey, J.W. (1990) “Data-based graphics: Visual display in the decades to come.” Statistical Science, 5.3. 327-339.

18