Quenouille (1949) invented the jackknife technique. The purpose of this was to reduce bias by looking at multiple samples of data in a methodical way. The name jackknife seems to have been coined by the well-known statistician John W. Tukey. Due mainly to the lack of computational power, the advances and utility of the jackknife method were restricted. Efron invented the bootstrap method in 1979 (see the following section for its applications) and established the connection with the jackknife method. In fact, these two methods have a lot in common and are generally put under the umbrella of resampling methods.

Suppose that we draw a random sample of size n from a probability distribution F, and we denote by the parameter of interest. Let be an estimator of , and here we don't have the probability distribution of for a given . Resampling methods will help in carrying out statistical inference when the probability distribution is unknown. A formal definition...

In this section, we will explore complex statistical functional. What is the statistical distribution of the correlation between two random variables? If normality assumption does not hold for the multivariate data, then what is an alternative way to obtain the standard error and confidence interval? Efron (1979) invented the bootstrap technique, which provides the solutions that enable statistical inference related to complex statistical functionals. In Chapter 1, Introduction to Ensemble Techniques, the permutation test, which repeatedly draws samples of the given sample and carries out the test for each of the resamples, was introduced. In theory, the permutation test requires number of resamples, where m and n are the number of observations in the two samples, though one does take their foot off the pedal after having enough resamples. The bootstrap method works in a similar way and is an important resampling method.

Let be an independent random...

The boot package is one of the core R packages, and it is optimized for the implementation of bootstrap methods. In the previous examples, we mostly used loops for carrying out the resampling technique. Here, we will look at how to use the boot R package.

The main structure of the boot function is as follows:

boot(data, statistic, R, sim = "ordinary", stype = c("i", "f", "w"), 
     strata = rep(1,n), L = NULL, m = 0, weights = NULL, 
     ran.gen = function(d, p) d, mle = NULL, simple = FALSE, ...,
     parallel = c("no", "multicore", "snow"),
     ncpus = getOption("boot.ncpus", 1L), cl = NULL)

The central arguments of the function are data, statistic, R, and stype. The data argument is the standard one, as with most R functions. The statistic is the most important argument for the implementation of the boot function and it is this function that will be applied on the bootstrap samples obtained...

We begin the bootstrap hypothesis testing problems with the t-test to compare means and the F-test to compare variances. It is understood that, since we are assuming normal distribution for the two populations under comparison, the distributional properties of the test statistics are well known. To carry out the nonparametric bootstrap for the t-statistic based on the t-test, we first define the function, and then run the bootstrap function boot on the Galton dataset. The Galton dataset is available in the galton data.frame from the RSADBE package. The galton dataset consists of 928 pairs of observations, with the pair consisting of the height of the parent and the height of their child. First, we define the t2 function, load the Galton dataset, and run the boot function as the following unfolds:

> t2 <- function(data,i) {
+   p <- t.test(data[i,1],data[i,2],var.equal=TRUE)$statistic
+   p
+ }
> data(galton)
> gt <- boot(galton,t2,R=100...

The US Crime dataset introduced in Chapter 1, Introduction to Ensemble Techniques, is an example of why the linear regression model might be a good fit. In this example, we are interested in understanding the crime rate (R) as a function of thirteen related variables such as average age, the southern state indicator, and so on. Mathematically, the linear regression model is as follows:

Here, are the p-covariates, is the intercept term, are the regression coefficients, and is the error term assumed to follow a normal distribution . The covariates can be written in a vector form and the ith observation can be summarized as , where . The n observations , are assumed to be stochastically independent. The linear regression model has been detailed in many classical regression books; see Draper and Smith (1999), for instance. A recent book that details the implementation of the linear regression model in R is Ciaburro (2018). As the reader might have guessed...

In the first section, we looked at the role of pseudovalues in carrying out inference related to survival data. The main idea behind the use of pseudovalues is to replace the incomplete observations with an appropriate (expected) value and then use the flexible framework of the generalized estimating equation. Survival analysis and the related specialized methods for it will be detailed in Chapter 10, Ensembling Survival Models, of the book. We will briefly introduce the notation here as required to set up the parameters. Let T denote the survival time, or the time to the event of interest, and we naturally have , which is a continuous random variable. Suppose that the lifetime cumulative distribution is F and the associated density function is f. Since the lifetimes T are incomplete for some of the observations and subject to censoring, we will not be able to properly infer about interesting parameters such as mean survival time or median survival time. Since...