Using SparkR for computing summary statistics
The describe (or summary) operation creates a new that contains count, mean, max, mean, and standard deviation values for a specified DataFrame or a list of numerical columns:
> sumstatsdf <- describe(df, "duration", "campaign", "previous", "age") > showDF(sumstatsdf)
Computing these values on a large Dataset can be computationally expensive. Hence, we present the individual computation of these statistical measures here:
> avgagedf <- agg(df, mean = mean(df$age)) > showDF(avgagedf) # Print this DF +-----------------+ | mean | +-----------------+ |40.02406040594348| +-----------------+
Next, we create a DataFrame that lists the minimum and maximum values and the range width:
> agerangedf <- agg(df, minimum = min(df$age), maximum = max(df$age), range_width = abs(max(df$age) - min(df$age))) > showDF(agerangedf)
Â
Next, we compute the sample variance and standard deviation as shown here:
> agevardf <- agg...