“School of Mathematics”

Back to Papers Home
Back to Papers of School of Mathematics

Paper   IPM / M / 16311
School of Mathematics
  Title:   MR plot: A big data tool for distinguishing distributions
  Author(s):  Majid Asadi (Joint with O. M. Ardakani, N. Ebrahimi, and E. S. Soofi)
  Status:   Published
  Journal: Stat. Anal. Data Min.
  Year:  2020
  Pages:   DOI: 10.1002/sam.11464
  Supported by:  IPM
  Abstract:
Big data enables reliable estimation of continuous probability density, cumulative distribution, survival, hazard rate, and mean residual functions (MRFs). We illustrate that plot of the MRF provides the best resolution for distinguishing between distributions. At each point, the MRF gives the mean excess of the data beyond the threshold. Graph of the empirical MRF, called here the MRplot, provides an effective visualization tool. A variety of theoretical and data driven examples illustrate that MR plots of big data preserve the shape of theMRF and complex models require bigger data. The MRF is an optimal predictor of the excess of the random variable.With a suitable prior, the expected MRF gives the Bayes risk in the form of the entropy functional of the survival function, called here the survival entropy.We show that the survival entropy is dominated by the standard deviation (SD) and the equality between the two measures characterizes the exponential distribution. The empirical survival entropy provides a data concentration statistic which is strongly consistent, easy to compute, and less sensitive than the SD to heavy tailed data. An application uses theNewYork City Taxi database with millions of trip times to illustrate the MR plot as a powerful tool for distinguishing distributions.

Download TeX format
back to top
Clients Logo
Clients Logo
Clients Logo
Clients Logo
Clients Logo
Clients Logo
Clients Logo
Clients Logo
scroll left or right