In Praise of Small Data

George Markowsky, In Praise of Small DataInternational Workshop on Big Data Analytics for Predictive Organization and Big Transformations Stanford, CA, USA, May 27, 2014 (in press)

Big Data tools can give “explanations” of complex and elaborate data sets. There is the danger that we might be content with the explanations that these tools produce. It is important to bear in mind that truly understanding something often requires simplifying the initial explanation. Indeed, the well known “Occam’s Razor” states roughly that the simplest explanation is the best. This paper examines two cases, Copernicus’s Solar System Model and a “Big Data” search for Paul Revere, where the initial models were too complex. It shows that simpler models are possible, and more fruitful for further research. Some general techniques for finding simpler explanations are discussed.