Data Mining

When I was a student, a long, long time ago, the term data mining described a frowned upon practice. Only bad people did data mining. It was a term used to describe the practice of doing a series of tests of significance on a set of data until a statistically significant effect was found. It was a form of sequential testing but it was a very unscientific, atheoretical form. Bad practice.

All that's changed. Data Mining today is a generally accepted practice, a tool that any good data analyst should have in their tool box. But today's version of data analysis isn't really the same as yesterday's version. There's some subtle but important distinctions to be made.

Wikipedia gives a simple definition for the term data mining

Data mining is the process of extracting patterns from data.

That's as good as any place else as a place to start defining the concept, but it's just a start. When we use the term data mining we're talking about more than just data, we're talking about a lot of data, a whole lot.

That's one of the ways that data mining differs from statistics. The field of statistics was developed as a tool to extract information from small amounts of data. Small sample statistics is the backbone of statistical theory and practice. We don't do that with data mining -- we're looking at extraction of information from huge collections of data.

Statistics is based on using probability models to fit data or to test hypotheses. Data mining is a process of uncovering models, not fitting models. A process of forming hypotheses, not testing them.

Here's a video of the first lecture a course on data mining given for first year grad students at Stanford and given simultaneously at the Google headquarters.



In this lecture he's mostly just defining the concept. I'll follow along in subsequent posts.