What data mining is, the KDD process, the main mining tasks, and the issues that make mining hard.
7.1 Overview and motivation
The amount of data that organisations collect grows much faster than people can study it, so organisations are often described as "data rich but information poor". Data mining addresses this gap by turning stored data into knowledge that can guide decisions.
ExampleShopMart records about 50 lakh bills every month. When the May bills are mined, they show that 12,000 customers bought school uniforms, and 7,800 of these customers also bought a tiffin box or a water bottle within the next two weeks. That is 7,800 ÷ 12,000 = 65 per cent of uniform buyers. Among all customers, only 8 per cent bought a tiffin box or bottle in the same period, so uniform buyers were 65 ÷ 8 ≈ 8.1 times more likely to buy one.
ExplanationNo manager would notice this pattern by reading individual bills. Once it has been found, however, ShopMart can offer a "back to school" bundle and place these products together on the shelves. The business gains directly from data it had already stored.
7.2 Definition
Data mining is the non-trivial extraction of implicit, previously unknown and potentially useful patterns from large amounts of data. It is one step in the larger process of knowledge discovery in databases (KDD).
ExampleThe statement "customers who buy rice also buy food" is trivial, because everyone already knows it. By contrast, the statement "customers aged 25 to 35 who buy baby formula at weekends spend 40 per cent more per visit" is non-trivial, was not known before, and is useful for targeting offers. It comes from comparing average bills: these customers spend ₹2,100 per visit, while other customers spend ₹1,500, and (2,100 − 1,500) ÷ 1,500 = 0.40, or 40 per cent more.
ExplanationEach word in the definition acts as a test. A pattern counts as a data mining result only if it is hidden in the data (implicit), new (previously unknown) and something the business can act on (useful).
7.3 KDD process
The figure shows the seven steps of knowledge discovery. The first four steps prepare the data, data mining itself is the fifth step, and the last two steps judge and present what the mining has found.
Example (a mobile operator predicting customer churn)In the cleaning step, the operator removes call records with negative durations and fills in missing ages. In the integration step, it joins the billing records, the call records and the complaint tickets using the customer ID. In the selection step, it keeps only the prepaid customers who were active in the last twelve months. In the transformation step, it calculates monthly averages, such as the number of calls per day and the average recharge amount. In the data mining step, it builds a decision tree that predicts whether a customer will leave within the next three months. In the evaluation step, it keeps only the rules that are more than 80 per cent accurate. For example, the rule "customers with three or more complaints and falling recharges are likely to leave" applies to 1,250 customers in the test data, and 1,050 of them actually left, so its accuracy is 1,050 ÷ 1,250 = 84 per cent and it is kept. Finally, in the presentation step, it builds a dashboard that lists high-risk customers for the retention team.
ExplanationMost of the effort in KDD, often 60 to 80 per cent, goes into the first four steps. If a mining algorithm is run on data that has not been prepared, it produces misleading rules.
7.4 Data mining functionalities (tasks)
Data mining tasks fall into two broad groups. Predictive tasks use known data to forecast unknown values, while descriptive tasks find patterns in the data as it already exists. The table below lists the main tasks with the question each one answers and an example.
| Task | Type | Question it answers | Example |
|---|---|---|---|
| Classification | Predictive | Which known class does this record belong to? | Approving or rejecting a loan application |
| Regression | Predictive | What numeric value will it have? | Predicting next month's sales for a store |
| Time-series analysis | Predictive | How will a series move over time? | Forecasting electricity demand hour by hour |
| Clustering | Descriptive | Which natural groups exist in the data? | Dividing customers into budget, regular and premium groups |
| Summarisation | Descriptive | What describes a class, or what separates two classes? | Profiling customers who spend over ₹50,000 a year |
| Association rules | Descriptive | Which items occur together? | Bread is bought with butter in 60% of the bills that contain bread |
| Sequence discovery | Descriptive | What happens in a particular order? | A customer buys a phone, then a cover, then earphones |
| Outlier analysis | Either | Which records are unusual? | A card used in Lucknow and in Dubai within one hour |
ExplanationThe right task always follows from the business question. If the question asks what will happen, a predictive task is needed. If it asks what patterns already exist, a descriptive task is needed.
7.5 Issues in data mining
Several issues make data mining difficult in practice. The first is scalability, because an algorithm that works well on 10,000 rows may never finish on 10 crore rows. The second is noisy and incomplete data, because wrong values such as an age of 250, or missing values such as blank incomes, distort the patterns that are found. The third is high dimensionality, because when a data set has thousands of columns, as a text collection with 50,000 different words does, distances between records lose their meaning.
There are also human and ethical issues. Privacy is a concern, because mining health or location data can reveal personal facts and raise legal problems. Interestingness is another, because an algorithm may produce 10,000 rules, most of them trivial, so measures are needed to rank them. Finally, user interaction matters, because analysts need to guide the mining with queries and visual feedback rather than accept whatever the algorithm returns.
ExplanationThese issues explain why the next two chapters concentrate on preparing and reducing the data before any mining algorithm is run.
Exam points
- Data mining is the non-trivial extraction of implicit, previously unknown and potentially useful patterns from large data.
- Data mining is one step of KDD; the steps are cleaning, integration, selection, transformation, mining, evaluation and presentation.
- Classification, regression and time-series analysis are predictive; clustering, summarisation, association rules and sequence discovery are descriptive.
- Outlier analysis finds unusual records, as in fraud detection.
- Most KDD effort goes into the preparation steps before mining.
Independent study notes for learning purposes. Not affiliated with or endorsed by any university, publisher or course provider. Product and company names are trademarks of their respective owners.