The six phases of an analytics project, the roles involved, and data preparation with Labs 9, 4 and 5.
The Data Analytics Lifecycle is a six-phase method that carries an analytics project from a business question to a model working in production. The six phases are Discovery, Data Preparation, Model Planning, Model Building, Communicate Results and Operationalize. The lifecycle is iterative: at several points the team checks whether it has enough information to move forward, and if it does not, it returns to an earlier phase.
Work moves clockwise from Discovery; the dashed arrows are the most common returns — from Data Preparation to Discovery when the data cannot answer the question, and from Model Planning to Data Preparation when the chosen method needs data that has not been prepared.
3.1 Need for a data analytics lifecycle
Without a defined process, analytics projects tend to drift. Teams solve a problem the business did not have, discover data problems only after the model is built, or produce results that never reach the people who must act on them. A lifecycle prevents this in several ways. It keeps every phase aligned with the business goal agreed at the start. It gives the work a repeatable structure that new team members can follow. It exposes risk early, because data gaps and infeasible ideas surface in Discovery and Data Preparation, before expensive modelling begins. It assigns clear responsibility and deliverables to each role and phase. It builds in checkpoints for iteration, and it ends with deployment, so the project produces something that is used rather than a report that is filed.
Worked example – the cost of finding a problem late. A project plans 1 week for Discovery, 4 weeks for Data Preparation, 1 week for Model Planning, 2 weeks for Model Building, 1 week for Communicate Results and 2 weeks for Operationalize — 11 weeks in total.
- Suppose the key data field turns out to be unreliable. If this is found during Discovery, the team loses about half a week re-scoping the problem.
- If it is found during Operationalize, the team must redo Data Preparation through Operationalize: 4 + 1 + 2 + 1 + 2 = 10 weeks of rework.
- The late discovery costs 10 / 0.5 = 20 times more effort. A disciplined lifecycle exists to catch such problems at the cheapest point.
3.2 Key roles for successful analytic projects
A successful project needs several roles, which in a small team may be played by fewer people. The Business User understands the domain and will use the results, so they advise on what is valuable and practical. The Project Sponsor starts and funds the project, sets priorities and judges the overall value. The Project Manager keeps milestones, schedule and quality on track. The Business Intelligence (BI) Analyst knows the existing data, KPIs, dashboards and reports, and brings business-domain knowledge from the data side. The Database Administrator (DBA) provisions and configures the database environment and controls access and security. The Data Engineer builds data pipelines and extracts, loads and transforms large datasets into the analytics sandbox. The Data Scientist chooses the analytical techniques, builds and evaluates the models, and makes sure the analytical objectives are met.
| Role | Main responsibility | Key phases |
|---|---|---|
| Business User | Defines value, uses results | 1, 5, 6 |
| Project Sponsor | Funds project, sets priorities | 1, 5 |
| Project Manager | Schedule, scope, quality | All |
| BI Analyst | Data sources, KPIs, reports | 1, 2, 5 |
| DBA | Database environment, access | 2, 6 |
| Data Engineer | Pipelines, ETL, sandbox | 2, 6 |
| Data Scientist | Methods, models, evaluation | 3, 4, 5 |
Example – staffing a hospital readmission project. The hospital's medical superintendent is the sponsor; a senior ward doctor is the business user; the IT department head is the project manager; the MIS officer who produces monthly bed-occupancy reports is the BI analyst; the hospital database manager is the DBA; one IT engineer is assigned as data engineer; and an analytics consultant is the data scientist. Seven roles are covered by seven people here, but in a college project one student might play the data engineer, BI analyst and data scientist roles together.
3.3 Phase 1 – Discovery
In Discovery the team learns the business domain, assesses the resources available (people, technology, time and data), and frames the business problem as an analytics problem. It identifies the key stakeholders and interviews the project sponsor to understand what success means. The team then writes initial hypotheses (IH) — testable statements about what the data may show — and identifies the data sources it will need. The phase ends when the team can state the problem, the success criteria and the data required.
Worked example – framing a measurable goal. A hospital discharges 10,000 patients a year, and 18% are readmitted within 30 days. The average readmission costs ₹40,000.
- Current readmissions = 18% × 10,000 = 1,800 per year, costing 1,800 × ₹40,000 = ₹7.2 crore.
- Goal agreed with the sponsor: reduce the rate to 15% within one year.
- Target readmissions = 15% × 10,000 = 1,500, so readmissions avoided = 1,800 − 1,500 = 300.
- Expected saving = 300 × ₹40,000 = ₹1.2 crore per year — the figure against which the project will be judged.
- Initial hypothesis: "Patients discharged on Friday with more than five medicines are readmitted more often." Data needed: discharge records, prescriptions and readmission dates.
Read more on the blog: Data Discovery.
3.4 Phase 2 – Data Preparation
Data Preparation is usually the longest phase and often takes half or more of the project's time. The team first creates an analytic sandbox — a separate workspace where data can be explored and changed without affecting production systems. Data is moved into it using ETL (extract, transform, load) or ELT/ETLT, where raw data is loaded first and transformed inside the sandbox so that nothing is lost. The team then works through five tasks in order: collection, quality assessment, cleaning, transformation and reduction.
Data collection. The team gathers data from all the sources identified in Discovery — databases, files, APIs and web pages — and records where each dataset came from and when. Read more on the blog: Data Collection.
Data quality assessment. Each dataset is profiled to measure completeness, validity, consistency, uniqueness and accuracy, as in the example in section 1.3. Read more on the blog: Data Quality Assessment.
Data cleaning. Cleaning fixes the problems found: missing values are filled or removed, duplicates are merged, invalid values corrected and outliers investigated.
Worked example – mean and median imputation. Daily sales (units) are 10, 12, —, 14, 9, where one day is missing. The mean of the four known values is (10 + 12 + 14 + 9) / 4 = 45 / 4 = 11.25, so the missing value is replaced by 11.25. The median of the known values (sorted: 9, 10, 12, 14) is (10 + 12) / 2 = 11; the median is preferred when the data contains outliers, because extreme values do not pull it.
Worked example – detecting outliers with the IQR rule. Monthly salaries (₹ thousand) are 22, 25, 27, 28, 30, 31, 33, 35, 120.
- The data is sorted and n = 9, so the median is 30. Q1 is the median of the lower half (22, 25, 27, 28) = (25 + 27) / 2 = 26. Q3 is the median of the upper half (31, 33, 35, 120) = (33 + 35) / 2 = 34.
- IQR = Q3 − Q1 = 34 − 26 = 8.
- Lower fence = Q1 − 1.5 × IQR = 26 − 12 = 14; upper fence = Q3 + 1.5 × IQR = 34 + 12 = 46.
- 120 > 46, so 120 is an outlier and must be checked — it may be a typing error for 12.0 or a genuine senior salary.
Read more on the blog: Data Cleaning.
Data transformation. Transformation converts data into a form suitable for modelling: scaling numeric variables to comparable ranges, encoding categories as numbers, creating new variables, and applying log transforms to skewed data.
Worked example – min-max normalisation to [0, 1]. Ages are 20, 30 and 50. With min = 20 and max = 50, x′ = (x − min) / (max − min):
- 20 → (20 − 20) / 30 = 0
- 30 → (30 − 20) / 30 = 10 / 30 = 0.333
- 50 → (50 − 20) / 30 = 1
Worked example – z-score standardisation. For the same ages, mean = (20 + 30 + 50) / 3 = 33.33. The squared deviations are (−13.33)² = 177.7, (−3.33)² = 11.1 and (16.67)² = 277.9, which sum to 466.7. The sample standard deviation is √(466.7 / 2) = √233.3 = 15.28. So z = (x − 33.33) / 15.28 gives 20 → −0.87, 30 → −0.22 and 50 → 1.09.
Read more on the blog: Data Transformation.
Data reduction. Data reduction produces a smaller representation of the data that gives the same, or almost the same, analytical result, saving storage and computing time. Dimensionality reduction reduces the number of attributes through feature selection or Principal Component Analysis (section 6.2). Numerosity reduction replaces the data with a smaller form, such as a regression equation, a histogram, cluster centres or a sample. Data compression encodes data in fewer bits, either losslessly (ZIP) or lossily (JPEG). Data cube aggregation summarises detail to a higher level, for example daily sales to monthly totals, and discretisation replaces continuous values with intervals.
Worked example – equal-width discretisation and aggregation. Ages 18, 22, 25, 34, 41, 47, 55, 63 are placed into 3 equal-width bins. Width = (63 − 18) / 3 = 15, giving bins [18, 33), [33, 48) and [48, 63]. The ages fall as {18, 22, 25}, {34, 41, 47} and {55, 63}, so eight numbers are reduced to three labels (young, middle, senior) with counts 3, 3 and 2. Similarly, 365 daily sales rows aggregated to monthly totals become 12 rows — a reduction of (365 − 12) / 365 ≈ 97%.
Blog lab: Lab 3 – Data Preparation and EDA
Data preparation labs (R) — do them in this order: collect the data (Lab 9), describe it (Lab 4), then clean and scale it (Lab 5).
Lab 9 (R) – Collecting data via web scraping, APIs and data connectors
Use the sources your instructor specifies; the URLs below are working examples. Always check a website's terms of use and robots.txt before scraping.
install.packages(c("rvest", "httr", "jsonlite", "DBI", "RSQLite"))
library(rvest); library(httr); library(jsonlite); library(DBI)
# ---- 1. Web scraping: read an HTML table ----
url <- "https://en.wikipedia.org/wiki/List_of_states_and_union_territories_of_India_by_population"
page <- read_html(url)
tables <- html_table(page, fill = TRUE)
states <- tables[[2]] # pick the table you need (check with length(tables))
head(states)
# Scrape text by CSS selector
headings <- page %>% html_elements("h2") %>% html_text2()
# ---- 2. REST API returning JSON ----
res <- GET("https://restcountries.com/v3.1/region/asia")
stop_for_status(res)
countries <- fromJSON(content(res, as = "text", encoding = "UTF-8"), flatten = TRUE)
asia <- data.frame(name = countries$name.common,
capital = sapply(countries$capital, `[`, 1),
population = countries$population)
head(asia[order(-asia$population), ])
# ---- 3. Data connector: a database through DBI ----
con <- dbConnect(RSQLite::SQLite(), "lab9.sqlite") # MySQL: RMariaDB::MariaDB(); PostgreSQL: RPostgres::Postgres()
dbWriteTable(con, "asia", asia, overwrite = TRUE)
dbGetQuery(con, "SELECT name, population FROM asia WHERE population > 100000000")
dbDisconnect(con)
# Save what you collected
write.csv(asia, "asia_countries.csv", row.names = FALSE)
Lab 4 (R) – Statistical operations: mean, median, mode, standard deviation
# Data from the user, or use a built-in dataset
input <- readline("Enter numbers separated by commas (blank = use mtcars$mpg): ")
x <- if (nchar(input) > 0) as.numeric(strsplit(input, ",")[[1]]) else mtcars$mpg
# R has no built-in mode for data, so define one (returns all modes on a tie)
get_mode <- function(v) {
tab <- table(v)
as.numeric(names(tab)[tab == max(tab)])
}
cat("Mean :", mean(x), "\n")
cat("Median :", median(x), "\n")
cat("Mode :", get_mode(x), "\n")
cat("Standard deviation:", sd(x), "\n") # sample SD (n - 1)
cat("Variance :", var(x), "\n")
summary(x)
# Column-wise statistics for a whole data frame
sapply(mtcars[, c("mpg", "hp", "wt")], function(col)
c(mean = mean(col), median = median(col), sd = sd(col)))
hist(x, main = "Distribution", col = "lightblue"); abline(v = mean(x), col = "red", lwd = 2)
Lab 5 (R) – Data pre-processing: (i) handling missing data, (ii) min-max normalisation
# Sample data with missing values (or use airquality, which has NAs)
df <- airquality
str(df)
# ---- (i) Handling missing data ----
colSums(is.na(df)) # count NAs per column
mean(!complete.cases(df)) * 100 # % rows with any NA
df_drop <- na.omit(df) # Option 1: delete rows with NA
df_mean <- df # Option 2: mean imputation
for (col in names(df_mean)) {
if (is.numeric(df_mean[[col]]))
df_mean[[col]][is.na(df_mean[[col]])] <- mean(df_mean[[col]], na.rm = TRUE)
}
df_med <- df # Option 3: median imputation (robust to outliers)
df_med$Ozone[is.na(df_med$Ozone)] <- median(df_med$Ozone, na.rm = TRUE)
colSums(is.na(df_mean)) # should be all 0
# ---- (ii) Min-max normalisation to [0, 1] ----
min_max <- function(v) (v - min(v)) / (max(v) - min(v))
df_norm <- as.data.frame(lapply(df_mean[, c("Ozone", "Solar.R", "Wind", "Temp")], min_max))
summary(df_norm) # every column now ranges 0 to 1
# Scale to a new range [new_min, new_max]
scale_to <- function(v, a, b) a + (v - min(v)) * (b - a) / (max(v) - min(v))
head(scale_to(df_mean$Temp, 0, 10))
3.5 Phase 3 – Model Planning
In Model Planning the team decides which methods, variables and tools the model will use. It explores the prepared data to understand relationships between variables and selects the key predictors, removing those that add nothing or that duplicate one another. It then chooses candidate techniques that suit the problem: regression to predict a number, classification to predict a category, clustering to find groups, association rules to find items that occur together, and time-series models to forecast over time. Finally it selects tools — R, Python, SAS, SQL analytics or Spark MLlib — according to data size and team skills. The output is a documented plan listing the candidate models and the variables each will use.
Worked example – choosing a predictor using correlation. For the attendance (x) and marks (y) data of section 1.7 — (60, 52), (70, 60), (80, 68), (90, 75), (100, 85) — the means are x̄ = 80 and ȳ = 68.
- Deviations x − x̄: −20, −10, 0, 10, 20. Deviations y − ȳ: −16, −8, 0, 7, 17.
- Sxy = Σ(x − x̄)(y − ȳ) = 320 + 80 + 0 + 70 + 340 = 810.
- Sxx = Σ(x − x̄)² = 400 + 100 + 0 + 100 + 400 = 1,000. Syy = Σ(y − ȳ)² = 256 + 64 + 0 + 49 + 289 = 658.
- r = Sxy / √(Sxx × Syy) = 810 / √658,000 = 810 / 811.2 = 0.9985.
- A correlation this close to 1 shows a strong linear relationship, so linear regression is planned as the model and attendance is kept as a predictor.
3.6 Phase 4 – Model Building
In Model Building the team builds and runs the models planned in Phase 3. It splits the data into training, validation and test sets so that the final accuracy is measured on data the model has never seen. It trains each candidate model on the training set, tunes its parameters on the validation set, and evaluates the final choice on the test set with measures such as accuracy, precision, recall, RMSE or R². It also checks that the model is robust and not overfitted. Tools include R, Python, SAS Enterprise Miner, SPSS Modeler, MATLAB and Weka, and the computing environment may need more power than earlier phases.
Worked example – splitting data and evaluating a classifier.
- A dataset of 1,000 records is split 70 : 15 : 15, giving 700 training, 150 validation and 150 test records.
- On a test set of 250 loan applications, the model's confusion matrix is: true positives (TP) = 80, false positives (FP) = 20, false negatives (FN) = 10, true negatives (TN) = 140.
- Accuracy = (TP + TN) / total = (80 + 140) / 250 = 220 / 250 = 88%.
- Precision = TP / (TP + FP) = 80 / 100 = 80%.
- Recall = TP / (TP + FN) = 80 / 90 = 88.9%.
- F1-score = 2 × P × R / (P + R) = 2 × 0.80 × 0.889 / (0.80 + 0.889) = 1.422 / 1.689 = 0.842.
3.7 Phase 5 – Communicate Results
In this phase the team compares the outcome with the success criteria set in Discovery and tells stakeholders honestly whether the project succeeded, failed or is inconclusive. It identifies the key findings, quantifies their business value, and presents them with clear visualisations and a narrative suited to each audience — a short summary for the sponsor, more technical detail for analysts. It also states the caveats, assumptions and limitations of the model, and recommends specific actions and next steps.
Worked example – quantifying value for the sponsor. Returning to the readmission project of section 3.3, the model is tested on 2,000 recent discharges.
- It flags 400 patients as high-risk; 160 of them are actually readmitted. Without the model, the readmission rate among all patients is 18%.
- Readmission rate among flagged patients = 160 / 400 = 40%, which is 40 / 18 = 2.2 times the average — the model concentrates risk well (this ratio is called lift).
- If follow-up calls prevent one in four of these 160 readmissions, 40 readmissions are avoided per 2,000 discharges, or 200 per 10,000 discharges a year.
- Value = 200 × ₹40,000 = ₹80 lakh per year, against a goal of ₹1.2 crore. The honest message to the sponsor is "two-thirds of the goal achieved; extending follow-up to medium-risk patients could close the gap."
3.8 Phase 6 – Operationalize
In the final phase the chosen model is deployed into the real business environment. The team usually runs a pilot first, on one ward, branch or region, to check that the model performs well under real conditions before a full rollout. It delivers four main outputs: a final report for the sponsor, a presentation for analysts and users, the code for engineers, and technical specifications for the data engineers and DBA who will maintain it. The model is then integrated into production systems and workflows and monitored continuously, because the data a model sees changes over time (model drift). When performance falls below an agreed level, the model is retrained or the team returns to an earlier phase.
Worked example – a retraining rule. The deployed model's accuracy is checked every month, and the team has agreed to retrain if accuracy drops by more than 5 percentage points from the 88% achieved at launch.
| Month | Accuracy | Drop from 88% | Action |
|---|---|---|---|
| 1 | 87.5% | 0.5 | None |
| 2 | 86.0% | 2.0 | None |
| 3 | 84.2% | 3.8 | Watch |
| 4 | 82.1% | 5.9 | Retrain |
In month 4 the drop of 88 − 82.1 = 5.9 points exceeds the threshold, so the model is retrained on recent data — completing the loop back to earlier phases.
Independent study notes written in my own words, for learning purposes. Not affiliated with any university or course provider. R, RStudio, Hadoop, Amazon S3, Tableau, Power BI and other product names are trademarks of their respective owners.