These 14 chapters cover the full Data Analytics syllabus in about 200 minutes of reading. Every topic is explained in full sentences with step-by-step worked examples, and the key ideas have diagrams. Each chapter below has a short overview and a list of what it covers, so you can start at the beginning or jump straight to the topic you need.
These are independent study notes, written in my own words and with my own examples, covering standard topics in data analytics. They are not official course material and are not affiliated with or endorsed by any university, course provider or company.
How to use these notes
- If you are new to the subject, read the chapters in order, because each one builds on the ones before it.
- Every chapter opens with an In this chapter list, so you can jump straight to any section.
- Look for the Worked example paragraphs: each one is a calculation you can check by hand before an exam.
- Each R lab sits right after the theory it practises; the table at the bottom of this page shows where to find every lab.
- The buttons at the bottom of each chapter take you to the previous chapter, the next chapter or back to this index.
Jump to a chapter
Introduction to Data Analytics
This chapter explains what data is, where it comes from and why organisations analyse it. It then looks at the platforms and tools that make analysis possible at scale, and ends with real applications across industries.
In this chapter:
- You will classify data as structured, semi-structured or unstructured, and as nominal, ordinal, discrete or continuous.
- You will measure data quality and size a Big Data platform with HDFS block and velocity calculations.
- You will follow the analytic process step by step and see how analysis differs from reporting.
- You will compare modern analytic tools and evaluate a fraud-detection application with precision and recall.
Introduction to R
R is the main tool used in the labs, and this chapter gets you working with it. It covers the R environment, loading and describing data, exploring it visually, and turning text into numbers that can be analysed.
In this chapter:
- You will compare RStudio, R Commander, Rattle and notebooks, and import and export CSV, Excel and text files.
- You will tell nominal, ordinal, interval and ratio attributes apart and use R's vectors, factors and data frames.
- You will compute mean, median, mode, variance, quartiles and IQR by hand and in R.
- You will see why data must be plotted before analysis (Anscombe's quartet) and calculate TF-IDF weights.
- You will complete R Labs 1, 2 and 3: numerical operations, data import/export and matrix operations.
Data Analytics Lifecycle
Successful analytics projects follow a repeatable method. This chapter walks through the six phases of the lifecycle, the people involved in each one, and the data-preparation work that usually takes most of a project's time.
In this chapter:
- You will learn why a lifecycle is needed and what each of the seven key project roles does.
- You will clean, transform and reduce data with worked examples of imputation, the IQR outlier rule and normalisation.
- You will plan and build a model, evaluate it with accuracy, precision, recall and F1, and communicate its business value.
- You will complete R Labs 9, 4 and 5: data collection, descriptive statistics and pre-processing.
Regression, Multivariate Analysis and Bayesian Modelling
Regression and Bayesian methods are two of the most widely used ways to model data. This chapter fits a regression line by hand, extends it to many variables, and then shows how Bayesian reasoning updates beliefs as evidence arrives.
In this chapter:
- You will fit a least-squares line step by step and interpret its slope, intercept and R².
- You will read a multiple regression and check adjusted R² and multicollinearity with VIF.
- You will apply Bayes' theorem, update a Beta prior and run inference in a Bayesian network.
- You will classify an e-mail with Naïve Bayes and complete R Lab 7 on simple linear regression.
Support Vector Machines, Time Series and Rule Induction
This chapter covers three different ways of learning from data. Support vector machines draw the widest possible boundary between classes, time-series models forecast values ordered in time, and rule induction produces models people can read.
In this chapter:
- You will find a maximum-margin boundary by hand and see how the kernel trick handles curved boundaries.
- You will forecast with moving averages, exponential smoothing and AR models, and see chaos in a nonlinear system.
- You will learn the 1R algorithm and measure the coverage and accuracy of IF–THEN rules.
- You will complete R Lab 12: building and evaluating a time-series model.
Neural Networks, PCA, Fuzzy Logic and Stochastic Search
This chapter brings together four soft-computing techniques. Neural networks learn from examples, PCA compresses many variables into a few, fuzzy logic reasons with vague concepts, and stochastic search finds good solutions to hard optimisation problems.
In this chapter:
- You will carry out one backpropagation step by hand and recognise underfitting and overfitting.
- You will run a step of competitive learning and compute principal components from a covariance matrix.
- You will build a fuzzy controller, extract a fuzzy rule from data and use a fuzzy decision tree.
- You will apply simulated annealing and a genetic algorithm, and complete R Lab 6 on PCA.
Mining Data Streams I
Data streams arrive too fast to store, so they must be analysed in a single pass with limited memory. This chapter explains how stream systems are organised and how to sample and filter a stream without keeping all of it.
In this chapter:
- You will compare streams with stored data and size a real payment stream.
- You will learn the parts of a Data Stream Management System and use sliding windows.
- You will see why sampling on the wrong key gives wrong answers, and how reservoir sampling works.
- You will insert and test keys in a Bloom filter and choose the best number of hash functions.
Mining Data Streams II
This chapter covers the classic algorithms for answering questions about a stream with very little memory. It finishes with real-time analytics platforms and two worked case studies: live sentiment analysis and stock-market prediction.
In this chapter:
- You will count distinct elements with the Flajolet–Martin algorithm.
- You will estimate the second frequency moment with the AMS algorithm.
- You will count ones in a window with DGIM and track popular items with a decaying window.
- You will design a real-time fraud check and follow the sentiment and stock-market case studies.
Frequent Itemsets
Frequent-itemset mining finds products, words or events that occur together. This chapter runs every major algorithm on the same five shopping baskets, so you can compare how each one works and how much memory it needs.
In this chapter:
- You will compute support, confidence and lift, and spot rules that look strong but are not.
- You will trace Apriori level by level and see how PCY and FP-Growth reduce the work.
- You will apply the SON and Toivonen limited-pass algorithms and count itemsets in a stream.
- You will predict a rating with collaborative filtering and complete R Labs 10 and 11.
Clustering
Clustering groups similar objects without being told the groups in advance. This chapter covers the main clustering methods, the special problems of high-dimensional and non-Euclidean data, and clustering for streams and parallel systems.
In this chapter:
- You will compute distances and build a dendrogram with single linkage.
- You will run k-means iteration by iteration and choose k with the elbow method.
- You will find subspace clusters with CLIQUE and projected clusters with PROCLUS.
- You will choose a clustroid, merge stream clusters, and complete R Lab 8 on the iris data.
Big Data Frameworks I
When data is too big for one machine, it must be stored and processed across a cluster. This chapter explains MapReduce, the Hadoop platform and its file system, and the Pig and Hive tools that make Hadoop easier to use.
In this chapter:
- You will trace a MapReduce word count and learn when a combiner is safe to use.
- You will plan the capacity of a Hadoop cluster and learn what YARN does.
- You will see how HDFS splits, replicates and places blocks across racks.
- You will trace a Pig Latin script and use Hive partitioning and bucketing.
Big Data Frameworks II
This chapter covers the storage systems that sit alongside Hadoop. It explains real-time access with HBase, the MapR distribution, how databases are split across servers, the four kinds of NoSQL database, and cloud object storage.
In this chapter:
- You will learn HBase's data model and design row keys that avoid hot-spotting.
- You will compare MapR-FS with HDFS and see why distributed metadata matters.
- You will compare range, hash and consistent-hash sharding with a rebalancing calculation.
- You will choose a NoSQL type, tune quorums under the CAP theorem, and see what S3's durability means.
Visualization I
Visualization turns data into pictures so that people can see patterns at a glance. This chapter explains how a visualization is built, what can go wrong, the limits of screens and computers, and how human vision shapes good design.
In this chapter:
- You will follow the visualization pipeline and map data values to pixels and colours.
- You will learn the main issues and user tasks, and calculate Tufte's lie factor.
- You will see how display space, rendering time and navigation limit what can be shown.
- You will understand pre-attentive processing, Gestalt laws and the limits of human vision.
Visualization II
This final chapter covers how to design and explore visualizations. It explains design principles, exploration of complex information, visual analysis and interaction techniques, and the systems used in practice, and ends with two R labs.
In this chapter:
- You will improve a chart's data–ink ratio and build a fisheye view by degree of interest.
- You will map data objects to visual objects and combine images, narrative and gestures.
- You will place a record on parallel coordinates and lay out a treemap.
- You will compare visualization systems and complete R Labs 13 (maps) and 14 (word cloud).
R labs at a glance
Labs keep their syllabus numbers but are placed after the topic they practise, so a few appear out of numeric order — for example Lab 9 comes before Lab 4, because data is collected before it is described.
| Lab | Topic | Chapter |
|---|---|---|
| 1 | User input and numerical operations | Chapter 2 |
| 2 | Import/export CSV, XLS, TXT | Chapter 2 |
| 3 | Matrix operations | Chapter 2 |
| 9 | Web scraping, APIs, data connectors | Chapter 3 |
| 4 | Mean, median, mode, standard deviation | Chapter 3 |
| 5 | Missing data, min-max normalisation | Chapter 3 |
| 7 | Simple linear regression | Chapter 4 |
| 12 | Time-series model and accuracy | Chapter 5 |
| 6 | PCA on a houses dataset | Chapter 6 |
| 10 | Association analysis and accuracy | Chapter 9 |
| 11 | Recommendation system and accuracy | Chapter 9 |
| 8 | K-means clustering on iris | Chapter 10 |
| 13 | Maps of the world, Indian states and districts | Chapter 14 |
| 14 | Text mining and word cloud | Chapter 14 |
Further reading
Many topics also have a longer post on the Introduction to Data Analytics hub page, linked from the relevant sections of each chapter.
Independent study notes written in my own words, for learning purposes. Not affiliated with any university or course provider. R, RStudio, Hadoop, Amazon S3, Tableau, Power BI and other product names are trademarks of their respective owners.