Introduction to Data Analytics

Chapter 1: Introduction to Data Analytics

← All chapters

Chapter 1 of 14
Introduction to Data Analytics

Sources, nature and classes of data, Big Data platforms, the analytic process, tools and applications.

In this chapter
  1. 1.1 Sources and nature of data
  2. 1.2 Classification of data – structured, semi-structured and unstructured
  3. 1.3 Characteristics of data
  4. 1.4 Introduction to Big Data platform
  5. 1.5 Need of data analytics
  6. 1.6 Evolution of analytic scalability
  7. 1.7 Analytic process and tools
  8. 1.8 Analysis vs reporting
  9. 1.9 Modern data analytic tools
  10. 1.10 Applications of data analytics

Data analytics is the process of examining data to find patterns, draw conclusions and support decisions. This chapter explains what data is, where it comes from, how it is classified, why organisations analyse it, and which platforms and tools make analysis possible at scale. Each topic ends with a worked numerical example so that the idea can be checked by calculation.

1.1 Sources and nature of data

Data is a collection of raw facts — numbers, words, images, sounds or signals — recorded about objects, events or people. A single value such as "38" carries little meaning on its own. It becomes information only when it is placed in context, for example "the patient's temperature is 38 °C", and it becomes knowledge when many such facts are analysed to reveal a pattern, for example "temperatures above 38 °C in this ward rose sharply after the water supply changed".

Sources of data. Data can be grouped by who collected it and how it was produced. Primary data is collected first-hand for a specific purpose through surveys, interviews, experiments or field observations; it is relevant and controlled but costly to gather. Secondary data was collected earlier by someone else — census tables, published reports, research papers and company archives — and is cheap to obtain but may not fit the new question exactly. Internal data is produced inside an organisation by its own systems, such as sales transactions, ERP and CRM records, payroll files and server logs. External data comes from outside the organisation, such as social media, open-government portals, market feeds and partner data. Finally, machine-generated data is produced automatically by sensors, GPS devices, RFID readers, CCTV cameras and web clickstreams, while human-generated data is created by people in the form of emails, documents, reviews, chat messages and photographs.

Nature of data. The nature of a variable decides which calculations are meaningful. Qualitative (categorical) data describes a quality. It is nominal when the categories have no order (blood group A, B, AB, O) and ordinal when they do have an order (grade A > B > C). Quantitative (numerical) data measures an amount. It is discrete when it can only take countable values (number of children) and continuous when it can take any value in a range (height, temperature). Data may also be static, changing rarely (date of birth), or dynamic, changing continuously (a share price), and it may be time-stamped, in which case the order of records matters.

Worked example – identifying the nature of each attribute. Consider this small student record:

Roll no. Name Gender Grade No. of backlogs Height (cm) CGPA
101 Asha F A 0 158.4 8.6
102 Ravi M B 2 171.2 7.1
103 Neha F A 1 162.0 8.2
  1. Roll no. is a number, but it is only an identifier; adding two roll numbers has no meaning, so it is nominal.
  2. Name and Gender are labels with no order, so they are nominal.
  3. Grade has a natural order (A is better than B), so it is ordinal; we may find the median grade but not the mean.
  4. No. of backlogs is counted in whole numbers, so it is discrete numerical; mean backlogs = (0 + 2 + 1) / 3 = 1.
  5. Height and CGPA can take any value in a range, so they are continuous numerical; mean height = (158.4 + 171.2 + 162.0) / 3 = 491.6 / 3 = 163.87 cm.

The example shows why classifying variables matters: the mean of roll numbers (102) is a valid calculation but a meaningless result.

1.2 Classification of data – structured, semi-structured and unstructured

Data is classified by how much organisation, or schema, it carries.

Structured data follows a fixed schema of rows and columns, where every record has the same fields and every field has a defined type. It lives in relational databases and spreadsheets and is queried with SQL. Bank transactions, inventory tables and student mark sheets are typical examples. Because its structure is known in advance, structured data is the easiest to store, search and analyse, but it is also the least flexible.

Semi-structured data has no rigid table schema, but it carries tags or markers that label its parts and describe a hierarchy. JSON documents, XML files, e-mail (with structured headers and a free-text body) and web server logs are semi-structured. Different records may have different fields, which makes the format flexible, and NoSQL databases such as MongoDB store it natively.

Unstructured data has no predefined model at all. Free text, images, audio, video, scanned documents and social-media posts are unstructured. It is the largest and fastest-growing category — industry estimates commonly place it at around 80% of all enterprise data — but it needs special techniques such as natural-language processing, image recognition and speech-to-text before it can be analysed.

Worked example – the same customer complaint in three forms.

  1. Structured — a table row: (cust_id = 5012, date = 2026-03-04, product = 'Phone X', rating = 2). A SQL query can average the ratings directly.
  2. Semi-structured — a JSON record: {"cust_id": 5012, "product": "Phone X", "rating": 2, "tags": ["battery", "refund"]}. The tags field may hold any number of values, and another record may not have it at all.
  3. Unstructured — a tweet: "Phone X battery dies by noon, really disappointed. Want my money back!" The rating and topic must first be extracted with text mining.

If a company receives 10,000 complaints a month, of which 2,000 arrive through the rating form, 1,500 through the app's JSON API and 6,500 as e-mails and social posts, then the share of each type is 2,000 / 10,000 = 20% structured, 1,500 / 10,000 = 15% semi-structured and 6,500 / 10,000 = 65% unstructured. Most of the useful signal therefore sits in the hardest-to-process category.

Read more on the blog: Data Classification.

1.3 Characteristics of data

The value of data depends on its quality. Good data is accurate (it correctly describes the real world), complete (no required values are missing), consistent (the same fact has the same value everywhere), timely (it is available when needed and is up to date), valid (it obeys the defined format and business rules), unique (each real-world entity is recorded once) and relevant (it fits the question being asked). Data that serves analytics well also has clarity (it is easy to understand), can be aggregated and summarised without losing meaning, and can be reused and refined for new purposes.

Worked example – measuring data quality. A customer table has 1,000 records. An audit finds that 50 records have no e-mail address, 20 records are duplicates of other customers, 30 phone numbers have only 9 digits instead of 10, and in a random sample of 200 records checked by phone, 190 addresses were correct.

  1. Completeness of e-mail = (1,000 − 50) / 1,000 = 950 / 1,000 = 95%.
  2. Uniqueness = (1,000 − 20) / 1,000 = 98%.
  3. Validity of phone numbers = (1,000 − 30) / 1,000 = 97%.
  4. Accuracy of addresses (estimated from the sample) = 190 / 200 = 95%.
  5. If the organisation's target is 98% on every dimension, completeness, validity and accuracy fail and must be fixed during data preparation (Chapter 2).

Read more on the blog: Properties of Data.

1.4 Introduction to Big Data platform

Big Data refers to datasets whose size, speed or variety exceeds the ability of conventional databases and single computers to store and process them in reasonable time. It is usually described by the 3 Vs. Volume is the sheer quantity of data, measured in terabytes and petabytes. Velocity is the speed at which data is generated and must be processed, from batch loads to real-time streams. Variety is the mix of structured, semi-structured and unstructured formats. Two further Vs are often added: Veracity, the trustworthiness of the data, and Value, the business benefit that justifies collecting it.

A Big Data platform is an integrated set of technologies that stores, processes and analyses such data on clusters of inexpensive computers. The best-known platform is the Hadoop ecosystem. HDFS (Hadoop Distributed File System) splits large files into blocks and stores several copies of each block on different machines, so a disk failure does not lose data. MapReduce and YARN run computations in parallel next to the data. Hive and Pig provide SQL-like and scripting access, HBase offers a column-oriented NoSQL store, and Apache Spark performs fast in-memory processing for analytics and machine learning. Cloud services such as Amazon EMR, Google BigQuery and Azure Synapse offer the same capabilities without owning hardware.

Worked example 1 – storage in HDFS. A 1 GB log file is stored in HDFS with a block size of 128 MB and a replication factor of 3.

  1. Number of blocks = 1,024 MB / 128 MB = 8 blocks.
  2. Copies stored = 8 blocks × 3 replicas = 24 block copies.
  3. Raw disk space used = 1 GB × 3 = 3 GB.
  4. Even if two of the three machines holding a block fail, one copy survives.

Worked example 2 – velocity. A smart city has 10,000 sensors, each sending one 200-byte reading every second.

  1. Data per second = 10,000 × 200 bytes = 2,000,000 bytes ≈ 2 MB/s.
  2. Data per day = 2 MB/s × 86,400 s = 172,800 MB ≈ 172.8 GB/day.
  3. Data per year ≈ 172.8 GB × 365 ≈ 63 TB/year, which is clearly beyond a single server.

Read more on the blog: Big Data Platform · 3 Vs of Big Data.

1.5 Need of data analytics

Organisations today collect far more data than any person can read, and that data is valuable only if it changes a decision. Data analytics is needed first to support evidence-based decisions, replacing guesswork with measured facts. It helps organisations understand customers by segmenting them, discovering their buying patterns and predicting who is likely to leave. It improves operational efficiency by revealing bottlenecks and waste in supply chains, production and staffing. It strengthens risk management through fraud detection and credit scoring, and it enables forecasting of demand, prices and resource needs. Taken together, these capabilities give a competitive advantage, reduce costs, and point to new products that customers want but do not yet have.

Analytics is usually described at four levels of increasing difficulty and value:

Type Question answered Example
Descriptive What happened? Monthly sales report
Diagnostic Why did it happen? Drill-down shows sales fell in one region
Predictive What will happen? Forecast of next quarter's demand
Prescriptive What should we do? Recommended price and stock level

Worked example – the value of churn analytics. A telecom company has 10,000 subscribers paying ₹500 per month, and 5% of them leave (churn) every month.

  1. Customers lost per month = 5% × 10,000 = 500.
  2. Revenue lost per month = 500 × ₹500 = ₹2,50,000.
  3. A predictive model identifies at-risk customers, and a targeted offer reduces churn to 4%. Customers lost = 4% × 10,000 = 400.
  4. Customers saved = 500 − 400 = 100, so revenue saved = 100 × ₹500 = ₹50,000 per month, or ₹6,00,000 per year.
  5. If the offer costs ₹100 per targeted customer and 800 customers are targeted, the cost is ₹80,000 per month, so this offer would lose money; the company should target fewer, higher-risk customers. Analytics shows not only the benefit but also when an action is not worth taking.

1.6 Evolution of analytic scalability

Analytic scalability is the ability of an analytics system to handle growing data and workload by adding resources. Early analysis was done by moving data out of a database onto a single analyst's machine, which became impossible as data grew. The next step was the enterprise data warehouse (EDW), a central, integrated store designed for reporting. In-database analytics then moved the computation to the data, running scoring and statistics inside the database instead of exporting rows. Massively parallel processing (MPP) databases split data across many nodes, each with its own processor and disk, and process the pieces simultaneously. Cloud computing made such capacity available on demand, grid computing pooled the spare power of many machines, and MapReduce and Hadoop brought parallel processing to cheap commodity hardware. Today, in-memory engines such as Spark keep data in RAM across a cluster for speeds many times faster than disk-based processing.

Worked example – why parallelism matters. An analyst must scan 10 TB of data, and one disk reads at 100 MB/s.

  1. Single machine: 10 TB = 10,000,000 MB, so time = 10,000,000 / 100 = 100,000 s ≈ 27.8 hours.
  2. MPP cluster of 100 nodes, each scanning 1/100 of the data in parallel: time = 100,000 / 100 = 1,000 s ≈ 16.7 minutes.
  3. Speed-up = 100,000 / 1,000 = 100×. In practice it is somewhat lower because of coordination overhead, but the principle is why every modern platform is distributed.

Read more on the blog: Evolution of Analytical Scalability.

1.7 Analytic process and tools

The analytic process is the sequence of steps that turns a business question into an answer that someone acts on. It begins by defining the problem: stating the question precisely and agreeing how success will be measured. The analyst then collects data from the relevant internal and external sources and prepares it by cleaning errors, handling missing values, integrating sources and transforming variables. Exploratory data analysis (EDA) follows, using summary statistics and charts to understand distributions and relationships. Next, the analyst builds models with statistical or machine-learning techniques and evaluates them on data they have not seen, so that accuracy is honestly measured. Finally the results are communicated and deployed, and the model is monitored so that it can be retrained when the data changes. The process is iterative: findings at a later step often send the analyst back to an earlier one.

Analytics is a loop: monitoring sends the analyst back to the problem1. Define problem2. Collect data3. Prepare data4. Explore (EDA)5. Build models6. Evaluate7. Deploy, report8. Monitorrefine or retrain
The analytic process · 8 steps, 1 feedback loop

The steps run left to right across the top and back along the bottom; the dashed line shows that monitoring results send the analyst back to redefine the problem or retrain the model.

Tools used at each step

Step Typical tools
Collection and storage SQL databases, Hadoop HDFS, NoSQL (MongoDB, Cassandra), cloud storage
Preparation Python (pandas), R (dplyr), Excel, Power Query, Talend
Analysis and modelling R, Python (scikit-learn), SAS, SPSS, Spark MLlib
Visualisation and reporting Tableau, Power BI, ggplot2, matplotlib

Worked example – the process applied. A college wants to know whether attendance affects marks.

  1. Problem: "Does attendance predict final marks, and by how much?"
  2. Data: attendance (%) and marks for 5 students — (60, 52), (70, 60), (80, 68), (90, 75), (100, 85).
  3. Preparation: no missing values; both variables numeric.
  4. EDA: mean attendance = (60 + 70 + 80 + 90 + 100) / 5 = 80%; mean marks = (52 + 60 + 68 + 75 + 85) / 5 = 340 / 5 = 68.
  5. Model: a straight line fitted by least squares (calculated in full in section 3.1) gives marks ≈ 0.81 × attendance + 3.2.
  6. Communicate: "Each extra 10% of attendance is associated with about 8.1 more marks."

1.8 Analysis vs reporting

Reporting and analysis are related but serve different purposes. Reporting organises data into summaries — tables, dashboards and scheduled reports — that tell people what is happening. It is usually routine, automated and pushed to users on a fixed schedule. Analysis interrogates data to explain why something happened and to recommend what to do. It is ad hoc, exploratory and pulled by a specific question, and it requires human judgement. Reporting raises questions; analysis answers them. The two work together: good reports point analysts to where they should look.

Basis Reporting Analysis
Purpose Monitor and inform Explain and recommend
Question What happened? Why did it happen? What next?
Approach Push — delivered on a schedule Pull — triggered by a question
Output Dashboards, tables, standard reports Insights, findings, recommendations
Nature Routine, mostly automated Ad hoc, needs human judgement
Value Raises questions Answers questions, drives action

Worked example – from report to analysis. The monthly sales report shows (₹ lakh):

Region February March
North 40 40
South 30 30
East 30 15
Total 100 85
  1. Reporting states the fact: total sales fell from 100 to 85, a change of (85 − 100) / 100 = −15%.
  2. Analysis breaks the fall down by region: North changed by 0, South by 0, East by 15 − 30 = −15. East explains 15 / 15 = 100% of the decline.
  3. Further analysis finds that a competitor opened two stores in the East region in March and cut prices by 10%.
  4. Recommendation: a targeted price response in the East only, instead of a company-wide discount that would sacrifice margin in the 70% of sales that were unaffected.

1.9 Modern data analytic tools

Modern analytic tools fall into a few families. Programming languages — R and Python — are free, have thousands of packages for statistics and machine learning, and give full control over every step. Database and query tools — SQL, Hive and Spark SQL — retrieve and aggregate data where it is stored. Big Data frameworks — Hadoop and Apache Spark — distribute processing across clusters. Business intelligence and visualisation tools — Tableau, Power BI and Qlik — let users build interactive dashboards by drag and drop. Statistical suites — SAS, SPSS and Stata — are widely used in industry and research for validated statistical procedures. Visual workflow tools — KNIME, RapidMiner and Orange — build analyses as diagrams without code, and spreadsheets such as Excel remain the most common tool for small datasets.

Tool Type Best for Cost
R Language Statistics, research, visualisation Free
Python Language Machine learning, automation, deep learning Free
Apache Spark Big Data engine Large-scale processing, streaming Free
Tableau / Power BI BI dashboards Interactive business reporting Paid (free tiers)
SAS / SPSS Statistical suite Regulated industries, survey analysis Paid
KNIME / RapidMiner Visual workflow Analytics without coding Free / paid

Read more on the blog: Modern Data Analytical Tools.

1.10 Applications of data analytics

Data analytics is applied wherever decisions depend on data. In healthcare it predicts disease risk and hospital readmission, analyses medical images and tracks epidemics. In banking and finance it detects fraudulent transactions, scores credit risk and drives algorithmic trading. In retail and e-commerce it powers market-basket analysis, recommendation systems and dynamic pricing. Manufacturing uses it for predictive maintenance and quality control, telecom for churn prediction and network optimisation, and transport for route optimisation and ride-demand forecasting. Governments apply it to smart-city planning, tax-evasion detection and crime-hotspot mapping; education uses it to predict student performance and dropout; agriculture uses it for crop-yield prediction and weather advisories; and sports and media use it for player analysis and content recommendation.

Worked example – evaluating a fraud-detection application. A bank processes 100,000 card transactions in a day, of which 200 are actually fraudulent. Its model flags 300 transactions as suspicious, and 180 of those are real frauds.

  1. True positives (TP) = 180; false positives (FP) = 300 − 180 = 120; false negatives (FN) = 200 − 180 = 20.
  2. Precision = TP / (TP + FP) = 180 / 300 = 60% — six in ten alerts are genuine.
  3. Recall = TP / (TP + FN) = 180 / 200 = 90% — the model catches nine in ten frauds.
  4. Accuracy = (100,000 − 120 − 20) / 100,000 = 99.86%. This looks excellent but is misleading, because a model that flags nothing would still score 99.8%. For rare events, precision and recall are the measures that matter.

Blog lab: Lab 2 – Working with data in R

☰ All chaptersChapter 2 →

Independent study notes written in my own words, for learning purposes. Not affiliated with any university or course provider. R, RStudio, Hadoop, Amazon S3, Tableau, Power BI and other product names are trademarks of their respective owners.