Python Libraries
for Data Analysis
Learn how NumPy, Pandas, Matplotlib, Seaborn and Scikit-learn work together to convert raw data into structured evidence, visual insight and defensible predictions.
Why Python uses a library ecosystem
Data analysis is not a single operation. It is a sequence of connected tasks: numerical calculation, data organization, inspection, transformation, visualization, modelling and evaluation.
Python provides a general-purpose programming language, while specialist libraries supply efficient tools for particular stages of analysis. NumPy handles fast numerical arrays. Pandas organizes labelled data into Series and DataFrames. Matplotlib provides the basic plotting grammar. Seaborn makes statistical relationships easier to see. Scikit-learn offers a consistent workflow for preprocessing, machine learning and model evaluation.
1. Installation and standard imports
Install the libraries once in the Python environment used by your notebook, IDE or command line. The installation command downloads the packages and their required dependencies.
pip install numpy pandas matplotlib seaborn scikit-learn
In Jupyter Notebook or Google Colab, a leading exclamation mark executes the command through the system shell:
!pip install numpy pandas matplotlib seaborn scikit-learn
Standard aliases reduce repetition and make code recognizable to other analysts:
import numpy as np import pandas as pd import matplotlib.pyplot as plt import seaborn as sns from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
ModuleNotFoundError.2. NumPy: calculation-ready arrays
Python lists are flexible containers, but numerical analysis benefits from a structure designed for compact storage and vectorized calculation. NumPy provides the ndarray, an n-dimensional array whose elements normally share one data type.
marks_list = [75, 82, 68, 90, 88] marks = np.array(marks_list) print(marks) print(type(marks)) print(marks.dtype)
An array is more than a converted list. It enables one operation to be applied across many values without writing an explicit loop. This approach is called vectorization.
updated_marks = marks + 5
scaled_marks = marks * 1.10
print("Original:", marks)
print("Updated:", updated_marks)
print("Scaled:", scaled_marks)
With a Python list, marks_list + [5] appends another element. With a NumPy array, marks + 5 adds five to every element. The meaning of an operator therefore depends on the data structure.
marks.sum(), marks.mean(), marks.min() and marks.max() summarize the array.marks.std() describes how far values typically vary around their mean.marks[marks >= 80] returns only elements meeting the condition.print("Sum:", marks.sum())
print("Mean:", marks.mean())
print("Maximum:", marks.max())
print("Minimum:", marks.min())
print("Standard deviation:", marks.std())
print("Marks at least 80:", marks[marks >= 80])
3. Multidimensional NumPy arrays
A two-dimensional array can represent students by rows and subjects by columns. Its shape reports the number of elements along each dimension, while ndim reports the number of dimensions.
student_marks = np.array([
[80, 75, 85],
[70, 68, 74],
[90, 88, 92]
])
print(student_marks.shape) # (3, 3)
print(student_marks.ndim) # 2
| Expression | Meaning | Result |
|---|---|---|
student_marks[0] | First row | All marks for the first student |
student_marks[1, 1] | Specific cell | Second student, second subject |
student_marks[:, 0] | First column | First-subject marks for all students |
student_marks.mean(axis=0) | Column means | Mean for each subject |
student_marks.mean(axis=1) | Row means | Mean for each student |
The axis argument determines the direction in which an operation collapses the array. Understanding axes is essential because the same aggregation can answer different questions depending on whether it summarizes rows or columns.
4. Pandas Series: one-dimensional labelled data
A Pandas Series combines values with an index. The index gives each observation a meaningful label, making the structure more expressive than a plain array when identity matters.
marks = pd.Series(
[78, 85, 90, 72, 88],
index=["Amit", "Neha", "Ravi", "Pooja", "Karan"]
)
print(marks["Ravi"]) # label-based access
print(marks.iloc[0:3]) # position-based access
print(marks[marks >= 80]) # Boolean filtering
Label-based alignment is a major Pandas advantage. When operations combine two Series, Pandas aligns values by index label rather than assuming that equal positions describe the same observation. This reduces some classes of manual matching error, but it also means analysts must inspect indexes carefully.
5. Pandas DataFrame: observations and variables
A DataFrame is a two-dimensional labelled table. Rows commonly represent observations such as students, customers or transactions. Columns represent variables such as age, marks, product, price or date.
student_data = {
"Name": ["Amit", "Neha", "Ravi", "Pooja", "Karan"],
"Age": [20, 21, 19, 20, 22],
"Marks": [78, 85, 90, 72, 88],
"Department": ["CSE", "CSE", "ECE", "ME", "CSE"]
}
df = pd.DataFrame(student_data)
print(df)
A DataFrame resembles an Excel worksheet or database table, but it is designed for reproducible, programmable analysis. Every selection, transformation and summary can be expressed as code and executed again when new data arrives.
Common construction patterns
| Source structure | Best interpretation | Typical use |
|---|---|---|
| Dictionary of lists | Each key becomes a column | Column-oriented data already held in Python |
| List of lists | Each inner list becomes a row | Matrix-like data with supplied column names |
| List of dictionaries | Each dictionary becomes a record | JSON-style or record-oriented input |
| Dictionary of Series | Series align by index | Combining labelled measures |
6. Inspect before analysing
Analytical questions should come after structural questions. Before calculating averages or creating charts, verify that the expected rows and columns arrived and that Pandas assigned sensible data types.
print(df.head(3)) # first three rows print(df.tail(2)) # final two rows print(df.shape) # rows and columns print(df.columns) # column names print(df.dtypes) # storage type of each column df.info() # non-null counts, types and memory print(df.describe())
info() compares non-null counts with the total row count.describe() is useful only after verifying that the columns represent the intended measurements.7. Selecting and filtering data
Selection identifies locations or columns. Filtering retains rows that satisfy one or more conditions.
df["Name"] # one column: Series df[["Name", "Marks"]] # multiple columns: DataFrame df.loc[0:2, ["Name", "Marks"]] # labels df.iloc[0:3, 0:3] # integer positions df[df["Marks"] > 80] df[df["Department"] == "CSE"]
Multiple Boolean conditions require element-wise operators. Enclose each condition in parentheses:
high_engagement = df[
(df["Marks"] >= 80) &
(df["Attendance"] >= 85)
]
cse_or_ece = df[
(df["Department"] == "CSE") |
(df["Department"] == "ECE")
]
| Operator | Meaning | Example |
|---|---|---|
& | AND | Both conditions must be true |
| | OR | At least one condition must be true |
~ | NOT | Reverses a Boolean condition |
8. Transforming, sorting and grouping
Raw columns become more useful when they are converted into features that support decisions or communication. For example, marks can be translated into a performance category.
df["Performance"] = np.where(
df["Marks"] >= 80,
"Good",
"Needs Improvement"
)
A custom function can express a rule with several outcomes:
def assign_grade(mark):
if mark >= 90:
return "A+"
elif mark >= 80:
return "A"
elif mark >= 60:
return "B"
else:
return "Fail"
df["Grade"] = df["Marks"].apply(assign_grade)
Sorting changes presentation order; it does not change the underlying values. Grouping changes the unit of analysis by summarizing many individual rows into categories.
ranked = df.sort_values("Marks", ascending=False)
department_summary = (
df.groupby("Department")["Marks"]
.agg(["count", "mean", "min", "max"])
.round(2)
)
print(department_summary)
9. Data quality and missing values
Missing data is not merely a technical inconvenience. The appropriate treatment depends on why the value is missing, how much data is affected and how the variable will be used.
print(missing_df.isnull().sum())
# Example: numerical imputation
missing_df["Age"] = missing_df["Age"].fillna(
missing_df["Age"].mean()
)
# Remove rows only when justified
clean_df = missing_df.dropna()
Caution: dropna() can silently remove valuable observations and may bias the remaining dataset. Record the rule and compare the dataset size before and after cleaning.
10. Matplotlib: selecting a chart for the question
A useful chart begins with an analytical question. Chart type should follow the relationship being examined.
| Chart | Question answered | Typical data |
|---|---|---|
| Line chart | How does a measure change over ordered time? | Month and sales |
| Bar chart | How do categories compare? | Department and average marks |
| Histogram | How is one numeric variable distributed? | Marks or age |
| Scatter plot | How do two numeric variables relate? | Study hours and marks |
months = ["Jan", "Feb", "Mar", "Apr", "May"]
sales = [120, 150, 140, 180, 210]
plt.plot(months, sales, marker="o")
plt.title("Monthly Sales")
plt.xlabel("Month")
plt.ylabel("Sales")
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
Titles, units, axis labels and legends carry analytical meaning. A reader should be able to identify what was measured, for whom, and over what period without guessing.
11. Seaborn: statistical relationships
Seaborn works with Pandas DataFrames and offers concise functions for common statistical graphics. Its default styling and semantic mappings make grouped relationships easier to read.
sns.scatterplot(
data=study_df,
x="Study_Hours",
y="Marks"
)
plt.show()
sns.boxplot(
data=department_df,
x="Department",
y="Marks"
)
plt.show()
sns.heatmap(
correlation_matrix,
annot=True,
cmap="coolwarm",
vmin=-1,
vmax=1
)
plt.show()
12. Scikit-learn: from description to prediction
Machine learning extends analysis by estimating outcomes for unseen cases. A valid workflow separates training data from testing data so that evaluation measures performance on observations not used to fit the model.
X contains predictor variables; y contains the outcome to predict.Training data fits the model; test data estimates generalization.
The estimator learns a relationship from the training observations.
Compare predictions with the unseen test outcomes.
Linear regression demonstration
X = df[["Study_Hours"]]
y = df["Marks"]
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.20,
random_state=42
)
model = LinearRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)
Simple linear regression estimates a straight-line relationship between a predictor and a numeric target:
The coefficient estimates the expected change in predicted marks associated with one additional study hour, assuming the fitted relationship is appropriate for the data.
Model evaluation
mae = mean_absolute_error(y_test, predictions)
mse = mean_squared_error(y_test, predictions)
r2 = r2_score(y_test, predictions)
print("MAE:", mae)
print("MSE:", mse)
print("R-squared:", r2)
| Metric | Meaning | Preferred direction |
|---|---|---|
| MAE | Average absolute distance between actual and predicted values | Lower is better |
| MSE | Average squared error; penalizes large errors more strongly | Lower is better |
| R² | Proportion of variation explained relative to a mean-only baseline | Closer to 1 is generally better |
A very small classroom dataset can demonstrate the code sequence, but it cannot establish real-world model validity. Reliable modelling also requires representative data, assumption checks, leakage prevention and context-appropriate validation.
Predicting a new observation
new_student = pd.DataFrame({
"Study_Hours": [7.5]
})
predicted_marks = model.predict(new_student)
print(predicted_marks)
The result is an estimate conditional on the pattern learned from the available data. Predictions should not be interpreted as guaranteed outcomes.
13. The complete analytical pipeline
Every stage should leave an auditable object: an array, DataFrame, cleaned table, summary, chart, fitted model or evaluation metric. This makes the workflow reproducible and easier to review.
Series versus DataFrame
| Feature | Series | DataFrame |
|---|---|---|
| Dimensions | One-dimensional | Two-dimensional |
| Structure | One labelled sequence | Rows and multiple labelled columns |
| Typical use | One measure or variable | Complete records with several variables |
| Created from | List, dictionary or array | Lists, dictionaries, arrays or files |
| Spreadsheet analogy | One column | A complete worksheet |
14. Integrated classroom exercise
Create the dataset below and reproduce the full workflow.
df = pd.DataFrame({
"Name": ["A", "B", "C", "D", "E"],
"Study_Hours": [2, 4, 5, 3, 6],
"Attendance": [65, 80, 90, 72, 88],
"Assignment_Score": [55, 70, 85, 62, 80],
"Marks": [48, 65, 82, 58, 78]
})
- Display the first three records.
- Calculate the average marks.
- Filter students whose marks are above 60.
- Add a Pass/Fail column using a threshold of 40.
- Sort the DataFrame by marks in descending order.
- Create a bar chart of student marks.
- Create a scatter plot of study hours against marks.
- Calculate the correlation matrix and display a heatmap.
- Train a linear regression model using study hours, attendance and assignment score.
- Evaluate the predictions and explain why the small sample limits interpretation.
Suggested solution sequence
print(df.head(3))
print("Average marks:", df["Marks"].mean())
above_60 = df[df["Marks"] > 60]
df["Result"] = np.where(df["Marks"] >= 40, "Pass", "Fail")
df = df.sort_values("Marks", ascending=False)
sns.barplot(data=df, x="Name", y="Marks")
plt.title("Marks by Student")
plt.show()
sns.scatterplot(data=df, x="Study_Hours", y="Marks")
plt.title("Study Hours and Marks")
plt.show()
numeric_df = df[
["Study_Hours", "Attendance", "Assignment_Score", "Marks"]
]
sns.heatmap(numeric_df.corr(), annot=True, cmap="coolwarm")
plt.show()
X = df[["Study_Hours", "Attendance", "Assignment_Score"]]
y = df["Marks"]