The fundamentals of machine learning in python

Written by Glenn Junsay Pansensoy · Programming · 166 min read

AI & ML Fundamentals – Machine Learning with Python 2026 | Poetic Bytes

Back to Home

Python 3.12+ · AI & ML Foundations 2026

AI & Machine Learning
with Python — From First Principles to Real Models

Name: AI & ML Fundamentals – Machine Learning with Python 2026
Author: Glenn Junsay Pansensoy

A complete, free foundation in artificial intelligence and machine learning. Learn the ideas, the math and the code — build algorithms from scratch, then use scikit-learn, TensorFlow and PyTorch, and finish with projects you can run in your browser.

ML learners

Runnable examples

Topic modules

4.8

Start the Course Explore ML Libraries Back to Home

Why this code?

This is the first complete ML pipeline you will see. It demonstrates loading a classic dataset, splitting into training/test sets, training a Random Forest classifier, and evaluating accuracy. Its significance lies in showing the entire workflow in a few lines – a pattern you will reuse for every supervised learning problem.

Why used: To give you a tangible example of ML in action – from data to prediction.
How it operates: load_iris() returns features (X) and labels (y). train_test_split randomly splits them so we can test on unseen data. RandomForestClassifier builds an ensemble of decision trees. fit() trains the model, predict() makes class predictions, and accuracy_score compares them with the true labels.
Effect on output: It prints a single number – accuracy (e.g., 1.00) – telling you how many test samples were correctly classified.
Role in program: It sets the stage for everything that follows – you will now learn each step in detail.

first_ml_model.py

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

iris = load_iris()
X, y = iris.data, iris.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = RandomForestClassifier(n_estimators=100)
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, y_pred):.2f}")

$ python first_ml_model.py
Accuracy: 1.00

🗺️ Course Map — 16 Modules, One Path

From "what is AI?" to building, evaluating and responsibly shipping models. Follow the modules in order, or jump to what you need.

M1 · AI & ML Foundations M2 · Math for ML M3 · Working With Data M4 · Types of ML M5 · The ML Workflow M6 · Build From Scratch M7 · Generalization M8 · The Python Toolbox M9 · Supervised Learning M10 · Unsupervised Learning M11 · Evaluation M12 · Deep Learning M13 · NLP & Language Models M14 · Reinforcement Learning M15 · Responsible AI & Production M16 · Capstone Projects

🧭 12-week roadmap 📖 Glossary 🎯 Self-check quiz ❓ FAQ

🎓 What you will be able to do

Explain the difference between AI, machine learning and deep learning — and where rule-based systems still win.
Read the maths that appears in ML papers and code: vectors, matrices, derivatives, probability.
Explore, clean and split a dataset without leaking information from the test set.
Build a neuron, a regression, a classifier, a decision tree, a clustering algorithm and a small neural network from scratch.
Recognise and fix overfitting; choose and interpret the right evaluation metric.
Train, compare and tune models with scikit-learn, and know when to reach for TensorFlow or PyTorch.
Describe how language models and attention work, and how reinforcement learning differs from supervised learning.
Check a model for bias, explain its decisions, and monitor it after deployment.

How to use this page

Every code block has a Run button — Python executes right in your browser, nothing to install. Below most blocks you will find an Expected output box showing what the code printed when it was tested (Python 3.13, NumPy 2.5, scikit-learn 1.9). Numbers can differ slightly in the last digits in your browser, and anything involving random numbers is seeded so it stays repeatable. Blocks marked run locally need TensorFlow, PyTorch or an internet download, so run those on your own computer or in Google Colab.

The best way to learn: predict the output first, run the code, then change one number and run it again.

Module 1

🌐 AI & ML Foundations

What "artificial intelligence" actually means, where it came from, and how machine learning fits inside it

🤖 What is Artificial Intelligence?

Artificial intelligence (AI) is the field of building systems that perform tasks which normally require human intelligence: perceiving the world, reasoning, learning, planning, and using language. A useful way to think about any AI system is as an agent — something that perceives its environment, decides, and acts to reach a goal, over and over in a loop.

perceive (sensors / data) → decide (model / rules / search) → act → new situation → repeat

Two labels you will hear constantly:

Narrow AI — systems built for one kind of task: spam filtering, route planning, speech recognition, image classification, chatbots. Every AI system in everyday use today is narrow, even when it is impressively capable inside its domain.
Artificial general intelligence (AGI) — a hypothetical system with human-level flexibility across almost any intellectual task. Researchers disagree about what exactly counts as AGI, whether current approaches lead there, and how long it might take.

🪆 AI ⊃ Machine Learning ⊃ Deep Learning

These three terms are nested, not interchangeable. Each is a subset of the one before it.

Artificial Intelligence — any technique that produces intelligent behaviour

Machine Learning — learns patterns from data

Deep Learning — many-layer neural networks

Modern generative AI (large language models, image generators) belongs mostly to deep learning. But AI also includes approaches with no learning at all — like the maze solver below — and machine learning includes many methods that are not neural networks, such as decision trees and k-nearest neighbours.

🧰 Five Ways to Build an Intelligent System

Approach	Where its behaviour comes from	Example	Main limit
Rule-based (symbolic)	Hand-written `if/then` rules and logic	Tax software, early chatbots, expert systems	Cannot handle situations nobody wrote a rule for
Search & planning	Exploring possible action sequences to reach a goal	GPS routing, chess engines, puzzle solvers	Search space explodes as problems grow
Machine learning	Patterns learned from labelled or unlabelled data	Spam filters, price prediction, churn models	Needs good data; can fail silently on new situations
Deep learning	Multi-layer neural networks learn their own features	Image recognition, speech, language models	Data- and compute-hungry; harder to interpret
Reinforcement learning	Trial and error guided by rewards	Game-playing agents, robot control	Needs huge numbers of trials; reward design is tricky

💡 Real products usually combine several. A voice assistant uses deep learning to recognise speech, rules to handle a few critical commands, and search to look things up.

Classic AI — no learning involved

🧭 An AI That Plans: Breadth-First Search

Before machine learning dominated, much of AI was search. This agent never learns anything — it systematically explores the maze, one step further at a time, until it finds the goal. Because it explores in widening rings, the first path it finds is guaranteed to be the shortest.

from collections import deque

# 0 = free cell, 1 = wall. The agent starts top-left and must reach bottom-right.
maze = [
    [0, 0, 1, 0, 0],
    [1, 0, 1, 0, 1],
    [0, 0, 0, 0, 0],
    [0, 1, 1, 1, 0],
    [0, 0, 0, 1, 0],
]
start, goal = (0, 0), (4, 4)

def solve(maze, start, goal):
    queue = deque([start])
    came_from = {start: None}
    while queue:
        r, c = queue.popleft()
        if (r, c) == goal:
            break
        for dr, dc in [(1, 0), (-1, 0), (0, 1), (0, -1)]:
            nr, nc = r + dr, c + dc
            inside = 0 <= nr < len(maze) and 0 <= nc < len(maze[0])
            if inside and maze[nr][nc] == 0 and (nr, nc) not in came_from:
                came_from[(nr, nc)] = (r, c)
                queue.append((nr, nc))
    if goal not in came_from:
        return None
    path, node = [], goal
    while node is not None:
        path.append(node)
        node = came_from[node]
    return path[::-1]

path = solve(maze, start, goal)
print("Shortest path length:", len(path) - 1, "steps")
print("Path:", path)

Expected output (click to compare with your run)

Shortest path length: 8 steps
Path: [(0, 0), (0, 1), (1, 1), (2, 1), (2, 2), (2, 3), (2, 4), (3, 4), (4, 4)]

How the search works

Why used: It shows intelligent behaviour (finding a shortest route) produced purely by an algorithm — the contrast that makes machine learning's "learn the rules from data" idea meaningful.
How it operates: A deque (queue) holds cells to explore. Take the oldest cell, look at its four neighbours, and queue every free, unvisited neighbour while recording where we came from in came_from. Once the goal is reached, follow came_from backwards to rebuild the path, then reverse it.
Effect on output: It prints the number of steps and the exact route, 8 steps through the maze.
Role in program: It is the "decide" step of the agent loop, implemented as search instead of learning.

✨ Try it: change a 0 to 1 to wall off the route and watch the path change — or block everything and see it return None.

📜 A Short History of AI

Knowing the history explains why the field swings between excitement and caution — and why the current boom is different from earlier ones.

1950

Alan Turing publishes Computing Machinery and Intelligence, proposing the "imitation game" (now called the Turing test) as a way to ask whether machines can think.

1956

The Dartmouth workshop brings researchers together; John McCarthy's term "artificial intelligence" names the field.

1957–58

Frank Rosenblatt introduces the perceptron, an early trainable neural-network model — the ancestor of the neuron you build in Module 6.

1966

Joseph Weizenbaum's ELIZA imitates a therapist using pattern-matching rules — a cousin of the rule-based chatbot in Module 6.

1970s & late 1980s

"AI winters": funding and interest drop after large promises go unmet.

1986

Rumelhart, Hinton and Williams popularise backpropagation, the algorithm that makes training multi-layer networks practical.

1997

IBM's Deep Blue defeats world chess champion Garry Kasparov — a triumph of search, not learning.

2012

AlexNet wins the ImageNet image-recognition challenge by a wide margin using GPUs and deep learning, starting the modern deep-learning boom.

2016

DeepMind's AlphaGo defeats Go champion Lee Sedol, combining deep neural networks, search and reinforcement learning.

2017

The paper Attention Is All You Need introduces the Transformer, the architecture behind today's language models (Module 13).

2022

ChatGPT's public release brings large language models to a mass audience, and generative AI becomes an everyday tool.

💡 Three ingredients drove the recent progress: much more data, much more computing power (especially GPUs), and better algorithms. Remove any one and the boom stalls.

🤖 So — What is Machine Learning?

Machine Learning (ML) is a subset of artificial intelligence that enables systems to learn from data and improve over time without being explicitly programmed. Instead of writing rules, you show the computer examples — and it learns the patterns.

📊 Why ML Matters in 2026

Machine learning is transforming every industry — from healthcare (diagnosing diseases from medical images) to finance (fraud detection and algorithmic trading) to retail (recommendation systems and demand forecasting). Python has become the dominant language for ML because of its rich ecosystem of libraries and its accessibility to both researchers and practitioners.

This course teaches you the core concepts that every AI and ML practitioner needs: the mathematics behind learning, data preparation, supervised and unsupervised learning, evaluation, neural networks, language models, reinforcement learning, and responsible deployment — using Python's NumPy, pandas, scikit-learn, TensorFlow and PyTorch. You'll practise on classic datasets and build projects you can show to employers.

16 topic modules 52 runnable examples 15,000+ learners 4.8 / 5 learner rating

🧠 ML vs. Traditional Programming

Traditional Programming

Rules (code) + Data → Answers

Machine Learning

Data + Answers → Rules (model)

In ML, we provide the computer with examples (data) and the correct answers (labels). The model learns the underlying patterns that map inputs to outputs — and can then make predictions on new, unseen data.

💡 This shift — from hand-coded rules to learned patterns — is what makes ML so powerful. The model discovers relationships that humans might never spot.

Module 2

🧮 Math for Machine Learning — Just Enough

You do not need a maths degree. Four ideas — vectors and matrices, derivatives, probability and statistics — explain almost everything you will meet. Each one below comes with code you can run.

📐 Linear Algebra — Data as Vectors and Matrices

A single sample (one house, one email, one image) is a vector: an ordered list of numbers, one per feature. A whole dataset is a matrix: one row per sample, one column per feature. Once data is in this shape, a computer can process thousands of samples with one operation.

dot(a, b) = a₁·b₁ + a₂·b₂ + … + aₙ·bₙ length |a| = √(a₁² + a₂² + … + aₙ²) cosine similarity = dot(a, b) / (|a| · |b|) (1 = same direction, 0 = unrelated, −1 = opposite)

import numpy as np

a = np.array([2.0, 1.0, 3.0])
b = np.array([1.0, 0.0, -1.0])

print("a + b      =", a + b)
print("3 * a      =", 3 * a)
print("dot(a, b)  =", np.dot(a, b))
print("length |a| =", round(float(np.linalg.norm(a)), 4))

cos_sim = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
print("cosine similarity =", round(float(cos_sim), 4))

Expected output (click to compare with your run)

a + b      = [3. 1. 2.]
3 * a      = [6. 3. 9.]
dot(a, b)  = -1.0
length |a| = 3.7417
cosine similarity = -0.189

Vectors in NumPy

Why used: The dot product is the core operation of machine learning — a neuron's weighted sum, a similarity score, an attention score are all dot products.
How it operates: a + b and 3 * a work element by element. np.dot multiplies matching positions and sums them. np.linalg.norm gives the length. Dividing the dot product by both lengths isolates the angle between the vectors — the cosine similarity.
Effect on output: The dot product is −1 (since 2·1 + 1·0 + 3·(−1) = −1) and the cosine similarity is about −0.19, meaning the vectors point in slightly opposing directions.
Role in program: It builds intuition for how "similar" is turned into a number — the basis of search, recommendations and embeddings.

A neural-network layer is just a matrix multiplication. When you multiply a data matrix by a weight matrix, every sample is transformed at once:

import numpy as np

# 4 samples, 3 features each: [size_m2, bedrooms, age_years]
X = np.array([[ 50, 1, 20],
              [ 80, 2, 10],
              [120, 3,  5],
              [200, 4,  1]], dtype=float)

w = np.array([1.5, 10.0, -0.8])     # one weight per feature
b = 20.0

scores = X @ w + b                  # (4, 3) @ (3,) -> (4,)
print("X shape:", X.shape, "| w shape:", w.shape)
print("scores:", scores)

# A "layer" of 2 neurons: the weight matrix has shape (3 inputs, 2 outputs)
W = np.array([[ 0.01, -0.02],
              [ 0.50,  0.10],
              [-0.04,  0.03]])
layer_out = X @ W                   # (4, 3) @ (3, 2) -> (4, 2)
print("layer output shape:", layer_out.shape)
print(layer_out.round(2))

Expected output (click to compare with your run)

X shape: (4, 3) | w shape: (3,)
scores: [ 89.  152.  226.  359.2]
layer output shape: (4, 2)
[[ 0.2  -0.3 ]
 [ 1.4  -1.1 ]
 [ 2.5  -1.95]
 [ 3.96 -3.57]]

Matrix multiplication and shapes

Why used: Vectorised matrix maths replaces slow Python loops and is exactly how GPUs speed up neural networks.
How it operates: X @ w multiplies a (4×3) matrix by a length-3 vector to give 4 scores — one per sample. X @ W with a (3×2) matrix gives a (4×2) result: two "neurons" applied to all 4 samples.
Effect on output: The shapes printed confirm the rule below, and the layer output has 2 values per sample.
Role in program: This is a layer of a neural network in miniature (a real layer would add a bias and an activation function).

💡 The shape rule: in (n, m) @ (m, k) the two inner numbers must match, and the outer numbers give the result's shape (n, k). Shape mismatches are the most common beginner error in ML code — print .shape often.

📉 Calculus — How Models Learn

Training a model means finding the parameters that make its error as small as possible. The derivative tells you the slope of the error at your current position; the gradient is the same idea for many parameters at once. Gradient descent walks downhill by repeatedly stepping against the slope.

new_value = old_value − learning_rate × slope (slope positive → move left; slope negative → move right; slope ≈ 0 → you have reached a minimum)

# f(x) = (x - 3)^2 has its lowest point at x = 3.
def f(x):
    return (x - 3) ** 2

def numerical_derivative(func, x, h=1e-6):
    return (func(x + h) - func(x - h)) / (2 * h)

def analytic_derivative(x):
    return 2 * (x - 3)

for x in [0.0, 2.0, 3.0, 5.0]:
    print(f"x={x:>4}  numeric={numerical_derivative(f, x):8.4f}  analytic={analytic_derivative(x):8.4f}")

# Gradient descent: repeatedly step AGAINST the slope
x = 10.0
learning_rate = 0.2
for step in range(11):
    if step % 2 == 0:
        print(f"step {step:2d}: x={x:.4f}  f(x)={f(x):.4f}")
    x -= learning_rate * analytic_derivative(x)

Expected output (click to compare with your run)

x= 0.0  numeric= -6.0000  analytic= -6.0000
x= 2.0  numeric= -2.0000  analytic= -2.0000
x= 3.0  numeric=  0.0000  analytic=  0.0000
x= 5.0  numeric=  4.0000  analytic=  4.0000
step  0: x=10.0000  f(x)=49.0000
step  2: x=5.5200  f(x)=6.3504
step  4: x=3.9072  f(x)=0.8230
step  6: x=3.3266  f(x)=0.1067
step  8: x=3.1176  f(x)=0.0138
step 10: x=3.0423  f(x)=0.0018

Derivatives and gradient descent

Why used: Every model you will train in this course — from linear regression to a language model — learns by this exact loop.
How it operates: The numerical derivative measures the slope by nudging x a tiny amount either side. It matches the exact formula 2(x−3). Gradient descent then starts far away at x = 10 and repeatedly subtracts learning_rate × slope.
Effect on output: The two derivative columns agree, and x glides from 10 toward 3 while f(x) falls toward 0. Steps get smaller as the slope flattens.
Role in program: It is the template for Programs 2, 3, 5 and 9 in Module 6.

Networks have many layers, so their error is a function of a function of a function. The chain rule lets us compute the slope through all those layers by multiplying local slopes. Applying the chain rule backwards through a network is what backpropagation is.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

h = 1e-6
z = 0.7
s = sigmoid(z)
analytic = s * (1 - s)                                   # the formula used in the neuron program
numeric = (sigmoid(z + h) - sigmoid(z - h)) / (2 * h)
print("sigmoid(0.7)         =", round(float(s), 6))
print("analytic derivative  =", round(float(analytic), 6))
print("numerical derivative =", round(float(numeric), 6))

# Chain rule: d/dw sigmoid(w * x) = sigmoid'(w * x) * x
w, x = 0.5, 2.0
chain = sigmoid(w * x) * (1 - sigmoid(w * x)) * x
numeric_chain = (sigmoid((w + h) * x) - sigmoid((w - h) * x)) / (2 * h)
print("chain rule result    =", round(float(chain), 6))
print("numerical check      =", round(float(numeric_chain), 6))

Expected output (click to compare with your run)

sigmoid(0.7)         = 0.668188
analytic derivative  = 0.221713
numerical derivative = 0.221713
chain rule result    = 0.393224
numerical check      = 0.393224

💡 Notice that the formula s × (1 − s) for the sigmoid's slope — the one used inside the neuron program in Module 6 — is confirmed by the numerical check.

🎲 Probability & Statistics — Reasoning Under Uncertainty

Real data is noisy, so ML is fundamentally about probabilities. Statistics gives you the vocabulary to describe a dataset; probability tells you how to update beliefs from evidence.

Mean / median — the typical value (the median ignores extreme outliers).
Variance / standard deviation — how spread out values are around the mean.
Normal distribution — the bell curve; about 68% of values fall within 1 standard deviation of the mean and about 95% within 2.
Conditional probability — P(A | B): the chance of A given that B happened.

import numpy as np

rng = np.random.default_rng(42)
heights = rng.normal(loc=165, scale=8, size=1000)      # simulated heights in cm

print("mean     :", round(float(heights.mean()), 2))
print("median   :", round(float(np.median(heights)), 2))
print("variance :", round(float(heights.var()), 2))
print("std dev  :", round(float(heights.std()), 2))

dist = np.abs(heights - heights.mean())
print("within 1 std:", round(float(np.mean(dist < heights.std())), 3), "(theory: about 0.683)")
print("within 2 std:", round(float(np.mean(dist < 2 * heights.std())), 3), "(theory: about 0.954)")

# Law of large numbers: the share of heads in fair coin flips approaches 0.5
flips = rng.integers(0, 2, size=100000)
for n in [10, 100, 1000, 100000]:
    print(f"{n:>6} flips -> proportion of heads = {flips[:n].mean():.4f}")

Expected output (click to compare with your run)

mean     : 164.77
median   : 165.05
variance : 62.56
std dev  : 7.91
within 1 std: 0.687 (theory: about 0.683)
within 2 std: 0.955 (theory: about 0.954)
    10 flips -> proportion of heads = 0.2000
   100 flips -> proportion of heads = 0.5600
  1000 flips -> proportion of heads = 0.4870
100000 flips -> proportion of heads = 0.5007

Describing data and the law of large numbers

Why used: Before modelling you must be able to summarise data, and you must know why small samples mislead.
How it operates: rng.normal simulates 1,000 heights. The code measures the centre and spread, then checks how many values land within 1 and 2 standard deviations. A second experiment flips a fair coin 100,000 times and tracks the share of heads for different sample sizes.
Effect on output: The measured shares land very close to the theoretical 68% and 95%. With 10 flips the share of heads can be far from 0.5, but it settles near 0.5 as the sample grows.
Role in program: It explains why we need big datasets and held-out test sets: small samples give unreliable conclusions.

Bayes' theorem shows how to update a belief when you see evidence — and exposes a trap called base-rate neglect.

P(A | B) = P(B | A) × P(A) / P(B)

# Bayes' theorem: P(A|B) = P(B|A) * P(A) / P(B)
p_fraud = 0.01                 # prior: 1% of transactions are fraud
p_flag_given_fraud = 0.99      # the detector catches 99% of fraud (recall)
p_flag_given_ok = 0.05         # but it also flags 5% of honest transactions

p_flag = p_flag_given_fraud * p_fraud + p_flag_given_ok * (1 - p_fraud)
p_fraud_given_flag = p_flag_given_fraud * p_fraud / p_flag

print(f"P(flagged)         = {p_flag:.4f}")
print(f"P(fraud | flagged) = {p_fraud_given_flag:.4f}")
print("Out of 1,000 flagged transactions, about", round(1000 * p_fraud_given_flag), "are truly fraud.")

Expected output (click to compare with your run)

P(flagged)         = 0.0594
P(fraud | flagged) = 0.1667
Out of 1,000 flagged transactions, about 167 are truly fraud.

💡 A detector that catches 99% of fraud sounds superb, yet when fraud is only 1% of transactions, most flagged transactions are still innocent (only about 1 in 6 is real fraud). This is exactly why precision and recall — not just accuracy — matter in Module 11, and why the Naive Bayes classifier in Module 6 starts from a prior.

🔗 Where the Maths Shows Up Later

Concept	You will see it in
Dot product	A neuron's weighted sum; attention scores; cosine similarity of text
Matrix multiplication	Every layer of a neural network; PCA
Derivative / gradient	Training linear and logistic regression, neural networks
Chain rule	Backpropagation
Mean & standard deviation	Feature scaling, drift detection
Normal distribution	Noise assumptions, weight initialisation
Bayes' theorem	Naive Bayes, reasoning about base rates
Expected value / averages	Loss functions (average error over the data)

✨ To go deeper later, the free book Mathematics for Machine Learning (Deisenroth, Faisal & Ong) covers all of this rigorously.

Module 3

🗃️ Working With Data

Practitioners spend most of their time on data, not algorithms. A model can only learn what the data shows it — "garbage in, garbage out."

🧱 The Vocabulary of Tabular Data

Term	Meaning	Example
Sample (row)	One observation	One customer
Feature (column)	An input measurement	age, income, city
Target / label	What you want to predict	churned (yes / no)
Numerical feature	A quantity you can do maths on	income, temperature
Categorical — nominal	Categories with no order	city, colour
Categorical — ordinal	Categories with a real order	free < basic < pro

Text, images, audio and time series are also data, but they need conversion into numbers first (Modules 12 and 13).

🔍 Exploratory Data Analysis (EDA)

Before training anything, look at your data. A good EDA answers: How big is it? What types are the columns? What is missing? What do the distributions look like? Are there impossible values or outliers? Which features relate to the target? Are the classes balanced?

import numpy as np
import pandas as pd

rng = np.random.default_rng(7)
n = 200
df = pd.DataFrame({
    "age": rng.integers(18, 70, size=n),
    "income": rng.normal(35000, 12000, size=n).clip(5000).round(0),
    "city": rng.choice(["Manila", "Cebu", "Davao", "Zamboanga"], size=n),
    "plan": rng.choice(["free", "basic", "pro"], size=n, p=[0.5, 0.3, 0.2]),
})
df["churned"] = ((df["income"] < 30000) & (rng.random(n) < 0.6)).astype(int)
df.loc[rng.choice(n, 12, replace=False), "income"] = np.nan    # inject missing values

print(df.head())
print("\nShape:", df.shape)
print("\nMissing values per column:")
print(df.isna().sum())
print("\nNumeric summary:")
print(df.describe().round(1))
print("\nPlan counts:")
print(df["plan"].value_counts())
print("\nChurn rate by plan:")
print(df.groupby("plan")["churned"].mean().round(3))
print("\nCorrelation with churn:")
print(df[["age", "income", "churned"]].corr()["churned"].round(3))

Expected output (click to compare with your run)

   age   income       city   plan  churned
0   67  10576.0      Davao  basic        1
1   50  31346.0  Zamboanga   free        0
2   53  24201.0       Cebu    pro        1
3   64  36969.0      Davao  basic        0
4   48  61937.0     Manila    pro        0

Shape: (200, 5)

Missing values per column:
age         0
income     12
city        0
plan        0
churned     0
dtype: int64

Numeric summary:
         age   income  churned
count  200.0    188.0    200.0
mean    44.2  33501.8      0.3
std     15.1  11168.5      0.4
min     18.0   5000.0      0.0
25%     31.0  25880.0      0.0
50%     44.0  34236.5      0.0
75%     57.0  41084.2      1.0
max     69.0  61937.0      1.0

Plan counts:
plan
free     97
basic    69
pro      34
Name: count, dtype: int64

Churn rate by plan:
plan
basic    0.290
free     0.227
pro      0.265
Name: churned, dtype: float64

Correlation with churn:
age       -0.019
income    -0.633
churned    1.000
Name: churned, dtype: float64

A first look at a dataset with pandas

Why used: EDA catches data problems — missing values, absurd ranges, imbalance — that would silently ruin a model.
How it operates: The code builds a synthetic customer table, then deliberately blanks 12 incomes. head and shape show size, isna().sum() counts gaps, describe summarises numeric columns, value_counts and groupby summarise categories, and corr measures linear relationships.
Effect on output: You see 12 missing incomes, the spread of each column, and that income has a strong negative correlation with churn (we generated the data that way; in real data, this is a pattern you would discover).
Role in program: It is the "collect and understand" half of the workflow before any modelling begins.

⚠️ Correlation is not causation. Low income correlating with churn does not prove low income causes churn — a hidden third factor could drive both.

🧹 Cleaning and Preparing Features

Missing values — drop the rows (if few), fill with the mean or median, fill with a model's estimate, or add a "was missing" indicator column.
Categorical encoding — one-hot encoding gives each nominal category its own 0/1 column; ordinal categories can map to ordered integers.
Scaling — standardisation (mean 0, standard deviation 1) or min-max scaling. It matters for distance-based and gradient-based models (k-NN, SVM, linear models, neural networks) but not for tree-based models.
Outliers — investigate them: sometimes errors to fix, sometimes the most important data points.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(0)
df = pd.DataFrame({
    "income": rng.normal(35000, 12000, 8).round(0),
    "city": ["Manila", "Cebu", "Davao", "Cebu", "Manila", "Davao", "Cebu", "Manila"],
    "plan": ["free", "basic", "pro", "free", "pro", "basic", "free", "basic"],
})

# 1) Nominal categories (no natural order) -> one-hot encoding
encoded = pd.get_dummies(df, columns=["city"], dtype=int)

# 2) Ordinal categories (a real order exists) -> integers that respect the order
encoded["plan"] = encoded["plan"].map({"free": 0, "basic": 1, "pro": 2})
print(encoded)

# 3) Scaling: learn the mean/std from TRAIN only, then apply to TEST (prevents leakage)
train, test = train_test_split(encoded, test_size=0.25, random_state=1)
scaler = StandardScaler().fit(train[["income"]])
print("\nmean and std learned from train:", scaler.mean_.round(1), scaler.scale_.round(1))
print("scaled train mean:", round(float(scaler.transform(train[["income"]]).mean()), 3) + 0.0)
print("scaled test mean :", round(float(scaler.transform(test[["income"]]).mean()), 3), "(not exactly 0, and that is normal)")

Expected output (click to compare with your run)

    income  plan  city_Cebu  city_Davao  city_Manila
0  36509.0     0          0           0            1
1  33415.0     1          1           0            0
2  42685.0     2          0           1            0
3  36259.0     0          1           0            0
4  28572.0     2          0           0            1
5  39339.0     1          0           1            0
6  50648.0     0          1           0            0
7  46365.0     1          0           0            1

mean and std learned from train: [37457.] [6772.]
scaled train mean: 0.0
scaled test mean : 1.044 (not exactly 0, and that is normal)

Encoding and scaling

Why used: Models need numbers, and many perform much better when numeric features share a similar scale.
How it operates: get_dummies expands city into one 0/1 column per city. The ordinal plan column maps to 0, 1, 2. The StandardScaler learns the mean and standard deviation from the training rows only, and then applies those same numbers to both train and test.
Effect on output: The encoded table is fully numeric. The scaled training income has mean 0, while the test income does not — as expected, because the test set was not used to compute the statistics.
Role in program: It introduces the rule that anything learned from data (like a mean) must be learned from training data alone — the subject of the next block.

✂️ Splitting Data — and the Leakage Trap

Set	Purpose	When you use it
Training	The model learns from it	Every time you call `fit()`
Validation	Compare models and tune settings	Repeatedly during development (or use cross-validation)
Test	One honest estimate of real-world performance	Once, at the very end

Data leakage happens when information from outside the training set sneaks into training, making results look better than they truly are. Common causes: scaling or selecting features using the whole dataset before splitting; duplicate rows on both sides of a split; features that secretly contain the answer (like "refund issued" when predicting churn); and, for time series, training on the future to predict the past (use time-ordered splits instead of random ones).

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RepeatedStratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(0)
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=5, random_state=0)
X = rng.normal(size=(100, 2000))      # 2,000 features of pure noise
y = rng.integers(0, 2, size=100)      # random labels: there is NOTHING to learn

# WRONG: choose the "best" features using ALL the data, then cross-validate
X_leaky = SelectKBest(f_classif, k=20).fit_transform(X, y)
leaky = cross_val_score(LogisticRegression(max_iter=1000), X_leaky, y, cv=cv).mean()

# RIGHT: do the feature selection inside each training fold with a Pipeline
pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=1000))
honest = cross_val_score(pipe, X, y, cv=cv).mean()

print(f"Leaky CV accuracy : {leaky:.2f}   <- looks impressive but is meaningless")
print(f"Honest CV accuracy: {honest:.2f}   <- near 0.50, as it must be for random labels")

Expected output (click to compare with your run)

Leaky CV accuracy : 0.78   <- looks impressive but is meaningless
Honest CV accuracy: 0.49   <- near 0.50, as it must be for random labels

Leakage in action

Why used: It is the most convincing demonstration that leakage is real, not theoretical.
How it operates: The data is pure noise with random labels, so no honest model can beat chance. The "wrong" approach picks the 20 features that best match the labels using all rows, then cross-validates — the held-out folds already influenced the choice. The "right" approach puts feature selection inside a Pipeline, so each fold selects features from its own training portion only.
Effect on output: The leaky score looks far better than chance (around 0.78) even though there is nothing to learn. The honest score sits near 0.50.
Role in program: It teaches the habit of wrapping every preprocessing step in a Pipeline, so cross-validation stays honest.

Module 4

🏷️ Types of Machine Learning

The three main paradigms — and when to use each one

📊

Supervised Learning

Learn a mapping from inputs to outputs using labelled data. Examples: classification (spam detection, image recognition) and regression (house price prediction, stock forecasting).

classificationregressionlabelled

🔮

Unsupervised Learning

Find hidden structure in unlabelled data. Examples: clustering (customer segmentation, grouping similar documents) and dimensionality reduction (visualisation, compression).

clusteringdimensionalityunlabelled

🎯

Reinforcement Learning

Learn by trial and error through rewards and penalties. Examples: game AI (AlphaGo), robotics, autonomous driving, recommendation systems.

rewardsagentsenvironments

Which one should you learn first?

Start with supervised learning — it's the most commonly used in industry and the foundation for most ML applications. Classification and regression problems appear everywhere: from spam filters to medical diagnosis.

Module 5

⚙️ The ML Workflow — 7 Steps to a Working Model

Every ML project follows the same essential pipeline

💡 The code snippets in steps 2, 4, 5, 6, and 7 below build on each other — run them in order, top to bottom. Variables like model and scaler stay available in your browser's Python session as you go, the same way cells in a notebook share state.

1. Define the Problem

What are you trying to predict? What is the business goal? Is it a classification or regression problem? What success looks like?

💡 Spend time here. A well-defined problem is half solved. Be specific: "Predict if a customer will churn in the next 30 days" is better than "Predict customer churn."

2. Collect and Prepare Data

Gather data from databases, APIs, or files. Handle missing values, encode categorical variables, scale numeric features, and split into training and test sets.

Data preparation code – why and how

This snippet covers the essential preprocessing steps: generating a sample dataset (standing in for a CSV you'd normally load), handling missing values by replacing them with the column mean, separating features (X) from target (y), splitting into training/test sets, and scaling features with StandardScaler.

Why used: Real-world data is messy; you must clean and transform it before feeding to a model. This code shows the standard pipeline.
How it operates: make_classification() generates a synthetic dataset right in the browser (in a real project you'd use pandas.read_csv() to load your own file instead). fillna() replaces missing values. train_test_split holds out 20% for testing. StandardScaler centres and scales features so they have mean=0 and variance=1 – crucial for many algorithms.
Effect on output: You get four variables: X_train_scaled, X_test_scaled, y_train, y_test. The scaling prevents features with larger magnitudes from dominating the model.
Role in program: This is the foundation – without proper data prep, the subsequent model training and evaluation would be unreliable.

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_classification

# Simulate a raw dataset, the way you might receive it from a CSV export
X_raw, y_raw = make_classification(n_samples=500, n_features=6, n_informative=4, random_state=42)
data = pd.DataFrame(X_raw, columns=[f"feature_{i}" for i in range(6)])
data["target"] = y_raw

# Simulate a few missing values, just like real-world data
rng = np.random.default_rng(42)
missing_idx = rng.choice(data.index, size=10, replace=False)
data.loc[missing_idx, "feature_0"] = np.nan
data.fillna(data.mean(numeric_only=True), inplace=True)

X = data.drop("target", axis=1)
y = data["target"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

print("Training samples:", X_train_scaled.shape[0])
print("Test samples:", X_test_scaled.shape[0])

3. Choose a Model

Select an algorithm based on your problem type and data size. For classification: Logistic Regression, Random Forest, XGBoost, or Neural Networks. For regression: Linear Regression, SVM, Gradient Boosting.

4. Train the Model

Feed the training data to the model. It learns the mapping from features to labels by adjusting its internal parameters.

Training a Random Forest

This code instantiates a RandomForestClassifier with 100 trees and then calls .fit() on the scaled training data from step 2 above. The .fit() method is where the learning happens – each tree is built on a random subset of the data, and their votes are combined to make a final prediction.

Why used: Random Forest is a powerful, robust algorithm that works well on tabular data and requires little tuning.
How it operates: During .fit(), the algorithm creates a forest of decision trees, each trained on a bootstrapped sample of the data. It also randomly selects a subset of features at each split, which reduces overfitting.
Effect on output: The model parameters (tree structures) are stored in the model object. You can now use it to make predictions on new data.
Role in program: This is the core – the trained model is the artifact that captures the learned patterns from your data.

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=100, max_depth=10, random_state=42)
model.fit(X_train_scaled, y_train)
print("Model trained on", X_train_scaled.shape[0], "samples.")

⚠️ Run step 2's snippet above first — this step reuses X_train_scaled and y_train from it.

5. Evaluate

Use metrics to measure performance on the test set. Accuracy, precision, recall, F1-score for classification. MAE, RMSE, R² for regression.

Model evaluation

This snippet shows how to obtain a classification report (precision, recall, f1-score per class) and a confusion matrix. These give you a detailed view of where the model is making errors.

Why used: Accuracy alone can be misleading, especially with imbalanced data. These metrics break down performance by class.
How it operates: model.predict() generates predictions on the test set. classification_report computes per-class metrics; confusion_matrix shows the counts of true positives, false positives, etc.
Effect on output: You see a textual report and a matrix – numbers that tell you which classes are confused. This helps you decide if the model is fit for purpose.
Role in program: Evaluation determines whether the model is good enough to deploy. It also guides further improvements (feature engineering, hyperparameter tuning).

from sklearn.metrics import classification_report, confusion_matrix

y_pred = model.predict(X_test_scaled)
print(classification_report(y_test, y_pred))
print("Confusion Matrix:", confusion_matrix(y_test, y_pred))

⚠️ Needs steps 2 and 4 above to have run first, so model and X_test_scaled exist.

6. Tune Hyperparameters

Adjust model settings (not learned from data) to improve performance. Use Grid Search or Random Search to find the best combination.

Hyperparameter tuning with GridSearchCV

This code performs an exhaustive search over a grid of hyperparameters (number of trees and maximum depth) using cross-validation to select the best combination.

Why used: Models have settings that you can't learn from the data (e.g., number of trees). Tuning them can significantly boost performance.
How it operates: GridSearchCV trains a model for every combination of parameters in the grid, using 5‑fold cross-validation to evaluate each. It then returns the best estimator and its parameters.
Effect on output: You get the optimal values (e.g., {'max_depth': 10, 'n_estimators': 200}) and the corresponding cross‑validated score.
Role in program: Tuning is the optimisation step – it squeezes extra performance out of the model before deployment.

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

params = {"n_estimators": [50, 100, 200], "max_depth": [5, 10, 20]}
grid = GridSearchCV(RandomForestClassifier(random_state=42), params, cv=5)
grid.fit(X_train_scaled, y_train)
print("Best params:", grid.best_params_)

⚠️ Needs step 2 above to have run first, so X_train_scaled and y_train exist.

7. Deploy

Save the model and integrate it into an application — a web app, an API, or a batch processing pipeline. Monitor its performance over time.

Model persistence

This snippet shows how to save your trained model (and scaler) to disk using joblib, and then load them later for inference on new data.

Why used: You don't want to retrain the model every time you need a prediction – saving it allows re-use.
How it operates: joblib.dump() serialises the Python objects to files (.pkl). Later, joblib.load() reconstructs the objects.
Effect on output: It creates physical files on disk that can be versioned, shared, and loaded into other scripts.
Role in program: Deployment is the delivery – without it, your model remains a research prototype.

import joblib

joblib.dump(model, "model.pkl")
joblib.dump(scaler, "scaler.pkl")

loaded_model = joblib.load("model.pkl")
loaded_scaler = joblib.load("scaler.pkl")

# A new sample must have the same 6 features as the training data
new_sample = [[0.5, -0.3, 1.2, 0.1, -0.7, 0.4]]
new_prediction = loaded_model.predict(loaded_scaler.transform(new_sample))
print("Prediction for new sample:", new_prediction)

⚠️ Needs steps 2 and 4 above to have run first, so model and scaler exist.

✨ Pro tip: Use joblib for scikit-learn models, torch.save() for PyTorch, and model.save() for TensorFlow/Keras.

Module 6

🧪 Build AI & ML From Scratch — No Libraries Doing the Thinking

Before you lean on scikit-learn or TensorFlow, build the core ideas yourself with plain Python and NumPy. Nine programs take you from a hand-written rule bot to a neural network trained with backpropagation, and every line is explained in order.

Program 1 of 9 — Classic AI, no ML

💬 A Rule-Based AI Chatbot

This is "AI" in its oldest and simplest sense: a program that behaves as if it understands you, using nothing but if/elif logic and pattern matching on keywords. There is no learning here — it is 100% hand-written rules. Understanding this program matters because it draws a clear line between rule-based AI (this program) and machine learning (the programs that follow), where the rules themselves are learned from data instead of typed by a human.

def chatbot_reply(message):
    text = message.lower().strip()

    if "hello" in text or "hi" in text:
        return "Hello! Ask me about Python or machine learning."
    elif "your name" in text:
        return "I'm PoeticBot, a rule-based chatbot with zero learning ability."
    elif "machine learning" in text:
        return "Machine learning is when a program learns patterns from data instead of being told exact rules."
    elif "bye" in text:
        return "Goodbye! Keep practising Python."
    else:
        return "I don't have a rule for that yet. Try 'hello', 'your name', or 'machine learning'."

# Simulate a short conversation
conversation = ["Hi there", "What is machine learning?", "What's your name?", "See you later"]

for user_message in conversation:
    print(f"User: {user_message}")
    print(f"Bot:  {chatbot_reply(user_message)}\n")

Line-by-line walkthrough

def chatbot_reply(message): — declares a function that takes one input, the visitor's raw message, and will return the bot's reply.
text = message.lower().strip() — normalises the input: lowercases it so "Hello" and "hello" match the same rule, and strips leading/trailing spaces.
if "hello" in text or "hi" in text: — this is the entire "understanding" step: it checks whether a keyword substring literally appears in the message. No meaning is parsed, only text matching.
return "Hello! ..." — if the check above is true, the function immediately exits and hands back a hard-coded string.
elif "your name" in text: — if the first condition failed, Python checks this next keyword rule.
elif "machine learning" in text: — another keyword rule, checked only if both prior ones failed.
elif "bye" in text: — a fourth rule for ending the conversation.
else: return "I don't have a rule..." — the fallback: if none of the keyword rules matched, the bot admits it has no answer. This is the built-in ceiling of rule-based AI — it can only respond to what a human anticipated.
conversation = [...] — builds a Python list of four sample messages to run through the bot, standing in for a real user typing them one at a time.
for user_message in conversation: — loops over each sample message in order.
print(f"User: {user_message}") — prints what the "user" said, for readability.
print(f"Bot: {chatbot_reply(user_message)}\n") — calls the function on that message and prints the bot's reply, with a trailing blank line between turns.

💡 Rule-based systems like this powered early chess programs, ELIZA (1966), and most phone-tree "AI" assistants. They're fast, predictable, and fully explainable — but they only know exactly what you typed into them. Machine learning exists to remove that ceiling.

Program 2 of 9 — The atom of deep learning

🧠 A Single Neuron Learning the AND Gate

Every neural network — including the ones behind GPT-style models — is built from this exact unit repeated millions of times: a weighted sum of inputs, squashed by an activation function, adjusted by comparing its guess to the right answer. This program builds one neuron with plain numpy and trains it, using gradient descent, to learn the logical AND gate purely from four examples.

import numpy as np

# Training data: every combination of two binary inputs, and the correct AND output
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([0, 0, 0, 1])

# Start with random weights and a random bias — the neuron knows nothing yet
np.random.seed(1)
weights = np.random.uniform(-1, 1, size=2)
bias = np.random.uniform(-1, 1)
learning_rate = 0.1

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# Train for 2,000 passes over the tiny dataset
for epoch in range(2000):
    for inputs, target in zip(X, y):
        z = np.dot(inputs, weights) + bias
        prediction = sigmoid(z)

        error = target - prediction
        gradient = error * prediction * (1 - prediction)

        weights += learning_rate * gradient * inputs
        bias += learning_rate * gradient

# Test the trained neuron on the same four cases
print("Learned weights:", weights)
print("Learned bias:", bias)
for inputs in X:
    z = np.dot(inputs, weights) + bias
    output = sigmoid(z)
    print(f"{inputs} -> {output:.3f} -> rounded: {round(output)}")

Line-by-line walkthrough

import numpy as np — loads NumPy so we can do vector math (dot products) instead of writing manual loops for the arithmetic.
X = np.array([[0,0],[0,1],[1,0],[1,1]]) — the four possible pairs of binary inputs; each row is one training example.
y = np.array([0, 0, 0, 1]) — the correct AND output for each row above (only 1,1 should output 1). This is the "supervision" — the labelled answer key.
np.random.seed(1) — fixes the random number generator so every run produces the exact same "random" starting weights, making the example reproducible.
weights = np.random.uniform(-1, 1, size=2) — assigns each of the two inputs a random starting importance (weight) between -1 and 1.
bias = np.random.uniform(-1, 1) — a third learnable number, added regardless of input, that lets the neuron shift its decision boundary.
learning_rate = 0.1 — controls how big a step the neuron takes when correcting itself after each mistake. Too high and it overshoots; too low and it learns too slowly.
def sigmoid(z): return 1 / (1 + np.exp(-z)) — the activation function. It squashes any real number into the range (0, 1), so the neuron's raw score becomes something interpretable as a probability.
for epoch in range(2000): — repeats the entire training set 2,000 times; each full pass is called an "epoch."
for inputs, target in zip(X, y): — inside each epoch, loops through the four training examples one at a time, pairing each input row with its correct label.
z = np.dot(inputs, weights) + bias — computes the neuron's raw score: multiply each input by its weight, sum them, then add the bias. This is the entire "forward pass."
prediction = sigmoid(z) — squashes that raw score into a 0–1 prediction.
error = target - prediction — how wrong the neuron was: positive if it under-predicted, negative if it over-predicted.
gradient = error * prediction * (1 - prediction) — this is calculus (the derivative of the sigmoid) telling us exactly how much and in which direction to nudge the weights to reduce the error. This is the "learning" in machine learning.
weights += learning_rate * gradient * inputs — updates each weight in proportion to how much that input contributed to the error, scaled by the learning rate.
bias += learning_rate * gradient — updates the bias the same way.
print("Learned weights:", weights) — after 2,000 epochs, shows the final weights the neuron discovered entirely on its own, with no rule ever typed by a human.
for inputs in X: ... print(...) — replays the four inputs through the trained neuron and prints its rounded prediction, which should now match the AND truth table exactly.

💡 This is the same core loop — forward pass, measure error, backpropagate the gradient, update weights — that trains networks with billions of parameters. Only the scale changes, not the idea.

Program 3 of 9 — Regression from first principles

📐 Linear Regression With Gradient Descent (No scikit-learn)

Earlier sections used LinearRegression and GradientBoostingRegressor from scikit-learn as ready-made tools. This program opens the hood: it fits a line y = m·x + b to noisy data by repeatedly nudging m and b downhill on the error surface — exactly what scikit-learn does internally, just visible line by line.

import numpy as np

# Generate noisy data that roughly follows y = 3x + 5
np.random.seed(0)
X = np.linspace(0, 10, 40)
noise = np.random.normal(0, 2, size=X.shape)
y = 3 * X + 5 + noise

# Start with a bad guess for slope (m) and intercept (b)
m, b = 0.0, 0.0
learning_rate = 0.01
n = len(X)

for epoch in range(1000):
    y_pred = m * X + b
    error = y_pred - y

    # Gradients of the mean-squared-error loss with respect to m and b
    grad_m = (2 / n) * np.dot(error, X)
    grad_b = (2 / n) * np.sum(error)

    m -= learning_rate * grad_m
    b -= learning_rate * grad_b

    if epoch % 200 == 0:
        mse = np.mean(error ** 2)
        print(f"Epoch {epoch:4d} | m={m:.3f} b={b:.3f} | MSE={mse:.3f}")

print(f"\nFinal line: y = {m:.2f}x + {b:.2f}  (true relationship: y = 3x + 5)")

Line-by-line walkthrough

X = np.linspace(0, 10, 40) — creates 40 evenly spaced x-values between 0 and 10, standing in for a feature column.
noise = np.random.normal(0, 2, size=X.shape) — generates random "measurement error" for each point, drawn from a normal distribution, so the data isn't a perfectly straight line — like real data never is.
y = 3 * X + 5 + noise — builds the target values from a known true relationship (slope 3, intercept 5) plus noise. We hide this formula from the algorithm; its job is to rediscover it.
m, b = 0.0, 0.0 — the model starts as the flat, wrong line y = 0. Both parameters will be learned from nothing.
learning_rate = 0.01 — how large a step to take on each update; matched to the scale of this data so training neither explodes nor crawls.
n = len(X) — the number of data points, used to average the error below.
for epoch in range(1000): — repeats the fit-and-correct cycle 1,000 times.
y_pred = m * X + b — computes what the current line predicts for every x, all at once via NumPy's vectorised math.
error = y_pred - y — the residual at every point: how far off the current line is, positive above the true point and negative below it.
grad_m = (2/n) * np.dot(error, X) — the calculus result for how the mean-squared error changes as m changes; np.dot(error, X) weights each residual by its x-position in one line.
grad_b = (2/n) * np.sum(error) — the same derivative for b, which is simply the average residual.
m -= learning_rate * grad_m — moves m a small step in the direction that reduces error. This is "descending" the error surface, hence gradient descent.
b -= learning_rate * grad_b — same update for the intercept.
if epoch % 200 == 0: — every 200th epoch, prints a progress snapshot so you can watch the line improve in real time.
mse = np.mean(error ** 2) — the mean squared error: the average of each residual, squared. Squaring penalises large misses harder than small ones and keeps errors from cancelling out.
print(f"\nFinal line: ...") — after training, prints the learned slope and intercept, which should land close to the true 3 and 5 despite the algorithm never being told those numbers.

💡 Compare the learned m and b to the true values 3 and 5 — they won't match exactly because of the injected noise, which is realistic: real data always has some. This gap between "learned" and "true" parameters is the entire reason evaluation metrics like RMSE and R² exist.

Program 4 of 9 — Classification by analogy

🧭 K-Nearest Neighbors From Scratch

KNN is the simplest classification algorithm to build by hand: to label a new point, measure its distance to every known point and let the closest k neighbours vote. There is no training phase, no weights, no gradients — just distance and a vote. It's an excellent second algorithm to implement because it makes "the model learns from data" concrete and geometric.

import numpy as np
from collections import Counter

# Labelled training points: 2D coordinates and a class (0 or 1)
X_train = np.array([[1, 2], [2, 3], [3, 3], [6, 5], [7, 7], [8, 6]])
y_train = np.array([0, 0, 0, 1, 1, 1])

def euclidean_distance(a, b):
    return np.sqrt(np.sum((a - b) ** 2))

def knn_predict(point, X_train, y_train, k=3):
    distances = [euclidean_distance(point, row) for row in X_train]
    nearest_indices = np.argsort(distances)[:k]
    nearest_labels = y_train[nearest_indices]
    vote = Counter(nearest_labels).most_common(1)[0][0]
    return vote

# Classify two brand-new, unlabelled points
new_points = np.array([[2, 2], [7, 6]])
for point in new_points:
    predicted_class = knn_predict(point, X_train, y_train, k=3)
    print(f"Point {point} -> predicted class {predicted_class}")

Line-by-line walkthrough

from collections import Counter — imports a tool for counting how often each value appears in a list, used later to tally votes.
X_train = np.array([[1,2], [2,3], ...]) — six known points in 2D space, each row an (x, y) coordinate.
y_train = np.array([0, 0, 0, 1, 1, 1]) — the class label already known for each of those six points — three belong to class 0, three to class 1.
def euclidean_distance(a, b): — defines straight-line distance between two points, the standard "how far apart" measurement in the plane.
return np.sqrt(np.sum((a - b) ** 2)) — subtracts the two points component-wise, squares each difference, sums them, and takes the square root — the Pythagorean theorem in one line.
def knn_predict(point, X_train, y_train, k=3): — the classifier itself: given a new point and the labelled data, predict its class using the k closest neighbours (default 3).
distances = [euclidean_distance(point, row) for row in X_train] — computes the distance from the new point to every single training point, producing a list of six distances.
nearest_indices = np.argsort(distances)[:k] — sorts those distances from smallest to largest and keeps the positions (indices) of the k smallest — i.e. the k closest neighbours.
nearest_labels = y_train[nearest_indices] — looks up the known class labels of exactly those k nearest neighbours.
vote = Counter(nearest_labels).most_common(1)[0][0] — counts how many of the k neighbours belong to each class and returns the class with the most votes — majority rule.
return vote — hands back the predicted class for the new point.
new_points = np.array([[2, 2], [7, 6]]) — two brand-new points with no label, one clearly near the class-0 cluster and one near the class-1 cluster.
for point in new_points: — loops through both unlabelled points to classify each in turn.
predicted_class = knn_predict(point, X_train, y_train, k=3) — runs the classifier defined above on this point.
print(f"Point {point} -> predicted class {predicted_class}") — reports the prediction, which should assign [2, 2] to class 0 and [7, 6] to class 1, matching the clusters they sit closest to.

💡 Try changing k to 1 or to 5 and re-running — with real, noisier data, a k that's too small overfits to individual outliers, while a k that's too large blurs the boundary between classes. This "how much do I trust nearby evidence" trade-off shows up in almost every ML algorithm in some form.

Program 5 of 9 — Classification by probability

🎯 Logistic Regression With Gradient Descent

Program 2 trained a single neuron on four examples. Here the same neuron — a weighted sum passed through a sigmoid — learns to separate two overlapping clouds of 200 points, using the full-dataset, vectorised form that real libraries use. The output is a probability that a point belongs to class 1, and the loss is log loss (cross-entropy), which punishes confident wrong answers very hard.

p = sigmoid(w·x + b) loss = −mean( y·log(p) + (1 − y)·log(1 − p) ) gradient wrt w = Xᵀ(p − y) / n

import numpy as np

np.random.seed(3)

# Two clusters of points: class 0 around (-1, -1) and class 1 around (+1, +1)
n = 100
X0 = np.random.randn(n, 2) + np.array([-1.0, -1.0])
X1 = np.random.randn(n, 2) + np.array([1.0, 1.0])
X = np.vstack([X0, X1])
y = np.array([0] * n + [1] * n)

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

w = np.zeros(2)
b = 0.0
learning_rate = 0.1

for epoch in range(501):
    p = sigmoid(X @ w + b)                                        # predicted probabilities
    loss = -np.mean(y * np.log(p + 1e-9) + (1 - y) * np.log(1 - p + 1e-9))

    grad_w = X.T @ (p - y) / len(y)
    grad_b = np.mean(p - y)

    w -= learning_rate * grad_w
    b -= learning_rate * grad_b

    if epoch % 100 == 0:
        accuracy = np.mean((p >= 0.5) == y)
        print(f"epoch {epoch:3d} | loss={loss:.4f} | accuracy={accuracy:.3f}")

print("weights:", w.round(3), "| bias:", round(float(b), 3))
new_point = np.array([0.8, 1.2])
print("P(class 1) for", new_point, "=", round(float(sigmoid(new_point @ w + b)), 3))

Expected output (click to compare with your run)

epoch   0 | loss=0.6931 | accuracy=0.500
epoch 100 | loss=0.2238 | accuracy=0.890
epoch 200 | loss=0.2115 | accuracy=0.890
epoch 300 | loss=0.2085 | accuracy=0.890
epoch 400 | loss=0.2075 | accuracy=0.890
epoch 500 | loss=0.2071 | accuracy=0.895
weights: [1.844 1.384] | bias: 0.097
P(class 1) for [0.8 1.2] = 0.962

Line-by-line walkthrough

np.random.seed(3) — fixes the random generator so the "random" clouds are identical on every run.
X0 = np.random.randn(n, 2) + np.array([-1.0, -1.0]) — 100 points drawn from a standard bell curve in 2D, then shifted so the cloud is centred at (−1, −1). This is class 0.
X1 = ... + np.array([1.0, 1.0]) — the same, centred at (+1, +1). This is class 1. The clouds are only 2 units apart per axis, so they overlap.
X = np.vstack([X0, X1]) — stacks both clouds into one 200×2 matrix of features.
y = np.array([0] * n + [1] * n) — the matching labels: one hundred 0s followed by one hundred 1s.
def sigmoid(z) — squashes any number into (0, 1) so the output can be read as a probability.
w = np.zeros(2); b = 0.0 — start knowing nothing: both weights and the bias are zero, so every point initially gets probability 0.5.
for epoch in range(501): — 501 full passes so the final epoch (500) is printed.
p = sigmoid(X @ w + b) — the forward pass for all 200 points at once: one weighted sum per row, squashed into a probability.
loss = -np.mean(...) — log loss. The tiny 1e-9 stops log(0) from producing infinity.
grad_w = X.T @ (p - y) / len(y) — the gradient of log loss with respect to the weights. Mathematically it simplifies to "average of (prediction − truth) × input," which is why log loss pairs so neatly with the sigmoid.
grad_b = np.mean(p - y) — the same gradient for the bias (its input is always 1).
w -= learning_rate * grad_w; b -= ... — step downhill, against the gradient.
if epoch % 100 == 0: — every 100 epochs, compute accuracy (p >= 0.5 means "predict class 1") and print progress.
print("weights:", ...) — shows the learned weights; both are positive because larger x or y values mean class 1.
new_point = ...; sigmoid(new_point @ w + b) — scores a brand-new point and prints the probability that it belongs to class 1.

💡 Accuracy stalls around 0.89–0.90 — and that is correct, not a bug. The two clouds overlap, so no straight line can classify every point. When a model hits a plateau, ask whether the data itself limits it before blaming the algorithm.

Program 6 of 9 — Learning rules from data

🌳 A Decision Tree From Scratch

A decision tree learns a flowchart of yes/no questions. At each step it asks: which question splits the data into the purest possible groups? Purity is measured with Gini impurity. This program grows a tree two levels deep to predict whether a student passes, and prints it as readable if/else rules — the machine-learned version of the hand-written chatbot in Program 1.

gini = 1 − p₁² − p₀² (0 = perfectly pure group, 0.5 = a 50/50 mix — the worst case for two classes)

import numpy as np

# Features: [hours_studied, hours_slept]    Label: 1 = passed, 0 = failed
X = np.array([[1, 8], [2, 6], [3, 9], [4, 7], [2, 4], [6, 8], [7, 6],
              [8, 7], [9, 9], [6, 3], [8, 4], [9, 2], [5, 6], [7, 5]], dtype=float)
y = np.array([0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1])
feature_names = ["hours_studied", "hours_slept"]

def gini(labels):
    if len(labels) == 0:
        return 0.0
    p1 = np.mean(labels)
    return 1.0 - p1 ** 2 - (1 - p1) ** 2

def best_split(X, y):
    best = (None, None, 1.0)                    # (feature index, threshold, weighted gini)
    for f in range(X.shape[1]):
        for t in np.unique(X[:, f]):
            left, right = y[X[:, f] <= t], y[X[:, f] > t]
            if len(left) == 0 or len(right) == 0:
                continue
            score = (len(left) * gini(left) + len(right) * gini(right)) / len(y)
            if score < best[2]:
                best = (f, t, score)
    return best

def build_tree(X, y, depth=0, max_depth=2):
    if depth == max_depth or gini(y) == 0:
        return {"leaf": int(round(np.mean(y)))}
    f, t, _ = best_split(X, y)
    if f is None:
        return {"leaf": int(round(np.mean(y)))}
    mask = X[:, f] <= t
    return {"feature": f, "threshold": t,
            "left": build_tree(X[mask], y[mask], depth + 1, max_depth),
            "right": build_tree(X[~mask], y[~mask], depth + 1, max_depth)}

def predict(tree, row):
    while "leaf" not in tree:
        branch = "left" if row[tree["feature"]] <= tree["threshold"] else "right"
        tree = tree[branch]
    return tree["leaf"]

def show(tree, indent=""):
    if "leaf" in tree:
        print(indent + "->", "PASS" if tree["leaf"] else "FAIL")
        return
    name = feature_names[tree["feature"]]
    print(f"{indent}if {name} <= {tree['threshold']}:")
    show(tree["left"], indent + "    ")
    print(f"{indent}else:")
    show(tree["right"], indent + "    ")

tree = build_tree(X, y)
show(tree)
preds = np.array([predict(tree, row) for row in X])
print("training accuracy:", round(float(np.mean(preds == y)), 3))
print("prediction for [6 hours studied, 5 hours slept]:", predict(tree, np.array([6.0, 5.0])))

Expected output (click to compare with your run)

if hours_studied <= 4.0:
    -> FAIL
else:
    if hours_slept <= 4.0:
        -> FAIL
    else:
        -> PASS
training accuracy: 1.0
prediction for [6 hours studied, 5 hours slept]: 1

Line-by-line walkthrough

X = np.array([[1, 8], ...]) — 14 students, each described by [hours studied, hours slept].
y = np.array([0, 0, ..., 1, 1]) — whether each student passed (1) or failed (0). The hidden truth: passing needs both enough study and enough sleep.
def gini(labels) — measures how mixed a group is. An empty group scores 0; otherwise p1 is the share of passes and the formula gives the impurity.
def best_split(X, y) — tries every possible question of the form "is feature f ≤ threshold t?" and keeps the best one.
for f in range(X.shape[1]) / for t in np.unique(X[:, f]) — loops over each feature and each distinct value as a candidate threshold.
left, right = y[X[:, f] <= t], y[X[:, f] > t] — splits the labels into the group that answers "yes" and the group that answers "no". Splits that leave one side empty are skipped.
score = (len(left) * gini(left) + len(right) * gini(right)) / len(y) — the weighted impurity after the split; larger groups count for more. Lower is better, and the best (feature, threshold, score) is remembered.
def build_tree(...) — builds the tree recursively. It stops and makes a leaf when the depth limit is reached or the group is already pure (gini == 0).
{"leaf": int(round(np.mean(y)))} — a leaf predicts the majority class (the mean of 0/1 labels rounded to 0 or 1).
mask = X[:, f] <= t and the two recursive calls — split the data on the winning question and grow a subtree on each side, one level deeper.
def predict(tree, row) — to classify a new student, start at the root and keep following the left or right branch until a leaf is reached.
def show(tree, indent) — prints the tree as nested if / else text, indenting each level.
tree = build_tree(X, y); show(tree) — learn and print the rules. The tree discovers on its own that studying 4 hours or less means failing, and for the rest, sleeping 4 hours or less still means failing.
preds = ...; np.mean(preds == y) — training accuracy (1.0 here, since the rules fit all 14 students).
predict(tree, np.array([6.0, 5.0])) — classifies a new student: 6 hours of study and 5 of sleep → pass.

💡 Left unchecked, a tree keeps splitting until every training point is perfectly classified — a recipe for overfitting (Module 7). That is why trees get a max_depth limit, and why random forests average many different trees instead of trusting one.

Program 7 of 9 — Unsupervised learning

🎯 K-Means Clustering From Scratch

So far every program had labels. K-Means has none: given only raw points, it discovers k groups by alternating two simple steps — assign each point to its nearest centre, then move each centre to the average of its points — until nothing changes.

import numpy as np

np.random.seed(5)

# Three blobs of 40 points each
true_centers = np.array([[0, 0], [6, 6], [0, 7]])
X = np.vstack([np.random.randn(40, 2) + c for c in true_centers])

k = 3
centroids = X[np.random.choice(len(X), k, replace=False)]    # random starting centroids

for iteration in range(10):
    # 1) Assignment step: every point joins its nearest centroid
    distances = np.linalg.norm(X[:, None, :] - centroids[None, :, :], axis=2)
    labels = np.argmin(distances, axis=1)

    # 2) Update step: move each centroid to the mean of its points
    new_centroids = np.array([X[labels == j].mean(axis=0) for j in range(k)])
    shift = np.linalg.norm(new_centroids - centroids)
    centroids = new_centroids

    inertia = np.sum((X - centroids[labels]) ** 2)
    print(f"iteration {iteration + 1:2d} | centroid shift={shift:.4f} | inertia={inertia:.2f}")
    if shift < 1e-6:
        print("Converged.")
        break

print("Final centroids:")
print(centroids.round(2))
print("Cluster sizes:", np.bincount(labels))

Expected output (click to compare with your run)

iteration  1 | centroid shift=1.6784 | inertia=222.91
iteration  2 | centroid shift=0.0000 | inertia=222.91
Converged.
Final centroids:
[[-0.14  6.92]
 [ 6.    6.17]
 [-0.08  0.21]]
Cluster sizes: [40 40 40]

Line-by-line walkthrough

np.random.seed(5) — makes the data and the random starting centres repeatable.
true_centers = np.array([[0, 0], [6, 6], [0, 7]]) — where we secretly place three groups. The algorithm will not be told this.
X = np.vstack([np.random.randn(40, 2) + c for c in true_centers]) — for each secret centre, draws 40 points scattered around it, then stacks all 120 points into one matrix.
k = 3 — we tell K-Means how many clusters to look for (choosing this well is covered in Module 10).
centroids = X[np.random.choice(len(X), k, replace=False)] — picks 3 random data points as the starting centres.
for iteration in range(10): — repeats assign-and-update, at most 10 times.
distances = np.linalg.norm(X[:, None, :] - centroids[None, :, :], axis=2) — a vectorised trick that computes the distance from every point to every centroid at once, producing a 120×3 table.
labels = np.argmin(distances, axis=1) — the assignment step: each point joins the centroid with the smallest distance.
new_centroids = np.array([X[labels == j].mean(axis=0) for j in range(k)]) — the update step: each centroid becomes the average position of the points assigned to it.
shift = np.linalg.norm(new_centroids - centroids) — how far the centres moved in total; zero means nothing changed.
inertia = np.sum((X - centroids[labels]) ** 2) — total squared distance from each point to its own centre. K-Means tries to minimise this.
if shift < 1e-6: ... break — stop once the centres have effectively stopped moving: the algorithm has converged.
print(centroids.round(2)) and np.bincount(labels) — report the final centres (close to the secret ones) and how many points each cluster holds (40 each here).

⚠️ K-Means depends on its random starting centres and can land in a poor solution on harder data. scikit-learn runs it several times (n_init) and keeps the best result. These blobs are well separated, so it converges in two iterations.

Program 8 of 9 — Probability meets text

✉️ A Naive Bayes Spam Filter From Scratch

This classifier counts how often each word appears in spam versus normal ("ham") messages and combines those counts using Bayes' theorem from Module 2. It is "naive" because it assumes words are independent of each other given the class — false in real language, yet surprisingly effective.

score(class) = log P(class) + Σ log P(word | class) pick the class with the highest score

import math
from collections import Counter, defaultdict

train = [
    ("win free money now", "spam"),
    ("free prize claim now", "spam"),
    ("limited offer win cash", "spam"),
    ("meeting at noon tomorrow", "ham"),
    ("lunch tomorrow with the team", "ham"),
    ("project report due today", "ham"),
]

class_counts = Counter(label for _, label in train)
word_counts = defaultdict(Counter)
for text, label in train:
    word_counts[label].update(text.split())

vocab = {w for counts in word_counts.values() for w in counts}

def predict(text):
    scores = {}
    for label in class_counts:
        score = math.log(class_counts[label] / len(train))            # log prior
        total_words = sum(word_counts[label].values())
        for word in text.split():
            # Laplace smoothing: add 1 so an unseen word never makes the probability 0
            p_word = (word_counts[label][word] + 1) / (total_words + len(vocab))
            score += math.log(p_word)
        scores[label] = score
    return max(scores, key=scores.get), scores

for message in ["free cash prize", "team meeting today", "win a free lunch"]:
    label, scores = predict(message)
    rounded = {k: round(v, 2) for k, v in scores.items()}
    print(f"{message!r:22} -> {label:4}  log-scores: {rounded}")

Expected output (click to compare with your run)

'free cash prize'      -> spam  log-scores: {'spam': -8.7, 'ham': -11.27}
'team meeting today'   -> ham   log-scores: {'spam': -11.18, 'ham': -9.19}
'win a free lunch'     -> spam  log-scores: {'spam': -12.48, 'ham': -14.11}

Line-by-line walkthrough

import math; from collections import Counter, defaultdict — math.log for the logarithms, and two helpers for counting.
train = [("win free money now", "spam"), ...] — six tiny labelled messages, three spam and three ham. Real filters train on thousands.
class_counts = Counter(label for _, label in train) — counts messages per class (3 spam, 3 ham); used to compute each class's prior probability.
word_counts = defaultdict(Counter) — a nested counter: for each class, how often each word appears.
for text, label in train: word_counts[label].update(text.split()) — splits each message into words and adds them to the right class's tally.
vocab = {w for counts in word_counts.values() for w in counts} — the set of every distinct word seen in training.
def predict(text): — scores a new message against each class.
score = math.log(class_counts[label] / len(train)) — begins with the log of the prior: how common this class is overall.
total_words = sum(word_counts[label].values()) — total word count for the class, the denominator when estimating word probabilities.
p_word = (word_counts[label][word] + 1) / (total_words + len(vocab)) — the probability of a word given the class, with Laplace smoothing (+1) so a word never seen in a class does not make the whole product zero.
score += math.log(p_word) — adding logs is the same as multiplying probabilities, but avoids numbers so tiny that the computer rounds them to zero.
return max(scores, key=scores.get), scores — picks the class with the highest score, and returns all scores for inspection.
for message in [...] — classifies three new messages, rounds the scores, and prints each. "free cash prize" and "win a free lunch" score higher as spam, and "team meeting today" as ham.

💡 The log-scores are negative numbers; the less negative one wins. With only six training messages this filter is fragile — notice how weak the evidence is for "win a free lunch." More data makes the word statistics trustworthy.

Program 9 of 9 — A real neural network

🧠 A Two-Layer Neural Network With Backpropagation

A single neuron can only draw one straight line. XOR ("one or the other, but not both") cannot be separated by any single line — so one neuron fails. Adding a hidden layer lets the network bend the boundary. This program implements the complete training loop of a neural network with only NumPy: forward pass, loss, backpropagation through both layers, and weight updates.

import numpy as np

np.random.seed(1)
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0], [1], [1], [0]], dtype=float)      # XOR: one neuron alone can never learn this

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

hidden = 4
W1 = np.random.randn(2, hidden)
b1 = np.zeros((1, hidden))
W2 = np.random.randn(hidden, 1)
b2 = np.zeros((1, 1))
lr = 0.5

for epoch in range(5001):
    # ---- forward pass ----
    h = np.tanh(X @ W1 + b1)                  # hidden layer
    out = sigmoid(h @ W2 + b2)                # output layer
    loss = -np.mean(y * np.log(out + 1e-9) + (1 - y) * np.log(1 - out + 1e-9))

    # ---- backward pass (chain rule, from the output back to the input) ----
    d_z2 = (out - y) / len(X)                 # gradient at the output (log-loss + sigmoid)
    dW2 = h.T @ d_z2
    db2 = d_z2.sum(axis=0, keepdims=True)
    d_h = d_z2 @ W2.T
    d_z1 = d_h * (1 - h ** 2)                 # derivative of tanh
    dW1 = X.T @ d_z1
    db1 = d_z1.sum(axis=0, keepdims=True)

    # ---- gradient descent update ----
    W1 -= lr * dW1
    b1 -= lr * db1
    W2 -= lr * dW2
    b2 -= lr * db2

    if epoch % 1000 == 0:
        print(f"epoch {epoch:4d} | loss={loss:.4f}")

print("\nPredictions after training:")
for inputs, p in zip(X, out):
    print(f"{inputs.astype(int)} -> {p[0]:.3f} -> {int(p[0] > 0.5)}")

Expected output (click to compare with your run)

epoch    0 | loss=1.1836
epoch 1000 | loss=0.0052
epoch 2000 | loss=0.0025
epoch 3000 | loss=0.0016
epoch 4000 | loss=0.0012
epoch 5000 | loss=0.0010

Predictions after training:
[0 0] -> 0.000 -> 0
[0 1] -> 0.999 -> 1
[1 0] -> 0.999 -> 1
[1 1] -> 0.002 -> 0

Line-by-line walkthrough

X and y — the four XOR input pairs and their answers: 0 for (0,0) and (1,1), 1 for (0,1) and (1,0).
def sigmoid(z) — the output activation, producing a probability.
W1 = np.random.randn(2, hidden), b1 — layer 1 weights connecting 2 inputs to 4 hidden neurons, plus their biases. Random weights are essential; if all weights started equal, every hidden neuron would learn the same thing.
W2 = np.random.randn(hidden, 1), b2 — layer 2 weights connecting the 4 hidden neurons to 1 output neuron.
lr = 0.5 — the learning rate for the weight updates.
h = np.tanh(X @ W1 + b1) — forward pass, layer 1: weighted sums for all four inputs at once, bent by the tanh activation. Without a non-linear activation, two layers would collapse into one and XOR would stay impossible.
out = sigmoid(h @ W2 + b2) — forward pass, layer 2: the hidden values are combined into the final probability.
loss = -np.mean(...) — log loss between the predictions and the answers.
d_z2 = (out - y) / len(X) — backward pass begins: for log loss with a sigmoid output, the error signal at the output is simply prediction − truth, averaged over the samples.
dW2 = h.T @ d_z2 and db2 — gradients for layer 2: how much each weight contributed to the error.
d_h = d_z2 @ W2.T — pushes the error signal back through layer 2's weights to find how much each hidden neuron was responsible.
d_z1 = d_h * (1 - h ** 2) — passes the signal back through the tanh activation using its derivative (1 − tanh²). This is the chain rule from Module 2.
dW1 = X.T @ d_z1 and db1 — gradients for layer 1.
W1 -= lr * dW1 … b2 -= lr * db2 — gradient descent: every weight and bias moves a small step against its gradient.
if epoch % 1000 == 0: — prints the loss every 1,000 epochs; it falls from above 1 to almost 0.
The final loop — runs the four inputs through the trained network. The outputs end up near 0, 1, 1, 0: XOR is solved.

💡 Frameworks like PyTorch and TensorFlow automate the backward pass (automatic differentiation), so you never write lines 9–13 by hand again. But having done it once, you know exactly what loss.backward() does.

From "built by hand" to "built with libraries"

Everything above — the neuron, the gradient descent, the distance-based voting — is exactly what scikit-learn, PyTorch, and TensorFlow do internally, just optimised in compiled code and wrapped behind .fit() and .predict(). Now that you've written the mechanics yourself, the Supervised Learning, Unsupervised Learning, and Libraries sections below will make a lot more sense — you already know what's happening under the hood.

Module 7

⚖️ Generalization — Overfitting, Underfitting & the Bias–Variance Trade-off

The goal of machine learning is not to memorise the training data but to perform well on data the model has never seen. This module is about why that is hard, and how to achieve it.

🎓 The Student Analogy

Imagine three students preparing for an exam. One barely studies and fails everything (underfitting). One memorises last year's answers word for word, aces the practice paper, but is lost when the questions change (overfitting). One learns the underlying ideas and does well on both (good generalization). A model's training error is the practice paper; its test error is the real exam.

Diagnosis	Training error	Test error	Also called	Typical fixes
Underfitting	High	High	High bias	More expressive model, better features, less regularisation, train longer
Overfitting	Low	Much higher	High variance	More data, simpler model, regularisation, early stopping, dropout
Good fit	Low	Similarly low	—	Keep going; monitor for drift

💡 Bias is error from a model that is too simple to capture the real pattern. Variance is error from a model so flexible that it chases noise and would change drastically with different training data. Making a model more complex lowers bias but raises variance — you are searching for the sweet spot between them.

📈 Watching a Model Overfit

The code below generates noisy data from a sine wave, then fits polynomials of increasing degree. A degree-1 polynomial is a straight line (too simple); a degree-15 polynomial has enough freedom to thread through every training point (too flexible). We measure the error on the training points and on 200 fresh test points.

import warnings
import numpy as np

warnings.simplefilter("ignore")           # very high-degree fits trigger numerical warnings
rng = np.random.default_rng(1)

def make_data(n):
    x = np.sort(rng.uniform(0, 1, n))
    y = np.sin(2 * np.pi * x) + rng.normal(0, 0.25, n)     # true signal + noise
    return x, y

x_train, y_train = make_data(20)
x_test, y_test = make_data(200)

def rmse(y_true, y_pred):
    return float(np.sqrt(np.mean((y_true - y_pred) ** 2)))

print("degree | train RMSE | test RMSE")
for degree in [1, 3, 5, 9, 15]:
    coeffs = np.polyfit(x_train, y_train, degree)
    train_err = rmse(y_train, np.polyval(coeffs, x_train))
    test_err = rmse(y_test, np.polyval(coeffs, x_test))
    print(f"{degree:6d} | {train_err:10.3f} | {test_err:9.3f}")

Expected output (click to compare with your run)

degree | train RMSE | test RMSE
     1 |      0.450 |     0.516
     3 |      0.276 |     0.241
     5 |      0.246 |     0.320
     9 |      0.207 |     2.736
    15 |      0.127 |   626.853

Train error vs test error

Why used: It reproduces the central picture of machine learning with nothing but NumPy.
How it operates: np.polyfit finds the polynomial of a given degree that best fits the 20 training points, and np.polyval evaluates it. RMSE (root mean squared error) is computed for the training set and the held-out test set.
Effect on output: Training error keeps falling as the degree rises (it can only improve with more freedom). Test error falls at first, is lowest around degree 3, and then climbs — dramatically by degree 9 and beyond. The model has memorised the noise.
Role in program: It shows that a low training error alone proves nothing — you must always judge a model on data it has not seen.

⚠️ A degree-15 fit on 20 points is numerically unstable, so the exact test error you see (hundreds, in the tested run) can differ a lot from machine to machine. The pattern — an exploding test error — is the lesson, not the specific number.

🪢 Regularisation — Penalising Complexity

Regularisation fights overfitting by adding a penalty for large weights to the loss, so the model prefers simpler explanations unless the data strongly argues otherwise.

loss = prediction error + α × penalty(weights) Ridge (L2): penalty = sum of squared weights → shrinks all weights toward zero Lasso (L1): penalty = sum of absolute weights → pushes some weights to exactly zero (built-in feature selection)

Here we have 60 samples but 40 features, and only 5 of those features actually matter — a classic overfitting setup.

import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score

rng = np.random.default_rng(0)
n, p = 60, 40                                    # few samples, many features
X = rng.normal(size=(n, p))
true_w = np.zeros(p)
true_w[:5] = [3, -2, 1.5, 4, -1]                 # only 5 features actually matter
y = X @ true_w + rng.normal(0, 2.0, size=n)

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.4, random_state=0)

models = {
    "Linear (no regularisation)": LinearRegression(),
    "Ridge (L2, alpha=10)": Ridge(alpha=10),
    "Lasso (L1, alpha=0.3)": Lasso(alpha=0.3),
}
for name, m in models.items():
    m.fit(X_tr, y_tr)
    train_r2 = r2_score(y_tr, m.predict(X_tr))
    test_r2 = r2_score(y_te, m.predict(X_te))
    print(f"{name:28s} train R2={train_r2:5.2f}   test R2={test_r2:5.2f}")

lasso = models["Lasso (L1, alpha=0.3)"]
print("Features Lasso kept:", int(np.sum(lasso.coef_ != 0)), "of", p)

Expected output (click to compare with your run)

Linear (no regularisation)   train R2= 1.00   test R2=-1.18
Ridge (L2, alpha=10)         train R2= 0.93   test R2= 0.60
Lasso (L1, alpha=0.3)        train R2= 0.89   test R2= 0.78
Features Lasso kept: 13 of 40

Ordinary vs Ridge vs Lasso

Why used: It demonstrates that adding a penalty can turn a useless model into a useful one.
How it operates: With 36 training rows and 40 features, plain linear regression has enough freedom to fit the training set perfectly, noise included. Ridge and Lasso are fit on the same data but are penalised for large weights. alpha controls how strong the penalty is.
Effect on output: Plain regression has a training R² of 1.00 but a negative test R² — worse than predicting the average. Ridge and Lasso score lower on the training set but far better on the test set, and Lasso keeps only a fraction of the 40 features.
Role in program: It shows the bias–variance trade-off in action: a little extra bias (training error) buys a large drop in variance (test error).

🛡️ Your Anti-Overfitting Toolkit

Hold out data — always evaluate on a test set the model never trained on.
Cross-validation — average over several splits for a steadier estimate (see Module 9).
More (and more varied) data — the most reliable cure.
Simpler models — fewer features, shallower trees, smaller networks.
Regularisation — L1/L2 penalties, tree depth limits, dropout in neural networks.
Early stopping — stop training when validation error stops improving.
Ensembles — averaging many models (random forests) reduces variance.

Module 8

📦 Python ML Libraries — Your Toolbox

The essential libraries for every ML project, from data prep to deep learning

📊

scikit-learn

The go-to library for classical ML — regression, classification, clustering, and dimensionality reduction. Simple, consistent API, and battle-tested.

pip install scikit-learnclassical ML

🧠

TensorFlow & Keras

Google's deep learning framework. Keras provides a high-level API for building and training neural networks. Best for production-scale deep learning.

pip install tensorflowdeep learning

🔥

PyTorch

Meta's deep learning library with dynamic computation graphs. Preferred by researchers for its flexibility and Pythonic feel. Growing fast in production.

pip install torchdynamic graphs

📈

Pandas & NumPy

The foundation of Python data science. NumPy for numerical arrays, Pandas for data manipulation and analysis. Used in every ML project.

pip install pandas numpydata prep

📉

Matplotlib & Seaborn

Visualisation libraries for plotting data distributions, model performance, feature importance, and more. Essential for EDA.

pip install matplotlib seabornvisualisation

🚀

XGBoost & LightGBM

Gradient boosting libraries that win competitions. XGBoost and LightGBM are the go-to models for tabular data — often outperforming deep learning.

pip install xgboostgradient boosting

Module 9

📊 Supervised Learning — In Depth

Classification and regression with real code examples

🏷️ Classification — Predicting Categories

Given input features, predict which category (class) the sample belongs to. Binary classification (spam vs. not spam), multi-class classification (iris species, digit recognition).

Classification with cross‑validation

This example uses cross‑validation to get a more robust estimate of model performance than a single train/test split. It also shows how to make predictions on a new sample.

Why used: Cross‑validation reduces variance in performance estimation and helps prevent overfitting to a particular split.
How it operates: cross_val_score performs 5‑fold CV: it splits the data into 5 parts, trains on 4 and validates on 1, rotating the validation fold. It returns an array of 5 accuracy scores.
Effect on output: You see the mean accuracy across folds, which gives a better sense of how the model will perform on unseen data. The final predict() demonstrates model usage.
Role in program: This reinforces the importance of reliable evaluation and shows how to use the model for inference.

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

iris = load_iris()
X, y = iris.data, iris.target

model = RandomForestClassifier(n_estimators=50, random_state=42)
scores = cross_val_score(model, X, y, cv=5)
print(f"Mean CV accuracy: {scores.mean():.3f}")

model.fit(X, y)
print("Predictions for new sample:", model.predict([[5.1, 3.5, 1.4, 0.2]]))

💡 cross_val_score gives a more reliable estimate of model performance than a single train/test split. Use it early in your workflow.

📈 Regression — Predicting Numbers

Given input features, predict a continuous numeric value. Examples: house prices, stock prices, temperature, sales volume.

Regression with Gradient Boosting

This snippet loads the diabetes progression dataset (built into scikit-learn, so it works fully offline with no download), trains a GradientBoostingRegressor, and evaluates it using RMSE (Root Mean Squared Error) and R² (coefficient of determination).

Why used: Regression is a fundamental task. Gradient Boosting is a powerful ensemble method that often wins Kaggle competitions.
How it operates: The model sequentially builds trees, each correcting the errors of the previous one. RMSE measures the average magnitude of errors (in the same units as the target). R² indicates how much variance is explained (1 = perfect).
Effect on output: You get two numbers: RMSE (lower is better) and R² (closer to 1 is better). These tell you how well the model predicts patient disease progression.
Role in program: It demonstrates a complete regression workflow and introduces key metrics for continuous targets.

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np

diabetes = load_diabetes()
X, y = diabetes.data, diabetes.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1, random_state=42)
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
print(f"RMSE: {np.sqrt(mean_squared_error(y_test, y_pred)):.3f}")
print(f"R²: {r2_score(y_test, y_pred):.3f}")

📏 Linear Regression — The Simplest Predictive Model

Linear regression predicts a number as a weighted sum of the features plus a constant. Its great virtue is that you can read the answer: each weight says how much the prediction changes when that feature goes up by one unit, holding the others fixed.

predicted price = w₁ × size + w₂ × bedrooms + b

import numpy as np
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(2)
size = rng.uniform(30, 200, 150)
bedrooms = rng.integers(1, 5, 150)
price = 800 * size + 15000 * bedrooms + 20000 + rng.normal(0, 15000, 150)

X = np.column_stack([size, bedrooms])
model = LinearRegression().fit(X, price)

print("learned coefficients:", model.coef_.round(1), "(true values: [800, 15000])")
print("learned intercept   :", round(float(model.intercept_), 1), "(true value: 20000)")
print("R2 on training data :", round(model.score(X, price), 3))
print("Prediction for 100 m2, 3 bedrooms ->", round(float(model.predict([[100, 3]])[0])))

Expected output (click to compare with your run)

learned coefficients: [  777.2 13928.3] (true values: [800, 15000])
learned intercept   : 23852.7 (true value: 20000)
R2 on training data : 0.877
Prediction for 100 m2, 3 bedrooms -> 143359

Reading a linear model

Why used: It is the baseline every other regression model must beat, and it is fully interpretable.
How it operates: The code generates prices from a known rule (800 per m², 15,000 per bedroom, a 20,000 base) plus noise, then LinearRegression().fit recovers the weights by minimising squared error. score returns R², the share of the variation in price the model explains.
Effect on output: The learned coefficients land near, but not exactly on, the true values — noise prevents a perfect recovery. R² is high, and the model predicts a price for a new 100 m², 3-bedroom home.
Role in program: It is the scikit-learn version of Program 3 from Module 6 — the same idea, one line of code.

🌲 Decision Trees — Models You Can Read

Program 6 built a tree by hand; scikit-learn grows far better ones. export_text prints the learned rules so you can audit exactly how a prediction is made — valuable in regulated or high-stakes settings.

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, export_text

iris = load_iris()
tree = DecisionTreeClassifier(max_depth=2, random_state=0).fit(iris.data, iris.target)
print(export_text(tree, feature_names=list(iris.feature_names)))
print("classes:", ", ".join(iris.target_names))

Expected output (click to compare with your run)

|--- petal width (cm) <= 0.80
|   |--- class: 0
|--- petal width (cm) >  0.80
|   |--- petal width (cm) <= 1.75
|   |   |--- class: 1
|   |--- petal width (cm) >  1.75
|   |   |--- class: 2

classes: setosa, versicolor, virginica

Printing the rules of a tree

Why used: Interpretability — you can follow a prediction from root to leaf and explain it to a non-expert.
How it operates: A tree limited to depth 2 is trained on the famous iris flower data. export_text converts the learned splits into indented rules.
Effect on output: Two questions about petal width separate the three species: a very narrow petal means setosa; otherwise, a width below about 1.75 cm means versicolor, and above it means virginica.
Role in program: It shows how a tiny, human-readable model can already be accurate on a simple problem.

🥊 Which Algorithm Should I Use?

Algorithm	Strengths	Needs scaling?	Watch out for
Linear / Logistic Regression	Fast, interpretable, great baseline	Yes (for regularised or gradient-based fits)	Only captures linear relationships
k-Nearest Neighbours	No training; simple to understand	Yes	Slow on big data; hurt by irrelevant features
Decision Tree	Readable rules; handles mixed data	No	Overfits easily on its own
Random Forest	Strong default for tabular data; robust	No	Larger and less interpretable than one tree
Gradient Boosting (XGBoost, LightGBM)	Often the most accurate on tabular data	No	More hyperparameters to tune; can overfit
Support Vector Machine	Powerful on small and medium data	Yes	Slow on very large datasets
Naive Bayes	Very fast; good for text	No	Independence assumption is unrealistic
Neural Networks	Best for images, audio, text; scale with data	Yes	Need lots of data and compute; hard to interpret

Tables like this give a starting point, but the only way to know is to test. Here six models are scored on a real medical dataset (breast-cancer tumour measurements built into scikit-learn) with 5-fold cross-validation. The models that need scaling are wrapped in a Pipeline so scaling stays inside each fold, avoiding the leakage described in Module 3.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.naive_bayes import GaussianNB

X, y = load_breast_cancer(return_X_y=True)

models = {
    "Logistic Regression": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
    "k-Nearest Neighbors": make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5)),
    "Decision Tree": DecisionTreeClassifier(max_depth=4, random_state=0),
    "Random Forest": RandomForestClassifier(n_estimators=100, random_state=0),
    "SVM (RBF kernel)": make_pipeline(StandardScaler(), SVC()),
    "Gaussian Naive Bayes": GaussianNB(),
}

print(f"{'Model':22s} mean accuracy (5-fold CV)")
for name, model in models.items():
    scores = cross_val_score(model, X, y, cv=5)
    print(f"{name:22s} {scores.mean():.3f}  (+/- {scores.std():.3f})")

Expected output (click to compare with your run)

Model                  mean accuracy (5-fold CV)
Logistic Regression    0.981  (+/- 0.007)
k-Nearest Neighbors    0.965  (+/- 0.010)
Decision Tree          0.916  (+/- 0.019)
Random Forest          0.963  (+/- 0.022)
SVM (RBF kernel)       0.974  (+/- 0.015)
Gaussian Naive Bayes   0.939  (+/- 0.015)

A fair head-to-head comparison

Why used: It replaces opinions about "the best algorithm" with evidence, using one procedure for every model.
How it operates: A dictionary holds the six models. For each, cross_val_score trains and tests on five different splits and returns five accuracies; the mean and standard deviation summarise them.
Effect on output: All six score well, but the simple, scaled logistic regression comes out on top and a single decision tree trails. The small standard deviations show the ranking is fairly stable, though differences of a point or so between neighbours are not meaningful.
Role in program: It encodes a professional habit: start with a simple baseline and make complex models earn their place.

💡 "More complex" does not mean "more accurate." On many datasets, a well-prepared simple model beats an elaborate one — and is easier to explain, debug and deploy.

Module 10

🔮 Unsupervised Learning — Finding Hidden Patterns

Clustering, dimensionality reduction, and anomaly detection

🧩 Clustering — Grouping Similar Samples

K-Means, DBSCAN, Hierarchical Clustering. Used for customer segmentation, document grouping, and image compression.

K‑Means clustering

This code generates synthetic blobs and applies K‑Means to group them into 4 clusters. It prints the cluster centers and the first 10 labels.

Why used: To illustrate the concept of clustering – finding natural groupings in data without labels.
How it operates: KMeans initialises cluster centers, then iteratively assigns points to the nearest center and recomputes the centers until convergence. fit_predict returns cluster labels for each sample.
Effect on output: You see the coordinates of the 4 cluster centers and the assigned cluster numbers for the first few points. This shows the algorithm's output.
Role in program: It gives a hands‑on feel for unsupervised learning and demonstrates a common use case.

from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs

X, _ = make_blobs(n_samples=300, centers=4, cluster_std=0.6, random_state=42)

kmeans = KMeans(n_clusters=4, random_state=42)
labels = kmeans.fit_predict(X)

print("Cluster centers:", kmeans.cluster_centers_)
print("Cluster labels:", labels[:10])

💡 K-Means requires you to specify the number of clusters upfront. Use the elbow method to choose K: plot inertia vs. number of clusters and look for the "elbow".

📉 Dimensionality Reduction — PCA

Principal Component Analysis (PCA) reduces the number of features while preserving most of the variance. Useful for visualisation, noise reduction, and speeding up other algorithms.

PCA for visualisation

This example reduces the 64‑dimensional digits dataset down to 2 principal components, so you can plot the data in 2D. It prints the explained variance ratio and the new shape.

Why used: High‑dimensional data is hard to visualise and can be slow for some algorithms. PCA compresses it.
How it operates: PCA finds the directions (principal components) that maximise variance. The first component captures the most variance, the second the next, and so on. transform projects the data onto these components.
Effect on output: You see the proportion of variance explained by each of the 2 components (e.g., 0.28, 0.13) and the reduced shape (1797, 2). This lets you judge how much information was preserved.
Role in program: It demonstrates a powerful preprocessing step that can speed up training and enable 2D visualisation.

from sklearn.decomposition import PCA
from sklearn.datasets import load_digits

digits = load_digits()
X, y = digits.data, digits.target

pca = PCA(n_components=2)
X_2d = pca.fit_transform(X)

print("Explained variance ratio:", pca.explained_variance_ratio_)
print("Reduced shape:", X_2d.shape)

✨ PCA is often a first step in exploratory data analysis — project high-dimensional data down to 2D or 3D for human-readable visualisation.

🔢 Choosing K — the Elbow Method and Silhouette Score

K-Means needs you to choose k, and the data does not announce the answer. Two tools help: inertia (how tight the clusters are — it always falls as k grows, so look for the "elbow" where gains flatten) and the silhouette score (from −1 to 1; how well each point fits its own cluster versus the next nearest; higher is better).

from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import silhouette_score

X, _ = make_blobs(n_samples=300, centers=4, cluster_std=0.7, random_state=42)

print("k | inertia  | silhouette")
for k in range(2, 8):
    km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X)
    print(f"{k} | {km.inertia_:8.1f} | {silhouette_score(X, km.labels_):.3f}")

Expected output (click to compare with your run)

k | inertia  | silhouette
2 |   9125.8 | 0.609
3 |   1840.9 | 0.789
4 |    277.5 | 0.855
5 |    251.0 | 0.739
6 |    225.6 | 0.599
7 |    200.4 | 0.457

Scanning k from 2 to 7

Why used: Picking k by eye is unreliable; these numbers give a defensible choice.
How it operates: The data was generated with 4 true groups, which the algorithm is not told about. The loop fits K-Means for each k and prints the inertia and the silhouette score.
Effect on output: Inertia plunges until k = 4, then barely improves — the elbow. The silhouette score also peaks at k = 4, matching the four hidden groups.
Role in program: It shows how to make an unsupervised choice without any labels to check against.

🌙 When K-Means Fails — DBSCAN

K-Means assumes round, similarly sized clusters. DBSCAN instead groups points that are densely packed together, so it can find clusters of any shape and labels isolated points as noise (-1) — and you do not choose the number of clusters. Here, two interleaved crescent moons defeat K-Means.

from sklearn.cluster import KMeans, DBSCAN
from sklearn.datasets import make_moons
from sklearn.metrics import adjusted_rand_score

X, y_true = make_moons(n_samples=300, noise=0.07, random_state=0)

km_labels = KMeans(n_clusters=2, n_init=10, random_state=0).fit_predict(X)
db_labels = DBSCAN(eps=0.2, min_samples=5).fit_predict(X)

print("K-Means agreement with the true moons:", round(adjusted_rand_score(y_true, km_labels), 3))
print("DBSCAN  agreement with the true moons:", round(adjusted_rand_score(y_true, db_labels), 3))
print("DBSCAN clusters found:", len(set(db_labels) - {-1}), "| noise points:", int((db_labels == -1).sum()))

Expected output (click to compare with your run)

K-Means agreement with the true moons: 0.234
DBSCAN  agreement with the true moons: 1.0
DBSCAN clusters found: 2 | noise points: 0

Crescents: K-Means vs DBSCAN

Why used: It demonstrates that the "best" algorithm depends on the shape of the data.
How it operates: make_moons produces two crescents with known true groups. Both algorithms cluster without seeing those groups; adjusted_rand_score then measures agreement with the truth (1.0 = perfect, about 0 = random). DBSCAN's eps is the neighbourhood radius and min_samples is how many neighbours make a point "dense."
Effect on output: K-Means cuts straight across both crescents and scores poorly. DBSCAN follows each crescent and recovers them exactly.
Role in program: It widens your toolbox beyond K-Means and highlights that clustering results always need interpretation.

🚨 Anomaly Detection — Finding the Odd One Out

Fraud, equipment faults and network intrusions are rare and varied, so there are seldom enough labelled examples to train a classifier. Isolation Forest exploits a simple fact: anomalies are few and different, so random splits isolate them in very few steps.

import numpy as np
from sklearn.ensemble import IsolationForest

rng = np.random.default_rng(0)
normal = rng.normal(0, 1, size=(300, 2))
outliers = rng.uniform(-6, 6, size=(10, 2))
X = np.vstack([normal, outliers])

iso = IsolationForest(contamination=10 / 310, random_state=0).fit(X)
flags = iso.predict(X)                    # -1 = anomaly, 1 = normal

print("Points flagged as anomalies:", int((flags == -1).sum()))
print("Of the 10 planted outliers, caught:", int((flags[-10:] == -1).sum()))

Expected output (click to compare with your run)

Points flagged as anomalies: 10
Of the 10 planted outliers, caught: 8

Isolating planted outliers

Why used: It is a standard unsupervised approach when labelled anomalies are scarce.
How it operates: 300 normal points form a tight bell-shaped cloud, and 10 outliers are scattered widely. contamination tells the model roughly what share of data to flag. predict returns −1 for anomalies.
Effect on output: It flags 10 points and catches most — but not necessarily all — of the planted outliers. Some outliers land inside the normal cloud by chance and are genuinely indistinguishable, and a couple of normal points get flagged instead (in the tested run it caught 8 of the 10).
Role in program: It reminds you that detectors trade off missed anomalies against false alarms, just like classifiers.

Other unsupervised tools worth knowing

Hierarchical clustering (builds a tree of nested groups), Gaussian mixture models (soft, probabilistic clusters), t-SNE and UMAP (non-linear 2D visualisation of high-dimensional data), and autoencoders (neural networks that learn compressed representations).

Module 11

📈 Evaluation Metrics — How to Know if Your Model Works

Accuracy isn't enough — understand the full picture

Metric	Classification	Regression
Accuracy	✓	—
Precision	✓	—
Recall	✓	—
F1-Score	✓	—
Confusion Matrix	✓	—
ROC-AUC	✓	—
MAE	—	✓
RMSE	—	✓
R²	—	✓

Choosing the Right Metric

For imbalanced classification (fraud detection, rare disease diagnosis), use precision/recall or F1-score — accuracy can be misleading (99% accuracy on a dataset with 1% fraud could just be predicting "not fraud" for everything). For regression, RMSE punishes large errors more than MAE.

📊 Confusion Matrix — The Complete Picture

Reading a confusion matrix

This snippet is self-contained: it trains a quick classifier on the iris dataset, then builds the confusion matrix and prints it as a labelled table. Each row is a true class and each column is a predicted class, so the diagonal holds the correct predictions and everything off the diagonal is a mistake.

Why used: A confusion matrix breaks accuracy down by class, showing exactly which classes get mixed up.
How it operates: confusion_matrix counts predictions for every (true, predicted) pair. Wrapping it in a pandas DataFrame with named rows and columns makes it easy to read. stratify keeps the class proportions equal in both splits.
Effect on output: You see that setosa is identified perfectly, and any error is a confusion between the two similar-looking species, versicolor and virginica.
Role in program: It guides improvement — it tells you which classes need more data or better features.

import pandas as pd
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import confusion_matrix

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.3, random_state=0, stratify=iris.target)

model = RandomForestClassifier(n_estimators=100, random_state=0)
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
names = list(iris.target_names)
cm = pd.DataFrame(confusion_matrix(y_test, y_pred),
                  index=["true " + n for n in names],
                  columns=["pred " + n for n in names])
print(cm)

Expected output (click to compare with your run)

                 pred setosa  pred versicolor  pred virginica
true setosa               15                0               0
true versicolor            0               15               0
true virginica             0                1              14

💡 The confusion matrix tells you where your model makes mistakes — which classes are being confused. This guides data collection and feature engineering.

🎯 Precision, Recall and F1 — Built by Hand

For a yes/no model, every prediction falls into one of four boxes: TP (said yes, was yes), FP (said yes, was no — a false alarm), FN (said no, was yes — a miss) and TN (said no, was no).

precision = TP / (TP + FP) "when the model says yes, how often is it right?" recall = TP / (TP + FN) "of all the real yeses, how many did the model catch?" F1 = 2 × precision × recall / (precision + recall) (the harmonic mean — low if either is low) accuracy = (TP + TN) / total

import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score

y_true = np.array([1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0])
y_pred = np.array([1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0])

tp = int(np.sum((y_pred == 1) & (y_true == 1)))
fp = int(np.sum((y_pred == 1) & (y_true == 0)))
fn = int(np.sum((y_pred == 0) & (y_true == 1)))
tn = int(np.sum((y_pred == 0) & (y_true == 0)))
print("TP, FP, FN, TN =", tp, fp, fn, tn)

precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)
print(f"from scratch : precision={precision:.3f}  recall={recall:.3f}  f1={f1:.3f}")
print(f"scikit-learn : precision={precision_score(y_true, y_pred):.3f}  "
      f"recall={recall_score(y_true, y_pred):.3f}  f1={f1_score(y_true, y_pred):.3f}")

Expected output (click to compare with your run)

TP, FP, FN, TN = 3 1 2 6
from scratch : precision=0.750  recall=0.600  f1=0.667
scikit-learn : precision=0.750  recall=0.600  f1=0.667

The formulas, checked against scikit-learn

Why used: Computing the metrics yourself removes the mystery, and confirming they match scikit-learn proves you understand them.
How it operates: Boolean masks like (y_pred == 1) & (y_true == 0) pick out each type of outcome; np.sum counts them. The three formulas then follow directly.
Effect on output: The counts are TP 3, FP 1, FN 2, TN 6. Precision is 0.75 (one false alarm among four "yes" calls), recall is 0.60 (two of five real positives missed), and both methods agree exactly.
Role in program: It prepares you for the next block, where accuracy alone proves misleading.

💡 Precision and recall trade off. Making a model more cautious (higher threshold) raises precision but lowers recall; making it more eager does the reverse. Choose by the cost of each mistake: for a cancer screening test a miss is worse than a false alarm (favour recall); for a spam filter deleting a real email is worse than letting spam through (favour precision).

🪤 The Accuracy Trap on Imbalanced Data

When only 2% of cases are positive, a "model" that always answers no is 98% accurate — and completely useless. This block shows that, and also introduces ROC-AUC: the probability that the model gives a randomly chosen positive a higher score than a randomly chosen negative (0.5 = coin flip, 1.0 = perfect ranking), which does not depend on any single threshold.

from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=5000, n_features=10, n_informative=5,
                           weights=[0.98, 0.02], flip_y=0, class_sep=1.0, random_state=0)
print("share of the positive class:", round(float(y.mean()), 3))

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)

def report(name, model):
    model.fit(X_tr, y_tr)
    pred = model.predict(X_te)
    auc = roc_auc_score(y_te, model.predict_proba(X_te)[:, 1])
    print(f"{name:16s} acc={accuracy_score(y_te, pred):.3f} precision={precision_score(y_te, pred, zero_division=0):.3f} "
          f"recall={recall_score(y_te, pred):.3f} f1={f1_score(y_te, pred):.3f} auc={auc:.3f}")

report("Always say 'no'", DummyClassifier(strategy="most_frequent"))
report("Random Forest", RandomForestClassifier(n_estimators=100, random_state=0))
report("RF (balanced)", RandomForestClassifier(n_estimators=100, class_weight="balanced", random_state=0))

Expected output (click to compare with your run)

share of the positive class: 0.02
Always say 'no'  acc=0.980 precision=0.000 recall=0.000 f1=0.000 auc=0.500
Random Forest    acc=0.989 precision=1.000 recall=0.433 f1=0.605 auc=0.968
RF (balanced)    acc=0.987 precision=0.727 recall=0.533 f1=0.615 auc=0.992

Three models, one imbalanced dataset

Why used: Fraud, disease, defects and churn are all rare-event problems where accuracy deceives.
How it operates: make_classification creates a dataset with 98% negatives. A DummyClassifier always predicts the majority class. A Random Forest is trained normally and then with class_weight="balanced", which makes mistakes on the rare class count for more.
Effect on output: The "always no" model gets 0.98 accuracy yet zero precision, recall and F1. The forests are only a little more accurate but catch a meaningful share of the positives, and the balanced forest trades some precision for higher recall.
Role in program: It hammers home that you must pick metrics that match the problem.

⚠️ Even the better models here miss roughly half the rare positives. Remedies include collecting more positive examples, adjusting the decision threshold, resampling (e.g. SMOTE), and anomaly-detection methods.

📏 Regression Metrics at a Glance

MAE = mean( |actual − predicted| ) average miss, in the target's own units RMSE = √ mean( (actual − predicted)² ) like MAE but punishes large misses harder R² = 1 − (model's squared error / error of always predicting the mean) 1 = perfect; 0 = no better than predicting the average; negative = worse than that

💡 Always compare a model to a trivial baseline (predict the mean, or always predict the majority class). A model that cannot beat the baseline has learned nothing useful, however impressive its raw score looks.

Module 12

🧠 Neural Networks & Deep Learning

You have already built a neuron and a two-layer network by hand. This module zooms out: how deep networks are organised, how they learn, and how to use them with modern frameworks.

🔗 From Neuron to Network

A neuron computes a weighted sum of its inputs and passes it through an activation function. A layer is many neurons side by side; a network stacks layers, so the output of one becomes the input of the next. "Deep" simply means many layers. Early layers learn simple patterns (edges in an image), and later layers combine them into complex ones (eyes, faces).

layer output = activation( inputs @ weights + bias ) network = layer₃( layer₂( layer₁(x) ) )

Without a non-linear activation, stacking layers would be pointless: a chain of linear steps is still just one linear step. Activations give networks the power to model curved, complex relationships.

import numpy as np

z = np.array([-3.0, -1.0, 0.0, 1.0, 3.0])
sigmoid = 1 / (1 + np.exp(-z))
tanh = np.tanh(z)
relu = np.maximum(0, z)
leaky = np.where(z > 0, z, 0.01 * z)

print("z         :", z)
print("sigmoid   :", sigmoid.round(3))
print("tanh      :", tanh.round(3))
print("ReLU      :", relu)
print("LeakyReLU :", leaky.round(2))

# softmax turns raw scores (logits) into probabilities that add up to 1
logits = np.array([2.0, 1.0, 0.1])
exps = np.exp(logits - logits.max())          # subtracting the max keeps the numbers stable
softmax = exps / exps.sum()
print("softmax   :", softmax.round(3), "| sum =", round(float(softmax.sum()), 3))

Expected output (click to compare with your run)

z         : [-3. -1.  0.  1.  3.]
sigmoid   : [0.047 0.269 0.5   0.731 0.953]
tanh      : [-0.995 -0.762  0.     0.762  0.995]
ReLU      : [0. 0. 0. 1. 3.]
LeakyReLU : [-0.03 -0.01  0.    1.    3.  ]
softmax   : [0.659 0.242 0.099] | sum = 1.0

Function	Output range	Typical use
ReLU max(0, z)	0 to ∞	The default for hidden layers: simple and fast
Leaky ReLU	−∞ to ∞	Like ReLU, but negative inputs keep a small slope
Tanh	−1 to 1	Zero-centred; common in older and recurrent networks
Sigmoid	0 to 1	Output layer for binary classification
Softmax	0 to 1, sums to 1	Output layer for multi-class classification
Linear (none)	−∞ to ∞	Output layer for regression

🔁 How a Network Learns

Every training step repeats the same four-part loop — exactly the loop of Program 9:

The training loop

Forward pass — push a batch of data through the layers to get predictions.
Loss — measure how wrong the predictions are with a loss function.
Backward pass — use the chain rule to compute the gradient of the loss with respect to every weight (backpropagation).
Update — nudge every weight a little against its gradient (gradient descent, or a smarter variant such as Adam).

Epoch — one full pass over the training data. Batch / mini-batch — the small chunk of data used for each update.
Learning rate — the step size; the single most important setting. Too high and training diverges; too low and it crawls.
Optimisers — SGD (plain gradient steps), SGD with momentum, and Adam (adapts the step size for each weight; a popular default).
Loss functions — mean squared error for regression, binary cross-entropy for yes/no, categorical cross-entropy for many classes.

💡 Frameworks compute the backward pass automatically (automatic differentiation). You define the forward pass; loss.backward() does the calculus.

🔢 Your First Neural Network: Recognising Handwritten Digits

scikit-learn includes a small neural network, MLPClassifier (a multi-layer perceptron), and a digits dataset of 1,797 images, each 8×8 pixels. That is enough to see a real network in action without leaving the browser.

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

digits = load_digits()
X_tr, X_te, y_tr, y_te = train_test_split(
    digits.data, digits.target, test_size=0.25, random_state=0, stratify=digits.target)

net = make_pipeline(StandardScaler(),
                    MLPClassifier(hidden_layer_sizes=(64, 32), max_iter=500, random_state=0))
net.fit(X_tr, y_tr)

print("Test accuracy:", round(net.score(X_te, y_te), 3))
print("Weight matrix shapes:", [w.shape for w in net[-1].coefs_])
print("Predicted:", net.predict(X_te[:10]))
print("True     :", y_te[:10])

Expected output (click to compare with your run)

Test accuracy: 0.964
Weight matrix shapes: [(64, 64), (64, 32), (32, 10)]
Predicted: [2 0 4 9 4 1 2 4 6 7]
True     : [2 0 4 9 4 1 2 4 6 7]

A two-hidden-layer network on digits

Why used: It is the quickest route from "I understand neurons" to "I trained a network that reads handwriting."
How it operates: Each image's 64 pixels are the input. Two hidden layers (64 and 32 neurons) feed an output layer of 10 neurons — one per digit. StandardScaler normalises the inputs, which neural networks need. stratify keeps digit proportions equal across the split.
Effect on output: The network classifies roughly 96% of unseen digits correctly. The printed weight shapes — (64, 64), (64, 32) and (32, 10) — are the matrices from the matrix-multiplication lesson in Module 2.
Role in program: It is a complete deep-learning workflow in miniature: scale, train, evaluate on held-out data.

🏗️ The Main Architectures

Architecture	Best for	Core idea
MLP (fully connected)	Tabular data, simple tasks	Every neuron connects to every neuron in the next layer
CNN (convolutional)	Images, video, audio spectrograms	Small filters slide across the input detecting local patterns, so the same edge detector works anywhere in the image
RNN / LSTM	Sequences (older approach to text and time series)	A hidden state carries memory from one step to the next
Transformer	Language, and increasingly images, audio and code	Attention lets every element look at every other element in parallel (see Module 13)
Autoencoder	Compression, denoising, anomaly detection	Squeeze the input into a small code, then reconstruct it
Generative models (GANs, diffusion, language models)	Creating new images, text, audio	Learn the distribution of the data well enough to sample new examples from it

🔥 TensorFlow/Keras and PyTorch

For anything beyond small experiments you will use a deep-learning framework, which provides automatic differentiation, GPU support and ready-made layers. The two dominant choices are PyTorch (flexible and very popular in research) and TensorFlow with Keras (a high-level, production-friendly API). Learn one well; the concepts transfer.

⚠️ TensorFlow and PyTorch are too large to run in the browser, so the two examples below have no Run button. Copy them into Google Colab (free GPUs) or run them on your own machine.

Keras — classify the real MNIST digits (28×28 images):

# Run locally or in Google Colab:  pip install tensorflow
import tensorflow as tf

(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()   # downloads on first use
x_train, x_test = x_train / 255.0, x_test / 255.0                           # scale pixels to 0-1

model = tf.keras.Sequential([
    tf.keras.Input(shape=(28, 28)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation="relu"),
    tf.keras.layers.Dropout(0.2),                       # randomly switches off 20% of neurons while training
    tf.keras.layers.Dense(10, activation="softmax"),    # one probability per digit 0-9
])
model.compile(optimizer="adam",
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])

model.fit(x_train, y_train, epochs=5, validation_split=0.1)
test_loss, test_acc = model.evaluate(x_test, y_test)
print("Test accuracy:", test_acc)

A Keras model, step by step

Why used: It is the standard "hello world" of deep learning, with only a few lines for a complete model.
How it operates: Sequential stacks layers in order. Flatten turns each 28×28 image into 784 numbers; Dense(128, relu) is a hidden layer; Dropout randomly disables neurons during training to fight overfitting; the final Dense(10, softmax) outputs ten digit probabilities. compile picks the optimiser and loss; fit runs the training loop.
Effect on output: Keras prints the loss and accuracy for every epoch, then the test accuracy; a network this small typically reaches the high 90s percent on MNIST.
Role in program: It is the framework version of Program 9 and the Module 12 digits example.

PyTorch — the training loop written out explicitly:

# Run locally or in Google Colab:  pip install torch
import torch
from torch import nn

torch.manual_seed(0)
X = torch.randn(200, 2)
y = ((X[:, 0] * X[:, 1]) > 0).float().unsqueeze(1)    # 1 when both inputs share the same sign

model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 1))
loss_fn = nn.BCEWithLogitsLoss()                      # sigmoid + log loss in one stable step
optimizer = torch.optim.Adam(model.parameters(), lr=0.05)

for epoch in range(200):
    optimizer.zero_grad()                             # clear gradients from the last step
    loss = loss_fn(model(X), y)                       # forward pass + loss
    loss.backward()                                   # backpropagation: computes every gradient
    optimizer.step()                                  # gradient descent update
    if epoch % 50 == 0:
        print(f"epoch {epoch:3d}  loss={loss.item():.4f}")

accuracy = ((torch.sigmoid(model(X)) > 0.5).float() == y).float().mean()
print("train accuracy:", round(accuracy.item(), 3))

The four-step loop in PyTorch

Why used: PyTorch exposes the training loop, so you can see forward, loss, backward and update as separate lines — the same loop you wrote by hand.
How it operates: A tiny network (Linear → ReLU → Linear) learns whether two numbers share the same sign — an XOR-like pattern. Each epoch clears old gradients, computes the loss, calls loss.backward() and takes an Adam step.
Effect on output: The loss should fall steadily over the 200 epochs and training accuracy should climb well above chance. (Exact values depend on your PyTorch version and hardware.)
Role in program: It shows what .backward() and .step() replace from Program 9.

🧰 Practical Tips for Training Networks

Scale your inputs (standardise, or divide pixels by 255).
Start small, and check that your network can overfit a tiny batch — if it cannot, there is a bug.
Watch training and validation loss together: validation loss rising while training loss falls means overfitting.
Use early stopping, dropout and weight decay to control overfitting.
Tune the learning rate first before touching anything else.
Prefer transfer learning: start from a pretrained model and fine-tune it instead of training from scratch.
Use a GPU for larger models — Google Colab and Kaggle offer free ones.

Module 13

💬 NLP & Language Models

Computers work with numbers, not words. Natural language processing is the story of turning text into numbers that preserve meaning — from simple word counts to the Transformers behind today's chatbots.

🔤 Step 1 — Text as Numbers: Bag of Words

The simplest representation: build a vocabulary of every word, then describe each document by how many times each word appears. Order is thrown away — hence "bag" of words. Similar documents get similar vectors, which the cosine similarity from Module 2 can measure.

from collections import Counter
import numpy as np

docs = ["the cat sat on the mat", "the dog sat on the log", "stock prices fell sharply today"]
tokens = [d.split() for d in docs]
vocab = sorted({w for t in tokens for w in t})
print("vocabulary:", vocab)

vectors = []
for d, t in zip(docs, tokens):
    counts = Counter(t)
    vec = np.array([counts[w] for w in vocab])
    vectors.append(vec)
    print(f"{d!r:36} -> {vec}")

def cosine(u, v):
    return float(u @ v / (np.linalg.norm(u) * np.linalg.norm(v)))

print("\ncosine(cat-doc, dog-doc)   =", round(cosine(vectors[0], vectors[1]), 3))
print("cosine(cat-doc, stock-doc) =", round(cosine(vectors[0], vectors[2]), 3))

Expected output (click to compare with your run)

vocabulary: ['cat', 'dog', 'fell', 'log', 'mat', 'on', 'prices', 'sat', 'sharply', 'stock', 'the', 'today']
'the cat sat on the mat'             -> [1 0 0 0 1 1 0 1 0 0 2 0]
'the dog sat on the log'             -> [0 1 0 1 0 1 0 1 0 0 2 0]
'stock prices fell sharply today'    -> [0 0 1 0 0 0 1 0 1 1 0 1]

cosine(cat-doc, dog-doc)   = 0.75
cosine(cat-doc, stock-doc) = 0.0

Counting words into vectors

Why used: It is the foundation of classic text classification and search.
How it operates: The vocabulary is the sorted set of all distinct words. For each document a Counter tallies word frequencies, and one count per vocabulary word forms a vector. cosine then compares two vectors by the angle between them.
Effect on output: The "cat" and "dog" sentences share words (the, sat, on) so their similarity is high, while the finance sentence shares nothing and scores exactly 0.
Role in program: It shows the core idea behind all text models: meaning becomes geometry.

💡 Bag of words cannot tell "dog bites man" from "man bites dog," and treats "good" and "great" as unrelated. The next steps address those weaknesses.

⚖️ Step 2 — TF-IDF and a Sentiment Classifier

TF-IDF improves word counts by down-weighting words that appear in almost every document (like "the") and up-weighting distinctive ones. Combined with logistic regression it makes a surprisingly strong text classifier. This toy example learns to separate positive from negative movie reviews from only 12 examples, so treat the probabilities as a demonstration, not as trustworthy sentiment analysis.

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

reviews = [
    ("great movie loved the acting", 1), ("fantastic story and wonderful music", 1),
    ("absolutely brilliant and moving", 1), ("a joy to watch loved it", 1),
    ("really enjoyable and funny", 1), ("wonderful performances great fun", 1),
    ("terrible plot and bad acting", 0), ("boring and far too long", 0),
    ("awful script i hated it", 0), ("a waste of time terrible", 0),
    ("dull boring and predictable", 0), ("bad movie awful ending", 0),
]
texts, labels = zip(*reviews)

model = make_pipeline(TfidfVectorizer(), LogisticRegression(C=10))
model.fit(texts, labels)

tests = ["wonderful acting loved it", "boring and terrible", "the plot was great fun"]
for t, p in zip(tests, model.predict_proba(tests)[:, 1]):
    print(f"{t!r:30} P(positive) = {p:.2f}")

vec, clf = model[0], model[1]
vocab = np.array(vec.get_feature_names_out())
order = np.argsort(clf.coef_[0])
print("most negative words:", list(vocab[order[:4]]))
print("most positive words:", list(vocab[order[-4:]]))

Expected output (click to compare with your run)

'wonderful acting loved it'    P(positive) = 0.79
'boring and terrible'          P(positive) = 0.15
'the plot was great fun'       P(positive) = 0.73
most negative words: ['awful', 'boring', 'bad', 'terrible']
most positive words: ['moving', 'wonderful', 'great', 'loved']

Learning sentiment from words

Why used: It is the standard strong baseline for text classification, and it is interpretable.
How it operates: A Pipeline chains TfidfVectorizer (text → weighted word vectors) and LogisticRegression. After fitting, each word has a learned coefficient: positive for words that signal a good review, negative for bad ones. C=10 weakens the regularisation, which suits this tiny dataset.
Effect on output: The model rates "wonderful acting loved it" clearly positive and "boring and terrible" clearly negative. The most extreme coefficients are exactly the words you would expect.
Role in program: It shows the full NLP pipeline — vectorise, train, predict — and how to inspect what the model learned.

⚠️ Words that never appeared in training (like "was" in the third test) contribute nothing, so predictions on text unlike the training data are unreliable. This is why real sentiment models train on thousands or millions of examples.

🧭 Step 3 — Embeddings: Meaning as Position

An embedding represents each word (or sentence) as a dense vector of a few hundred numbers, learned so that items used in similar contexts end up close together in space. "King" and "queen" land near each other, far from "banana." Instead of the rigid, mostly-zero count vectors above, embeddings capture similarity of meaning — "good" and "great" become neighbours.

Modern models create contextual embeddings, where the vector for "bank" differs in "river bank" and "bank account" depending on the surrounding words.

🔦 Step 4 — Attention and the Transformer

Reading a sentence, you do not give every word equal weight: to understand "it" in "The cat chased the mouse because it was hungry," you look back at the relevant noun. Self-attention gives a model the same ability. Every token creates three vectors — a query (what am I looking for?), a key (what do I contain?) and a value (what do I pass on?). Comparing queries to keys gives attention weights, which blend the values.

Attention(Q, K, V) = softmax( Q·Kᵀ / √d ) · V

import numpy as np

def softmax(x):
    e = np.exp(x - x.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

np.random.seed(0)
tokens = ["the", "cat", "sat"]
d = 4                                      # tiny embedding size
X = np.random.randn(len(tokens), d)        # one vector per token

Wq, Wk, Wv = (np.random.randn(d, d) for _ in range(3))
Q, K, V = X @ Wq, X @ Wk, X @ Wv

scores = Q @ K.T / np.sqrt(d)              # how strongly each token "looks at" every other token
weights = softmax(scores)                  # each row sums to 1
output = weights @ V                       # every token becomes a weighted mix of the value vectors

print("attention weights (row = token looking, column = token looked at):")
for tok, row in zip(tokens, weights.round(2)):
    print(f"{tok:>4}: {row}")
print("row sums:", weights.sum(axis=1).round(2))
print("output shape:", output.shape)

Expected output (click to compare with your run)

attention weights (row = token looking, column = token looked at):
 the: [0. 1. 0.]
 cat: [0.06 0.04 0.9 ]
 sat: [0.07 0.91 0.01]
row sums: [1. 1. 1.]
output shape: (3, 4)

Line-by-line walkthrough

def softmax(x) — turns each row of raw scores into positive weights that sum to 1. Subtracting the row maximum first keeps the exponentials from overflowing.
tokens = ["the", "cat", "sat"]; d = 4 — a three-word sentence, with each word represented by a tiny 4-number embedding.
X = np.random.randn(len(tokens), d) — random embeddings stand in for learned ones (a real model learns these during training).
Wq, Wk, Wv = (np.random.randn(d, d) for _ in range(3)) — three weight matrices that project each embedding into its query, key and value roles. A real model learns these too.
Q, K, V = X @ Wq, X @ Wk, X @ Wv — computes the query, key and value vectors for all tokens at once.
scores = Q @ K.T / np.sqrt(d) — the dot product of every query with every key gives a 3×3 grid of "how relevant is token j to token i." Dividing by √d keeps the numbers from growing with the vector size.
weights = softmax(scores) — each row becomes a set of attention weights that sum to 1.
output = weights @ V — each token's new representation is a weighted mix of all the value vectors. Tokens now carry information from the others they paid attention to.
The prints show the weights as a table and confirm each row sums to 1.

⚠️ The weights here come from random matrices, so which word attends to which is meaningless — only the mechanism is real. In a trained model, the matrices are learned so that attention focuses on genuinely relevant words.

A Transformer stacks many attention layers (with several "heads" attending in parallel), plus feed-forward layers and normalisation. Because attention processes all tokens at once, Transformers train efficiently on GPUs — the key to scaling them up.

🤖 Step 5 — Large Language Models

A large language model (LLM) is a Transformer trained on a vast amount of text with one deceptively simple goal: predict the next token. To do that well across billions of examples, the model must absorb grammar, facts, reasoning patterns and writing styles.

Tokens — text is split into sub-word pieces (not always whole words) before being fed to the model.
Pre-training — learning to predict the next token over a huge text corpus.
Fine-tuning & alignment — further training on curated examples and human feedback so the model follows instructions helpfully and safely.
Generation — the model produces text one token at a time, sampling from its predicted probabilities; a "temperature" setting controls how adventurous the sampling is.
Context window — how much text the model can consider at once.
Retrieval-augmented generation (RAG) — look up relevant documents first and give them to the model, grounding its answer in real sources.

⚠️ Language models can be wrong while sounding certain. They can state false facts ("hallucinate"), reflect biases in their training text, and miss recent events. Verify important claims against reliable sources, and never paste sensitive personal data into tools you do not control.

To try a pretrained model yourself, the Hugging Face transformers library gives access to thousands of them in a few lines (run locally or in Colab; it downloads a model on first use):

# Run locally or in Google Colab:  pip install transformers torch
from transformers import pipeline

classifier = pipeline("sentiment-analysis")           # downloads a small pretrained model on first use
results = classifier(["I love this course!", "This was a waste of time."])
for r in results:
    print(r["label"], round(r["score"], 3))

Module 14

🎮 Reinforcement Learning

The third great paradigm: instead of learning from examples, an agent learns by trying things and seeing what happens.

🧩 The Reinforcement Learning Setup

An agent sits in an environment. At each step it observes a state, chooses an action, and receives a reward and a new state. Its goal is not the best immediate reward but the largest total reward over time. The rule that picks actions is its policy.

	Supervised	Unsupervised	Reinforcement
Learns from	Labelled examples	Unlabelled data	Rewards from interaction
Feedback	The correct answer, each time	None	Delayed, often just "good" or "bad"
Goal	Predict the label	Find structure	Maximise long-term reward

Two core tensions: exploration vs exploitation (try new actions, or repeat what already works?) and the credit assignment problem (when a reward arrives late, which earlier action deserves the credit?).

From scratch — Q-learning

🗺️ A Q-Learning Agent in a Grid World

The agent starts at the top-left of a 4×4 grid. Reaching the goal (G) earns +10, falling into the trap (X) costs −10, and every other step costs −1 so that wandering is discouraged. It keeps a table Q of how good each action is in each cell, and improves the table from experience.

Q(s, a) ← Q(s, a) + α × [ reward + γ × max Q(next state) − Q(s, a) ] α = learning rate γ = discount factor (how much future rewards matter) ε = chance of a random exploratory move

import numpy as np

np.random.seed(0)
SIZE = 4
GOAL, TRAP = (3, 3), (1, 2)
ACTIONS = [(-1, 0), (1, 0), (0, -1), (0, 1)]          # up, down, left, right
LETTERS = ["U", "D", "L", "R"]

def step(state, action):
    r, c = state
    dr, dc = ACTIONS[action]
    r, c = min(max(r + dr, 0), SIZE - 1), min(max(c + dc, 0), SIZE - 1)
    if (r, c) == GOAL:
        return (r, c), 10, True
    if (r, c) == TRAP:
        return (r, c), -10, True
    return (r, c), -1, False

Q = np.zeros((SIZE, SIZE, 4))                          # one value per (row, col, action)
alpha, gamma, epsilon = 0.1, 0.95, 0.2

for episode in range(2000):
    state, done, steps = (0, 0), False, 0
    while not done and steps < 200:
        if np.random.rand() < epsilon:
            action = np.random.randint(4)              # explore
        else:
            action = int(np.argmax(Q[state]))          # exploit
        next_state, reward, done = step(state, action)
        best_next = 0 if done else np.max(Q[next_state])
        Q[state][action] += alpha * (reward + gamma * best_next - Q[state][action])
        state = next_state
        steps += 1

state, path = (0, 0), [(0, 0)]
while state not in (GOAL, TRAP) and len(path) < 20:
    state, _, _ = step(state, int(np.argmax(Q[state])))
    path.append(state)
print("Learned path:", path)
print("Reached the goal:", path[-1] == GOAL)

print("\nBest action in each cell (G = goal, X = trap):")
for r in range(SIZE):
    cells = []
    for c in range(SIZE):
        if (r, c) == GOAL:
            cells.append("G")
        elif (r, c) == TRAP:
            cells.append("X")
        else:
            cells.append(LETTERS[int(np.argmax(Q[r, c]))])
    print(" ".join(cells))

Expected output (click to compare with your run)

Learned path: [(0, 0), (1, 0), (2, 0), (2, 1), (3, 1), (3, 2), (3, 3)]
Reached the goal: True

Best action in each cell (G = goal, X = trap):
D D L D
D D X D
R D D D
R R R G

How the agent learns

Why used: Q-learning is the foundation of value-based reinforcement learning, and a small grid makes every number inspectable.
How it operates: step defines the environment: it moves the agent (staying inside the grid) and returns the new position, reward and whether the episode has ended. For 2,000 episodes the agent picks a random action with probability ε (explore) or the best-known action (exploit), then applies the update rule above. best_next is zero at the end of an episode.
Effect on output: After training, following the best action in each cell leads from the start to the goal while avoiding the trap. The printed map shows the learned best move in every cell — U/D/L/R for up, down, left, right.
Role in program: It demonstrates learning purely from rewards, with no labelled examples anywhere.

💡 Real-world RL — game-playing systems, robot control, recommendation tuning, and the human-feedback stage that aligns chatbots — replaces this table with a neural network (deep reinforcement learning), because most problems have far too many states to list.

Module 15

🛡️ Responsible AI & Taking Models to Production

A model that scores well in a notebook is not yet a good system. Models affect real people — loan decisions, medical triage, hiring, moderation — and they change once the world changes around them.

⚖️ Core Principles

⚖️

Fairness

Does the model work equally well for different groups of people? Bias enters through unrepresentative data, historical prejudice captured in labels, and the choice of what to predict.

🔍

Transparency & explainability

Can you explain why a decision was made? Vital when people are affected and when regulators or users ask for reasons.

🔒

Privacy & security

Collect only the data you need, protect it, and remember that models can memorise and leak parts of their training data.

🧑‍⚖️

Accountability & human oversight

A person or team must be responsible for outcomes, with humans in the loop for high-stakes decisions and a way to contest results.

🧱

Reliability & safety

Test how the model behaves on unusual inputs, and plan for failure — especially where mistakes can cause harm.

🔎 Overall Accuracy Can Hide Unequal Performance

A single accuracy number averages over everyone, so a small group can be served badly while the headline looks fine. The standard check is simple: evaluate separately for each group that matters. This stylised example has a large group A and a small group B whose underlying pattern differs.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
# Stylised example: Group A is large, Group B is small, and the rule that links x to y
# is different for B (its positives start at a higher x than group A's do)
xa = rng.normal(0, 1, 900)
ya = (xa + rng.normal(0, 0.5, 900) > 0).astype(int)
xb = rng.normal(0, 1, 100)
yb = (xb + rng.normal(0, 0.5, 100) > 1.0).astype(int)

X = np.concatenate([xa, xb]).reshape(-1, 1)
y = np.concatenate([ya, yb])
group = np.array(["A"] * 900 + ["B"] * 100)

X_tr, X_te, y_tr, y_te, g_tr, g_te = train_test_split(
    X, y, group, test_size=0.3, random_state=0, stratify=group)
pred = LogisticRegression().fit(X_tr, y_tr).predict(X_te)

print("Overall accuracy:", round(float(np.mean(pred == y_te)), 3))
for g in ["A", "B"]:
    mask = g_te == g
    print(f"Group {g} accuracy: {np.mean(pred[mask] == y_te[mask]):.3f}  (n={mask.sum()})")

Expected output (click to compare with your run)

Overall accuracy: 0.807
Group A accuracy: 0.826  (n=270)
Group B accuracy: 0.633  (n=30)

Per-group evaluation

Why used: Fairness problems are invisible in aggregate metrics, so disaggregated evaluation is the first and cheapest audit.
How it operates: Group A (900 people) and group B (100) follow different rules relating x to the outcome. One logistic regression is trained on everyone, so it mostly learns group A's rule. The code then computes accuracy within each group on the test set.
Effect on output: Overall accuracy looks decent, but group B's accuracy is noticeably lower — and its test sample is small (30 people), so that figure is itself uncertain.
Role in program: It is a template for the audit you should run on any model that affects people.

⚠️ This is a deliberately simplified illustration, not a complete fairness analysis. Real fairness work also considers different definitions of fairness (which can conflict with one another), how the labels were produced, legal requirements, and consultation with the affected communities.

💡 Peeking Inside a Model: Permutation Importance

To see which features a model relies on, shuffle one feature's values in the test set and measure how much the score drops. A big drop means the model depended on that feature. It works with any model, which is why it is a favourite first tool for explanation (SHAP and LIME are more detailed alternatives).

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

data = load_breast_cancer()
X_tr, X_te, y_tr, y_te = train_test_split(data.data, data.target, test_size=0.3, random_state=0)
model = RandomForestClassifier(n_estimators=60, random_state=0).fit(X_tr, y_tr)
print("test accuracy:", round(model.score(X_te, y_te), 3))

result = permutation_importance(model, X_te, y_te, n_repeats=5, random_state=0)
top = result.importances_mean.argsort()[::-1][:5]
print("Top 5 features by permutation importance:")
for i in top:
    print(f"  {data.feature_names[i]:25s} {result.importances_mean[i]:.4f}")

Expected output (click to compare with your run)

test accuracy: 0.965
Top 5 features by permutation importance:
  worst radius              0.0164
  worst perimeter           0.0164
  worst area                0.0152
  worst concavity           0.0129
  worst concave points      0.0105

Which features matter?

Why used: Explanations build trust, expose silly shortcuts (like a feature that secretly encodes the answer) and help debugging.
How it operates: A random forest is trained on tumour measurements. permutation_importance shuffles each feature five times on the test set and records the average accuracy drop. The five largest are printed.
Effect on output: The top features are size and shape measurements such as worst radius and worst perimeter. The importances are small numbers because many features carry overlapping information, so shuffling just one of them costs little accuracy — the model can lean on its correlated neighbours.
Role in program: It illustrates the question to ask of every model: is it right for the right reasons?

🚀 From Notebook to Production (MLOps)

Save the whole pipeline — preprocessing and model together, so production applies identical steps. Use joblib.dump(pipeline, "model.joblib"), and record library versions.

Serve it — wrap the model in an API (for example with FastAPI or Flask), or run it in batch on a schedule.

Version everything — code, data, model files and experiment settings, so any result can be reproduced.

Monitor — track input data, predictions and, when the truth arrives, accuracy. Alert when they change.

Retrain and redeploy — models decay as the world changes, so plan for regular updates and safe rollbacks.

The most common reason deployed models decay is drift: data drift (the inputs change — a new customer demographic) and concept drift (the relationship between inputs and outcome changes — consumer behaviour shifts). A statistical test can compare incoming data with the training data:

import numpy as np
from scipy.stats import ks_2samp

rng = np.random.default_rng(0)
train_income = rng.normal(35000, 8000, 2000)
this_week = rng.normal(35000, 8000, 500)        # same distribution as training
after_campaign = rng.normal(42000, 8000, 500)   # the customer base has changed

for name, sample in [("this week", this_week), ("after campaign", after_campaign)]:
    stat, p = ks_2samp(train_income, sample)
    status = "DRIFT - investigate or retrain" if p < 0.01 else "ok"
    print(f"{name:15s} KS statistic={stat:.3f}  p-value={p:.4f}  -> {status}")

Expected output (click to compare with your run)

this week       KS statistic=0.038  p-value=0.6023  -> ok
after campaign  KS statistic=0.339  p-value=0.0000  -> DRIFT - investigate or retrain

Detecting data drift

Why used: Models fail silently in production — nothing crashes, predictions are just gradually worse — so you need automatic alarms.
How it operates: The Kolmogorov–Smirnov test (ks_2samp) measures whether two samples plausibly came from the same distribution. We compare training incomes with a "normal" week and a week after a marketing campaign that attracted wealthier customers. A very small p-value triggers the warning.
Effect on output: The normal week passes; the post-campaign week is flagged as drift.
Role in program: It is a minimal version of the monitoring that production ML systems run continuously.

✅ Before You Ship a Model — Checklist

✔Is a model needed at all, or would simple rules work as well?
✔Does it beat a simple baseline, on a test set that was untouched until the end?
✔Have I checked for leakage and for duplicated data between train and test?
✔Does the metric match the real cost of mistakes?
✔Have I evaluated separately for the groups of people affected?
✔Can I explain its behaviour well enough for the people it affects?
✔Is personal data handled lawfully, minimised and protected?
✔Is there human oversight for high-stakes decisions, and a way to appeal?
✔Is monitoring in place, and a plan to retrain and roll back?

Module 16

🏗️ Capstone Projects You Can Run Right Here

Four complete, end-to-end projects. Each one runs in your browser, and each ends with ideas to make it your own portfolio piece.

🏠

House Price Predictor

Regression: compare a linear baseline against forests and boosting, and read the feature importances.

regressionscikit-learnpandas

Open project →

📧

Spam Message Classifier

Text classification with TF-IDF and Naive Bayes, evaluated with precision and recall.

classificationNLPNaive Bayes

Open project →

🛒

Customer Segmentation

K-Means on recency, frequency and spend (RFM), then turn clusters into business language.

clusteringK-Meansscaling

Open project →

🧠

Digit Recogniser

A neural network reads handwritten digits; then you analyse exactly which digits it confuses.

neural networkserror analysisimages

Open project →

About the data in these projects

So that every project runs offline in your browser, the house-price, spam and customer datasets here are small synthetic examples generated or typed in the code — not real records. That makes them ideal for learning the workflow, but not for drawing real-world conclusions. Each project says how to move to real data (for example, the California Housing dataset, or a public spam corpus).

Project 1 · Regression

🏠 House Price Predictor

Goal: predict a house's price from its area, bedrooms, age and distance to the city centre. Success metric: mean absolute error (MAE, in currency units) and R². Baseline: plain linear regression.

import numpy as np
import pandas as pd
from sklearn.ensemble import GradientBoostingRegressor, RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(10)
n = 1000
df = pd.DataFrame({
    "area_m2": rng.uniform(25, 250, n),
    "bedrooms": rng.integers(1, 6, n),
    "age_years": rng.integers(0, 50, n),
    "distance_km": rng.uniform(0.5, 25, n),
})
# A non-linear pricing rule plus noise (a stand-in for a real housing dataset)
df["price"] = (1200 * df["area_m2"] + 8000 * df["bedrooms"] - 900 * df["age_years"]
               - 6000 * np.sqrt(df["distance_km"]) * (df["area_m2"] / 100)
               + rng.normal(0, 12000, n))

X, y = df.drop(columns="price"), df["price"]
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0)

models = {
    "Linear Regression": LinearRegression(),
    "Random Forest": RandomForestRegressor(n_estimators=80, random_state=0),
    "Gradient Boosting": GradientBoostingRegressor(random_state=0),
}
print(f"{'Model':18s} {'MAE':>9s} {'RMSE':>9s} {'R2':>6s}")
for name, m in models.items():
    pred = m.fit(X_tr, y_tr).predict(X_te)
    rmse = float(np.sqrt(np.mean((y_te - pred) ** 2)))
    print(f"{name:18s} {mean_absolute_error(y_te, pred):9.0f} {rmse:9.0f} {r2_score(y_te, pred):6.3f}")

rf = models["Random Forest"]
print("\nRandom Forest feature importances:")
for name, imp in sorted(zip(X.columns, rf.feature_importances_), key=lambda t: -t[1]):
    print(f"  {name:12s} {imp:.3f}")

Expected output (click to compare with your run)

Model                    MAE      RMSE     R2
Linear Regression      10273     13142  0.961
Random Forest          11276     14047  0.955
Gradient Boosting      10807     13520  0.959

Random Forest feature importances:
  area_m2      0.906
  age_years    0.042
  distance_km  0.033
  bedrooms     0.018

What this project teaches

Why used: It walks the full regression workflow: build features, split, compare models fairly, interpret.
How it operates: The price comes from a known formula with some curvature (a distance effect that depends on area) plus random noise. Three models train on 80% of the data and are scored on the other 20%. feature_importances_ shows how much each feature helped the forest's splits.
Effect on output: All three models score within a whisker of each other, and the plain linear baseline is competitive — because our synthetic rule is close to linear. Area dominates the importances.
Role in program: It demonstrates an honest lesson: fancy models only win when the data has structure that simple models miss. Always check the baseline.

✨ Make it yours: switch to real data with the snippet below (run locally — it downloads the dataset), add feature scaling and cross-validation, and try GridSearchCV from Module 5 on the best model.

# Run locally: needs internet the first time (downloads the dataset)
from sklearn.datasets import fetch_california_housing

housing = fetch_california_housing(as_frame=True)
X, y = housing.data, housing.target        # target = median house value in units of $100,000
print(X.shape)                             # (20640, 8): 20,640 districts, 8 features
print(list(X.columns))
# Now reuse the same train/test split and model-comparison code from above.

Project 2 · Text classification

📧 Spam Message Classifier

Goal: label short messages as spam or ham. Success metric: precision and recall on spam (a deleted real message is costly, so precision matters). The pipeline is TF-IDF features into Multinomial Naive Bayes.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline

spam = [
    "win a free prize now", "claim your free cash reward today", "urgent your account needs verification",
    "limited offer buy now and save", "congratulations you won a free gift card", "free entry win cash now",
    "cheap loans approved instantly", "act now exclusive deal just for you", "you have been selected for a free holiday",
    "earn money fast from home", "final notice claim your prize", "get rich quick with this offer",
]
ham = [
    "are we still meeting for lunch tomorrow", "please review the attached project report", "can you send me the notes from class",
    "dinner at my place on saturday", "the meeting is moved to three pm", "thanks for helping me with the assignment",
    "see you at the library later", "happy birthday hope you have a great day", "i will call you after work",
    "reminder to submit the form by friday", "did you finish reading the chapter", "let us plan the trip next week",
]
texts = spam + ham
labels = [1] * len(spam) + [0] * len(ham)

X_tr, X_te, y_tr, y_te = train_test_split(texts, labels, test_size=0.3, random_state=3, stratify=labels)
model = make_pipeline(TfidfVectorizer(), MultinomialNB())
model.fit(X_tr, y_tr)

print(classification_report(y_te, model.predict(X_te), target_names=["ham", "spam"], zero_division=0))
for msg in ["claim your free prize now", "can we move the meeting to friday", "exclusive cash offer just for you"]:
    p = model.predict_proba([msg])[0, 1]
    print(f"{msg!r:40} P(spam) = {p:.2f}")

Expected output (click to compare with your run)

              precision    recall  f1-score   support

         ham       0.80      1.00      0.89         4
        spam       1.00      0.75      0.86         4

    accuracy                           0.88         8
   macro avg       0.90      0.88      0.87         8
weighted avg       0.90      0.88      0.87         8

'claim your free prize now'              P(spam) = 0.78
'can we move the meeting to friday'      P(spam) = 0.28
'exclusive cash offer just for you'      P(spam) = 0.60

What this project teaches

Why used: It is the classic first NLP project, and it connects directly to the Naive Bayes you built from scratch in Module 6.
How it operates: 24 hand-written messages (12 spam, 12 ham) are split 70/30 with stratify, so both classes appear in train and test. The Pipeline turns text into TF-IDF vectors and trains Naive Bayes. classification_report prints precision, recall and F1 per class.
Effect on output: The report scores only 8 test messages, so each mistake moves the numbers a lot — one missed spam changes recall by 25 points. Three new messages then get a spam probability.
Role in program: It shows the end-to-end text workflow, and why tiny test sets give shaky conclusions.

⚠️ With 24 messages, treat the scores as a demonstration of the method, not evidence the filter works. For a real filter, train on thousands of messages from a public corpus (such as the SMS Spam Collection from the UCI repository), use cross-validation, and re-check precision regularly because spammers adapt.

Project 3 · Unsupervised learning

🛒 Customer Segmentation

Goal: discover groups of customers from behaviour alone, using the classic RFM features: Recency (days since last order), Frequency (number of orders) and Monetary value (total spend).

import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(4)

def group(n, recency, frequency, monetary):
    return pd.DataFrame({
        "recency_days": rng.normal(recency, 8, n).clip(1),
        "orders": rng.poisson(frequency, n),
        "spend": rng.normal(monetary, monetary * 0.2, n).clip(5),
    })

customers = pd.concat([group(150, 10, 12, 900),     # frequent, big spenders
                       group(200, 40, 4, 250),      # occasional buyers
                       group(120, 120, 1, 60)],     # lapsed customers
                      ignore_index=True)

X = StandardScaler().fit_transform(customers)       # scale so no feature dominates
km = KMeans(n_clusters=3, n_init=10, random_state=0).fit(X)
customers["segment"] = km.labels_

print("silhouette score:", round(silhouette_score(X, km.labels_), 3))
profile = customers.groupby("segment").agg(
    customers=("spend", "size"),
    recency_days=("recency_days", "mean"),
    orders=("orders", "mean"),
    spend=("spend", "mean"),
).round(1)
print(profile)

Expected output (click to compare with your run)

silhouette score: 0.695
         customers  recency_days  orders  spend
segment                                        
0              202          39.3     3.9  246.2
1              148          10.9    12.6  899.2
2              120         119.9     1.1   59.8

What this project teaches

Why used: Segmentation is one of the most common business uses of clustering, and it shows why scaling is mandatory for distance-based methods.
How it operates: The code simulates three customer types, then removes the type label. StandardScaler puts days, orders and currency on one scale (otherwise spend, in the hundreds, would swamp everything). K-Means with k = 3 finds the groups, and a groupby profile turns each cluster into averages you can read.
Effect on output: Three clear segments emerge with sizes close to the true 200 / 150 / 120: frequent big spenders, occasional buyers and lapsed customers. Cluster numbers are arbitrary labels; the meaning comes from reading the profile table.
Role in program: It shows the last, human step of unsupervised learning: naming and acting on the groups.

✨ Make it yours: use the elbow and silhouette scan from Module 10 to choose k instead of assuming 3, and suggest one different marketing action for each segment. In real data, clusters are rarely this cleanly separated.

Project 4 · Neural network

🧠 Digit Recogniser With Error Analysis

Goal: classify 8×8 handwritten digit images, then go beyond a single accuracy number and find out which digits the network confuses. For the full 28×28 MNIST version in Keras, see Module 12.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

digits = load_digits()
X_tr, X_te, y_tr, y_te = train_test_split(
    digits.data, digits.target, test_size=0.25, random_state=0, stratify=digits.target)

net = make_pipeline(StandardScaler(),
                    MLPClassifier(hidden_layer_sizes=(64,), max_iter=500, random_state=0))
net.fit(X_tr, y_tr)
pred = net.predict(X_te)
print("Test accuracy:", round(float(np.mean(pred == y_te)), 3))

# Error analysis: which pairs of digits does the network mix up most?
cm = confusion_matrix(y_te, pred)
np.fill_diagonal(cm, 0)
worst = np.dstack(np.unravel_index(np.argsort(cm.ravel())[::-1][:3], cm.shape))[0]
for true_digit, pred_digit in worst:
    print(f"true {true_digit} predicted as {pred_digit}: {cm[true_digit, pred_digit]} times")

# Draw one test image as text (pixel values range from 0 to 16)
def ascii_digit(pixels):
    for row in pixels.reshape(8, 8):
        print("".join("#" if v > 8 else "." for v in row))

print("\nA test image the network read as", pred[0], "(true label:", y_te[0], ")")
ascii_digit(X_te[0])

Expected output (click to compare with your run)

Test accuracy: 0.973
true 8 predicted as 1: 2 times
true 9 predicted as 3: 2 times
true 9 predicted as 5: 1 times

A test image the network read as 2 (true label: 2 )
...##...
..####..
.....#..
.....#..
.....#..
....##..
...####.
...####.

What this project teaches

Why used: Error analysis is how practitioners improve models: look at the mistakes, not just the score.
How it operates: A one-hidden-layer network trains on scaled pixels. The confusion matrix has its diagonal zeroed so that only mistakes remain; sorting the remaining counts reveals the most common confusions. The final helper prints an image as text, # for dark pixels.
Effect on output: Accuracy is high (around 97%), and the handful of errors are concentrated in a few digit pairs. Seeing the actual pixels of a test image makes the input concrete.
Role in program: It closes the course with the habit that matters most: inspect your model's failures.

✨ Make it yours: add noise to the test pixels and watch accuracy fall; try hidden_layer_sizes=(128, 64); then rebuild it in Keras or PyTorch on full MNIST and compare.

📁 Turning a Project Into a Portfolio Piece

Start with a clear question and say who would use the answer.
Use real, properly licensed data (Kaggle, UCI, Hugging Face datasets, government open data) and cite it.
Show a baseline and your reasoning for each choice, not just the final number.
Include error analysis, limitations and ethics — what could go wrong and for whom.
Make it reproducible: fixed random seeds, a requirements.txt, and a README anyone can follow.
Publish it on GitHub with a short write-up of what you learned.

🧭 A 12-Week Study Roadmap

A realistic plan for about 5–7 hours a week. Adjust the pace freely — understanding beats speed.

Weeks	Focus	Modules	Checkpoint — you can…
1	Big picture and Python tools	1, 4, 8	…explain AI vs ML vs deep learning, and use NumPy and pandas
2–3	Math you need	2	…compute a dot product and a derivative, and explain gradient descent
4	Data and the workflow	3, 5	…explore a dataset, split it without leakage, and run a full pipeline
5–6	Build it yourself	6, 7	…write gradient descent and k-means from scratch, and spot overfitting
7–8	Supervised & unsupervised with scikit-learn	9, 10	…compare models with cross-validation and pick k for clustering
9	Evaluation done properly	11	…choose a metric to fit the problem and read a confusion matrix
10–11	Deep learning, language and RL	12, 13, 14	…train a small network in a framework and explain attention
12	Responsible AI and a capstone	15, 16	…audit a model per group, and publish one project on GitHub

📚 Where to Go Next — Respected Free and Paid Resources

Resource	Best for
scikit-learn user guide and tutorials (scikit-learn.org)	The reference for classical ML; excellent explanations and examples
Machine Learning Specialization — Andrew Ng (Coursera / DeepLearning.AI)	A gentle, well-structured first course
Practical Deep Learning for Coders — fast.ai (free)	Learning deep learning by building, top-down
Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow — Aurélien Géron	The most popular practical book
Mathematics for Machine Learning — Deisenroth, Faisal & Ong (free PDF)	The maths, rigorously but approachably
Dive into Deep Learning — d2l.ai (free)	Interactive deep-learning textbook with code in PyTorch, JAX and more
Deep Learning — Goodfellow, Bengio & Courville (free online)	Theory and foundations
Reinforcement Learning: An Introduction — Sutton & Barto (free online)	The standard RL text
Hugging Face courses (huggingface.co/learn)	Transformers, LLMs and modern NLP
Kaggle Learn and competitions	Short practical lessons and real datasets to practise on

⚠️ Check each resource's current availability and pricing — course platforms change their offerings.

📖 Glossary of Core Terms

Plain-language definitions of the vocabulary used throughout this course

Activation function: A non-linear function applied to a neuron's weighted sum (ReLU, sigmoid, tanh, softmax). Without it, a network could only learn linear relationships.
Algorithm vs model: The algorithm is the learning procedure (e.g. gradient descent on a tree-building rule); the model is the trained result — the parameters that make predictions.
Attention: A mechanism that lets each element of a sequence weigh how relevant every other element is, forming the core of the Transformer.
Backpropagation: Applying the chain rule backwards through a network to compute the gradient of the loss with respect to every weight.
Bias (statistical): Systematic error from a model that is too simple to capture the real pattern. Not the same as the bias term (intercept) in a neuron, nor social bias in data.
Classification: Predicting a category (spam / not spam, digit 0–9).
Clustering: Grouping similar samples without labels.
Confusion matrix: A table counting predictions for every (true class, predicted class) pair.
Cross-validation: Evaluating a model by training and testing on several different splits of the data and averaging the results.
Data leakage: Information from outside the training data (often the test set or the future) influencing training, making results look better than they are.
Deep learning: Machine learning with neural networks that have many layers.
Drift: A change over time in the input data (data drift) or in the relationship between inputs and outcomes (concept drift) that degrades a deployed model.
Embedding: A learned vector representation in which similar items sit close together.
Epoch: One complete pass through the training data.
Feature: An input variable (column) used to make a prediction.
Feature engineering: Creating or transforming features so a model can learn more easily.
Gradient descent: An optimisation method that repeatedly moves parameters a small step against the gradient of the loss.
Hyperparameter: A setting chosen before training (learning rate, tree depth, number of clusters), as opposed to parameters learned from data.
Label / target: The value a supervised model is trying to predict.
Learning rate: The size of each gradient-descent step.
Loss function: A number measuring how wrong the model's predictions are; training tries to minimise it.
Overfitting: Learning the training data, noise included, so well that performance on new data suffers.
Parameter: A value the model learns from data, such as a weight.
Pipeline: A chain of preprocessing and modelling steps packaged together so they are applied consistently and without leakage.
Precision: Of everything the model flagged as positive, the share that truly was.
Recall: Of all the truly positive cases, the share the model caught.
Regression: Predicting a continuous number (a price, a temperature).
Regularisation: Penalising model complexity (for example large weights) to reduce overfitting.
Reinforcement learning: Learning to act by trial and error to maximise cumulative reward.
Supervised learning: Learning from examples that include the correct answers.
Token: A piece of text (a word or part of a word) that a language model reads and predicts.
Transformer: A neural-network architecture built on attention, behind most modern language models.
Underfitting: A model too simple to capture the pattern, so it performs poorly even on its training data.
Unsupervised learning: Finding structure in data that has no labels.
Variance (model): Sensitivity to the particular training sample; a high-variance model changes a lot if trained on different data.

🎯 Self-Check Quiz

Ten questions across the course. Pick an answer to see the explanation straight away — nothing is stored or sent anywhere.

❓ ML FAQ — Common Questions

Honest answers for learners

Do I need a maths degree to learn ML?

No. You need basic algebra and statistics — the same level you'd get from a high school math education. The Python libraries (scikit-learn, TensorFlow) handle the heavy mathematics for you. As you advance, you'll naturally deepen your understanding.

Which ML library should I learn first?

scikit-learn — it's the most accessible, has a consistent API, and covers 90% of classical ML tasks. Once you're comfortable with scikit-learn, learn TensorFlow or PyTorch for deep learning.

What's the difference between AI, ML, and deep learning?

AI is the broadest — any system that exhibits intelligent behaviour. ML is a subset of AI where systems learn from data. Deep learning is a subset of ML using neural networks with many layers — particularly powerful for image, audio, and text data.

How much data do I need to train a model?

It depends on the problem. Simple models (logistic regression) can work with hundreds of samples. Deep learning often requires tens of thousands to millions of samples. Start with small datasets (iris, housing, MNIST) to learn the workflow, then scale up.

What is overfitting and how do I prevent it?

Overfitting happens when your model learns the training data too well — including noise — and fails to generalise to new data. Prevent it with: cross-validation, regularisation (L1/L2), early stopping, simpler models, and more training data.

Can I do ML on my laptop?

Yes — for most tasks. scikit-learn models run on any laptop. Deep learning with small datasets (MNIST, CIFAR-10) works on CPU. For larger models, use Google Colab (free GPU) or cloud services.

What is the difference between a parameter and a hyperparameter?

Parameters are learned from the data during training — a neuron's weights, or the thresholds in a decision tree. Hyperparameters are settings you choose beforehand — the learning rate, the depth of a tree, the k in k-means — and you tune them using validation data, never the test set.

My model is 99% accurate. Why do people say it may be useless?

If 99% of cases are negative, a model that always answers "negative" scores 99% while finding nothing. On imbalanced problems check precision, recall, F1 and ROC-AUC, and always compare against a simple baseline. Module 11 shows this with code.

Is deep learning always better than traditional machine learning?

No. For tables of mixed numeric and categorical data, random forests and gradient boosting are often as accurate or better, and faster and easier to explain. Deep learning shines on images, audio, text and very large datasets.

How do ChatGPT-style language models work?

They are Transformers trained on huge amounts of text to predict the next token, then further trained to follow instructions and reflect human feedback. They generate one token at a time. They can sound confident while being wrong, so verify important facts. Module 13 builds up the idea step by step.

How long does it take to learn machine learning?

With 5–7 hours a week, the 12-week roadmap in this guide takes you through the fundamentals and a first project. Becoming proficient takes longer — months to years of building projects with real, messy data.

Will AI replace programmers and data scientists?

Nobody can predict this precisely. AI tools are already changing how people work, mostly by automating parts of jobs rather than whole occupations at once. Understanding the fundamentals — how models learn, fail and should be checked — makes you better at using these tools and at spotting their mistakes.

🗣️ Learner Voices

No comments yet — start the ML conversation!

Tags: artificial intelligencemachine learningpython mldeep learningneural networksscikit-learntensorflowpytorchsupervised learningunsupervised learning

Comments

Share this essay

Facebook X LinkedIn WhatsApp Telegram Reddit Email

About | Privacy Policy | Terms of Service | Cookie Policy
© 2026 Glenn Junsay Pansensoy. All rights reserved.

Published on poetic-bytes.blogspot.com  ·  #artificialintelligence #machinelearning #pythonml #deeplearning #neuralnetworks

Machine Learning Fundamentals

AI & Machine Learningwith Python — From First Principles to Real Models

Why this code?

🗺️ Course Map — 16 Modules, One Path

🎓 What you will be able to do

How to use this page

🌐 AI & ML Foundations

🤖 What is Artificial Intelligence?

🪆 AI ⊃ Machine Learning ⊃ Deep Learning

🧰 Five Ways to Build an Intelligent System

🧭 An AI That Plans: Breadth-First Search

How the search works

📜 A Short History of AI

🤖 So — What is Machine Learning?

📊 Why ML Matters in 2026

🧠 ML vs. Traditional Programming

🧮 Math for Machine Learning — Just Enough

📐 Linear Algebra — Data as Vectors and Matrices

Vectors in NumPy

Matrix multiplication and shapes

📉 Calculus — How Models Learn

Derivatives and gradient descent

🎲 Probability & Statistics — Reasoning Under Uncertainty

Describing data and the law of large numbers

🔗 Where the Maths Shows Up Later

🗃️ Working With Data

🧱 The Vocabulary of Tabular Data

🔍 Exploratory Data Analysis (EDA)

A first look at a dataset with pandas

🧹 Cleaning and Preparing Features

Encoding and scaling

✂️ Splitting Data — and the Leakage Trap

Leakage in action

🏷️ Types of Machine Learning

Supervised Learning

Unsupervised Learning

Reinforcement Learning

Which one should you learn first?

⚙️ The ML Workflow — 7 Steps to a Working Model

1. Define the Problem

2. Collect and Prepare Data

Data preparation code – why and how

3. Choose a Model

4. Train the Model

Training a Random Forest

5. Evaluate

Model evaluation

6. Tune Hyperparameters

Hyperparameter tuning with GridSearchCV

7. Deploy

Model persistence

🧪 Build AI & ML From Scratch — No Libraries Doing the Thinking

💬 A Rule-Based AI Chatbot

Line-by-line walkthrough

🧠 A Single Neuron Learning the AND Gate

Line-by-line walkthrough

📐 Linear Regression With Gradient Descent (No scikit-learn)

Line-by-line walkthrough

🧭 K-Nearest Neighbors From Scratch

Line-by-line walkthrough

🎯 Logistic Regression With Gradient Descent

Line-by-line walkthrough

🌳 A Decision Tree From Scratch

Line-by-line walkthrough

🎯 K-Means Clustering From Scratch

Line-by-line walkthrough

✉️ A Naive Bayes Spam Filter From Scratch

Line-by-line walkthrough

🧠 A Two-Layer Neural Network With Backpropagation

Line-by-line walkthrough

From "built by hand" to "built with libraries"

⚖️ Generalization — Overfitting, Underfitting & the Bias–Variance Trade-off

🎓 The Student Analogy

📈 Watching a Model Overfit

Train error vs test error

🪢 Regularisation — Penalising Complexity

Ordinary vs Ridge vs Lasso

🛡️ Your Anti-Overfitting Toolkit

📦 Python ML Libraries — Your Toolbox

scikit-learn

AI & Machine Learning
with Python — From First Principles to Real Models