ML from Scratch — Episode 3: Logistic Regression

What is Logistic Regression?

Logistic regression is a ML model that predicts an output in the range of 0 to 1 — interpreted as the probability of belonging to a class.

Hypothesis Function

We use the sigmoid function to keep predictions between 0 and 1:

$$\hat{y} = \sigma(z) = \frac{1}{1+e^{-z}}$$

where z = wx + b.

Why Binary Cross Entropy over MSE?

BCE gives a convex loss surface, whereas MSE with sigmoid gives a non-convex one — making it difficult for gradient descent to reach the global minimum.

$$L = -\frac{1}{m} \sum \left[ y \log(\hat{y}) + (1-y) \log(1-\hat{y}) \right]$$

Gradients

Applying the chain rule to the loss function in three steps:

$$\frac{\partial L}{\partial w} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} \cdot \frac{\partial z}{\partial w}$$

The sigmoid and cross entropy cancel cleanly, giving:

$$\frac{\partial L}{\partial w} = \frac{1}{m}\sum(\hat{y} - y) \cdot x$$

$$\frac{\partial L}{\partial b} = \frac{1}{m}\sum(\hat{y} - y)$$

Weight Update

$$w = w - \alpha \cdot \frac{\partial L}{\partial w}$$

$$b = b - \alpha \cdot \frac{\partial L}{\partial b}$$