I used to nod along when people said gradient descent is just walking downhill. Then one day I sat down with a blank editor and realized I couldn't write the update step from memory. That was my cue: I didn't trust that I understood it. So I did what any self-respecting developer in 2026 would do — I vibe coded a demo to play with it.
Vibe coding means using an AI assistant to generate code from natural language. It has become a popular way to prototype quickly, but it only works well if you test the output. Instead of blindly copying, I treated the AI as a sparring partner: I described what I wanted, examined the code, and ran it with different parameters. The result was a small playground that deepened my understanding of gradient descent more than any textbook.
What Gradient Descent Actually Does
Gradient descent is an optimization algorithm that minimizes a loss function by iteratively moving in the direction of the steepest descent. For a function f(θ), the update rule is:
θ_t+1 = θ_t - η ∇f(θ_t)
where η (eta) is the learning rate, a hyperparameter that controls the step size. The gradient ∇f points uphill, so we subtract it to go downhill. That is the core idea. Source: Wikipedia: Gradient descent
I had read this formula dozens of times. Yet when I tried to implement it for a simple linear regression, I mixed up the sign, forgot to divide by batch size, and misjudged how learning rate affects convergence. Reading is passive. Tuning parameters and watching the optimizer move is active understanding.
Why Vibe Coding a Demo Is a Legit Learning Tool
Vibe coding gets a bad reputation because some people trust AI output blindly. But when you use it as a sparring partner, it is a great educational tool. You describe the behavior you want, the AI writes the code, and then you immediately run it and visually inspect the results. That is the opposite of blind trust — it is the scientific method.
Here is what I asked for:
"Write a Python script that visualizes gradient descent on a convex function, with sliders for learning rate and starting point."
The AI gave me a script using standard scientific libraries. I ran it, saw the curve and the trajectory of the optimizer, and then I started breaking it. I changed the learning rate to 1.5 and watched it explode. I made the function non-convex and watched it get stuck in a local minimum. Those "aha" moments are impossible from a static textbook.
Step-by-Step: Build Your Own Gradient Descent Playground
Let me share the core parts of the script so you can recreate it.
1. Define the function and its gradient
We will start with a simple parabola f(x)=x², whose gradient is 2x.
import numpy as np
import matplotlib.pyplot as plt
def f(x):
return x**2
def grad_f(x):
return 2*x
2. Implement the gradient descent update
def gradient_descent(start_x, learning_rate, steps):
x = start_x
trajectory = [x]
for _ in range(steps):
x = x - learning_rate * grad_f(x)
trajectory.append(x)
return np.array(trajectory)
3. Visualize the journey
trajectory = gradient_descent(start_x=8, learning_rate=0.1, steps=30)
xs = np.linspace(-10, 10, 200)
plt.plot(xs, f(xs), 'b-')
plt.scatter(trajectory, f(trajectory), c='r', zorder=5)
plt.show()
That shows a blue curve with red dots hopping down to the minimum. Satisfying.
But the real magic happens when you make it interactive. Instead of hardcoding parameters, wrap the function in a simple notebook with interactive widgets, so you can slide the learning rate and starting position. The chart updates instantly. I spent an hour just sliding the learning rate around.
What I Learned from Playing with the Demo
1. Learning rate is a delicate balance
If η is too small, convergence is painfully slow. If it is too large, the algorithm overshoots and diverges. For f(x)=x², the optimum learning rate is exactly 1/2, but in general you do not know it. That is why modern optimizers like Adam or RMSprop use adaptive learning rates. Source: An overview of gradient descent optimization algorithms
The table below summarizes what I observed:
| Learning Rate | Behavior | Intuition |
|---|---|---|
| 0.01 | Converges slowly | Step too small |
| 0.1 | Converges quickly | Balanced |
| 0.5 | Oscillates | Optimal for x² |
| 1.5 | Diverges | Step too large and overshoots |
2. Local minima are real, and they hurt
Next I changed the function to f(x)=x⁴-4x²+3, which has two minima and one maximum. With a starting point of 0, the optimizer got stuck in one of the valleys. With -0.5, it fell into the other one. This is a classic problem in non-convex optimization, and it is why practitioners use momentum and random restarts. My demo showed me that "moving downhill" is not enough when you have multiple valleys.
3. The gradient tells you nothing about the global landscape
A key insight: the gradient only gives local information. If you start at the right peak, you will never cross the ridge to the left valley. This is why random restarts are used in practice. I now genuinely understand why data scientists hate local minima.
Practical Tips for Your Own Vibe-Coded Demos
- Always test edge cases: Let the learning rate go above 1.5 and watch the algorithm explode. Start at the exact minimum (x=0) and see it stay there. This builds trust in your own understanding.
- Use interactive widgets: A simple Jupyter notebook with sliders turns abstract math into a tactile experience. You remember things you have played with better than things you have read.
- Don't trust the AI output blindly: The first version of the code I got had a sign error in the update step. If I had not known the formula, I would have accepted it. Always sanity-check the math before running.
- Extend the demo: Try batch gradient descent with a random function, or add momentum. The more you change, the more you learn.
Final Thought
I now trust myself a little more. Vibe coding did not replace the need to understand — it created a playground where understanding could grow. You do not have to be a mathematician to code gradient descent, but you do have to play with it. A demo like this runs in five minutes, but the intuition you gain lasts forever.
So if you have ever said "I know that" about an algorithm and then froze when asked to write it from scratch, build yourself a toy. It is the best way to find out what you actually know.
Comments