Jethro Odeyemi
AI Research at VIDO, UofS
You May Also Enjoy
Can Attention Work Without the sqrt(d) Scaling?
12 minute read
Published:
The division by $\sqrt{d_k}$ inside the attention softmax might be the most-copied magic constant in deep learning. It appears in essentially every Transformer implementation ever written, and yet the original paper justifies it in a single footnote, prefaced in the main text by a hedge: “We suspect that for large values of $d_k$, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients” [1]. A suspicion, not a theorem. So I wanted to work through it properly: why is the $\sqrt{d_k}$ there, what actually breaks if you delete it, and — since several production models have effectively deleted it — what does it take for attention to work without it? I ran a few quick experiments along the way, and every number below is measured, not hypothetical. Read more
The Unreasonable Effectiveness of Scale
5 minute read
Published:
Scaling laws describe the relationship between a model’s performance and the scale of three key ingredients: the number of model parameters, the size of the dataset, and the amount of computational power used for training. The core finding is that as you increase these resources, the model’s performance improves in a predictable, power-law fashion. Read more
A Technical Deep Dive into Exploding Gradients
5 minute read
Published:
I remember one of the experiences I had duing my MS in Computer Science at Georgia Tech while working on a CNN for protein data. I was feeding raw protein data as an image, with pixel values in the standard 0-255 range, directly into the network. My model’s accuracy was stuck below 20%, and the loss was oscillating wildly. After hours of debugging, I traced the issue to its source: I had neglected to normalize my input data, leading to a classic case of “exploding gradients.” Read more
Why Randomized Optimization Needs Quantum Computing
5 minute read
Published:
Randomized optimization algorithms like Genetic Algorithms (GA), Simulated Annealing (SA), and Randomized Hill Climbing (RHC) are powerful tools for solving problems where traditional gradient-based methods fail. These “black-box” problems are common in fields like logistics, engineering design, and machine learning, where the optimization landscape is complex, non-differentiable, or riddled with local minima. Read more