Jethro Odeyemi

Jethro Odeyemi

AI Research at VIDO, UofS

Blog posts

2026

Can Attention Work Without the sqrt(d) Scaling?

12 minute read

Published:

The division by $\sqrt{d_k}$ inside the attention softmax might be the most-copied magic constant in deep learning. It appears in essentially every Transformer implementation ever written, and yet the original paper justifies it in a single footnote, prefaced in the main text by a hedge: “We suspect that for large values of $d_k$, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients” [1]. A suspicion, not a theorem. So I wanted to work through it properly: why is the $\sqrt{d_k}$ there, what actually breaks if you delete it, and — since several production models have effectively deleted it — what does it take for attention to work without it? I ran a few quick experiments along the way, and every number below is measured, not hypothetical. Read more

2025

The Unreasonable Effectiveness of Scale

4 minute read

Published:

Scaling laws describe the relationship between a model’s performance and the scale of three key ingredients: the number of model parameters, the size of the dataset, and the amount of computational power used for training. The core finding is that as you increase these resources, the model’s performance improves in a predictable, power-law fashion. Read more

A Technical Deep Dive into Exploding Gradients

4 minute read

Published:

I remember one of the experiences I had duing my MS in Computer Science at Georgia Tech while working on a CNN for protein data. I was feeding raw protein data as an image, with pixel values in the standard 0-255 range, directly into the network. My model’s accuracy was stuck below 20%, and the loss was oscillating wildly. After hours of debugging, I traced the issue to its source: I had neglected to normalize my input data, leading to a classic case of “exploding gradients.” Read more

Why Randomized Optimization Needs Quantum Computing

5 minute read

Published:

Randomized optimization algorithms like Genetic Algorithms (GA), Simulated Annealing (SA), and Randomized Hill Climbing (RHC) are powerful tools for solving problems where traditional gradient-based methods fail. These “black-box” problems are common in fields like logistics, engineering design, and machine learning, where the optimization landscape is complex, non-differentiable, or riddled with local minima. Read more

Why Backprop Isn’t Magic: The Challenge of Local Minima

7 minute read

Published:

Backpropagation is the cornerstone algorithm powering much of the deep learning revolution. Coupled with gradient descent, it allows us to train incredibly complex neural networks on vast datasets. However, it’s not a silver bullet. One of the fundamental challenges that can prevent backpropagation from finding the best possible solution is the presence of local minima in the optimization landscape. Read more

DeepSeek R1: Pioneering Reasoning in Large Language Models Through Reinforcement Learning

3 minute read

Published:

The development of reasoning capabilities in large language models (LLMs) is a complex yet pivotal frontier in AI research. DeepSeek R1 represents a major leap in this space, introducing innovative methodologies for reasoning-oriented model training. In this post, we’ll explore what makes DeepSeek R1 significant, its architectural innovations, and its implications for the future of AI. Read more

Unveiling a $500 Billion Leap in AI: Trump’s Private Sector Investment Plan

3 minute read

Published:

President Donald Trump is set to announce a monumental private sector initiative aimed at bolstering the United States’ artificial intelligence (AI) infrastructure with an investment of up to $500 billion. This ambitious plan involves leading tech companies like OpenAI, SoftBank, and Oracle, under a collaborative venture named “Stargate.” Read more

Supervised Learning Showdown: kNN, SVM, Neural Networks, and Boosted Trees

10 minute read

Published:

In this post, we dive into the world of supervised learning, comparing the performance of four popular algorithms: k-Nearest Neighbors (kNN), Support Vector Machines (SVM), Neural Networks (NN), and Decision Trees with Boosting (specifically, AdaBoost). We’ll analyze their effectiveness on two distinct datasets, highlighting their strengths and weaknesses. Read more

Algorithm — Generate Parentheses: Python and C++ Solutions

4 minute read

Published:

In this blog post, we’ll explore LeetCode’s “Generate Parentheses” problem and present two clean backtracking solutions—one in Python and one in C++. This classic problem is an excellent demonstration of how to use recursion to systematically explore all valid possibilities. Read more

2024

Algorithm — Reverse Only Letters: A Python Solution

2 minute read

Published:

In many programming interviews, candidates encounter challenges that test their ability to manipulate strings efficiently. One such problem involves reversing a string with a twist: only the letters should be reversed, while non-letter characters remain in their original positions. In this blog post, we’ll explore this problem and present an optimized Python solution. Read more