Generative AI is all the rage today. It has a history, though — one where ML tasks were more narrowly defined using deep neural networks. I've been following this space since 2017 and sharing my learning with friends and colleagues. Here is a smattering of notebooks from the past.
Deep Learning From Above
I chose the title after listening to Feynman's very funny talk
Los Alamos From Below.
My intent was to understand the high-level concepts of deep learning. I start with a
cluster of points in a 2D space. I first produce the best-fit line using linear algebra,
then recreate the same line using PyTorch's nn.module. This illustrates the ML training
loop — forward and backward passes that reduce the training loss and arrive at the weights
characterizing the line as an intercept and a slope. The deep learning method is more
general and can be applied widely to other functions.
Transfer Learning
I was inspired by Jeremy Howard's FastAI offering. In this notebook I explore the FastAI deep learning library. He built a layered architecture that looks like traditional software — high-level, mid-level and lower-level APIs. For a simple use case, we can build a solution using the highest layer. I illustrate transfer learning: we start with a pretrained classifier model and fine-tune it for sentiment analysis using the FastAI learner class.
Convolutional Neural Network
This notebook explores some of the inspirations behind deep neural networks in general and CNNs in particular. We review the similarities between biological neurons and artificial neurons. An artificial neural network is built by interconnecting very many artificial neurons. We describe the Hubel and Wiesel experiment that inspired filters and hierarchical abstractions in CNNs. Later, Zeiler and Fergus deepened our understanding of what goes on behind a CNN. We use PyTorch to identify handwritten digits with the famous MNIST dataset.
Sequence Modeling
This one is close to my heart. During the pandemic, a school colleague started a talk series
where folks from the '97 IIT Kanpur batch shared their learnings with batchmates. Everyone in
the audience had a technical bent, but life had taken them in different directions. I started
with the value proposition: ML can be thought of as a function approximator, mapping any input
x to an output y. I introduce the ML training loop, then use
fairseq to learn an "add" function. It learns
to translate a string of numbers into their sum — given 2+5, it produces 7 without doing
normal addition. It's surprising how accurate it gets. It also shows how it will occasionally
hallucinate, since it isn't building any logical model of addition.
ChatGPT
This was shortly after ChatGPT burst onto the scene and there was a strong desire to understand the magic behind its success. I scoured the web and found Hugging Face to be a great resource. Lewis Tunstall from Hugging Face released Natural Language Processing with Transformers around the same time. I used the Hugging Face transformers library to illustrate various tasks, then covered the history of GPT by leveraging Lewis's work — the notion of Attention, the Transformer architecture, and post-training via RLHF, a key ingredient of the quality at that time. Finally I showcased some samples that felt like magic.