Andrej Karpathy Reproduces GPT-2 in Latest Tutorial
In a marathon video on his YouTube channel, Andrej Karpathy reproduced GPT-2 in just over four hours. The OpenAI co-founder, who left earlier this year, has spent numerous hours creating tutorial videos for his viewers, including how-to videos on decoding models. In his latest lecture, Karpathy recreated the smallest version of GPT-2, which has 124 million parameters. In the four hour long video, Karpathy broke down the entire process, starting completely from scratch. “First, we build the GPT-2 network, then we optimise its training to be really fast, then we set up the training run following the GPT-2 and GPT-3 paper and their hyperparameters, then we hit run, and come back the next morning to see our results, and enjoy some amusing model generations,” he wrote. https://twitter.com/karpathy/status/1799949853289804266 With the GPT-2 recreation, Karpathy believes the team was very close to GPT-3’s 124M model. “Our “overnight” run even gets very close to the GPT-3 (124M) model. This video builds on the Zero To Hero series and at times references previous videos. You could also see this video as building my nanoGPT repo, which by the end is about 90% similar,” he said. Karpathy has long been a proponent for democratising knowledge on AI, specifically the science behind LLMs. Karpathy, who had been responsible for deep learning and computer vision at OpenAI, left the company for a second time in February this year. However, he has been actively involved in the AI community, creating tutorials and breakdown videos on different models, even more so after his departure. Most recently, he released a project titled llm.c, where users can train LLMs using only C without having to rely on PyTorch and cPython. Even prior, he had released an hour-long lecture explaining the intricacies of LLMs and how they function. Shortly after his departure from OpenAI, he had also released a tutorial on understanding tokenisation, also taking the opportunity to analyse Google’s Gemma tokeniser following its launch. View all stories Top Earthquake Prediction Tools To Watch in 2025 QS World University Rankings 2026: Top 10 Indian Universities Meet Dr. M C Mary Kom at The Rising 2025 – Conference | Bengaluru Top 10 Indian Companies with Highest Women Workforce 2025 DeepSeek Vs ChatGPT – Why World is Falling for this Chinese AI Model
With the GPT-2 recreation, Karpathy believes the team was very close to GPT-3’s 124M model.
In a marathon video on his YouTube channel, Andrej Karpathy reproduced GPT-2 in just over four hours.
["Andrej Karpathy", "OpenAI"]
["Go", "ChatGPT", "Andrej Karpathy", "OpenAI", "AI", "PyTorch", "computer vision", "Python", "GPT", "deep learning", "R"]
["AI", "deep learning", "computer vision", "ChatGPT", "OpenAI", "PyTorch", "Python", "R", "Go", "GPT"]
Donna is a technology journalist at AIM, hoping to explore AI and its implications in local communities, as well as its intersections with the space, defence, education and civil sectors.