Andrej Karpathy Turns WSJ Front Page Article into a Music Video
In a unique experiment blending generative AI tools, former OpenAI co-founder Andrej Karpathy used the Wall Street Journal’s front page to produce a music video on August 1, 2024. The process involved several AI applications to achieve the final product. August 1, 2024: The Music VideoFun hack just stitching up gen AI tools :), in this case to create a music video for today.– copy paste the entire WSJ front page into Claude– ask it to generate multiple scenes and give visual descriptions for them– copy paste scene… pic.twitter.com/UeRO838Nhs— Andrej Karpathy (@karpathy) August 2, 2024 First, Karpathy copied the entire front page of the WSJ into the AI language model Claude. Claude generated multiple scenes and provided visual descriptions for each. These descriptions were then fed into Ideogram AI, an image generation tool, to create corresponding visuals. Next, the generated images were uploaded into RunwayML’s Gen 3 Alpha to convert each image into a 10-second video segment. Concurrently, Claude was tasked with generating lyrics that captured the essence of the day’s news. These lyrics were inputted into Suno AI to produce the accompanying music. The final step involved stitching the video segments and music together using iMovie. The project showcases the potential of generative AI tools to create multimedia content in a seamless and innovative manner. Andrej Karpathy has been exploring various generative AI tools to create videos. Recently, he combined these tools to produce visual stories. As a demonstration, he took the opening sentences of Pride and Prejudice and turned them into a video. I'm playing around with generative AI tools and stitching them together into visual stories. Here I took the first few sentences of Pride and Prejudice and made it into a video.The gen stack used for this one:– @AnthropicAI Claude took the first chapter, generated the scenes… pic.twitter.com/vX64avfRUY— Andrej Karpathy (@karpathy) July 4, 2024 Andrej Karpathy recently announced the launch of his new venture, Eureka Labs, an AI+Education company dedicated to creating an AI-native learning environment. Eureka Labs aims to improve education by integrating generative AI with traditional teaching methods. Karpathy, who previously held key positions at OpenAI and Tesla, described Eureka Labs as “a new kind of school that is AI native.” Karpathy’s recent experimentation with generative AI tools may be part of his effort to develop new learning tools. These tools could enable students to learn through AI-generated videos and music, enhancing their learning experiences.
The generated images were uploaded into RunwayML’s Gen 3 Alpha to convert each image into a 10-second video segment.
In a unique experiment blending generative AI tools, former OpenAI co-founder Andrej Karpathy used the Wall Street Journal's front page to produce a music video on August 1, 2024.
["AI Video Generation Models", "Andrej Karpathy"]
["Anthropic", "Andrej Karpathy", "OpenAI", "AI", "programming_languages:R", "RPA", "ML", "llm_models:Claude", "AI Video Generation Models", "Aim", "generative AI", "R"]
["AI", "ML", "generative AI", "OpenAI", "Anthropic", "Aim", "R", "RPA", "llm_models:Claude", "programming_languages:R"]
Siddharth is a media graduate who loves to explore tech through journalism and putting forward ideas worth pondering about in the era of artificial intelligence.