Google Introduces Dreamix, A Text-Conditioned Video Editing Model
Google Research recently released Dreamix, an AI model that will be able to perform text-based motion and appearance editing of general videos. While there have been several image-editing tools released based on diffusion models, Dreamix is the first diffusion-based method for video editing. The model essentially functions in three ways: With only a video and a text prompt, the model is capable of editing videos while preserving fidelity to colour, posture, object size, and camera pose, ensuring that the resulting video is temporally consistent. Similarly, for a small collection of images with the same subject given as input, the model can generate new videos with the subject in motion. Additionally, with only an image and a text input, the model will be able to create videos while preserving the visual fidelity to object location and background. To ensure high quality output and fidelity to the input image and prompt, the team will be to enable a text-conditioned video diffusion model (VDM) with two main ideas: first, rather than using pure-noise as initialisation of the model, they used a degraded version of the original video by downscaling it and adding noise, keeping only low spatio-temporal information; and second, they further improved the fidelity to the original video by introducing a mixed fine-tuning approach, wherein the VDMs are also fine-tuned on the collection of individual frames of the input video while discarding their temporal order by masing the temporal attention. Additionally, the researchers also proposed a new framework for image animation for applications that include animating the objects and background in an image, creating dynamic camera motion, and more. This is done by employing simple image processing operations, such as frame replication or geometric image transformation, to generate a coarse video. The video is then passed through the Dreamix video editor for editing. On comparing this method against the state-of-the-art baselines, their model showed superior results on factors like quality, fidelity, and alignment.
Dreamix is the first diffusion-based method for video editing.
Google Research recently released Dreamix, an AI model that will be able to perform text-based motion and appearance editing of general videos.
["Google", "Google Research"]
["Go", "TPU", "programming_languages:R", "AI", "programming_languages:Go", "diffusion models", "Google", "R", "Google Research"]
["AI", "TPU", "R", "Go", "diffusion models", "programming_languages:R", "programming_languages:Go"]
Ayush is interested in knowing how technology shapes and defines our culture, and our understanding of the world. He believes in exploring reality at the intersections of technology and art, science, and politics.