OpenAI has introduced Sora, an AI model that can generate videos from text instructions. Sora uses a diffusion model, transforming static noise into a video over several steps. It can develop entire videos or extend existing ones, maintaining the subject’s consistency even when temporarily out of view. The model uses a transformer architecture, similar to GPT models, and builds on past research in DALL·E and GPT models. It can also animate still images and extend or fill missing frames in existing videos. OpenAI tests Sora with red teamers and visual artists to assess potential risks and gather feedback.
Introduction to Sora: The Text-to-Video AI Model
OpenAI introduces Sora, an AI model that can generate videos from text instructions. Sora can create videos up to a minute long while maintaining visual quality and adherence to the user’s prompt. The model is designed to understand and simulate the physical world in motion, aiming to aid in problem-solving that requires real-world interaction.
Sora’s Research Techniques and Capabilities
Sora is a diffusion model that generates a video by starting with static noise and gradually transforming it by removing it over many steps. The model can generate entire videos at once or extend generated videos to make them longer. This capability addresses the challenging problem of ensuring a subject remains consistent even when it goes out of view temporarily.
Sora uses a transformer architecture similar to GPT models, allowing superior scaling performance. Videos and images are represented as collections of smaller data units called patches, akin to tokens in GPT. This unified data representation allows for training diffusion transformers in a broader range of visual data, spanning different durations, resolutions, and aspect ratios.
Building on past research in DALL·E and GPT models, Sora uses the recaptioning technique from DALL·E 3, which involves generating highly descriptive captions for the visual training data. This allows the model to faithfully follow the user’s text instructions in the rendered video. In addition to generating a video solely from text instructions, Sora can take an existing still image and generate a video from it, animating its contents with accuracy and attention to detail. The model can also extend an existing video or fill in missing frames.
Sora’s Strengths and Weaknesses
Sora can generate complex scenes with multiple characters, specific types of motion, and accurate details of the subject and background. The model understands what the user has asked for in the prompt and how those things exist in the physical world.
However, Sora has some limitations. It may struggle with accurately simulating the physics of a complex scene and may not understand specific instances of cause and effect. For example, a person might take a bite out of a cookie, but afterward, the cookie may not have a bite mark. The model may also confuse spatial details of a prompt, such as mixing up left and right, and may struggle with precise descriptions of events that take place over time, like following a specific camera trajectory.
Safety Measures for Sora
OpenAI is taking several safety steps before making Sora available in its products. The organization is working with red teamers and domain experts in misinformation, hateful content, and bias, who will be adversarially testing the model. OpenAI is also developing tools to detect misleading content, such as a detection classifier that can identify when Sora generates a video. The organization plans to include C2PA metadata if the model is deployed in an OpenAI product.
OpenAI is also leveraging existing safety methods built for its products that use DALL·E 3, which apply to Sora. For example, a text classifier will check and reject text input prompts that violate usage policies. Robust image classifiers review the frames of every video generated to ensure adherence to usage policies before it’s shown to the user.
OpenAI is engaging with policymakers, educators, and artists worldwide to understand their concerns and identify positive use cases for this new technology. The organization acknowledges that it cannot predict all the beneficial ways people will use the technology or abuse it. Therefore, learning from real-world use is critical to creating and releasing increasingly safe AI systems over time.
