Edityourself Achieves Seamless Audio-Driven Video Editing with DiT and Precise Lip Sync

Researchers are tackling the challenge of seamlessly editing pre-recorded talking head videos, a task current generative video models struggle with. John Flynn, Wolfgang Paier, and Dimitar Dinev, from Pipio AI, alongside Sam Nhut Nguyen, Hayk Poghosyan, Manuel Toribio et al, present EditYourself, a new framework leveraging Diffusion Transformers to enable transcript-based modification of existing footage. This innovation allows for the addition, removal, and retiming of spoken content while preserving crucial elements like motion, speaker identity, and accurate lip synchronisation. By enabling precise and coherent restructuring of video performances, EditYourself represents a significant step towards practical generative video tools for professional post-production workflows.

By enabling precise and coherent restructuring of video performances, EditYourself represents a significant step towards practical generative video tools for professional post-production workflows.

Audio-driven video editing with preserved coherence

Scientists have unveiled EditYourself, a novel framework for audio-driven video-to-video (V2V) editing, addressing a critical gap in generative video technology. Current generative models excel at creating new video content from text and images, but struggle with the nuanced task of editing pre-recorded videos where alterations to the spoken script demand preservation of motion, temporal coherence, speaker identity, and accurate lip synchronisation. A key contribution is the development of a two-stage training scheme that allows inference on speech audio across diverse text, image, and video inputs, ensuring accurate lip synchronisation without requiring audio feature downsampling or being limited by video frame rates. Experiments demonstrate that EditYourself introduces a reference-based identity conditioning mechanism, termed Forward, Backward RoPE Conditioning, coupled with TeaCache-aware inference, to stabilise appearance and temporal coherence in long videos.

Evaluations against state-of-the-art Image-to-Video and V2V lip-sync benchmarks reveal that the method achieves superior visual quality and synchronisation accuracy. The research tackles the problem of visual dialog editing, V2V editing driven by changes to the spoken dialogue, going beyond simple lip synchronisation to enable complete audio replacement and support core post-production operations. By utilising a transcript-centric workflow, creators gain an intuitive and expressive interface for precise word-level modifications, such as removing filler words or updating facts post-recording, and opening possibilities for integration with AI agents for automated video editing. Ultimately, this breakthrough paves the way for rapid content updates, personalised video variants, and a more streamlined video production process.

Audio-driven video editing via diffusion training offers promising

The study pioneered a two-stage training scheme to enable inference on speech audio across varying text, image, and video inputs, maintaining accurate lip synchronisation alongside a windowed audio conditioning strategy. This strategy precisely aligns speech and video without downsampling audio features, proving robust across differing video frame rates. To address long video generation challenges, scientists developed a reference-based identity conditioning mechanism, termed Forward, Backward RoPE Conditioning, coupled with TeaCache-aware inference. This innovative combination stabilises appearance and temporal coherence over extended durations, preventing visual inconsistencies.

Evaluations against recent I2V and V2V lip-sync benchmarks demonstrated that the method achieves state-of-the-art visual quality and synchronisation accuracy. This work harnessed latent-space editing, supporting core post-production operations such as inserting, removing, and retiming video segments, while preserving visual continuity. The approach enables a shift from a “script-perfect-before-shooting” paradigm to a “shoot once, refine later” model, facilitating rapid updates and personalised variants, and integration with LLM-based AI agents for automated video editing.

Audio guides precise transcript-based video editing streamlines post-production

The research addresses a critical gap in existing generative video technologies, which typically excel at creating novel content but struggle with editing pre-recorded videos. The baseline network utilises the LTX-0.9.7 DiT and associated Video-VAE, operating with 14 billion parameters in a compressed latent space achieved through a 32×32×8 compression rate. Videos are generated in a two-pass fashion, beginning with denoising on a coarser representation, followed by learned upsampling and a higher-resolution denoising pass. Crucially, the LTX-Video model was pre-trained using a multi-task objective encompassing T2V, I2V, keyframe generation, and spatial and temporal inpainting, achieved by masking tokens and assigning them distinct conditioning timesteps.

This pre-training strategy provides a robust foundation for subsequent audio-driven editing capabilities. Researchers adopted the Flow Matching paradigm, encoding video samples into a latent representation x0 and defining a linear probability path to interpolate between this representation and a noise distribution x1. The DiT model, vθ, was trained to predict the velocity field transforming noise back into data, minimising the base training objective LFM = Et,x1,x0,c[∥vθ(xt, t, c) −(x1 −x0)∥2 2]. To incorporate audio and identity conditioning, this objective was modified, with the expanded training loss detailed in Equation 8 of the work.

At inference, new videos are generated by solving the probability flow ODE, integrating the velocity field from t = 1 to t = 0 using a first-order Euler solver over 40 steps. To facilitate audio conditioning, the team introduced additional cross-attention layers into the transformer blocks, utilising pre-extracted Whisper-small features caudio ∈RL×B×C. These features are processed by a learned projection and pooling module, producing lip-sync embeddings at the latent video frame rate. The Audio Projection module and associated cross-attention layers introduce approximately 2 billion additional learnable parameters. To address potential audio-video misalignment, a phase-shifted grid sampling strategy was implemented, ensuring consistent window semantics across videos with varying frame rates and preserving the original audio feature rate. This approach avoids the pitfalls of interpolation, which can discard high-frequency information and introduce temporal inconsistencies.

EditYourself enables audio-driven video manipulation with remarkable precision

Scientists have developed EditYourself, a new framework for editing pre-recorded videos based on audio input. This DiT-based system allows for transcript-based modifications to talking head videos, seamlessly adding, removing, or retiming spoken content while preserving natural motion and speaker identity. Forward-Backward RoPE Conditioning maintains stable identity and appearance throughout extended edits, operating efficiently within latent space. Researchers acknowledge the potential for misuse of this technology, particularly concerning visual forgeries and misinformation. They advocate for a multi-layered approach to responsible deployment, including legal frameworks for content ownership and technical safeguards like identity verification and digital watermarking. Future work should focus on content provenance and synthetic media detection to mitigate these risks. The authors also note the valuable contributions of the Lightricks LTX-Video team, whose open-source model weights aided development.