Trajectory-guided Anime Video Synthesis via Effective Motion Learning

Cartoon and anime motion production is traditionally labor-intensive, requiring detailed animatics and extensive in-betweening from keyframes. To streamline this process, we propose a novel framework that synthesizes motion directly from a single colored keyframe, guided by user-provided trajectories. Addressing the limitations of prior methods, which struggle with anime due to reliance on optical flow estimators and models trained on natural videos, we introduce an efficient motion representation specifically adapted for anime, leveraging CoTracker to capture sparse frame-to-frame tracking effectively. To achieve our objective, we design a two-stage learning mechanism: the first stage predicts sparse motion from input frames and trajectories, generating a motion preview sequence via explicit warping; the second stage refines these previews into high-quality anime frames by fine-tuning ToonCrafter, an anime-specific video diffusion model. We train our framework on a novel animation video dataset comprising more than 500,000 clips. Experimental results demonstrate significant improvements in animating still frames, achieving better alignment with user-provided trajectories and more natural motion patterns while preserving anime stylization and visual quality. Our method also supports versatile applications, including motion manga generation and 2D vector graphic animations. The data and code will be released upon acceptance.

We introduce an efficient sparse motion representation that records displacements on a sparse array of keypoints. Using a high-quality point-based tracker such as CoTracker, this representation effectively captures most frame-to-frame tracking and morphing profiles in anime. With this representation, we then develop our framework in two stages. In the first stage, we propose a video diffusion model to predict the motion representation directly from the conditions, including the input frame, user trajectories, and text prompts. To support training, we also propose a large-scale anime motion dataset with a novel motion sampling procedure to approximate user trajectories from CoTracker motion estimations. At the end of this stage, we warp the initial frame with the model predictions to construct the subsequent frames as a motion preview. Although this sequence may contain distortions and uncertainties, it still serves as an explicit motion prior for the next stage. Thus, in the second stage, we leverage the generative prior of an anime-specific video diffusion model, ToonCrafter, to refine the motion preview into a high-quality anime video clip. Specifically, we finetune the model to interpret motion guidance implicitly from the user-provided trajectories and explicitly from the warped motion preview. This approach enables the synthesis of complete anime frames with improved details, realistic and trajectory-conforming motion, and robust handling of occlusions.