Omnimatte

Omnimatte is a new matte/mask generation model by Erika Lu, who developed it in collaboration with Google AI researchers during two internships there. Unlike other state-of-the-art segmentation networks, Omnimatte creates masks for both objects and their “effects” like shadows or dust clouds in videos, enabling editors to easily add layers of content between the background and a foreground subject in a realistic way. Forrester Cole and Tali Dekel explain how the model works in detail (with lots of gifs!) in a post on the Google AI Blog.