#11: Adobe and Google's new video AI tools, Stanford's HYPE for GANs, and a conversation with books

We’re back to our regularly scheduled programming in today’s Dynamically Typed, with lots of news.

In productized AI, there are new AI-enabled video tools in Adobe After Effects and Google Photos, and an article about the art community’s growing acceptance for AI as an art medium. In machine learning tech, there’s Stanford’s new HYPE metric to measure the quality of generative models, Google’s disbanding of their AI ethics board, three new ML resources, and much more. Let’s dive in.

Productized Artificial Intelligence 🔌

Adobe After Effects’s new content-aware filling for videos. (Adobe)

Adobe After Effects now has a version of Photoshop’s content-aware fill feature for video. Editors can use any existing tools to crop an unwanted object out of a video (like the cars in the lower screenshot above), and After Effects will then do its best to fill the gaps in a natural-looking way (the empty road in the upper screenshot). Adobe says the feature is powered by its Sensei AI platform but doesn’t go into much detail about how its generative model works. Nonetheless, it’s a very cool example of productized AI: it’s built into an existing mainstream tool that doesn’t require video editors to learn anything about the underlying machine learning, and it’ll save them a lot of manual work in tasks like painting away microphones, removing bystanders in stock footage, etc. Frederic Lardinois for TechCrunch: Adobe brings content-aware fill for videos to After Effects.

Google Photos can now automatically extract “special moments” from videos. Google annotated clips from a collection public videos with labels like “blowing out birthday candle” and “cat wags tail” and trained a temporal action location network (TALNet) to automatically detect these in videos uploaded to Google Photos. The app then presents these to users as shareable animations, such as these example gifs. More:

• Google AI blog: Capturing Special Video Moments with Google Photos
• Research paper by Yu-Wei Chao et al.: Rethinking the Faster R-CNN Architecture for Temporal Action Localization (arXiv)

Francesca Gavin wrote about how the art community is embracing—and paying for—AI-generated art. She mentions GAN-generated pieces like Obvious’ Edmond de Belamy , which was (controversially) auctioned off for $432,500 last year, and Mario Klingemann’s Memories of Passersby I , which I featured in DT #9. One artwork I hadn’t heard of before is Closed Loop by Jake Elwes, a two-channel projection of one network that generates an image based on a short description and another network that describes the generated image. These then go back and forth, describing and visualizing each other’s outputs. Elwes has a very mesmerizing video of this on his website. For more AI art, see Gavin’s article: Artificial intelligence: the art world’s weird and wonderful new medium.

Machine Learning Technology 🎛

Images generated using StyleGAN, plotted by their HYPE score. (Stanford)

Sharon Zhou et al. at Stanford have proposed HYPE, a new metric for evaluating GANs. Generative Adversarial Networks have made some incredible progress in generating realistic-looking images in the last few years (see for example StyleGAN in DT #4). One problem, however, is that “there is currently no standardized, validated evaluation” to objectively compare the quality of images generated by different models. Zhou proposes Human eYe Perceptual Eveluation, or HYPE (an amazing name for a metric in this space), which is a measure of how quickly human evaluators can tell that a generated image is fake (HYPE-Time), and whether they can tell if it’s fake at all (HYPE-Infinity). More:

• Official whitepaper: HYPE: Human eYe Perceptual Evaluation of Generative Models (arXiv)
• Stanford’s website for HYPE, which mentions that code is coming soon: hype.stanford.edu

Wayve demonstrated their self-driving car in an urban environment. A few months ago the company showed off an autonomous vehicle that used end-to-end deep learning to translate video inputs directly to driving commands (see DT #6), driving on a simple rural road. They’ve refined their models and added satellite navigation as additional input, and they’re now able to drive on public urban roads in Cambridge, UK. Their car initially learns policies by imitating human drivers, and then uses reinforcement learning to improve these policies by using safety driver interventions as rewards. With only 20 hours of non-simulated training, their results are very impressive. More here:

• Wayve’s blog post: Learning to Drive like a Human
• Mike Butcher for TechCrunch: Wayve claims ‘world first’ in driving a car autonomously with only its AI and a SatNav

Li Yuan at the New York Times did a feature on Chinese data labeling companies. A lot of modern machine learning is done in a supervised setting, requiring labeling enormous datasets. In China, where labor is relatively cheap compared to Europe and the United States, “data labeling factories” like Ruijin Technology Company do exactly that:

“We’re the construction workers in the digital world. Our job is to lay one brick after another,” said Yi Yake, co-founder of a data labeling factory in Jiaxian, a city in central Henan province. “But we play an important role in A.I. Without us, they can’t build the skyscrapers.”

The companies are popping up in areas outside of large cities where labor and office spaces are cheap, enabling them to pay relatively high wages for the area while still providing labeling services at extremely low prices. Yuan interviewed some workers at the factories: some find the work too boring, while others prefer it over “the same movement, day after day” that they’d otherwise do at an assembly line. Read the full piece here: How Cheap Labor Drives China’s A.I. Ambitions.

Google has disbanded its AI ethics board less than two weeks after launch. Just a few weeks ago, Google announced that it had formed ATEAC, a council in charge of enforcing the company’s AI ethics guidelines (see DT #10). One member of the board, Kay Cole James, is involved with a foundation known for its anti-LGBT views; another, Dyan Gibbens, is CEO of a company that built drones for the US military. Thousands of Googlers signed a petition against their positions on the board, and two other board members also voiced their protest (one even publicly resigned). In response, Google decided to disband ATEAC and to go “back to the drawing board.” James wrote a Washington Post op-ed about her disappointment at the decision, which I think is worth reading, but take it with a grain of salt: people like Kara Swisher think that ATEAC was always always “a rubber stamp PR move” and that James was never qualified to serve on it. More by Jilian D’Onfro for Forbes: Google Scraps Its AI Ethics Board Less Than Two Weeks After Launch In The Wake Of Employee Protest.

Quick ML resource links ⚡️

• tf-encrypted is an experimental layer on top of TensorFlow to do machine learning on encrypted data. Links: code (GitHub); paper (arXiv)
• Dragonfly is a library for Bayesian optimization, or as they call it, “tuning hyperparameters without grad students.” Links: code (GitHub); paper (arXiv)
• Dent Reality is a company that does indoor augmented reality positioning and wayfinding without the need for additional hardware; they are now ready to deploy their tech. Links: Dent Reality; demo (YouTube)

Tech and Startups 🚀

Amazon is planning to launch Project Kuiper, a global satellite-powered internet network. Just like the competing initiatives by SpaceX and OneWeb (see DT #9), Amazon’s project involves sending a few thousand small satellites into orbit to “provide low-latency, high-speed broadband connectivity to unserved and underserved communities around the world.” Alan Boyle at GeekWire wrote a great article comparing Amazon’s project to its competitors: Amazon to offer broadband access from orbit with 3,236-satellite ‘Project Kuiper’ constellation.

Cool Things ✨

Talk to Books’ response to “Who is Harry Potter’s mother?” (Google)

Google’s Talk to Books tool answers your questions using passages from books. The tool is trained on human conversation, so when you ask it something like “Who is Harry Potter’s mother?” or “What’s your favorite scary movie?” it’ll find passages from books that it thinks would be a natural response to your question. You can try out the experimental AI at Talk to Books.

Thanks for reading! As always, let me know what you thought of this issue using the buttons below or by sending me a message. If you’re new here, subscribe for a new issue of Dynamically Typed every second Sunday!