#53: Runway ML's Green Screen, new tools from EMNLP, and tracking illegal cattle farming with AI

Hey everyone, welcome to Dynamically Typed #53! It has one again been a very busy two weeks on my news feeds, so let’s dive straight in.

For productized AI, Runway ML launched the first (well, maybe) web-based automated video background removal tool; and Apple open-sourced a fork of TensorFlow 2 optimized for their new M1 Macs. On the ML research side, I read a very cool new Distill paper about understanding vision in reinforcement learning agents; GitHub is embracing fast.ai’s nbdev; and two great new resources came out of the EMNLP 2020: a 274-slide tutorial on high-performance NLP, and the Language Interpretability Tool (LIT). Next, for climate AI, a new paper accepted to the CCAI track at NeurIPS 2020 uses high-resolution satellite imagery to track illegal cattle farming; and Google’s AI blog has a feature about their Tree Canopy Lab. Finally, in cool things: the NYT created an amazing interactive story about GAN-generated fake faces; and Google engineers used GANs to create fantastical creatures.

Productized Artificial Intelligence 🔌

• 🟩 Runway ML, the “app store” of easy-to-use machine learning models for creators (see DT #18), added a new Green Screen feature, which it says is “[the] first real-time web tool for cutting objects out of videos. Using machine learning, it makes rotoscoping (a.k.a. masking) a lot faster and a lot less painful.” It looks very cool, but take their claim of being first with a grain of salt: Kaleido, the folks behind DT-favorite remove.bg, also launched an ML-powered automatic video background removal tool called unscreen earlier this year (#35). However, for Runway ML, Green Screen represents yet another well-integrated feature for their already extensive AI creativity product, which is not something unscreen can match as a single-use tool. Along with Photoshop’s new AI features (#51), this is yet another example of how quickly deep learning vision models are becoming easy to use for everyone.
• 🍎 Apple has forked TensorFlow 2 to optimize it for their new crazy-fast M1 Macs! This came as a pretty big surprise, and it makes the new M1 Macs even more attractive to ML developers: for the first time, this’ll enable using the internal GPU to train TensorFlow models on Mac laptops, leadings to ~5x speedups (!) compared to the previous generation. I’ll probably hold until for the next generation — by which time Apple’s optimizations should also be upstreamed to the main TensorFlow branch instead of only being available on their own fork — but it’s clear that even now these laptops are already huge game changers.

Machine Learning Research 🎛

• 👾 Cool new Distill paper from Hilton et al. (2020): Understanding RL Vision. The authors train a reinforcement learning agent to play a procedurally-generated video game based on single frames as input, and then develop an interactive interface (embedded in the article!) to study what different parts of the network learn. Using Circuits editing (see DT #37), they then make the agent blind to e.g. left-moving enemies in the game, and experimentally show that this indeed makes it fail more often by missing such enemies. “Our results depend on levels in CoinRun being procedurally-generated, leading us to formulate a diversity hypothesis for interpretability. [Interpretable features tend to arise (at a given level of abstraction) if and only if the training distribution is diverse enough (at that level of abstraction).] If it is correct, then we can expect RL models to become more interpretable as the environments they are trained on become more diverse.” As always, the full article is a great Sunday long read.
• 📓 Hamel Husain wrote in a post on the GitHub blog that the company is going to assist in developing fast.ai’s nbdev, a literate programming environment for Python. Donald Knuth describes literate programming as “a move away from writing computer programs in the manner and order imposed by the computer, and instead enables programmers to develop programs in the order demanded by the logic and flow of their thoughts.” I linked to nbdev last year (DT #28) but haven’t seen it in the wild much since then. I’m also not yet convinced that this notebooks-centric approach scales to making large software projects easier to follow. (Maybe that’s just because I’m used to the current standard structures for Python projects?) But the fact that GitHub is embracing nbdev so enthusiastically makes me curious about how it’ll develop in the coming years, and whether it’ll start to pop up more outside of fast.ai’s courses/projects.
• ⚡️ Fantastic slides by Ilharco et al. (2020) for the EMNLP 2020 tutorial on High Performance Natural Language Processing (PDF), which got a lot of love on Twitter. Alexander Rush: “Every slide is current to the minute. Amazing set of diagrams.” Jeff Dean: “hundreds of slides that will transform your understanding!”
• ⚡️ Also presented at EMNLP 2020, the Language Interpretability Tool (LIT) is an open-source platform for visualizing and understanding NLP models. It builds on top of Google’s previous What-If Tool, and supports “local explanations, including salience maps, attention, and rich visualizations of model predictions, as well as aggregate analysis including metrics, embedding spaces, and flexible slicing.” James Wexler and Ian Tenney introduce the tool in a post on the Google AI Blog, which also includes a few demos.

I’ve also collected all 75+ ML research tools previously featured in Dynamically Typed on a Notion page for quick reference. ⚡️

Artificial Intelligence for the Climate Crisis 🌍

• 🐄 In collaboration with the NGO Global Witness, which investigates human rights abuses surrounding conflicts over natural resources, Laradji et al. (2020) built a machine learning model to track illegal cattle ranching based on high-resolution (40cm) satellite imagery. “Cattle farming is responsible for 8.8% of greenhouse gas emissions worldwide. … While some regulations are in place for preserving the Amazon against deforestation, these are being flouted in various ways, hence the need to scale and automate the monitoring of cattle ranching activities.” The paper has been accepted to the Tackling Climate Change with ML Workshop at NeurIPS 2020, and training code is available on GitHub at IssamLaradji/cownter_strike.
• 🌳 For Google’s The Keyword blog, Alicia Cormie wrote about how Los Angeles’ first-ever City Forest Officer Rachel Malarich is using the company’s Tree Canopy Lab to decide where to plant trees. Tree Canopy Lab uses satellite imagery to automatically map a city’s tree cover density in different areas, which Cormie uses to find the communities most in need of more trees. Trees in cities are awesome: they improve air quality and reduce the effects of extreme climate change-induced heat. Plant more trees in cities! It’s great to see an ML-powered project being used to help with this.

Cool Things ✨

• 🙍‍♀️ If you’re going to click one link from today’s DT, make it this one. The New York Times’ Kashmir Hill and Jeremy White wrote an amazing new visual article about GAN-generated human faces. It details how projects like This Person Does Not Exist (see DT #8) and Rosebud.AI (#37) work, has embedded examples of latent space sliders that generate faces with different properties, and also includes some tips on spotting images with fake faces.
• 🐩 Related: Andeep Singh Toor and Fred Bertsch wrote a post for the Google AI blog about Using GANs to Create Fantastical Creatures. I like the bunny-wolf, although it is more than slightly terrifying. The model, which “automatically creates a fully fleshed out rendering from a user-supplied creature outline,” is available as a downloadable demo called Chimera Painter.

Thanks for reading! As usual, you can let me know what you thought of today’s issue using the buttons below or by replying to this email. If you’re new here, check out the Dynamically Typed archives or subscribe below to get a new issues in your inbox every second Sunday.

If you enjoyed this issue of Dynamically Typed, why not forward it to a friend? It’s by far the best thing you can do to help me grow this newsletter. 🎨