#46: Snapchat's machine learning platform, AI at Apple, and an energy consumption tool for ML research

Hey everyone, welcome to Dynamically Typed #43! Today’s feature story is on productized AI: I wrote about SnapML—Snapchat’s machine learning platform—and its potential as a platform for creative ML developers to distribute their models to an audience of millions. Quick links in that section are an in-depth feature on AI at Apple and a new Unscreen release. Beyond productized AI, I’ve got a blogpost about GPT-3 + AGI and a cool new 3D dataset of London for ML research, plus a training power consumption tool and a new clouds dataset for climate change AI.

(Also: this is the first time I’ve made a custom header image for a feature story—right below this, for SnapML. Let me know what you think!)

Productized Artificial Intelligence 🔌

SnapML is a software stack for building Lenses that use machine learning models to interact with the Snapchat camera. You can build and train a model in any ONNX-compatible framework (like TensorFlow or PyTorch) and drop it straight into Snapchat’s Lens Studio as a SnapML component. SnapML can then apply some basic preprocessing to the camera feed, run it through the model, and format the outputs in a way that other Lens Studio components can understand. A segmentation model outputs a video mask, an object detection outputs bounding boxes, a style transfer model outputs a new image, etc. You even have control over how the model runs: once every frame, in the background, or triggered by a user action. (More details in the docs.)

Matthew Moelleman has written some great in-depth coverage on SnapML for the Fritz AI Heartbeat blog, including a technical overview and a walkthrough of making a pizza segmentation Lens. As he notes, SnapML has the potential to be super interesting as a platform:

Perhaps most importantly, because these models can be used directly in Snapchat as Lenses, they can quickly become available to millions of users around the world.

Indeed, if you create an ML-powered Snapchat filter in Lens Studio, you can easily publish and share it using a Snapcode, which users can scan to instantly use the Lens in their snaps. I don’t think any other platform has such a streamlined (no-code!) system for distributing trained ML models directly to a user base of this size. Early SnapML user Hart Woolery, also speaking to Heartbeat:

It’s a game-changer. At least within the subset of people working on live-video ML models. This now becomes the easiest way for ML developers to put their work in front of a large audience. I would say it’s analogous to how YouTube democratized video publishing. It also lowers the investment in publishing, which means developers can take increased risks or test more ideas at the same cost.

Similar to YouTube, the first commercial applications of SnapML have been marketing-related: there’s already a process for submitting sponsored Lenses, which can of course include an ML component. It’s not too hard to imagine that some advertising agencies will specialize in building SnapML models that, for example, segment Coke bottles or classify different types of Nike shoes in a Lens. I bet you can bootstrap a pretty solid company around that pitch.

Another application could be Lenses that track viral challenges: count and display how many pushups someone does, or whether they’re getting all the steps right for a TikTok dance. Snapchat is building many of these things itself, but the open platform leaves lots of room for creative ML engineers to innovate—and even get a share of this year’s $750,000 Official Lens Creators fund. (See some of the creations that came out of the fund here.)

The big question for me is whether and how Snapchat will expand these incentives for creating ML-powered Lenses. The Creators fund tripled in size from 2019 to 2020; will we see it grow again next year? Or are we going to get an in-app Snapchat store for premium Lenses with revenue sharing for creators? In any case, I think this will be a very exciting space to follow over the next few years.

Quick productized AI links 🔌

• 🍎 Samuel Axon wrote an in-depth feature on machine learning at Apple for Ars Technica, with input from two executives at the company: John Giannandrea (SVP for ML and AI Strategy) and Bob Borchers (VP of Product Marketing). From handwriting recognition to battery charging optimization, AI has—Software 2.0-style—steadily been eating its way into more and more of the iOS software stack, far beyond just powering the obvious things like Siri speech recognition and camera roll semantic search. Of course, Giannandrea and Borchers also talk a lot about Apple’s focus on on-device ML and their “neural engine” accelerator chips. It’s a long article, but a must-read if you’re into productized AI.
• 📹 From the folks behind DT-favorite remove.bg (DT #3, #5, #12, #16), automatic video background removal tool Unscreen (#35) now has a Pro version that supports full HD, MP4 export, and unlimited-length videos. It’s web-only for now but an API is in the works. I’ve really enjoyed following this team’s progress over the past almost two years, and it’s great to see they’re continuing to execute to successfully.

Machine Learning Research 🎛

• 🤖 Does GPT-3, OpenAI’s latest iteration of their gargantuan language model (DT #42, #44) mean we’re imminently close to artificial general intelligence (AGI) like some of the Twitter hype has been suggesting? Reinforcement learning researcher Julian Togelius says no: in A very short history of some times we solved AI, he argues that we’ve been moving the goalpost fo AGI for decades. “Algorithms for search, optimization, and learning that were once causing headlines about how humanity was about to be overtaken by machines are now powering our productivity software. And games, phone apps, and cars. Now that the technology works reliably, it’s no longer AI (it’s also a bit boring).” Forgive the long quotes, but I share Togelius’ views on AGI almost exactly, and he communicates them very succinctly: “So when will we get to real general artificial intelligence? Probably never. Because we’re chasing a cloud, which looks solid from a distance but scatters in all directions as we drive into it.” For his more optimistic conclusion, read the full blog post.
• 🏙 Cool new dataset by Zhou et al. (2020): HoliCity: A City-Scale Data Platform for Learning Holistic 3D Structures. Covering a 20-square-kilometer area of central London, it aligns 6,300 high-resolution panorama photos with a 3D CAD model of the city, “with the ultimate goal of supporting real-world applications including city-scale reconstruction, localization, mapping, and augmented reality.” The dataset’s website includes a few samples with interactive sliders between their RGB, plane, CAD, and semantic views.

Artificial Intelligence for the Climate Crisis 🌍

• 💡 New paper from Henderson et al. (2020): Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning. The authors propose a framework and open-source software tool for measuring the energy use of ML models trained on Linux systems with Intel chips and NVIDIA GPUs. They also use electricityMap to estimate carbon emissions based on energy use, but because datacenters increasingly generate their own solar or wind power, and their electricity’s carbon intensity may therefore not be the same as that of the grid in their region, I’d take those numbers with a grain of salt. Anyway, I first came across the tool about half a year ago, and I’m glad it has picked up some steam since then. Consider giving it a star on GitHub and using it for your next paper or integrating into your company’s tooling!
• ⛅️ New dataset from Nielsen et al. (2020): CloudCast: A Satellite-Based Dataset and Baseline for Forecasting Clouds is “the first publicly available dataset with high-resolution cloud types on a high temporal granularity to the authors’ best knowledge.” The dataset has over 70k images, each annotated with 10 different cloud types, that were recoded in 15-minute increments from January 1st, 2017 to December 31st, 2018. Sounds like it could be very useful for solar PV nowcasting models (DT #18, #40).

Thanks for reading! As usual, you can let me know what you thought of today’s issue using the buttons below or by replying to this email. If you’re new here, check out the Dynamically Typed archives or subscribe below to get a new issues in your inbox every second Sunday.

If you enjoyed this issue of Dynamically Typed, why not forward it to a friend? It’s by far the best thing you can do to help me grow this newsletter. 🪐