Dynamically Typed

#46: Snapchat's machine learning platform, AI at Apple, and an energy consumption tool for ML research

Hey everyone, welcome to Dynamically Typed #43! Today’s feature story is on productized AI: I wrote about SnapML—Snapchat’s machine learning platform—and its potential as a platform for creative ML developers to distribute their models to an audience of millions. Quick links in that section are an in-depth feature on AI at Apple and a new Unscreen release. Beyond productized AI, I’ve got a blogpost about GPT-3 + AGI and a cool new 3D dataset of London for ML research, plus a training power consumption tool and a new clouds dataset for climate change AI.

(Also: this is the first time I’ve made a custom header image for a feature story—right below this, for SnapML. Let me know what you think!)

Productized Artificial Intelligence 🔌

SnapML is a software stack for building Lenses that use machine learning models to interact with the Snapchat camera. You can build and train a model in any ONNX-compatible framework (like TensorFlow or PyTorch) and drop it straight into Snapchat’s Lens Studio as a SnapML component. SnapML can then apply some basic preprocessing to the camera feed, run it through the model, and format the outputs in a way that other Lens Studio components can understand. A segmentation model outputs a video mask, an object detection outputs bounding boxes, a style transfer model outputs a new image, etc. You even have control over how the model runs: once every frame, in the background, or triggered by a user action. (More details in the docs.)

Matthew Moelleman has written some great in-depth coverage on SnapML for the Fritz AI Heartbeat blog, including a technical overview and a walkthrough of making a pizza segmentation Lens. As he notes, SnapML has the potential to be super interesting as a platform:

Perhaps most importantly, because these models can be used directly in Snapchat as Lenses, they can quickly become available to millions of users around the world.

Indeed, if you create an ML-powered Snapchat filter in Lens Studio, you can easily publish and share it using a Snapcode, which users can scan to instantly use the Lens in their snaps. I don’t think any other platform has such a streamlined (no-code!) system for distributing trained ML models directly to a user base of this size. Early SnapML user Hart Woolery, also speaking to Heartbeat:

It’s a game-changer. At least within the subset of people working on live-video ML models. This now becomes the easiest way for ML developers to put their work in front of a large audience. I would say it’s analogous to how YouTube democratized video publishing. It also lowers the investment in publishing, which means developers can take increased risks or test more ideas at the same cost.

Similar to YouTube, the first commercial applications of SnapML have been marketing-related: there’s already a process for submitting sponsored Lenses, which can of course include an ML component. It’s not too hard to imagine that some advertising agencies will specialize in building SnapML models that, for example, segment Coke bottles or classify different types of Nike shoes in a Lens. I bet you can bootstrap a pretty solid company around that pitch.

Another application could be Lenses that track viral challenges: count and display how many pushups someone does, or whether they’re getting all the steps right for a TikTok dance. Snapchat is building many of these things itself, but the open platform leaves lots of room for creative ML engineers to innovate—and even get a share of this year’s $750,000 Official Lens Creators fund. (See some of the creations that came out of the fund here.)

The big question for me is whether and how Snapchat will expand these incentives for creating ML-powered Lenses. The Creators fund tripled in size from 2019 to 2020; will we see it grow again next year? Or are we going to get an in-app Snapchat store for premium Lenses with revenue sharing for creators? In any case, I think this will be a very exciting space to follow over the next few years.

Quick productized AI links 🔌

Machine Learning Research 🎛

Artificial Intelligence for the Climate Crisis 🌍

Thanks for reading! As usual, you can let me know what you thought of today’s issue using the buttons below or by replying to this email. If you’re new here, check out the Dynamically Typed archives or subscribe below to get a new issues in your inbox every second Sunday.

If you enjoyed this issue of Dynamically Typed, why not forward it to a friend? It’s by far the best thing you can do to help me grow this newsletter. 🪐