#34: Google's app for detecting fake news memes, an AI for logical reasoning, and Microsoft's library for training trillion-parameter models

Hey everyone, welcome to Dynamically Typed #34! Today’s feature stories include a fake news detection app for journalists by Google and Storyful; Microsoft’s distributed systems tool for training absolutely massive machine learning models; and a fun logical reasoning demo by the Allen Institute for AI. On an editorial note: I’ve started drifting into a format of doing one feature story per category, plus a bunch of quick one-to-two-sentence links. This way I can still share more in-depth thoughts on a few stories, but also cover a lot of other interesting stuff that otherwise wouldn’t make the cut. Is this something you all like? Let me know!

Productized Artificial Intelligence 🔌

Google and Storyful have launched Source, an app to help journalists fact-check online images and screenshots.

Journalists and fact-checkers face huge challenges in sorting accurate information from fast-spreading misinformation. But it’s not just about the words we read. Viral images and memes flood our feeds and chats, and often they’re out-of-context or fake.

Traditionally, journalists and fact checkers often use reverse image search to see if a viral image was real or fake news, but this is difficult because the recency bias in online search means that reprints from the past few hours or days often dominate search results. The Source app instead finds an image’s “public history”—even if it has been heavily altered since its original publication—and highlights whether it has previously shown up on whitelisted or blacklisted domains. It also automatically extracts and translates text from images, making it easier for journalists to catalogue and search them. The app is a good example of productized AI because it takes a few machine learning tasks that have been solved to the point of being accessible as simple API calls (in this case: reverse image search, optical character recognition, and machine translation) and wraps them up into an easy-to-use tool for non-expert users.

A while ago I toyed around with the idea of making a tool to extract text from screenshots of tweets, and using that to trace back whether the original tweet still exists (or if it ever did). I wanted to then hook that up to a twitter bot that you could tag to check whether a screenshot of a tweet was real or fake. I never got around to building it, partially because of restrictions on Twitter’s API, but mostly because of privacy concerns (I didn’t want to enable/encourage doxxing). But this automatic tweet verifying does seem like it could be very useful for journalists using Source, so I’m excited to see whether Google and Storyful will add features like this to the app as they develop it further.

You can read more about Source in this story by Irene Jay Liu on Google’s The Keyword blog: The new tool helping Asian newsrooms detect fake images.

Quick productized AI links 🔌

• 🥦 Amazon has expanded its Go line of automated shops into its first fully-fledged grocery store. Haddie Golden for The Guardian: ‘Just walk out’: Amazon debuts its first supermarket with no checkout lines. How long until all Whole Foods stores (which Amazon owns) work this way? (Thanks for the link, Dad!)
• 🧫 A group of scientists has succeeded in using machine learning to find a new E. coli antibiotic. Paper in Cell by Stokes et al. (2020): A Deep Learning Approach to Antibiotic Discovery; also see this Andrej Karpathy tweet.
• 🤖 Facebook has published a paper about Temporal Interaction EmbeddingS (TIES), a “production-ready model at Facebook-scale networks” for “capturing rogue social interactions”—for catching networks of bots/fake accounts.

Machine Learning Research 🎛

ZeRO’s memory usage and the end of one training iteration. (Microsoft Research)

Rangan Majumder and Junhua Wang wrote about ZeRO and DeepSpeed, Microsoft Research’s library for optimizing gargantuan deep learning models. This is what they used to train Turing-NLG, which at 17 billion parameters is the largest known language model to date—compare that to the 1.5 billion parameters in OpenAI’s GPT-2 model, which I’ve written about extensively (DT #8, #27). ZeRO achieves this by partitioning the model that’s being trained in groups corresponding to sets of layers, with:

• Optimizer state partitioning, where optimizer variables (such as the momentum in Adam) are divided between the memories of different GPUs
• Gradient partitioning, where gradients are divided between GPUs.
• Parameter partitioning, where the model’s weights are divided between GPUs.

This allows them to do data parallelism (e.g. having different GPUs train on different subsets of the data) on much larger models than was previously possible, at higher speed and throughput. The exact process by which this happens is quite complex, but the authors created a nice five-minute animation to explains step by step how a complete training iteration happens in this setting. For the video and more details, check out the Microsoft Research Blog post, ZeRO & DeepSpeed: New system optimizations enable training models with over 100 billion parameters, and microsoft/DeepSpeed on GitHub.

Quick ML research + resource links 🎛 (see all 56 resources)

• 🧩 The Abstraction and Reasoning Corpus (ARC) from Keras creator François Chollet’s The Measure of Intelligence paper (see DT #26) now has a Kaggle competition: “create an AI capable of solving reasoning tasks it has never seen before.” Check out the discussion posts (they’re great) and let me know if you have ideas for the challenge you’d like to try out together!
• ⚡️ Microsoft’ open-source repo of best practices, code samples and documentation for computer vision (mostly on Azure): microsoft/computervision-recipes.
• 🤔 Yoshua Bengio wants machine learning conferences to start incentivizing Slow Science: Time to rethink the publication process in machine learning.
• 🐦 Plumerai released Larq Compute Engine, my coworkers’ new open-source project for deploying binarized neural networks to hardware and for using them for efficient inference. Check out the launch blog post (which I helped write!) and help us get the word out by giving larq/compute-engine a star on GitHub. :)

Artificial Intelligence for the Climate Crisis 🌍

Quick climate AI links 🌍

• 🔌 The International Energy Agency (IEA) published a detailed online report on energy use by datacenters; Data centres and energy – from global headlines to local headaches? In the future, it’ll become increasingly important to schedule machine learning experiments and model training to run at times that datacenters have access to lots of low-carbon electricity (e.g. wind, solar, hydro, nuclear)—are there tools to do this based on weather predictions and datacenter locations? Should we build one?
• ❄️ Cool paper by Prabha et al. (2020) on arXiv: Lake Ice Monitoring with Webcams and Crowd-Sourced Images, which introduces both a dataset and state-of-the-art model for observing freezing patterns over time—without having to worry about clouds blocking satellite imaging.

Cool Things ✨

ROVER correctly predicts a logical conclusion over the given rules and statement. (Allen Institute for AI)

The Allen Institute for AI has an online demo of their logical reasoner called ROVER, which “determines whether statements are true or false based on rules given in natural language.” You can pick from a few pre-made sets of rules or make your own (e.g. “I like marbles. Marbles are blue.”) and give ROVER some statements to evaluate based on those rules (“I like some blue things. I like all blue things.”), and it’ll give you its best guess as to their correctness (“True” and “False,” respectively, both with 99% confidence). ROVER is capable of reasoning over much more complex rule sets, too:

We find transformers appear to learn rule-based reasoning with high (99%) accuracy on these datasets, and in a way that generalizes to test data requiring substantially deeper chaining than in the training data (95%+ scores). We also demonstrate that the models transfer well to two hand-authored rulebases, and to rulebases paraphrased into more natural language.

You can try it for yourself on the Allen AI website or check out the paper by Clark et al. (2020) on arXiv: Transformers as Soft Reasoners over Language.

Quick cool things links ✨

• 🎥 Denis Shiryaev upscaled a classic black-and-white movie to 4K and 60 frames per second using a generative adversarial network, and it looks amazing: Arrival of a Train at La Ciotat (The Lumière Brothers, 1896). Also see the original video and an updated colorized version using DeOldify (see DT #15).

Thanks for reading! As usual, you can let me know what you thought of today’s issue using the buttons below or by replying to this email. If you’re new here, check out the Dynamically Typed archives or subscribe below to get a new issues in your inbox every second Sunday.

If you enjoyed this issue of Dynamically Typed, why not forward it to a friend? It’s by far the best thing you can do to help me grow this newsletter. ☔️