Dynamically Typed

#34: Google's app for detecting fake news memes, an AI for logical reasoning, and Microsoft's library for training trillion-parameter models

Hey everyone, welcome to Dynamically Typed #34! Today’s feature stories include a fake news detection app for journalists by Google and Storyful; Microsoft’s distributed systems tool for training absolutely massive machine learning models; and a fun logical reasoning demo by the Allen Institute for AI. On an editorial note: I’ve started drifting into a format of doing one feature story per category, plus a bunch of quick one-to-two-sentence links. This way I can still share more in-depth thoughts on a few stories, but also cover a lot of other interesting stuff that otherwise wouldn’t make the cut. Is this something you all like? Let me know!

Productized Artificial Intelligence 🔌

Google and Storyful have launched Source, an app to help journalists fact-check online images and screenshots.

Journalists and fact-checkers face huge challenges in sorting accurate information from fast-spreading misinformation. But it’s not just about the words we read. Viral images and memes flood our feeds and chats, and often they’re out-of-context or fake.

Traditionally, journalists and fact checkers often use reverse image search to see if a viral image was real or fake news, but this is difficult because the recency bias in online search means that reprints from the past few hours or days often dominate search results. The Source app instead finds an image’s “public history”—even if it has been heavily altered since its original publication—and highlights whether it has previously shown up on whitelisted or blacklisted domains. It also automatically extracts and translates text from images, making it easier for journalists to catalogue and search them. The app is a good example of productized AI because it takes a few machine learning tasks that have been solved to the point of being accessible as simple API calls (in this case: reverse image search, optical character recognition, and machine translation) and wraps them up into an easy-to-use tool for non-expert users.

A while ago I toyed around with the idea of making a tool to extract text from screenshots of tweets, and using that to trace back whether the original tweet still exists (or if it ever did). I wanted to then hook that up to a twitter bot that you could tag to check whether a screenshot of a tweet was real or fake. I never got around to building it, partially because of restrictions on Twitter’s API, but mostly because of privacy concerns (I didn’t want to enable/encourage doxxing). But this automatic tweet verifying does seem like it could be very useful for journalists using Source, so I’m excited to see whether Google and Storyful will add features like this to the app as they develop it further.

You can read more about Source in this story by Irene Jay Liu on Google’s The Keyword blog: The new tool helping Asian newsrooms detect fake images.

Quick productized AI links 🔌

Machine Learning Research 🎛

ZeRO’s memory usage and the end of one training iteration. (Microsoft Research)

ZeRO’s memory usage and the end of one training iteration. (Microsoft Research)

Rangan Majumder and Junhua Wang wrote about ZeRO and DeepSpeed, Microsoft Research’s library for optimizing gargantuan deep learning models. This is what they used to train Turing-NLG, which at 17 billion parameters is the largest known language model to date—compare that to the 1.5 billion parameters in OpenAI’s GPT-2 model, which I’ve written about extensively (DT #8, #27). ZeRO achieves this by partitioning the model that’s being trained in groups corresponding to sets of layers, with:

This allows them to do data parallelism (e.g. having different GPUs train on different subsets of the data) on much larger models than was previously possible, at higher speed and throughput. The exact process by which this happens is quite complex, but the authors created a nice five-minute animation to explains step by step how a complete training iteration happens in this setting. For the video and more details, check out the Microsoft Research Blog post, ZeRO & DeepSpeed: New system optimizations enable training models with over 100 billion parameters, and microsoft/DeepSpeed on GitHub.

Quick ML research + resource links 🎛 (see all 56 resources)

Artificial Intelligence for the Climate Crisis 🌍

Quick climate AI links 🌍

Cool Things ✨

ROVER correctly predicts a logical conclusion over the given rules and statement. (Allen Institute for AI)

ROVER correctly predicts a logical conclusion over the given rules and statement. (Allen Institute for AI)

The Allen Institute for AI has an online demo of their logical reasoner called ROVER, which “determines whether statements are true or false based on rules given in natural language.” You can pick from a few pre-made sets of rules or make your own (e.g. “I like marbles. Marbles are blue.”) and give ROVER some statements to evaluate based on those rules (“I like some blue things. I like all blue things.”), and it’ll give you its best guess as to their correctness (“True” and “False,” respectively, both with 99% confidence). ROVER is capable of reasoning over much more complex rule sets, too:

We find transformers appear to learn rule-based reasoning with high (99%) accuracy on these datasets, and in a way that generalizes to test data requiring substantially deeper chaining than in the training data (95%+ scores). We also demonstrate that the models transfer well to two hand-authored rulebases, and to rulebases paraphrased into more natural language.

You can try it for yourself on the Allen AI website or check out the paper by Clark et al. (2020) on arXiv: Transformers as Soft Reasoners over Language.

Quick cool things links ✨

Thanks for reading! As usual, you can let me know what you thought of today’s issue using the buttons below or by replying to this email. If you’re new here, check out the Dynamically Typed archives or subscribe below to get a new issues in your inbox every second Sunday.

If you enjoyed this issue of Dynamically Typed, why not forward it to a friend? It’s by far the best thing you can do to help me grow this newsletter. ☔️