#32: Google's chatbot gets punny, GitHub finds good first issues, and Tinder targets toxicity

February 2, 2020

Hey everyone, welcome to Dynamically Typed #32! We’ve hit another base-2 milestone: this is edition number 2⁵; the previous one, 2⁴ (DT #16), came out last summer when I had just moved to Amsterdam; and the next one, 2⁶, will be in April 2021. 🤓

Here’s what I have for you in today’s newsletter. For productized AI, I’m covering GitHub’s new AI-powered good first issues feature, as well as the different ways social media companies are using AI to combat toxicity on their platforms. For ML research, there’s Google’s new Meena chatbot, an economics perspective on AI research, and an author’s thoughts on writing for Distill. Finally, for climate AI, there seems to be a big influx of venture capital money in Silicon Valley. Happy reading!

Productized Artificial Intelligence 🔌

Documentation fixes are a common example of “good first issues,” as indicated by the purple tag on this issue on the Larq repository. (Larq is the open-source BNN training library I work on at Plumerai.)

Tiferet Gazit wrote about how GitHub made its AI-powered good first issue feature. GitHub is an online platform for collaborating on open- (and closed-) source codebases, where users can write issues to notify repository maintainers of bugs or desired features in their software projects. Maintainers can label low-hanging fruit issues—like documentation updates, error reporting fixes, or output formatting bugs—as __ a good first issue that new contributors can tackle to familiarize themselves with the project’s codebase.

GitHub recently launched a feature that automatically detects such good first issues in open-source codebases. Initially it worked by just looking for (synonyms of) the good-first-issue or documentation tags set manually by repository maintainers, but that did not yield enough issues for GitHub’s social news feed:

Relying on these labels, however, means that only about 40 percent of the repositories we recommend have easy issues we can surface. Moreover, it leaves maintainers with the burden of triaging and labeling issues. Instead of relying on maintainers to manually label their issues, we wanted to use machine learning to broaden the set of issues we could surface.

That’s where deep learning comes in: GitHub used this initial sample of good first issues detected based on explicitly-set labels as training data for a natural language classifier that predicts whether an issue qualifies as good first issue based on its title and body text. With this new model, which gets run on all new open-source issues once a day, GitHub is now “able to surface issues in about 70 percent of repositories we recommend to users”—a big improvement!

Gazit’s blog post on the feature goes in depth on a lot of technical details, including data pre-processing/denoising, the deployment setup, and the coverage vs. accuracy tradeoff, that we know typically take much more time on a productized AI project than the actual model creation and training do. It’s great to see these aspects being highlighted. Read the full post on The GitHub Blog: How we built the good first issues feature.

Quick productized AI links 🔌

💬 To combat the low reporting rates of the harassment happening on its platform, Tinder is now using artificial intelligence to detect potentially toxic chat messages and ask users whether they want to report the sender. An interesting challenge here is that the preceding conversation—or lack thereof—has a big effect on whether a message comes across a flirty or creepy. The number of reported chats increased by 37% after the feature launched. Arielle Pardes for WIRED : Tinder Asks ‘Does This Bother You’?
🤬 Related: Instagram and Twitter are also using AI to detect potential bullying and hateful content as users write it, and ask them whether they’re sure they want to post it. Sara Harrison for WIRED : Twitter and Instagram Unveil New Ways to Combat Hate—Again

Machine Learning Research 🎛

“Meena executes a multi-turn joke in an open-domain setting. We were unable to find this in the data.” (Adiwardana et al., 2020)

“Meena executes a multi-turn joke in an open-domain setting. We were unable to find this in the data.” (Adiwardana et al., 2020)

Google AI has a new open-domain chatbot: Meena. It’s based on one encoder and thirteen decoder networks, constructed from neural architecture search-discovered evolved transformer blocks, and, like los of modern natural language processing/generation research, the model behind Meena is gargantuan :

The Meena model has 2.6 billion parameters and is trained on 341 GB of text, filtered from public domain social media conversations. Compared to an existing state-of-the-art generative model, OpenAI GPT-2 [see DT #8, #13, #22, #27, #28], Meena has 1.7x greater model capacity and was trained on 8.5x more data.

Although the model is trained on the standard automated perplexity metric—which intuitively “represents the number of choices the model is trying to choose from when producing the next token”—the authors introduce a new human evaluation metric to assess their model. The new metric asks reviewers to rate a bot’s responses on sensibleness (Does this response make sense in the context of the preceding conversation?) and specificity (Is this not something generic like “haha” or “that’s nice” you could reply in lots of situations?), and averages them. Under this SSA metric, Meena scores 79%, compared to 86% for human conversations and 56% for the next-best competing chatbot.

One cool thing that drew me to this work: they preprocess their data for neural network consumption using byte-pair encoding (BPE) by Sennrich et al. (2016)—who was one of my professors in Edinburgh, for the (quite a mouthful) natural language processing, generation, and machine translation course!

Artificial Intelligence for the Climate Crisis 🌍

I didn’t find any new climate change AI projects that have progressed far enough to cover in the newsletter for this edition. However, there is still lots of exciting stuff happening in the space.

I’ve started listening to Jason Jacobs’ My Climate Journey podcast, which I’ve really enjoyed so far. It’s been great to hear what climate challenges lots of different industries are dealing with (beyond the obvious ones like energy and transportation), and to start thinking about how AI/ML might be relevant there.

It also looks like more and more of Silicon Valley is warming up (ha) to working on climate-related problems. On Twitter these past two weeks, Ravi Mikkelsen, Katie Fehrenbacher, Ann Bordetsky, and again Jason Jacobs have been talking about the surge of venture capital investments into climate tech. (And away from crypto.) Hopefully, this means we’ll only start to see more and more climate change AI companies and products over the next few years!

Quick climate AI links 🌍

☁️ Cumulo is a new “benchmark dataset for training and evaluating global cloud classification models.” Understanding cloud types, formations, and movements will be key for things like solar power output prediction (mitigation) but also for identifying severe weather events (adaptation).

Thanks for reading! As usual, you can let me know what you thought of today’s issue using the buttons below or by replying to this email. If you’re new here, check out the Dynamically Typed archives or subscribe below to get a new issues in your inbox every second Sunday.

If you enjoyed this issue of Dynamically Typed, why not forward it to a friend? It’s by far the best thing you can do to help me grow this newsletter. 🦆

Dynamically Typed

#32: Google's chatbot gets punny, GitHub finds good first issues, and Tinder targets toxicity

Productized Artificial Intelligence 🔌

Machine Learning Research 🎛

Artificial Intelligence for the Climate Crisis 🌍

Productized AI

ML Research

Climate AI

Cool Things