Article by AI based on youtube video transcript: Why AI doesn't speak every language

Introduction to Language Models

Okay. I want you to check this out because it represents a big challenge for large language models like GPT-3 and now GPT-4. But it is not code. It is a list that countries around the world are grappling with.

How Large Language Models Learn

Before we get into the problems with large language models, let's review at a basic level how they work. You've probably heard of ChatGPT. It's not really a model but an app that sits on top of a large language model. In this case, a version of GPT. One thing models like ChatGPT do is natural language processing. It's used in everything from telephone customer service to auto completion.

The Analogy of the Used Bookstore

GPT is like a used bookstore. It's never left the room and only learned through the books that are at the store. Essentially, these large language models, they scan a lot of text and try to learn a language. They can check their process by covering up the answers and then seeing if they got it right.

Limitations in Language Representation

Then they can use that knowledge to recognize sentiment, summarize, translate, and generate responses or recommendations based on the analyzed data. This is an amazing ability but that's because it's read a lot of stuff. You can ask ChatGPT to rephrase something to Shakespeare but that's because it's read all the Shakespeare.

The Issue of High-Resource vs. Low-Resource Languages

So this is a print out of Common Crawl from 2008 to the present. Common crawl basically means that they go over all the websites and index them. And on this list, they put every language that they think that they've indexed. Here, you notice right away all the English. Every crawl is like more than 40% just English.

The Efforts of Researchers to Expand Language Data

There was a paper that stated that of the 7,000 languages spoken globally about 20 of those languages make up the bulk of NLP research. This is Ruth-Ann Armstrong. She's a researcher who I interviewed and she's doing something that a lot of researchers are trying to do... make new data sets.

The Importance of Transparency and Data Quantity

But there is still a catch. It performs reasonably well. But it's worth it to build a language-specific model that has been specifically trained and evaluated for that language. So the problem here isn't performance it’s transparency and it's the amount of data.

The Role of Open-Source in Multilingual AI

Some people are trying to build a library next to the bookstore. This is Paris, where the French have a supercomputer that wasn't being used a lot. Thomas Wolf is a co-founder of Hugging Face which is like a hub for AI research on the Internet and they ended up working on Big Science’s BLOOM... a project to create an open-source multilingual model.

The Need for Trust and Inclusivity in Language Technologies

So if you want to have this model everywhere you need to be able to trust them. So if you trust Microsoft, that's fine. But if you don't trust them... yeah. It's our language. So we speak— we are Catalan speakers. So whereas because of a small language or of a moderately small language because you may have languages that have a sizable amount of speakers in the real world but that have very, very little digital footprint. So they are bound to just... disappear.