The pursuit of language use in non-humans has plagued our imaginations for generations. In recent years some great apes have been taught sign language or shown other novel forms of intelligent communication. We’ve also developed a number of computer applications designed to imitate language use, and while those of us who have used Siri or Alexa can attest to their growing functionality, the impression that they are independently producing intelligent responses in natural language is still lacking.
Recently, however, OpenAI, a San Francisco based research group, has released a new Natural Language Processing artificial intelligence called GPT-3 (or Generative Pre-trained Transformer 3), and while it certainly isn’t a human, there’s no denying its capability to sound like one.
Autocomplete Gets Some Cosmetic Work Done
The premise of GPT-3 is simple; the program completes a text response based off of any written prompt provided by the user. It’s the same technology that powers your phone’s autocomplete or
Google’s search engine. What makes the AI so amazing though is the sophistication of its responses, and the extreme versatility of its capabilities
GPT-3 can produce impressively coherent results. Given the proper directions, it can craft a written piece in any genre or style, even being powerful enough to imitate the voices of real-life writers. One website hosts a beta tester’s experiments with the AI, collecting everything from a fictional letter denying Indiana Jones’ tenure to rewritten versions of classic poems—all composed by the program. The AI has been prompted to imitate public figures, too, with some success; in one case, a beta tester had the program write a short speech in the style of Scott Barry Kaufman, to which the writer later responded, “it definitely sounds like something I would say”.
Perhaps even more important than its ability to sound human, however, is the breadth of its potential application. Being able to return text from any prompt, GPT-3 has already proven capabilities beyond what one would expect. Some uses are logical, like writing an article for a news publication, or powering a text-based role playing game that you can play in your browser. Other beta testers discovered interesting features, however, such as an ability to code web interfaces based solely off of a verbal description of the end product (this tweet, posted by user @sharifshameem, shows GPT-3 coding the google homepage from a text description).
A Look Under the Artificial Hood
GPT-3 is powered by neural networks and deep learning, which means two things. Firstly, neural networks are computer programs loosely based on the structure of the human brain and common in the field of artificial intelligence. They contain ‘neurons’, or nodes which represent predetermined pieces of input or output information. From there, the program works through sets of data, trying to discover underlying patterns. This is why neural networks are often thought of as “learning” or “teaching themselves”, because they mature and develop on their own by independently processing data—and this is where deep learning comes in.
Deep learning is a specific form of machine learning where a neural network is given only raw data and has to figure out what features of that data are and are not significant on its own. The program isn’t told what to look for, instead making its own connections. This can yield a more robust end product than targeted machine learning, but requires a far larger and more varied pool of training information to get there.
Luckily, one of the most striking features of GPT-3 is the sheer amount of data that it has been trained on. GPT-3 has chewed through around 45 terabytes of information, according to its developers, including the entirety of Wikipedia (which, by the way, made up only 3% of it’s total training). It spent a portion of its development freely crawling the internet, reading everything from newspapers to blog posts to classic literature to twitter feeds. The premise is that, if exposed to enough information, a program could grow to recognize more than just the fundamental patterns in our language. With enough training, an AI could delineate the patterns present in specific cultural contexts—in other words, the ways that we actually use language, and not just the nitty-gritty of how it works. GPT-3, it seems, is proof that this is possible.
A Stain on the Chrome Man’s Reputation
GPT-3 is far from perfect. While it’s certainly produced some eerily human-sounding text, those are only the successes, and it’s equally skilled at producing nonsense. One beta user gave the program a Turing Test and discovered that it struggles with basic arithmetic, quickly forgets what it’s already said, and answers nonsense questions as if they were legitimate. It may be able to passably imitate Shakespeare, but it also thinks that the sun has one eye, and that a pencil is heavier than a toaster.
In addition to nonsensical outputs, concerns have been raised over problematic ones, too. Because GPT-3 was trained on the unfiltered breadth of the internet, it has picked up some of the toxic and biased patterns harboured within it, and this occasionally comes out in its responses. When tasked with discussing certain sensitive or controversial topics, the AI has been recorded producing text that would be considered hateful or prejudiced if written by a human.
The Design Behind the Wizard’s Curtain
All of this stems from a basic fact of GPT-3’s design; GPT-3 is not a writer. GPT-3 is a statistician.
Looking at some of the program’s compositions, it’s tempting to decide that what we are witnessing is a cognisant force, capable of planning ahead and deliberate choices in diction and form. In reality, all GPT-3 is trained to do is guess what word is most likely to come next. Taking into account a particular context and, given a word or phrase to start with, the AI simply reflects on all of the patterns it’s learned through its training and guesses what word a human would be statistically most likely to write next. While this reflects the way that we learn language in a shallow way, it’s an approach still devoid of many of the other fundamental skills needed for complex language use, such as spatial awareness, logical reasoning, memory, sense of time, or a sense of self.
All in all, GPT-3 is not likely to replace our reporters or novelists or programmers anytime soon. While it is doubtless an important step along the road of developing an artificial intelligence capable of using language as competently as we do, there’s still a lot of work to be done in rounding out its capabilities. Despite this, it is still powerful enough to find a home improving the technology which powers everything from search engines to video games. And, perhaps most importantly, it is powerful enough to trick us. While at times it may sound like a rambling, biased, or misinformed human, the fact remains that it’s still able to sound like a human. And it’s only getting better at it.
References
https://medium.com/mytake/natural-language-processing-ft-siri-2bc7b854a2a3
https://enterprisersproject.com/article/2019/7/deep-learning-explained-plain-english
https://openai.com/blog/gpt-3-apps/
https://www.nytimes.com/2020/11/24/science/artificial-intelligence-ai-gpt3.html
