https://www.theverge.com/c/24300623/ai-companions-replika-openai-chatgpt-assistant-romance
thoughtful essay
Naturally, people presented with a technology that uses language with human fluency and invited to interact with it as if it were human immediately started ascribing to it other human faculties. Often, this meant mistaking ChatGPT’s confident style and expert vocabulary for the sort of accuracy they would signal if coming from a human. Others spooked themselves by cajoling the system into saying it was sentient, in love with them, or desiring freedom and power, not realizing it was mirroring back human-written fantasies contained in its training data. These errors were by no means limited to laypeople. More subtle forms of anthropomorphization persist, even among those working on AI. For example, it’s common to say models are progressing from toddler-level intelligence to high-school and PhD levels, when these measures of human education are a poor fit for systems that can explain quantum physics in iambic pentameter but struggle with basic arithmetic.
These systems can generate language that seems astonishingly human, Shanahan wrote, but the fundamentally alien process they use to do so has important implications for how we should understand their words. To use Shanahan’s example, when you ask a person, “What country is to the south of Rwanda?” and they answer “Burundi,” they are communicating a fact they believe to be true about the external world. When you pose the question to a language model, what you are really asking is, “Given the statistical distribution of words in the vast public corpus of text, what are the words most likely to follow the sequence ‘what country is to the south of Rwanda?’” Even if the system responds with the word “Burundi,” this is a different sort of assertion with a different relationship to reality than the human’s answer, and to say the AI “knows” or “believes” Burundi to be south of Rwanda is a category mistake that will lead to errors and confusion.
There is another perspective, not necessarily contradictory, which is that how we treat AI matters not because AI has any intrinsic moral standing but because of its potential to change us.
People often respond to the perceived weaknesses of AI by pointing to similar shortcomings in humans, but these comparisons can be a sort of reverse anthropomorphism that equates what are, in reality, two different phenomena. For example, AI errors are often dismissed by pointing out that people also get things wrong, which is superficially true but elides the different relationship humans and language models have to assertions of fact. Similarly, human relationships can be illusory — someone can misread another person’s feelings — but that is different from how a relationship with a language model is illusory. There, the illusion is that anything stands behind the words at all — feelings, a self — other than the statistical distribution of words in a model’s training data.
