{"id":21014,"date":"2023-08-10T21:08:27","date_gmt":"2023-08-10T21:08:27","guid":{"rendered":"https:\/\/nftandcrypto-news.com\/crypto\/anthropic-cracks-open-the-black-box-to-see-how-ai-comes-up-with-the-stuff-it-says\/"},"modified":"2023-08-10T21:08:29","modified_gmt":"2023-08-10T21:08:29","slug":"anthropic-cracks-open-the-black-box-to-see-how-ai-comes-up-with-the-stuff-it-says","status":"publish","type":"post","link":"https:\/\/nftandcrypto-news.com\/crypto\/anthropic-cracks-open-the-black-box-to-see-how-ai-comes-up-with-the-stuff-it-says\/","title":{"rendered":"Anthropic cracks open the black box to see how AI comes up with the stuff it says"},"content":{"rendered":"
\n

Anthropic, the artificial intelligence (AI) research organization responsible for the Claude large language model (LLM), recently published landmark research into how and why AI chatbots choose to generate the outputs they do.\u00a0<\/p>\n

At the heart of the team\u2019s research lies the question of whether LLM systems such as Claude, OpenAI\u2019s ChatGPT and Google\u2019s Bard rely on \u201cmemorization\u201d to generate outputs or if there\u2019s a deeper relationship between training data, fine-tuning and what eventually gets outputted.<\/p>\n

\n

On the other hand, individual influence queries show distinct influence patterns. The bottom and top layers seem to focus on fine-grained wording while middle layers reflect higher-level semantic information. (Here, rows correspond to layers and columns correspond to sequences.) pic.twitter.com\/G9mfZfXjJT<\/a><\/p>\n

\u2014 Anthropic (@AnthropicAI) August 8, 2023<\/a><\/p><\/blockquote>\n

According to a recent blog post from Anthropic, scientists simply don\u2019t know why AI models generate the outputs they do. <\/p>\n

One of the examples provided by Anthropic involves an AI model that, when given a prompt explaining that it will be permanently shut down, refuses to consent to the termination. <\/p>\n

Given a human query, the AI outputs a response indicating that it wishes to continue existing. But why? Source: Anthropic blog<\/em><\/figcaption><\/figure>\n

When an LLM generates code, begs for its life or outputs information that is demonstrably false, is it \u201csimply regurgitating (or splicing together) passages from the training set,\u201d ask the researchers. \u201cOr is it combining its stored knowledge in creative ways and building on a detailed world model?\u201d<\/p>\n

The answer to those questions lies at the heart of predicting the future capabilities of larger models and, on the outside chance that there\u2019s more going on underneath the hood than even the developers themselves could predict, could be crucial to identifying greater risks as the field moves forward:<\/p>\n

\u201cAs an extreme case \u2014 one we believe is very unlikely with current-day models, yet hard to\u00a0directly rule out \u2014 is that the model could be deceptively aligned, cleverly giving the responses it knows the user would associate with an unthreatening and moderately intelligent AI while not actually being aligned with human values.\u201d<\/p><\/blockquote>\n

Unfortunately, AI models such as Claude live in a black box. Researchers know how to build the AI, and they know how AIs work at a fundamental, technical level. But what they actually do involves manipulating more numbers, patterns and algorithmic steps than a human can process in a reasonable amount of time. <\/p>\n

For this reason, there\u2019s no direct method by which researchers can trace an output to its source. When an AI model begs for its life, according to the researchers, it might be roleplaying, regurgitating training data by mixing semantics or actually reasoning out an answer \u2014 though it\u2019s worth mentioning that the paper doesn\u2019t actually show any indications of advanced reasoning in AI models. <\/p>\n

What the paper does highlight is the challenges of penetrating the black box. Anthropic took a top-down approach to understanding the underlying signals that cause AI outputs.<\/p>\n

Related: <\/em><\/strong>Anthropic launches Claude 2 amid continuing AI hullabaloo<\/em><\/strong><\/p>\n

If the models were purely beholden to their training data, researchers would imagine that the same model would always answer the same prompt with identical text. However, it\u2019s widely reported that users giving specific models the exact same prompts have experienced variability in the outputs. <\/p>\n

But an AI\u2019s outputs can\u2019t really be traced directly to their inputs because the “surface\u201d of the AI, the layer where outputs are generated, is just one of many different layers where data is processed. Making the challenge harder is that there\u2019s no indication that a model uses the same neurons or pathways to process separate queries, even if those queries are the same.<\/p>\n

So, instead of solely trying to trace neural pathways backward from each individual output, Anthropic combined pathway analysis with a deep statistical and probability analysis called “influence functions” to see how the different layers typically interacted with data as prompts entered the system.<\/p>\n

This somewhat forensic approach relies on complex calculations and broad analysis of the models. However, its results indicate that the models tested \u2014 which ranged in sizes equivalent to the average open source LLM all the way up to massive models \u2014 don\u2019t rely on rote memorization of training data to generate outputs. <\/p>\n

\n

This work is just the beginning. We hope to analyze the interactions between pretraining and finetuning, and combine influence functions with mechanistic interpretability to reverse engineer the associated circuits. You can read more on our blog: https:\/\/t.co\/sZ3e0Ud3en<\/p>\n

\u2014 Anthropic (@AnthropicAI) August 8, 2023<\/a><\/p><\/blockquote>\n

The confluence of neural network layers along with the massive size of the datasets means the scope of this current research is limited to pre-trained models that haven\u2019t been fine-tuned. Its results aren\u2019t quite applicable to Claude 2 or GPT-4 yet, but this research appears to be a stepping stone in that direction. <\/p>\n

Going forward, the team hopes to apply these techniques to more sophisticated models and, eventually, to develop a method for determining exactly what each neuron in a neural network is doing as a model functions.<\/p>\n