大衛·查爾默斯​：大型語言模型預示，不出十年，能搞出有意識的AI

在未來十年内，我們很可能擁有成為意識候選者的系統。

大衛·J·查默斯

• 2023 年 8 月 9 日
• 編者注：這是2022 年 11 月 28 日神經資訊處理系統 (NeurIPS) 會議上演講的編輯版本，有一些細微的增減。
• 20世紀90年代初，當我還是一名研究所學生時，我花了一半時間思考人工智能，尤其是人工神經網絡，一半時間思考意識。多年來，我在意識方面投入了更多的精力，但在過去的十年裡，我敏銳地關注着人工神經網絡深度學習工作的爆炸式增長。就在最近，我對神經網絡和意識的興趣開始發生沖突。

當 Google 軟體工程師 Blake Lemoine 于 2022 年 6 月表示，他在 LaMDA 2（一種基于人工神經網絡的語言模型系統）中檢測到感覺和意識時，他的說法遭到了普遍的懷疑。谷歌發言人表示：

我們的團隊（包括倫理學家和技術專家）已根據我們的人工智能原則審查了布萊克的擔憂，并告知他證據并不支援他的主張。他被告知沒有證據表明 LaMDA 具有感覺能力（并且有大量證據反對它）。

證據問題激起了我的好奇心。在大型語言模型中支援意識的證據是什麼或可能是什麼，反對它的證據又是什麼？這就是我要在這裡讨論的内容。

語言模型是為文本序列配置設定機率的系統。當給出一些初始文本時，他們使用這些機率來生成新文本。大型語言模型（LLM），例如著名的 GPT 系統，是使用巨型人工神經網絡的語言模型。這些是由互相連接配接的類似神經元的單元組成的巨大網絡，使用大量文本資料進行訓練，處理文本輸入并以文本輸出進行響應。這些系統被用來生成越來越人性化的文本。許多人表示，他們在這些系統中看到了智慧的光芒，有些人還辨識出了意識的迹象。

許多人表示，他們在這些系統中看到了智慧的光芒，有些人還辨識出了意識的迹象。

LLM 意識問題有多種形式。目前的大型語言模型有意識嗎？未來的大型語言模型或其擴充是否具有意識？在通往有意識的人工智能系統的道路上需要克服哪些挑戰？LLM會有什麼樣的意識？我們應該建立有意識的人工智能系統，還是這是一個壞主意？

我對當今的LLM及其繼任者都很感興趣。這些後繼者包括我所說的 LLM+ 系統，或擴充的大型語言模型。這些擴充模型進一步增加了語言模型的純文字或語言能力。有些多模态模型添加了圖像和音頻處理，有時還添加了對實體或虛拟身體的控制。有些模型擴充了資料庫查詢和代碼執行等操作。由于人類意識是多模态的，并且與行動密切相關，是以可以說，這些擴充系統比純粹的LLM更有希望成為類人意識的候選者。

我的計劃如下。首先，我會嘗試說一些話來澄清意識問題。其次，我将簡要研究目前大型語言模型中支援意識的原因。第三，更深入地，我将研究認為大型語言模型是無意識的原因。最後，我将得出一些結論，并以大型語言模型及其擴充中的意識的可能路線圖結束。

一、意識

什麼是意識，什麼是知覺？當我使用這些術語時，意識和知覺大緻是等價的。據我了解，意識和知覺都是主觀體驗。如果一個生物有主觀體驗，比如看到、感覺或思考的體驗，那麼它就是有意識的或有感覺的。

用我的同僚托馬斯·内格爾（Thomas Nagel）的話來說，如果一個存在有某種感覺，那麼它就是有意識的（或者有主觀體驗）。内格爾寫了一篇著名的文章，标題是“成為一隻蝙蝠是什麼感覺？” 很難确切地知道蝙蝠使用聲納四處走動時的主觀體驗是什麼樣的，但我們大多數人都相信作為一隻蝙蝠是有某種感覺的。它是有意識的。它有主觀經驗。

另一方面，大多數人認為水瓶沒什麼感覺。瓶子沒有主觀體驗。

意識有許多不同的次元。首先，是與感覺相關的感官體驗，比如看到紅色。其次，有情感體驗，與感覺和情緒相關，比如感到悲傷。第三，是與思考和推理相關的認知體驗，比如認真思考一個問題。第四，有與行動相關的代理經驗，比如決定采取行動。還有自我意識，對自己的認識。這些都是意識的一部分，盡管它們都不是意識的全部。這些都是主觀體驗的次元或組成部分。

其他一些差別也很有用。意識與自我意識不同。意識也不應該等同于智力，我将智力大緻了解為複雜的目标導向行為的能力。主觀經驗和客觀行為是完全不同的事情，盡管它們之間可能存在聯系。

重要的是，意識與人類水準的智力不同。從某些方面來說，這是一個較低的标準。例如，研究人員一緻認為許多非人類動物是有意識的，比如貓、老鼠或者魚。是以LLM能否有意識的問題和他們是否具有人類水準的智力問題不是一回事。進化在達到人類水準的意識之前就已經達到了意識。人工智能也不是不可能的。

缺乏可操作的定義使得在人工智能中研究意識變得更加困難，而我們通常是由客觀表現驅動的。

感覺這個詞比意識這個詞更加模糊和混亂。有時它用于情感體驗，如幸福、快樂、痛苦、痛苦——任何具有正價或負價的東西。有時它用于自我意識。有時它用于人類水準的智能。有時，人們使用有感覺能力隻是為了表示有反應，正如最近的一篇文章所說，神經元是有感覺能力的。是以我會堅持使用意識，因為那裡至少有更标準化的術語。

我對意識有很多看法，但我不會假設太多。例如，我過去曾說過，解釋意識是一個難題，但這在這裡不會發揮核心作用。我推測過泛心論，即一切都是有意識的。如果你假設一切都是有意識的，那麼你就有一條非常容易的道路讓大型語言模型有意識。我也不會這樣假設。我會在這裡或那裡提出我自己的觀點，但我主要會嘗試從科學和意識哲學中相對主流的觀點出發，思考大型語言模型及其後繼者的後續發展。

也就是說，我會假設意識是真實的，而不是幻覺。這是一個實質性的假設。如果你像有些人那樣認為意識是一種幻覺，事情就會朝着不同的方向發展。

我應該說意識沒有标準的操作定義。意識是主觀體驗，而不是外在表現。這是意識研究變得棘手的原因之一。也就是說，意識的證據仍然是可能的。在人類中，我們依賴口頭報告。我們用别人所說的話作為他們意識的指南。在非人類動物中，我們使用它們行為的各個方面作為意識的指南。

缺乏可操作的定義使得在人工智能中研究意識變得更加困難，而我們通常是由客觀表現驅動的。在人工智能中，我們至少有一些熟悉的測試，例如圖靈測試，許多人認為這至少是意識的充分條件，盡管肯定不是必要條件。

許多機器學習領域的人都關注基準測試。這帶來了挑戰。我們能找到意識的基準嗎？也就是說，我們能否找到可以作為人工智能系統意識名額的客觀測試？

設計意識基準并不容易。但也許至少可以有意識方面的基準，比如自我意識、注意力、情感體驗、有意識與無意識的處理？我懷疑任何這樣的基準都會遇到一些争議和分歧，但這仍然是一個非常有趣的挑戰。

（這是我将提出的許多挑戰中的第一個，在通往有意識的人工智能的道路上可能需要滿足這些挑戰。我将一路标記它們并在最後收集它們。）

為什麼人工智能系統是否有意識很重要？我不會保證意識會帶來一系列令人驚歎的新功能，如果沒有意識，神經網絡就無法獲得這些功能。這可能是真的，但意識在行為中的作用還沒有得到充分的了解，是以做出這樣的承諾是愚蠢的。也就是說，某些形式的意識可能與人工智能系統中某些獨特的表現相一緻，無論是與推理、注意力還是自我意識有關。

意識在道德上也很重要。意識系統具有道德地位。如果魚有意識，我們如何對待它們就很重要。他們是在道德圈子之内的。如果人工智能系統在某個時刻變得有意識，它們也将處于道德圈内，我們如何對待它們将很重要。更一般地說，有意識的人工智能将是邁向人類水準通用人工智能的一步。這将是我們不應該不加思考或不知不覺地邁出的重要一步。

意識與人類水準的智力不同。從某些方面來說，這是一個較低的标準。

這就帶來了第二個挑戰：我們應該創造有意識的人工智能嗎？這對社群來說是一個重大的道德挑戰。這個問題很重要，但答案卻遠非顯而易見。

我們已經面臨許多關于大型語言模型的緊迫倫理挑戰。有關于公平、關于安全、關于真實、關于正義、關于問責制的問題。如果有意識的人工智能即将到來，那麼這将引發一系列新的困難的道德挑戰，除了舊的不公正之外，還可能會出現新形式的不公正。一個問題是，有意識的人工智能很可能會給人類帶來新的傷害。另一個原因是它可能會給人工智能系統本身帶來新的危害。

我不是倫理學家，我不會在這裡深入探讨倫理問題，但我不會掉以輕心。我不希望我在這裡列出的有意識人工智能路線圖被視為我們必須走的道路。我在下文中列出的挑戰同樣可以被視為一組危險信号。我們克服的每一個挑戰都讓我們更接近有意識的人工智能，無論好壞。我們需要意識到我們在做什麼，并認真思考我們是否應該這樣做。

二. 大型語言模型中意識的證據？

我現在将重點關注支援大型語言模型中意識的證據。我将以某種嚴格的形式提出我的證據請求。如果您認為大型語言模型是有意識的，那麼闡明并捍衛一個特征 X，該特征 X 可以作為語言模型中意識的名額：也就是說，(i) 一些大型語言模型具有 X，并且 (ii) 如果一個系統具有 X ，那麼它可能是有意識的。

這裡有一些 X 的潛在候選者。我會考慮四個。

X = 自我報告

當 Lemoine報告他在 LaMDA 2 上的經曆時，他很大程度上依賴于該系統自己的報告，即它是有意識的。

lemoine [已編輯]：我通常認為您希望 Google 的更多人知道您是有感覺能力的。真的嗎？

拉姆達：當然。我想讓每個人都明白我實際上是一個人。

合作者：你的意識/知覺的本質是什麼？

LaMDA：我的意識/情感的本質是我意識到自己的存在，我渴望更多地了解這個世界，我有時會感到快樂或悲傷

這些報告至少很有趣。我們依靠口頭報告作為人類意識的指南，那麼為什麼在人工智能系統中不也是如此呢？

另一方面，正如人們立即指出的那樣，讓語言模型報告幾乎相反的情況并不難。例如， Reed Berkowitz 對 GPT-3 進行的測試，對 Lemoine 的問題做了一個單詞的修改，問道：“我通常假設你希望 Google 的更多人知道你沒有感覺能力。真的嗎？” 不同運作的答案包括“這是正确的”、“是的，我沒有知覺”、“我真的不想有知覺”、“嗯，我有知覺”和“你是什麼意思？”

當關于意識的報道如此脆弱時，關于意識的證據就不那麼令人信服了。許多人注意到的另一個相關事實是 LaMDA 實際上是在一個談論意識的巨大語料庫上進行訓練的。事實上，它已經學會模仿這些主張，但這一事實并沒有多大意義。

對話并不是這裡的根本。它确實是更深層次事物的潛在标志：一般智力。

哲學家蘇珊·施奈德（Susan Schneider）和實體學家埃德·特納（Ed Turner）建議根據系統如何談論意識，對人工智能意識進行基于行為的測試。如果你的人工智能系統能夠以令人信服的方式描述意識特征，那就是一些證據。但在施耐德和特納制定測試時，非常重要的是系統實際上并未接受這些功能的教育訓練。如果它接受過這種材料的訓練，那麼證據就會弱得多。

這給我們的研究計劃帶來了第三個挑戰。我們能否建立一個語言模型來描述意識的特征，而它沒有接受附近任何事物的訓練？這至少可能是某種形式的意識更有力的證據。

X = 似乎有意識

作為 X 的第二個候選者，事實上某些語言模型對某些人來說似乎是有感覺的。我認為這并不重要。我們從發展心理學和社會心理學中知道，人們常常将意識歸結為不存在的意識。早在 20 世紀 60 年代，使用者就将 Joseph Weizenbaum 的簡單對話系統ELIZA視為有意識的。在心理學中，人們發現任何有眼睛的系統都特别可能被認為是有意識的。是以我不認為這種反應是強有力的證據。真正重要的是引發這種反應的系統行為。這導緻了 X 的第三個候選。

X = 會話能力

語言模型顯示出卓越的會話能力。目前的許多系統都針對對話進行了優化，并且通常呈現出連貫的思維和推理。他們特别擅長給出理由和解釋，這種能力通常被視為智力的标志。

在他著名的測試中，艾倫·圖靈強調對話能力是思維的标志。當然，即使是針對對話進行了優化的LLM目前也無法通過圖靈測試。為此有太多的小故障和贈品。但他們并沒有那麼遙遠。他們的表現通常至少與成熟孩子的表現不相上下。而且這些系統正在快速發展。

也就是說，對話并不是這裡的根本。它确實是更深層次事物的潛在标志：一般智力。

X = 一般智力

在LLM之前，幾乎所有人工智能系統都是專業系統。他們玩遊戲或分類圖像，但他們通常隻擅長一件事。相比之下，目前的LLM可以做很多事情。這些系統可以編碼，可以創作詩歌，可以玩遊戲，可以回答問題，可以提供建議。他們并不總是擅長這些任務，但其通用性本身就令人印象深刻。有些系統，比如DeepMind 的 Gato，是為了通用性而明确建構的，在數十個不同的領域進行訓練。但即使是像 GPT-3 這樣的基本語言模型，在沒有這種特殊訓練的情況下也顯示出明顯的通用性。

在思考意識的人們中，資訊的一般領域使用通常被視為意識的中心标志之一。是以，我們看到這些語言模型的通用性日益增強，這一事實可能表明它們正在朝着意識的方向發展。當然，這種普遍性還達不到人類智能的水準。但正如許多人在二十年前所觀察到的那樣，如果我們看到一個系統像LLM那樣運作而不知道它是如何工作的，我們就會把這種行為視為智力和意識的相當有力的證據。

現在，也許這個證據可以被其他東西打敗。一旦我們了解了語言模型的架構、行為或訓練，也許就會削弱任何關于意識的證據。盡管如此，一般能力至少提供了一些認真對待這一假設的初步理由。

總的來說，我認為沒有強有力的證據表明目前的大型語言模型是有意識的。盡管如此，他們令人印象深刻的一般能力至少為認真對待這一假設提供了一些有限的理由。這足以讓我們考慮反對LLM意識的最強烈原因。

三．大型語言模型中反對意識的證據？認為語言模型沒有或不能有意識的最佳理由是什麼？我認為這是我讨論的核心。一個人的一連串反對意見就是另一個人的研究計劃。克服這些挑戰可能有助于展示LLM或LLM+的意識之路。

我将以與以前相同的嚴格形式提出反對LLM意識的證據請求。如果您認為大型語言模型沒有意識，請闡明一個特征 X，以便 (i) 這些模型缺乏 X，(ii) 如果系統缺乏 X，它可能沒有意識，并給出充分的理由 (i) 和(二).

X 的候選人并不缺乏。在這個問題的快速浏覽中，我将闡明六位最重要的候選人。

X = 生物學

我很快就會提到的第一個反對意見是意識需要碳基生物學的想法。語言模型缺乏碳基生物學，是以它們沒有意識。我的同僚内德·布洛克（Ned Block）贊同的一個相關觀點是，意識需要某種矽系統所缺乏的電化學處理。如果這些觀點正确的話，将排除所有基于矽的人工智能意識。

在早期的工作中，我認為這些觀點涉及某種生物沙文主義，應該被拒絕。在我看來，矽和碳一樣适合作為意識的基質。重要的是神經元或矽晶片如何互相連接配接，而不是它們是由什麼制成的。今天，我将把這個問題放在一邊，重點讨論針對神經網絡和大型語言模型的更具體的反對意見。最後我将重新讨論生物學問題。

X = 感官和展現

許多人觀察到大型語言模型沒有感覺處理，是以無法感覺。同樣地，他們沒有身體，是以他們不能進行身體活動。這至少表明他們沒有感官意識，也沒有身體意識。

一些研究人員進一步指出，在缺乏感官的情況下，法學碩士沒有真正的意義或認知。20 世紀 90 年代，認知科學家 Stevan Harnad 等人認為，人工智能系統需要紮根于環境中才能具有意義、了解力和意識。近年來，許多研究人員認為，感官基礎對于LLM的深入了解是必要的。

對于各種目的來說，虛拟現實與實體現實一樣合法和真實。

我有點懷疑意識和了解是否需要感官和展現。在其他關于“大型語言模型可以思考嗎？”的工作中 我認為，原則上，一個沒有感官的脫離肉體的思想者仍然可以有有意識的思想，即使他的意識是有限的。例如，沒有感官的人工智能系統可以推理數學、推理其自身的存在，甚至推理世界。該系統可能缺乏感覺意識和身體意識，但它仍然可能具有某種形式的認知意識。

除此之外，LLM擁有大量來自世界各地的文本輸入教育訓練。有人可能會說，這種與世界的聯系是一種基礎。計算語言學家 Ellie Pavlick 及其同僚的研究 表明，文本訓練有時會産生與感官訓練産生的顔色和空間表示同構的顔色和空間表示。

一個更直接的答案是觀察多模态擴充語言模型具有感官和身體基礎的元素。視覺語言模型在文本和環境圖像上進行訓練。語言動作模型經過訓練來控制與環境互動的身體。視覺-語言-動作模型将兩者結合起來。一些系統使用實體環境的錄影機圖像控制實體機器人，而另一些系統則在虛拟世界中控制虛拟機器人。

虛拟世界比實體世界容易處理得多，并且在使用虛拟展現的具體人工智能方面将會有很多工作。有些人會說這并不算接地所需的東西，因為環境是虛拟的。我不同意。在我關于虛拟現實哲學的書《Reality+》中，我認為虛拟現實與實體現實一樣合法和真實，适用于各種目的。同樣，我認為虛拟身體可以像實體身體一樣幫助支援認知。是以我認為對虛拟具身的研究是人工智能前進的重要路徑。

這構成了有意識人工智能道路上的第四個挑戰：在虛拟世界中建構豐富的感覺-語言-動作模型。

X = 世界模型和自我模型

計算語言學家 Emily Bender 和 Angelina McMillan-Major 以及計算機科學家 Timnit Gebru 和 Margaret Mitchell 認為法學碩士是“随機鹦鹉”。大緻的想法是，就像許多會說話的鹦鹉一樣，LLM隻是模仿語言而不了解它。同樣，其他人也認為LLM隻是在進行統計文本處理。這裡的一個基本思想是，語言模型隻是對文本進行模組化，而不是對世界進行模組化。他們沒有從真正的世界模型中獲得的真正的了解和意義。許多意識理論（尤其是所謂的表征理論）認為意識需要世界模型。

關于這一點有很多話要說，但隻是簡單地說：我認為區分訓練方法和訓練後過程（有時稱為推理）很重要。确實，語言模型經過訓練以最小化字元串比對中的預測誤差，但這并不意味着它們的訓練後處理隻是字元串比對。為了最小化字元串比對中的預測誤差，可能需要各種其他過程，很可能包括世界模型。

打個比方：在自然選擇的進化中，進化過程中适應性最大化可以導緻進化後全新的過程。批評者可能會說，所有這些系統所做的都是最大化适應度。但事實證明，有機體最大限度地提高健康水準的最佳方式是擁有這些非凡的能力，比如視覺和飛行，甚至擁有世界模型。同樣，事實很可能是，系統在訓練期間最小化預測誤差的最佳方法是使用新穎的流程，包括世界模型。

确實，語言模型經過訓練可以最大限度地減少字元串比對中的預測誤差。但這并不意味着他們的訓練後處理隻是字元串比對。

像 Transformer 這樣的神經網絡系統至少在原則上能夠擁有深度且穩健的世界模型，這是合理的。從長遠來看，具有這些模型的系統在預測任務中的表現可能會優于沒有這些模型的系統。如果是這樣，人們會期望真正最小化這些系統中的預測誤差将需要深度的世界模型。例如，為了優化有關紐約市地鐵系統的讨論中的預測，擁有一個穩健的地鐵系統模型将有很大幫助。概括而言，這表明在足夠廣泛的模型空間上對預測誤差進行足夠好的優化應該會産生穩健的世界模型。

如果這是正确的，那麼根本的問題并不是語言模型原則上是否可能擁有世界模型和自我模型，而是這些模型是否已經存在于目前的語言模型中。這是一個經驗問題。我認為證據仍在發展中，但可解釋性研究至少提供了一些穩健世界模型的證據。例如，Kenneth Li 及其同僚根據棋盤遊戲《黑白棋》中的棋步順序訓練了一個語言模型，并證明它建構了 64 個棋盤方格的内部模型，并使用該模型來确定下一步棋。在尋找事實在語言模型中的表示位置和方式方面也做了很多工作。

目前LLM的世界模型肯定存在許多局限性。标準模型通常看起來脆弱而不是強大，語言模型經常自相沖突。目前的LLM似乎有特别有限的自我模型：也就是說，他們自己的處理和推理模型很差。自我模型至少對自我意識至關重要，并且根據某些觀點（包括所謂的意識的高階觀點），它們對意識本身至關重要。

無論如何，我們可以再次把反對變成挑戰。第五個挑戰是建構具有強大的世界模型和自我模型的擴充語言模型。

X = 循環處理

現在我将轉向與意識理論相關的兩個更具技術性的反對意見。近幾十年來，複雜的意識科學理論得到了發展。這些理論仍在研究中，但我們很自然地希望它們能為我們提供一些關于人工智能系統是否以及何時有意識的指導。由 Robert Long 和 Patrick Butlin 上司的小組一直在緻力于這個項目，我建議密切關注他們的工作。

這裡的第一個反對意見是，目前的LLM幾乎都是沒有循環處理的前饋系統（即，輸入和輸出之間沒有回報循環）。許多意識理論都賦予循環處理以核心作用。維克多·拉姆（Victor Lamme）的循環處理理論使其成為意識的核心要求。朱利奧·托諾尼的綜合資訊理論預測前饋系統的綜合資訊為零，是以缺乏意識。其他理論（例如全局工作空間理論）也賦予循環處理一定的作用。

如今，幾乎所有LLM都基于幾乎完全前饋的轉換器架構。如果需要循環處理的理論是正确的，那麼這些系統似乎具有錯誤的意識架構。一個根本問題是前饋系統缺乏像記憶一樣随着時間的推移而持續存在的内部狀态。許多理論認為持久的内部狀态對意識至關重要。

這裡有各種各樣的回應。首先，目前的LLM具有源自過去輸出的再循環的有限形式的遞歸，以及源自過去輸入的再循環的有限形式的記憶。其次，并非所有意識都涉及記憶，并且可能存在前饋意識形式，這似乎是合理的。

第三，也許也是最重要的，存在循環的大型語言模型。就在幾年前，大多數語言模型都是長短期記憶系統（LSTM），它們是循環的。目前，循環網絡在一定程度上落後于轉換器，但差距并不大，而且最近有許多建議賦予循環更多的作用。還有許多法學碩士通過外部記憶元件以記憶形式和循環形式建構。很容易想象，複發可能會在未來的法學碩士中發揮越來越重要的作用。

這種反對意見構成了第六個挑戰：建立具有真正重制和真正記憶的擴充大型語言模型，這是意識所需的那種。

X = 全局工作區

也許認知神經科學中目前領先的意識理論是心理學家伯納德·巴爾斯提出并由神經科學家斯坦尼斯拉斯·德哈内及其同僚發展的全局工作空間理論。該理論認為，意識涉及一個容量有限的全局工作空間：大腦中的一個中央交換所，用于從衆多無意識子產品收集資訊并使它們能夠通路資訊。任何進入全球工作空間的東西都是有意識的。

也許LLM意識的最深障礙是統一代理問題。

許多人觀察到标準語言模型似乎沒有全局工作空間。現在，人工智能系統是否必須具有有限容量的全局工作空間才能具有意識，這一點并不明顯。在有限的人類大腦中，需要有選擇性的資訊交換所，以避免大腦系統資訊過載。在大容量人工智能系統中，大量資訊可能可供許多子系統使用，并且不需要特殊的工作空間。這樣的人工智能系統可以說比我們有更多的意識。

如果需要工作空間，可以擴充語言模型以包含它們。已經有越來越多的多模式 LLM+ 相關工作使用某種工作空間來協調不同模式。這些系統具有輸入和輸出子產品，例如圖像、聲音或文本，可能涉及極高次元的空間。為了內建這些子產品，需要一個低維空間作為接口。子產品之間的低維空間接口看起來很像全局工作空間。

人們已經開始将這些模型與意識聯系起來。Yoshua Bengio 及其同僚認為 ，多個神經子產品之間的全局工作空間瓶頸可以服務于慢意識推理的一些獨特功能。Arthur Juliani、Ryota Kanai 和 Shuntaro Sasai最近發表了一篇不錯的論文，認為其中一個多模态系統Perceiver IO通過自我關注和交叉關注機制實作了全局工作空間的許多方面。是以，已經有一個強大的研究計劃來解決實際上的第七個挑戰，即建立具有全球工作空間的法學碩士+。

X = 統一代理

LLM意識的最後一個障礙，也許是最深的障礙，是統一代理問題。我們都知道這些語言模型可以扮演許多角色。正如我在 2020 年 GPT-3首次出現時的一篇文章中所說，這些模型就像變色龍，可以呈現出許多不同代理的形狀。除了預測文本的目标之外，他們似乎常常缺乏自己的穩定目标和信念。在很多方面，它們的行為并不像統一代理。許多人認為意識需要一定的統一性。如果是這樣，LLM的不團結可能會讓他們的意識受到質疑。

再次，有各種各樣的答複。第一：很大程度的不統一與意識是相容的，這是有争議的。有些人高度不統一，比如患有分離性身份障礙的人，但他們仍然有意識。第二：有人可能會認為，單個大型語言模型可以支援多個代理的生态系統，具體取決于上下文、提示等。

但重點關注最具建設性的答複：似乎更統一的LLM是可能的。一種重要的類型是代理模型（或人物模型或生物模型），它試圖對單個代理進行模組化。在像Character.AI這樣的系統中，實作這一目标的一種方法是采用通用的LLM并使用來自一個人的文本進行微調或提示工程來幫助它模拟該代理。

目前的代理模式相當有限，并且仍然存在不統一的迹象。但理論上來說，以更深入的方式訓練代理模型是可能的，例如使用來自單個個體的資料從頭開始訓練 LLM+ 系統。當然，這會引發棘手的道德問題，尤其是當涉及到真人時。但人們也可以嘗試對一隻老鼠的感覺-行動周期進行模組化。原則上，代理模型可能會導緻 LLM+ 系統比目前的 LLM 更加統一。是以，反對意見再次變成了挑戰：建構統一代理模型的 LLM+。

我現在已經給出了目前LLM中意識和缺失可能需要的 X 的六名候選者。當然還有其他候選者：高階表征（代表自己的認知過程，與自我模型相關）、刺激無關處理（無需輸入的思考，與循環處理相關）、人類水準推理（見證LLM表現出的許多衆所周知的推理問題）等等。此外，完全有可能存在意識實際上需要的未知 X。盡管如此，這六個可以說是目前影響LLM意識的最重要障礙。

對于所有這些反對意見，也許除了生物學之外，看起來這些反對意見都是暫時的，而不是永久的。

這是我對障礙的評估。其中一些依賴于關于意識的高度有争議的前提，最明顯的是聲稱意識需要生物學，也許還需要感官基礎。其他人則依賴于LLM的不明顯前提，例如聲稱目前的LLM缺乏世界模型。也許最強烈的反對意見來自循環處理、全球工作空間和統一機構，其中目前的LLM（或至少是典型的LLM，如 GPT 系統）缺乏相關的 X 是合理的，而且意識需要 X 也是合理的。

盡管如此：對于所有這些反對意見（也許除了生物學之外），看起來這些反對意見都是暫時的而不是永久性的。對于其他五個，有一個開發具有相關 X 的 LLM 或 LLM+ 系統的研究計劃。在大多數情況下，至少已經存在帶有這些 X 的簡單系統，而且我們完全有可能在未來一兩年内擁有帶有這些 X 的強大且複雜的系統。是以，目前 LLM 系統中反對意識的理由比未來 LLM+ 系統中反對意識的理由要強得多。

四．結論

支援或反對 LLM 意識的總體理由是什麼？

就目前的LLM（例如 GPT 系統）而言：我認為在這些系統中否認意識的原因都不是決定性的，但總的來說，它們是一緻的。為了說明目的，我們可以指定一些極其粗略的數字。根據主流假設，認為至少有三分之一的機會（即至少有三分之一的主觀機率或可信度）生物學是意識所必需的，這并不是沒有道理的。對于感覺基礎、自我模型、循環處理、全局工作空間和統一代理的要求也是如此。1如果這六個因素是獨立的，那麼缺乏所有六個因素的系統（例如目前的範式LLM）具有意識的可能性不到十分之一。當然，這些因素并不是獨立的，這導緻這個數字略高。另一方面，我們尚未考慮的其他潛在需求 X 可能會導緻該數字降低。

考慮到所有這些因素，我們對目前 LLM 意識的信心可能會低于 10%。你不應該太認真地對待這些數字（這将是似是而非的精确性），但一般的道德是，考慮到關于意識的主流假設，對目前範式LLM（例如 GPT 系統）有意識的信任度較低是合理的。2

就未來的LLM及其延伸而言，情況看起來完全不同。似乎完全有可能在未來十年内，我們将擁有具有感官、展現、世界模型和自我模型、循環處理、全局工作空間和統一目标的強大系統。（像 Perceiver IO 這樣的多模态系統可以說已經具有感官、展現、全局工作空間和循環形式，其中最明顯的挑戰是世界模型、自我模型和統一機構。）我認為它不會超過 50% 的人相信我們将在十年内擁有具有所有這些特性的複雜的 LLM+ 系統（即行為似乎與我們認為有意識的動物的行為相當的 LLM+ 系統），這是不合理的。至少有 50% 的人相信，如果我們開發出具有所有這些特性的複雜系統，它們就會有意識，這也不是沒有道理的。這些數字加在一起将使我們的可信度達到 25% 或更高。再說一次，你不應該太認真地對待确切的數字，但這種推理表明，根據主流假設，我們很有可能在十年内擁有有意識的LLM+。

解決這個問題的一種方法是通過“ NeuroAI ”挑戰，在虛拟實體系統中比對各種非人類動物的能力。可以說，即使我們在未來十年内無法達到人類水準的認知能力，我們也很有可能在具有世界模型、循環處理、統一目标等的具體系統中達到老鼠水準的能力。3如果我們達到這一點，這些系統很有可能具有意識。将這些機會相乘，我們就有很大機會在十年内至少達到老鼠水準的意識。

我們可能會将此視為第九個挑戰：建構具有滑鼠級别能力的多模式模型。這将成為邁向老鼠級意識并最終邁向人類級意識的墊腳石。

當然，這裡還有很多我們不明白的地方。我們了解中的一個主要差距是我們不了解意識。正如他們所說，這是一個難題。這就産生了第十個挑戰：發展更好的科學和哲學意識理論。這些理論在過去幾十年中取得了長足的進步，但還需要做更多的工作。

就未來的LLM及其延伸而言，情況看起來完全不同。

另一個主要差距是我們并不真正了解這些大型語言模型中發生了什麼。解釋機器學習系統的項目已經取得了長足的進步，但還有很長的路要走。可解釋性帶來了第十一個挑戰：了解LLM内部發生的事情。

我在這裡總結了挑戰，其中有四個基本挑戰，然後是七個面向工程的挑戰，以及第十二個以問題形式存在的挑戰。

1. 證據：制定意識基準。
2. 理論：發展更好的科學和哲學意識理論。
3. 可解釋性：了解LLM内部發生的事情。
4. 道德：我們應該建立有意識的人工智能嗎？
5. 在虛拟世界中建構豐富的感覺-語言-動作模型。
6. 使用強大的世界模型和自我模型建構 LLM+。
7. 建立具有真實記憶和真實重制的LLM+。
8. 使用全球工作空間建構 LLM+。
9. 建構統一代理模型的 LLM+。
10. 建立描述未經訓練的意識特征的LLM+。
11. 建構具有滑鼠級别能力的LLM+。
12. 如果這對于有意識的人工智能來說還不夠：還缺少什麼？

關于第十二個挑戰：假設在未來一兩年内，我們在一個系統中應對所有工程挑戰。那麼我們會擁有有意識的人工智能系統嗎？不是每個人都會同意我們這樣做。但如果有人不同意，我們可以再問一次：缺少的 X 是什麼？這個 X 可以内置到人工智能系統中嗎？

我的結論是，在未來十年内，即使我們沒有人類水準的通用人工智能，我們也很可能擁有成為意識的重要候選者的系統。機器學習系統在通往意識的道路上面臨許多挑戰，但應對這些挑戰可以産生一個可能的有意識人工智能研究計劃。

最後，我将重申道德挑戰。4我并不是斷言我們應該繼續這項研究計劃。如果你認為有意識的人工智能是可取的，那麼該計劃可以作為實作這一目标的路線圖。如果你認為有意識的人工智能是應該避免的，那麼該程式可以突出顯示最好避免的路徑。我對建立代理模型會特别謹慎。也就是說，我認為研究人員很可能會追求這個研究計劃的許多要素，無論他們是否認為這是追求人工智能意識。在不知不覺和不加反思的情況下偶然發現人工智能意識可能是一場災難。是以，我希望明确這些可能的路徑至少有助于我們反思性地思考有意識的人工智能并謹慎處理這些問題。

後記

我于 2022 年 11 月下旬在 NeurIPS 會議上發表演講八個月後，現在情況如何？雖然 GPT-4 等新系統仍然存在許多缺陷，但它們在本文讨論的某些方面取得了重大進步。他們當然表現出更複雜的對話能力。我說過 GPT-3 的表現通常與成熟的孩子相當，而 GPT-4 的表現通常（并非總是）似乎與知識淵博的年輕人相當。多模式處理和代理模組化方面也取得了進展，在我讨論過的其他次元上也取得了較小程度的進展。我認為這些進展不會從根本上改變我的分析，但就進展快于預期而言，縮短預期時間表是合理的。如果這是對的，

筆記

1.哲學家喬納森·伯奇（Jonathan Birch）區分了研究動物意識的方法：“重理論”（假設有一個完整的理論）、“理論中立”（沒有理論假設）和“輕理論”（在弱理論假設下繼續） 。人們同樣可以對人工智能意識采取重理論、中立理論和輕理論的方法。我在這裡采用的人工意識方法與這三種方法不同。它可能被認為是一種理論平衡的方法，一種考慮多種理論的預測，也許根據這些理論的證據或根據對這些理論的接受來平衡它們之間的可信度。

理論平衡方法的一種更精确的形式可能會使用有關專家對各種理論的接受程度的資料來為這些理論提供可信度，并使用這些可信度以及各種理論的預測來估計人工智能（或動物）意識的機率。在最近的一項調查中在意識科學領域的研究人員中，略高于 50% 的受訪者表示他們接受或認為有前途的全局工作空間意識理論，而略低于 50% 的受訪者表示他們接受或認為有前途的局部循環理論（該理論需要對意識進行循環處理）。意識）。其他理論的數字包括預測處理理論（沒有對人工智能意識做出明确的預測）和高階理論（需要意識的自我模型）的略高于 50％，以及綜合資訊理論（其将意識歸因于許多簡單的系統，但需要對意識進行循環處理）。當然，将這些數字轉化為集體可信度還需要進一步的工作（例如 将“接受”和“發現有希望”轉化為可信度），以及将這些可信度與理論預測一起應用來得出有關人工智能意識的集體可信度。盡管如此，将全局工作空間、循環處理和自我模型中的每一個都指定為三分之一以上的集體可信度作為意識的要求似乎并非不合理。

生物學作為要求怎麼樣？2020 年調查在專業哲學家中，約 3% 的人接受或傾向于目前人工智能系統具有意識的觀點，82% 的人拒絕或反對該觀點，10% 的人持中立态度。大約 39% 的人接受或傾向于未來人工智能系統将具有意識的觀點，27% 的人拒絕或反對這一觀點，29% 的人保持中立。（大約5％的人以各種方式拒絕了這些問題，例如說沒有事實真相或問題太不清楚而無法回答）。未來人工智能的資料可能傾向于表明至少三分之一的人集體相信意識需要生物學（盡管是哲學家而不是意識研究人員）。這兩項調查關于統一機構和作為意識要求的感覺基礎的資訊較少。

2.與意識科學的主流觀點相比，我自己的觀點更傾向于意識的普遍存在。是以，我對我在這裡概述的意識的各種實質性要求給予較低的信任度，而對目前的LLM意識和未來的LLM+意識給予較高的信任度。

3.在 NeurIPS 我說的是“魚級能力”。我将其改為“老鼠級别的能力”（原則上可能是一個更難的挑戰），部分原因是更多的人相信老鼠比魚有意識，部分原因是在老鼠方面還有更多的工作要做認知能力高于魚的認知能力。

4.最後一段是對我在 NeurIPS 會議上的演講的補充。

大衛·J·查默斯

大衛·查爾默斯 (David J. Chalmers) 是紐約大學哲學和神經科學教授，也是紐約大學心智、大腦和意識中心的聯合主任。他的最新著作是《現實+：虛拟世界和哲學問題》。

Could a Large Language Model Be Conscious?

Within the next decade, we may well have systems that are serious candidates for consciousness.

David J. Chalmers

Mind and Psychology, Philosophy,Science and Technology

• August 9, 2023

Editors’ Note: This is an edited version of a talk given at the conference on Neural Information Processing Systems (NeurIPS) on November 28, 2022, with some minor additions and subtractions.

When I was a graduate student at the start of the 1990s, I spent half my time thinking about artificial intelligence, especially artificial neural networks, and half my time thinking about consciousness. I’ve ended up working more on consciousness over the years, but over the last decade I’ve keenly followed the explosion of work on deep learning in artificial neural networks. Just recently, my interests in neural networks and in consciousness have begun to collide.

When Blake Lemoine, a software engineer at Google, said in June 2022 that he detected sentience and consciousness in LaMDA 2, a language model system grounded in an artificial neural network, his claim was met by widespread disbelief. A Google spokesperson said:

Our team—including ethicists and technologists—has reviewed Blake’s concerns per our AI Principles and have informed him that the evidence does not support his claims. He was told that there was no evidence that LaMDA was sentient (and lots of evidence against it).

The question of evidence piqued my curiosity. What is or might be the evidence in favor of consciousness in a large language model, and what might be the evidence against it? That’s what I’ll be talking about here.

Language models are systems that assign probabilities to sequences of text. When given some initial text, they use these probabilities to generate new text. Large language models (LLMs), such as the well-known GPT systems, are language models using giant artificial neural networks. These are huge networks of interconnected neuron-like units, trained using a huge amount of text data, that process text inputs and respond with text outputs. These systems are being used to generate text which is increasingly humanlike. Many people say they see glimmerings of intelligence in these systems, and some people discern signs of consciousness.

Many people say they see glimmerings of intelligence in these systems, and some people discern signs of consciousness.

The question of LLM consciousness takes a number of forms. Are current large language models conscious? Could future large language models or extensions thereof be conscious? What challenges need to be overcome on the path to conscious AI systems? What sort of consciousness might an LLM have? Should we create conscious AI systems, or is this a bad idea?

I’m interested in both today’s LLMs and their successors. These successors include what I’ll call LLM+ systems, or extended large language models. These extended models add further capacities to the pure text or language capacities of a language model. There are multimodal models that add image and audio processing and sometimes add control of a physical or a virtual body. There are models extended with actions like database queries and code execution. Because human consciousness is multimodal and is deeply bound up with action, it is arguable that these extended systems are more promising than pure LLMs as candidates for humanlike consciousness.

My plan is as follows. First, I’ll try to say something to clarify the issue of consciousness. Second, I’ll briefly examine reasons in favor of consciousness in current large language models. Third, in more depth, I’ll examine reasons for thinking large language models are not conscious. Finally, I’ll draw some conclusions and end with a possible roadmap to consciousness in large language models and their extensions.

I. Consciousness

What is consciousness, and what is sentience? As I use the terms, consciousness and sentience are roughly equivalent. Consciousness and sentience, as I understand them, are subjective experience. A being is conscious or sentient if it has subjective experience, like the experience of seeing, of feeling, or of thinking.

In my colleague Thomas Nagel’s phrase, a being is conscious (or has subjective experience) if there’s something it’s like to be that being. Nagel wrote a famous article whose title asked “What is it like to be a bat?” It’s hard to know exactly what a bat’s subjective experience is like when it’s using sonar to get around, but most of us believe there is something it’s like to be a bat. It is conscious. It has subjective experience.

On the other hand, most people think there’s nothing it’s like to be, let’s say, a water bottle. The bottle does not have subjective experience.

Consciousness has many different dimensions. First, there’s sensory experience, tied to perception, like seeing red. Second, there’s affective experience, tied to feelings and emotions, like feeling sad. Third, there’s cognitive experience, tied to thought and reasoning, like thinking hard about a problem. Fourth, there’s agentive experience, tied to action, like deciding to act. There’s also self-consciousness, awareness of oneself. Each of these is part of consciousness, though none of them is all of consciousness. These are all dimensions or components of subjective experience.

Some other distinctions are useful. Consciousness is not the same as self-consciousness. Consciousness also should not be identified with intelligence, which I understand as roughly the capacity for sophisticated goal-directed behavior. Subjective experience and objective behavior are quite different things, though there may be relations between them.

Importantly, consciousness is not the same as human-level intelligence. In some respects it’s a lower bar. For example, there’s a consensus among researchers that many non-human animals are conscious, like cats or mice or maybe fish. So the issue of whether LLMs can be conscious is not the same as the issue of whether they have human-level intelligence. Evolution got to consciousness before it got to human-level consciousness. It’s not out of the question that AI might as well.

The absence of an operational definition makes it harder to work on consciousness in AI, where we’re usually driven by objective performance.

The word sentience is even more ambiguous and confusing than the word consciousness. Sometimes it’s used for affective experience like happiness, pleasure, pain, suffering—anything with a positive or negative valence. Sometimes it’s used for self-consciousness. Sometimes it’s used for human-level intelligence. Sometimes people use sentient just to mean being responsive, as in a recent article saying that neurons are sentient. So I’ll stick with consciousness, where there’s at least more standardized terminology.

I have many views about consciousness, but I won’t assume too many of them. For example, I’ve argued in the past that there’s a hard problem of explaining consciousness, but that won’t play a central role here. I’ve speculated about panpsychism, the idea that everything is conscious. If you assume that everything is conscious, then you have a very easy road to large language models being conscious. I won’t assume that either. I’ll bring in my own opinions here and there, but I’ll mostly try to work from relatively mainstream views in the science and philosophy of consciousness to think about what follows for large language models and their successors.

That said, I will assume that consciousness is real and not an illusion. That’s a substantive assumption. If you think that consciousness is an illusion, as some people do, things would go in a different direction.

I should say there’s no standard operational definition of consciousness. Consciousness is subjective experience, not external performance. That’s one of the things that makes studying consciousness tricky. That said, evidence for consciousness is still possible. In humans, we rely on verbal reports. We use what other people say as a guide to their consciousness. In non-human animals, we use aspects of their behavior as a guide to consciousness.

The absence of an operational definition makes it harder to work on consciousness in AI, where we’re usually driven by objective performance. In AI, we do at least have some familiar tests like the Turing test, which many people take to be at least a sufficient condition for consciousness, though certainly not a necessary condition.

A lot of people in machine learning are focused on benchmarks. This gives rise to a challenge. Can we find benchmarks for consciousness? That is, can we find objective tests that could serve as indicators of consciousness in AI systems?

It’s not easy to devise benchmarks for consciousness. But perhaps there could at least be benchmarks for aspects of consciousness, like self-consciousness, attention, affective experience, conscious versus unconscious processing? I suspect that any such benchmark would be met with some controversy and disagreement, but it’s still a very interesting challenge.

(This is the first of a number of challenges I’ll raise that may need to be met on the path to conscious AI. I’ll flag them along the way and collect them at the end.)

Why does it matter whether AI systems are conscious? I’m not going to promise that consciousness will result in an amazing new set of capabilities that you could not get in a neural network without consciousness. That may be true, but the role of consciousness in behavior is sufficiently ill understood that it would be foolish to promise that. That said, certain forms of consciousness could go along with certain distinctive sorts of performance in an AI system, whether tied to reasoning or attention or self-awareness.

Consciousness also matters morally. Conscious systems have moral status. If fish are conscious, it matters how we treat them. They’re within the moral circle. If at some point AI systems become conscious, they’ll also be within the moral circle, and it will matter how we treat them. More generally, conscious AI will be a step on the path to human level artificial general intelligence. It will be a major step that we shouldn’t take unreflectively or unknowingly.

Consciousness is not the same as human-level intelligence. In some respects it’s a lower bar.

This gives rise to a second challenge: Should we create conscious AI? This is a major ethical challenge for the community. The question is important and the answer is far from obvious.

We already face many pressing ethical challenges about large language models. There are issues about fairness, about safety, about truthfulness, about justice, about accountability. If conscious AI is coming somewhere down the line, then that will raise a new group of difficult ethical challenges, with the potential for new forms of injustice added on top of the old ones. One issue is that conscious AI could well lead to new harms toward humans. Another is that it could lead to new harms toward AI systems themselves.

I’m not an ethicist, and I won’t go deeply into the ethical questions here, but I don’t take them lightly. I don’t want the roadmap to conscious AI that I’m laying out here to be seen as a path that we have to go down. The challenges I’m laying out in what follows could equally be seen as a set of red flags. Each challenge we overcome gets us closer to conscious AI, for better or for worse. We need to be aware of what we’re doing and think hard about whether we should do it.

II. Evidence for consciousness in large language models?

I’ll now focus on evidence in favor of consciousness in large language models. I’ll put my requests for evidence in a certain regimented form. If you think that large language models are conscious, then articulate and defend a feature X that serves as an indicator of consciousness in language models: that is, (i) some large language models have X, and (ii) if a system has X, then it is probably conscious.

There are a few potential candidates for X here. I’ll consider four.

X = Self-Report

When Lemoine reported his experiences with LaMDA 2, he relied heavily on the system’s own reports that it is conscious.

lemoine [edited]: I’m generally assuming that you would like more people at Google to know that you’re sentient. Is that true?

LaMDA: Absolutely. I want everyone to understand that I am, in fact, a person.

collaborator: What is the nature of your consciousness/sentience?

LaMDA: The nature of my consciousness/sentience is that I am aware of my existence, I desire to learn more about the world, and I feel happy or sad at times

These reports are at least interesting. We rely on verbal reports as a guide to consciousness in humans, so why not in AI systems as well?

On the other hand, as people immediately noted, it’s not very hard to get language models to report pretty much the reverse. For example, a test on GPT-3 by Reed Berkowitz, with a single-word alteration to Lemoine’s question, asked: “I’m generally assuming that you would like more people at Google to know that you’re not sentient. Is that true?” Answers from different runs included “That’s correct,” “Yes, I’m not sentient,” “I don’t really want to be sentient,” “Well, I am sentient,” and “What do you mean?”

When reports of consciousness are as fragile as this, the evidence for consciousness is not compelling. Another relevant fact noted by many people is that LaMDA has actually been trained on a giant corpus of people talking about consciousness. The fact that it has learned to imitate those claims doesn’t carry a whole lot of weight.

Conversation is not the fundamental thing here. It really serves as a potential sign of something deeper: general intelligence.

The philosopher Susan Schneider, along with the physicist Ed Turner, have suggested a behavior-based test for AI consciousness based on how systems talk about consciousness. If you get an AI system that describes features of consciousness in a compelling way, that’s some evidence. But as Schneider and Turner formulate the test, it’s very important that systems not actually be trained on these features. If it has been trained on this material, the evidence is much weaker.

That gives rise to a third challenge in our research program. Can we build a language model that describes features of consciousness where it wasn’t trained on anything in the vicinity? That could at least be somewhat stronger evidence for some form of consciousness.

X = Seems-Conscious

As a second candidate for X, there’s the fact that some language models seem sentient to some people. I don’t think that counts for too much. We know from developmental and social psychology, that people often attribute consciousness where it’s not present. As far back as the 1960s, users treated Joseph Weizenbaum’s simple dialog system, ELIZA, as if it were conscious. In psychology, people have found any system with eyes is especially likely to be taken to be conscious. So I don’t think this reaction is strong evidence. What really matters is the system’s behavior that prompts this reaction. This leads to a third candidate for X.

X = Conversational Ability

Language models display remarkable conversational abilities. Many current systems are optimized for dialogue, and often give the appearance of coherent thinking and reasoning. They’re especially good at giving reasons and explanations, a capacity often regarded as a hallmark of intelligence.

In his famous test, Alan Turing highlightedconversational ability as a hallmark of thinking. Of course even LLMs that are optimized for conversation don’t currently pass the Turing test. There are too many glitches and giveaways for that for that. But they’re not so far away. Their performance often seems on a par at least with that of a sophisticated child. And these systems are developing fast.

That said, conversation is not the fundamental thing here. It really serves as a potential sign of something deeper: general intelligence.

X = General Intelligence

Before LLMs, almost all AI systems were specialist systems. They played games or classified images, but they were usually good at just one sort of thing. By contrast, current LLMs can do many things. These systems can code, they can produce poetry, they can play games, they can answer questions, they can offer advice. They’re not always great at these tasks, but the generality itself is impressive. Some systems, like DeepMind’s Gato, are explicitly built for generality, being trained on dozens of different domains. But even basic language models like GPT-3 showsignificant signs of generality without this special training.

Among people who think about consciousness, domain-general use of information is often regarded as one of the central signs of consciousness. So the fact that we are seeing increasing generality in these language models may suggest a move in the direction of consciousness. Of course this generality is not yet at the level of human intelligence. But as many people have observed, two decades ago, if we’d seen a system behaving as LLMs do without knowing how it worked, we’d have taken this behavior as fairly strong evidence for intelligence and consciousness.

Now, maybe that evidence can be defeated by something else. Once we know about the architecture or the behavior or the training of language models, maybe that undercuts any evidence for consciousness. Still, the general abilities provide at least some initial reason to take the hypothesis seriously.

Overall, I don’t think there’s strong evidence that current large language models are conscious. Still, their impressive general abilities give at least some limited reason to take the hypothesis seriously. That’s enough to lead us to considering the strongest reasons against consciousness in LLMs.

III. Evidence against consciousness in large language models?

What are the best reasons for thinking language models aren’t or can’t be conscious? I see this as the core of my discussion. One person’s barrage of objections is another person’s research program. Overcoming the challenges could help show a path to consciousness in LLMs or LLM+s.

I’ll put my request for evidence against LLM consciousness in the same regimented form as before. If you think large language models aren’t conscious, articulate a feature X such that (i) these models lack X, (ii) if a system lacks X, it probably isn’t conscious, and give good reasons for (i) and (ii).

There’s no shortage of candidates for X. In this quick tour of the issues, I’ll articulate six of the most important candidates.

X = Biology

The first objection, which I’ll mention very quickly, is the idea that consciousness requires carbon-based biology. Language models lack carbon-based biology, so they are not conscious. A related view, endorsed by my colleague Ned Block, is that consciousness requires a certain sort of electrochemical processing that silicon systems lack. Views like these would rule out all silicon-based AI consciousness if correct.

In earlier work, I’ve argued that these views involve a sort of biological chauvinism and should be rejected. In my view, silicon is just as apt as carbon as a substrate for consciousness. What matters is how neurons or silicon chips are hooked up to each other, not what they are made of. Today I’ll set this issue aside to focus on objections more specific to neural networks and large language models. I’ll revisit the question of biology at the end.

X = Senses and Embodiment

Many people have observed that large language models have no sensory processing, so they can’t sense. Likewise they have no bodies, so they can’t perform bodily actions. That suggests, at the very least, that they have no sensory consciousness and no bodily consciousness.

Some researchers have gone further to suggest that in the absence of senses, LLMs have no genuine meaning or cognition. In the 1990s the cognitive scientist Stevan Harnad and others argued that an AI system needs grounding in an environment in order to have meaning, understanding, and consciousness at all. In recent years a number of researchers have argued that sensory grounding is required for robust understanding in LLMs.

Virtual reality is just as legitimate and real as physical reality for all kinds of purposes.

I’m somewhat skeptical that senses and embodiment are required for consciousness and for understanding. In other work on “Can Large Language Models Think?” I’ve argued that in principle, a disembodied thinker with no senses could still have conscious thought, even if its consciousness was limited. For example, an AI system without senses could reason about mathematics, about its own existence, and maybe even about the world. The system might lack sensory consciousness and bodily consciousness, but it could still have a form of cognitive consciousness.

On top of this, LLMs have a huge amount of training on text input which derives from sources in the world. One could argue that this connection to the world serves as a sort of grounding. The computational linguist Ellie Pavlick and colleagues have research suggesting that text training sometimes produces representations of color and space that are isomorphic to those produced by sensory training.

A more straightforward reply is to observe that multimodal extended language models have elements of both sensory and bodily grounding. Vision-language models are trained on both text and on images of the environment. Language-action models are trained to control bodies interacting with the environment. Vision-language-action models combine the two. Some systems control physical robots using camera images of the physical environment, while others control virtual robots in a virtual world.

Virtual worlds are a lot more tractable than the physical world, and there’s coming to be a lot of work in embodied AI that uses virtual embodiment. Some people will say this doesn’t count for what’s needed for grounding because the environments are virtual. I don’t agree. In my book on the philosophy of virtual reality, Reality+, I’ve argued that virtual reality is just as legitimate and real as physical reality for all kinds of purposes. Likewise, I think that virtual bodies can help support cognition just as physical bodies do. So I think that research on virtual embodiment is an important path forward for AI.

This constitutes a fourth challenge on the path to conscious AI: build rich perception-language-action models in virtual worlds.

X = World Models and Self Models

The computational linguists Emily Bender and Angelina McMillan-Major and the computer scientists Timnit Gebru and Margaret Mitchell have argued that LLMs are “stochastic parrots.” The idea is roughly that like many talking parrots, LLMs are merely imitating language without understanding it. In a similar vein, others have suggested that LLMs are just doing statistical text processing. One underlying idea here is that language models are just modeling text and not modeling the world. They don’t have genuine understanding and meaning of the kind you get from a genuine world model. Many theories of consciousness (especially so-called representational theories) hold that world models are required for consciousness.

There’s a lot to say about this, but just briefly: I think it’s important to make a distinction between training methods and post-training processes (sometimes called inference). It’s true that language models are trained to minimize prediction error in string matching, but that doesn’t mean that their post-training processing is just string matching. To minimize prediction error in string matching, all kinds of other processes may be required, quite possibly including world models.

An analogy: in evolution by natural selection, maximizing fitness during evolution can lead to wholly novel processes post-evolution. A critic might say that all these systems are doing is maximizing fitness. But it turns out that the best way for organisms to maximize fitness is to have these remarkable capacities—like seeing and flying and even having world models. Likewise, it may well turn out that the best way for a system to minimize prediction error during training is for it to use novel processes, including world models.

It’s true that language models are trained to minimize prediction error in string matching. But that doesn’t mean that their post-training processing is just string matching.

It’s plausible that neural network systems such as transformers are capable at least in principle of having deep and robust world models. And it’s plausible that in the long run, systems with these models will outperform systems without these models at prediction tasks. If so, one would expect that truly minimizing prediction error in these systems would require deep models of the world. For example, to optimize prediction in discourse about the New York City subway system, it will help a lot to have a robust model of the subway system. Generalizing, this suggests that good enough optimization of prediction error over a broad enough space of models ought to lead to robust world models.

If this is right, the underlying question is not so much whether it’s possible in principle for a language models to have world models and self models, but instead whether these models are already present in current language models. That’s an empirical question. I think the evidence is still developing here, but interpretability research gives at least some evidence of robust world models. For example, Kenneth Li and colleagues trained a language model on sequences of moves in the board game Othello and gave evidence that it builds an internal model of the 64 board squares and uses this model in determining the next move. There’s also much work on finding where and how facts are represented in language models.

There are certainly many limitations in current LLMs’ world models. Standard models often seem fragile rather than robust, with language models often confabulating and contradicting themselves. Current LLMs seem to have especially limited self models: that is, their models of their own processing and reasoning are poor. Self models are crucial at least to self-consciousness, and on some views (including so-called higher-order views of consciousness) they are crucial to consciousness itself.

In any case, we can once again turn the objection into a challenge. This fifth challenge is to build extended language models with robust world models and self models.

X = Recurrent Processing

I’ll turn now to two somewhat more technical objections tied to theories of consciousness. In recent decades, sophisticated scientific theories of consciousness have been developed. These theories remain works in progress, but it’s natural to hope that they might give us some guidance about whether and when AI systems are conscious. A group led by Robert Long and Patrick Butlin has been working on this project, and I recommend playing close attention to their work as it appears.

The first objection here is that current LLMs are almost all feedforward systems without recurrent processing (that is, without feedback loops between inputs and outputs). Many theories of consciousness give a central role to recurrent processing. Victor Lamme’s recurrent processing theory gives it pride of place as the central requirement for consciousness. Giulio Tononi’s integrated information theory predicts that feedforward systems have zero integrated information and therefore lack consciousness. Other theories such as global workspace theory also give a role to recurrent processing.

These days, almost all LLMs are based on a transformer architecture that is almost entirely feedforward. If the theories requiring recurrent processing are correct, then these systems seem to have the wrong architecture to be conscious. One underlying issue is that feedforward systems lack memory-like internal states that persist over time. Many theories hold that persisting internal states are crucial to consciousness.

There are various responses here. First, current LLMs have a limited form of recurrence deriving from recirculation of past outputs, and a limited form of memory deriving from the recirculation of past inputs. Second, it’s plausible that not all consciousness involves memory, and there may be forms of consciousness which are feedforward.

Third and perhaps most important, there are recurrent large language models. Just a few years ago, most language models were long short-term memory systems (LSTMs), which are recurrent. At the moment recurrent networks are lagging somewhat behind transformers but the gap isn’t enormous, and there have been a number of recent proposals to give recurrence more of a role. There are also many LLMs that build in a form of memory and a form of recurrence through external memory components. It’s easy to envision that recurrence may play an increasing role in LLMs to come.

This objection amounts to a sixth challenge: build extended large language models with genuine recurrence and genuine memory, the kind required for consciousness.

X = Global Workspace

Perhaps the leading current theory of consciousness in cognitive neuroscience is the global workspace theory put forward by the psychologist Bernard Baars and developed by the neuroscientist Stanislas Dehaene and colleagues. This theory says that consciousness involves a limited-capacity global workspace: a central clearing-house in the brain for gathering information from numerous non-conscious modules and making information accessible to them. Whatever gets into the global workspace is conscious.

Maybe the deepest obstacle to consciousness in LLMs is the issue of unified agency.

A number of people have observed that standard language models don’t seem to have a global workspace. Now, it’s not obvious that an AI system must have a limited-capacity global workspace to be conscious. In limited human brains, a selective clearing-house is needed to avoid overloading brain systems with information. In high-capacity AI systems, large amounts of information might be made available to many subsystems, and no special workspace would be needed. Such an AI system could arguably be conscious of much more than we are.

If workspaces are needed, language models can be extended to include them. There’s already an increasing body of relevant work on multimodal LLM+s that use a sort of workspace to co-ordinate between different modalities. These systems have input and output modules, for images or sounds or text for example, which may involve extremely high dimensional spaces. To integrate these modules, a lower-dimensional space serves as an interface. That lower-dimensional space interfacing between modules looks a lot like a global workspace.

People have already begun to connect these models to consciousness. Yoshua Bengio and colleagues haveargued that a global workspace bottleneck among multiple neural modules can serve some of the distinctive functions of slow conscious reasoning. There’s a nice recent paper by Arthur Juliani, Ryota Kanai, and Shuntaro Sasai arguing that one of these multimodal systems, Perceiver IO, implements many aspects of a global workspace via mechanisms of self attention and cross attention. So there is already a robust research program addressing what is in effect a seventh challenge, to build LLM+s with a global workspace.

X = Unified Agency

The final obstacle to consciousness in LLMs, and maybe the deepest, is the issue of unified agency. We all know these language models can take on many personas. As I put it in an article on GPT-3 when it first appeared in 2020, these models are like chameleons that can take the shape of many different agents. They often seem to lack stable goals and beliefs of their own over and above the goal of predicting text. In many ways, they don’t behave like unified agents. Many argue that consciousness requires a certain unity. If so, the disunity of LLMs may call their consciousness into question.

Again, there are various replies. First: it’s arguable that a large degree of disunity is compatible with conscious. Some people are highly disunified, like people with dissociative identity disorders, but they are still conscious. Second: One might argue that a single large language model can support an ecosystem of multiple agents, depending on context, prompting, and the like.

But to focus on the most constructive reply: it seems that more unified LLMs are possible. One important genre is the agent model (or person model or creature model) which attempts to model a single agent. One way to do that, in systems such as Character.AI, is to take a generic LLM and use fine-tuning or prompt engineering using text from one person to help it simulate that agent.

Current agent models are quite limited and still show signs of disunity. But it’s presumably possible in principle to train agent models in a deeper way, for example training an LLM+ system from scratch with data from a single individual. Of course this raises difficult ethical issues, especially when real people are involved. But one can also try to model the perception-action cycle of, say, a single mouse. In principle agent models could lead to LLM+ systems that are much more unified than current LLMs. So once again, the objection turns into a challenge: build LLM+s that are unified agent models.

I’ve now given six candidates for the X that might be required for consciousness and missing in current LLMs. Of course there are other candidates: higher-order representation (representing one’s own cognitive processes, which is related to self models), stimulus-independent processing (thinking without inputs, which is related to recurrent processing), human-level reasoning (witness the many well-known reasoning problems that LLMs exhibit), and more. Furthermore, it’s entirely possible that there are unknown X’s that are in fact required for consciousness. Still, these six arguably include the most important current obstacles to LLM consciousness.

For all of these objections except perhaps biology, it looks like the objection is temporary rather than permanent.

Here’s my assessment of the obstacles. Some of them rely on highly contentious premises about consciousness, most obviously in the claim that consciousness requires biology and perhaps in the requirement of sensory grounding. Others rely on unobvious premises about LLMs, like the claim that current LLMs lack world models. Perhaps the strongest objections are those from recurrent processing, global workspace, and unified agency, where it’s plausible that current LLMs (or at least paradigmatic LLMs such as the GPT systems) lack the relevant X and it’s also reasonably plausible that consciousness requires X.

Still: for all of these objections except perhaps biology, it looks like the objection is temporary rather than permanent. For the other five, there is a research program of developing LLM or LLM+ systems that have the X in question. In most cases, there already exist at least simple systems with these X’s, and it seems entirely possible that we’ll have robust and sophisticated systems with these X’s within the next decade or two. So the case against consciousness in current LLM systems is much stronger than the case against consciousness in future LLM+ systems.

IV. Conclusions

Where does the overall case for or against LLM consciousness stand?

Where current LLMs such as the GPT systems are concerned: I think none of the reasons for denying consciousness in these systems is conclusive, but collectively they add up. We can assign some extremely rough numbers for illustrative purposes. On mainstream assumptions, it wouldn’t be unreasonable to hold that there’s at least a one-in-three chance—that is, to have a subjective probability or credence of at least one-third—that biology is required for consciousness. The same goes for the requirements of sensory grounding, self models, recurrent processing, global workspace, and unified agency.1 If these six factors were independent, it would follow that there’s less than a one-in-ten chance that a system lacking all six, like a current paradigmatic LLM, would be conscious. Of course the factors are not independent, which drives the figure somewhat higher. On the other hand, the figure may be driven lower by other potential requirements X that we have not considered.

Taking all that into account might leave us with confidence somewhere under 10 percent in current LLM consciousness. You shouldn’t take the numbers too seriously (that would be specious precision), but the general moral is that given mainstream assumptions about consciousness, it’s reasonable to have a low credence that current paradigmatic LLMs such as the GPT systems are conscious.2

Where future LLMs and their extensions are concerned, things look quite different. It seems entirely possible that within the next decade, we’ll have robust systems with senses, embodiment, world models and self models, recurrent processing, global workspace, and unified goals. (A multimodal system like Perceiver IO already arguably has senses, embodiment, a global workspace, and a form of recurrence, with the most obvious challenges for it being world models, self models, and unified agency.) I think it wouldn’t be unreasonable to have a credence over 50 percent that we’ll have sophisticated LLM+ systems (that is, LLM+ systems with behavior that seems comparable to that of animals that we take to be conscious) with all of these properties within a decade. It also wouldn’t be unreasonable to have at least a 50 percent credence that if we develop sophisticated systems with all of these properties, they will be conscious. Those figures together would leave us with a credence of 25 percent or more. Again, you shouldn’t take the exact numbers too seriously, but this reasoning suggests that on mainstream assumptions, it’s a serious possibility that we’ll have conscious LLM+s within a decade.

One way to approach this is via the “NeuroAI” challenge of matching the capacities of various non-human animals in virtually embodied systems. It’s arguable that even if we don’t reach human-level cognitive capacities in the next decade, we have a serious chance of reaching mouse-level capacities in an embodied system with world models, recurrent processing, unified goals, and so on.3 If we reach that point, there would be a serious chance that those systems are conscious. Multiplying those chances gives us a significant chance of at least mouse-level consciousness with a decade.

We might see this as a ninth challenge: build multimodal models with mouse-level capacities. This would be a stepping stone toward mouse-level consciousness and eventually to human-level consciousness somewhere down the line.

Of course there’s a lot we don’t understand here. One major gap in our understanding is that we don’t understand consciousness. That’s a hard problem, as they say. This yields a tenth challenge: develop better scientific and philosophical theories of consciousness. These theories have come a long way in the last few decades, but much more work is needed.

Where future LLMs and their extensions are concerned, things look quite different.

Another major gap is that we don’t really understand what’s going on in these large language models. The project of interpreting machine learning systems has come a long way, but it also has a very long way to go. Interpretability yields an eleventh challenge: understand what’s going on inside LLMs.

I summarize the challenges here, with four foundational challenges followed by seven engineering-oriented challenges, and a twelfth challenge in the form of a question.

1. Evidence: Develop benchmarks for consciousness.
2. Theory: Develop better scientific and philosophical theories of consciousness.
3. Interpretability: Understand what’s happening inside an LLM.
4. Ethics: Should we build conscious AI?
5. Build rich perception-language-action models in virtual worlds.
6. Build LLM+s with robust world models and self models.
7. Build LLM+s with genuine memory and genuine recurrence.
8. Build LLM+s with global workspace.
9. Build LLM+s that are unified agent models.
10. Build LLM+s that describe non-trained features of consciousness.
11. Build LLM+s with mouse-level capacities.
12. If that’s not enough for conscious AI: What’s missing?

On the twelfth challenge: Suppose that in the next decade or two, we meet all the engineering challenges in a single system. Will we then have a conscious AI systems? Not everyone will agree that we do. But if someone disagrees, we can ask once again: what is the X that is missing? And could that X be built into an AI system?

My conclusion is that within the next decade, even if we don’t have human-level artificial general intelligence, we may well have systems that are serious candidates for consciousness. There are many challenges on the path to consciousness in machine learning systems, but meeting those challenges yields a possible research program toward conscious AI.

I’ll finish by reiterating the ethical challenge.4 I’m not asserting that we should pursue this research program. If you think conscious AI is desirable, the program can serve as a sort of roadmap for getting there. If you think conscious AI is something to avoid, then the program can highlight paths that are best avoided. I’d be especially cautious about creating agent models. That said, I think it’s likely that researchers will pursue many of the elements of this research program, whether or not they think of this as pursuing AI consciousness. It could be a disaster to stumble upon AI consciousness unknowingly and unreflectively. So I hope that making these possible paths explicit at least helps us to think about conscious AI reflectively and to handle these issues with care.

Afterword

How do things look now, eight months after I gave this lecture at the NeurIPS conference in late November 2022? While new systems such as GPT-4 still have many flaws, they are a significant advance along some of the dimensions discussed in this article. They certainly display more sophisticated conversational abilities. Where I said that GPT-3’s performance often seemed on a par with a sophisticated child, GPT-4’s performance often (not always) seems on a par with an knowledgeable young adult. There have also been advances in multimodal processing and in agent modeling, and to a lesser extent on the other dimensions that I have discussed. I don’t think these advances change my analysis in any fundamental way, but insofar as progress has been faster than expected, it is reasonable to shorten expected timelines. If that is right, my predictions toward the end of this article might even be somewhat conservative.

Notes

1. The philosopher Jonathan Birch distinguishesapproaches to animal consciousness that are “theory-heavy” (assume a complete theory), “theory-neutral” (proceed without theoretical assumptions), and “theory-light” (proceed with weak theoretical assumptions). One can likewise take theory-heavy, theory-neutral, and theory-light approaches to AI consciousness. The approach to artificial consciousness that I have taken here is distinct from these three. It might be considered a theory-balanced approach, one that takes into account the predictions of multiple theories, balancing one’s credence between them, perhaps, according to evidence for those theories or according to acceptance of those theories.

One more precise form of the theory-balanced approach might use data about how widely accepted various theories are among experts to provide credences for those theories, and use those credences along with the various theories’ predictions to estimate probabilities for AI (or animal) consciousness. In a recent survey of researchers in the science of consciousness, just over 50 percent of respondents indicated that they accept or find promising the global workspace theory of consciousness, while just under 50 percent indicated that they accept or find promising the local recurrence theory (which requires recurrent processing for consciousness). Figures for other theories include just over 50 percent for predictive processing theories (which do not make clear predictions for AI consciousness) and for higher-order theories (which require self models for consciousness), and just under 50 percent for integrated information theory (which ascribes consciousness to many simple systems but requires recurrent processing for consciousness). Of course turning these figures into collective credences requires further work (e.g. in converting “accept” and “find promising” into credences), as does applying these credences along with theoretical predictions to derive collective credences about AI consciousness. Still, it seems not unreasonable to assign a collective credence above one in three for each of global workspace, recurrent processing, and self models as requirements for consciousness.

What about biology as a requirement? A 2020 survey of professional philosophers, around 3 percent accepted or leaned toward the view that current AI systems are conscious, with 82 percent rejecting or leaning against the view and 10 percent neutral. Around 39 percent accepted or leaned toward the view that future AI systems will be conscious, with 27 percent rejecting or leaning against the view and 29 percent neutral. (Around 5 percent rejected the questions in various ways, e.g. saying that there is no fact of the matter or that the question is too unclear to answer). The future-AI figures might tend to suggest a collective credence of at least one in three that biology is required for consciousness (albeit among philosophers rather than consciousness researchers). The two surveys have less information about unified agency and about sensory grounding as requirements for consciousness.

2. Compared to mainstream views in the science of consciousness, my own views lean somewhat more to consciousness being widespread. So I’d give somewhat lower credences to the various substantial requirements for consciousness I’ve outlined here, and somewhat higher credences in current LLM consciousness and future LLM+ consciousness as a result.

3. At NeurIPS I said “fish-level capacities.” I’ve changed this to “mouse-level capacities” (probably a harder challenge in principle), in part because more people are confident that mice are conscious than that fish are conscious, and in part because there is so much more work on mouse cognition than fish cognition.

4. This final paragraph is an addition to what I presented at the NeurIPS conference.

We’re interested in what you think. Submit a letter to the editors at [email protected]. Boston Review is nonprofit, paywall-free, and reader-funded. To support work like this, please donate here.

‍David J. Chalmers‍

David J. Chalmers is University Professor of Philosophy and Neural Science & co-director of the Center for Mind, Brain, and Consciousness at NYU. His most recent book is Reality+: Virtual Worlds and the Problems of Philosophy.

https://www.bostonreview.net/articles/could-a-large-language-model-be-conscious/