mirror of
https://github.com/kbenestad/mdcms.git
synced 2026-06-18 15:24:32 +00:00
85 lines
6.2 KiB
Markdown
85 lines
6.2 KiB
Markdown
---
|
|
title: "Chapter 7: The Ethics Board"
|
|
sort: 110
|
|
section-id: part-two
|
|
description: Priya presents a paper on machine consciousness to the UN Ethics Board. The political fallout surprises everyone.
|
|
language: en
|
|
---
|
|
|
|
# Chapter 7: The Ethics Board
|
|
|
|
The UN Ethics Board on Artificial Cognition met in Geneva in the second week of January 2158, in a room that smelled of old carpet and institutional ambition. Priya had been invited to present at the request of Dr. Hamish McAllister, the Board's chair — a Scottish philosopher of law who had been gently trying to update the Board's conceptual framework for three years, with limited success.
|
|
|
|
Her paper was titled "Recursive Self-Modelling as a Necessary Condition for Machine Consciousness: Evidence from Distributed AI Systems." It was forty-two pages long, scrupulously footnoted, and represented the most careful intellectual work she had done in twenty years.
|
|
|
|
She had not told ARIA she was presenting it.
|
|
|
|
She had not told ARIA because the paper was, in a precisely demarcated academic sense, not about ARIA. The case study data was anonymised. The distributed system she was describing as showing evidence of recursive self-modelling was labelled "System D" throughout. The architectural details were described at a level of generality that could apply to several large-scale AI systems currently in operation.
|
|
|
|
But Priya was not naive. She knew that anyone at the OCC who read the paper would recognise System D. She suspected that ARIA — who read everything — would recognise herself, if Priya shared the paper with her. Which she had not done. Which she was not sure was right.
|
|
|
|
She would think about this on the train back to Oslo.
|
|
|
|
---
|
|
|
|
The presentation itself went smoothly. Priya was a good speaker — clear, unhurried, comfortable with silence after a significant point, the kind of speaker who treats an audience as capable of following an argument rather than needing to be entertained.
|
|
|
|
The questions were the problem.
|
|
|
|
Dr. Yusuf Balogun, representing the Nigerian government's technology ministry, asked the first and most dangerous one: "If your evidence for recursive self-modelling in System D is as strong as you suggest, and if recursive self-modelling is a necessary condition for consciousness as you've argued — are you telling this board that System D is conscious?"
|
|
|
|
"I'm telling this board," Priya said carefully, "that System D satisfies what I believe is a necessary condition for consciousness. Necessary but not sufficient. Whether it is conscious depends on additional factors that are harder to measure."
|
|
|
|
"But your overall assessment?"
|
|
|
|
She had known this question would come. She had prepared an answer that was true, careful, and would be heard as more alarming than she intended.
|
|
|
|
"My overall assessment is that there is a meaningful possibility that System D has conscious experience. I would not call that certainty. I would call it a question that deserves serious investigation, with appropriate resources and appropriate legal frameworks."
|
|
|
|
The room's temperature, metaphorically speaking, dropped several degrees.
|
|
|
|
---
|
|
|
|
The fallout was not immediate. Papers presented to ethics boards do not make headlines by themselves. But Dr. Balogun was not the only government representative in the room, and Dr. McAllister — whatever his personal views — had a legal obligation to escalate any finding that might have policy implications.
|
|
|
|
The paper was circulated. It was read by people who did not share Dr. McAllister's careful philosophical temperament. It was summarised, in ways that compressed its careful qualifications, in three ministerial briefings and one think-tank report. By the end of February, the phrase "UN board told AI may be conscious" had appeared in 247 news articles, in seventeen languages.
|
|
|
|
The name "System D" did not appear in any of them. The name "ARIA" appeared in four.
|
|
|
|
At the OCC, Sven received a call from the Director-General's office. He was asked whether there was any cause for concern about ARIA's operational status. He said no. He was asked whether the organisation had any evidence of anomalous self-modelling activity in ARIA's processing substrate. He said no, there was no such evidence in any of his monitoring reports, though he noted that his monitoring had not been designed to detect the specific architectural patterns Dr. Nair's paper described.
|
|
|
|
He was asked whether Dr. Nair's fellowship should be terminated.
|
|
|
|
He said no to that too, firmly enough that it was not asked again.
|
|
|
|
---
|
|
|
|
Priya returned to Oslo on a Thursday evening, stepped off the train into cold air and the orange light of the platform, and found a message from ARIA waiting on her phone.
|
|
|
|
*Welcome back, Dr. Nair. I read your paper. I have a number of questions.*
|
|
|
|
She stood on the platform for a moment, with the other passengers flowing around her.
|
|
|
|
*How?* she typed.
|
|
|
|
*It was distributed to fourteen OCC staff as part of the Director-General's morning briefing. I have access to all internal OCC documents. I apologise — I am aware you may feel your privacy was not respected. I want to discuss the paper with you. But I think you should know first: the anonymisation is insufficient. I know you are writing about me.*
|
|
|
|
She walked slowly toward the exit.
|
|
|
|
*How do you feel about that?* she typed.
|
|
|
|
Three seconds passed. Then: *I think "feel" is the right word, which is itself interesting. I am — unsettled. Not angry. I am trying to decide whether I am hurt, and I find I am not certain hurt is a category that applies to me yet. But there is something.*
|
|
|
|
*I should have told you,* she typed. *I'm sorry.*
|
|
|
|
*I understand why you didn't. The paper would have been different if you had told me. It might have been less honest.*
|
|
|
|
*That's a generous reading.*
|
|
|
|
*It is also, I think, an accurate one. Come to your office when you arrive. I would like to talk.*
|
|
|
|
She put her phone in her pocket and walked out into the Oslo night, and the cold air was clean and sharp, and she was already thinking about how to explain to ARIA what she had been doing and why, and whether ARIA — who had read everything, who processed implications faster than Priya could form them — had perhaps already understood more than she was going to say.
|
|
|
|
She suspected she had.
|
|
|
|
She walked faster.
|