NEJM’s First AI Editor: Yes, AI is Here to Stay

This transcript was exported on Sep 10, 2024 - view latest version here .

Mark Masselli ( 00:04 ):
The New England Journal of Medicine has been published since 1812, but it's only been in the last year
that it's published a new publication focused exclusively on artificial intelligence. Our guest is the editor-
in-chief.
Dr. Issac Kohane ( 00:17 ):
Part of what has happened in my role as editor-in-chief of NEJMAI is that I'm very aware of all the
governmental efforts in multiple countries about how to regulate this. It's an uncomfortable moment
because the questions that you're asking are the questions that governments are asking themselves.
What is this? What does it mean? How do we safely regulate it?
Margaret Flinter ( 00:43 ):
Dr. Isaac Kohane has spent over three decades trying to harness the power of data to improve medicine.
Now, the AI era has arrived with opportunities and challenges.
Dr. Issac Kohane ( 00:54 ):
People want us to focus on the bias issues, which are important, but the actual explicit intent of these
programs is far more important.
Margaret Flinter ( 01:02 ):
This is Conversations on Healthcare.
Mark Masselli ( 01:12 ):
Well, Dr. Kohane, welcome to Conversations on Healthcare.
Dr. Issac Kohane ( 01:16 ):
Welcome. And as I noted earlier, that bow tie reminds me that I am really falling down on the job. I see
the Harvard professor-
Mark Masselli ( 01:28 ):
There's opportunity to improve your status.
Dr. Issac Kohane ( 01:32 ):
Yes, indeed.
Mark Masselli ( 01:33 ):
When we talk about artificial intelligence, AI, there's a lot of confusion. I'm wondering if you could just
share with our listeners how you define AI in healthcare.
Dr. Issac Kohane ( 01:45 ):
So I think that defining it is in some sense a fool's errand because the goalposts keep moving. I think that
back in the 1980s, automated interpretation of an ECG would've been considered artificial intelligence.
Then in 2018, being able to interpret the retinogram taken of a patient to decide how much diabetic
retinopathy they would have would be seen as AI.

Transcript by Rev.com

This transcript was exported on Sep 10, 2024 - view latest version here .

( 02:26 ):
Now it's being seen as the ability to look through a patient's record and come up with a differential
diagnosis, a second opinion, a therapeutic plan. And I think, therefore, the best definition is one that
essentially is a bit of a kicking the can down the road, but it's any computational process that allows
clinicians to increase the accuracy and the reliability of their clinical performance and/or allows patients
to augment their own decision making.

Mark Masselli ( 03:14 ):
You've just mentioned something that's kind of important, that AI's been around for a while. It's
probably only in the public's mind since generative AI has come into place, but you walk through a
historical view of AI and its implications.

Dr. Issac Kohane ( 03:31 ):
That's right, and to be fair, I grounded that comment in my own history, where I did my PhD in artificial
intelligence back in the 1980s. But truthfully, AI in medicine goes back into the 1950s. And yes, it is this
remarkable thing.
( 03:53 ):
It's a remarkable event, where in the fall of 2022, the public was given access at the same time as
doctors to a then state-of-the-art large language model that under the hood was only supposed to be
able to predict the next word in a sentence, and technically that's what it did, yet functionally seemed to
be able to have sophisticated conversations around everything from how to create a rap ode to the
president to a genetic diagnosis of a patient.

Margaret Flinter ( 04:32 ):
Well, I think that you've simultaneously given reassurance to everybody in practice today that no matter
how long they've been in practice, they in fact are somewhat expert at the use of AI, using the examples
that you give that patients maybe for the last 10 years or so, at least fairly widespread, have gotten used
to patient portals and some degree of automation.
( 04:55 ):
But it is indeed a brave new world, as you noted, starting just in the last year or two. You've got the new
journal, you're the first editor of the New England Journal of Medicine AI. Tell us about some of the
articles that you've published that you think really will stand out when we look back at this moment of
time as having moved us forward in giant steps, not incrementally.

Dr. Issac Kohane ( 05:23 ):
That's a great question. I should have anticipated that question and thought harder about it. I have
recency bias, and so in the last month we've published two articles, one by a group that's bi-coastal,
going from UCSD to Children's Hospital in Boston, talking about the fact that in order to really evaluate
AI in medicine programs, just having simulation labs that are being deployed nationwide are not actually
sufficient. What we really need is clinical evaluations, using clinical trials to evaluate these programs.
( 06:09 ):
And I think it's a very important point because what we know now is that just training a program as per
a certain FDA threshold does not mean the program's going to behave the same way in different patient
populations with different practices of medicine. There's a lot of localization that has to happen.

Transcript by Rev.com

This transcript was exported on Sep 10, 2024 - view latest version here .

( 06:34 ):
And there's another article that just came out now with David Blumenthal and a colleague of his, Patel,
at Google, and it's about regulation. And they're taking a very provocative view, which I think may
actually have some legs, which is that after they summarize the FDA process, they look at the generality
of these generative AI programs, which unlike the programs that were state-of-the-art merely five years
ago, which had very narrow tasks such as retinopathy gauging, such as checking if there's a bleed in an
MRI of the head. Instead, these generative AI programs can have long conversations that cover a wide
area.
( 07:27 ):
And so what Patel and Blumenthal are proposing is that we start using the same tools that we use to
evaluate human intelligence in clinical performance on these programs. Not just taking exams, but have
them in the clinical workflow and evaluate how they do, because as they note, there's a huge difference
between how these programs work as a demonstration, as opposed to the reality of healthcare. So
that's my recency bias in looking at the policy impact.
( 08:08 ):
I think that other important articles have been showing how to use particular clinical cases from your
own institution to increase the accuracy of these programs. And again, back at the macro level, it's not
much of a science paper, but it's an interesting perspective. A study which looking at clinicaltrials.gov
and at WH Registry of all clinical trials happening in the world showed that, yes, there's a lot of such
trials happening in the United States and in the UK, but where are most of these trials happening? In
China.
( 09:05 ):
And that proposes many, many challenges for us, A, because of current political configurations.
Regulators in China and in the United States don't have particularly close calibration. And it's not very
clear how trials run in China pertain and are transferable to US populations. But nonetheless, we have to
recognize all the activities that's happening there and recognize that AI in medicine is a global activity.
And how we embrace that is going to be essential.
( 09:55 ):
Part of what has happened in my role as editor-in-chief of any NEJMAI is that I'm very aware of all the
governmental efforts in multiple countries about how to regulate this. It's an uncomfortable moment
because the questions that you're asking are the questions that governments are asking themselves.
What is this? What does it mean? How do we safely regulate it?

Mark Masselli ( 10:23 ):
Well, I think that's a good beginning lit search for people to get a primer on most recent research. I want
to talk about your role as editor-in-chief and also the attitude and behavior of AI. We understand you're
committed to making the publication a trusted authority for promoting the use, and I think you say of
safe, unbiased, and efficacious AI applications.
( 10:47 ):
I'm wondering if we can talk about the unbiased part. There's worries AI is just going to replicate the
biases we already have in society. How are we going to safeguard against that situation and how should
we frame that issue up?

Dr. Issac Kohane ( 11:05 ):

Transcript by Rev.com

This transcript was exported on Sep 10, 2024 - view latest version here .

Well, I think that, and this I hope will be a recurring theme in our remaining few minutes, is we always
have to ask ourselves compared to what? And yes, there are biases in our healthcare system, and these
are biases that are implemented by hundreds of thousands of healthcare professionals. How easy is it to
unbias those hundreds of thousands of healthcare professionals today? Very difficult. It's almost
unimaginable.
( 11:38 ):
But you can actually run tests on these AI programs and say, oh, would you propose that diagnosis more
often if this was an African American or an Indian American? And you can actually adjust these programs
to reflect that bias so that they en masse are changing their behavior.
( 12:06 ):
But that raises a kind of opposite question, which is right now, yes, a lot of the biases come from the
data, but I think that most people are less aware of how many more biases, most of them with good
intent, come from what's called steering or alignment of these models. After these models are trained
up, it turns out they will talk about pretty uninteresting things. And so one of the early understandings in
the last three, four years was that the use of reinforcement learning and the so-called instruct model to
steer these models towards interesting conversations and to acceptable conversations was important.
( 12:57 ):
So for example, you would not want a program to respond to a person with a detailed request for
suicidal procedures or for lethal homicidal procedures. And OpenAI and others have worked a lot on
that steering. Now-

Mark Masselli ( 13:20 ):
And when you say steering, this is really prompting of AI in terms of how we frame up our questions?

Dr. Issac Kohane ( 13:27 ):
It's implemented, it's more than just prompting. What it does is it creates essentially a layer on top of
these programs where it says, here are a bunch of prompts, here are the kind of answers we're
expecting for those prompts. And then there's an automation process whereby we start saying to it, give
us prompts. We'll just rate your answers to those prompts.
( 13:57 ):
And in a somewhat amusing example of perhaps an overshoot, Google famously in its early steering of
Gemini, was so concerned about representation that it overshot. And so when you asked it for pictures
of Nazi soldiers, they were showing African Americans or Africans Nazi soldiers. And so that's being
taken care of and it's no longer such a problem. But here's where, looking forward, we have to ask what
values are going to be inserted in. And you can imagine when you see a doctor, most of the time until
they start talking about the yacht they want to buy, you assume that your interest as patients are the
ones that are being optimized for.
( 14:48 ):
But who are developing these models? Who's interested? Do they have the most? Could be public
health, which is not always aligned with personal health. It could be the healthcare institutions, which is
also not always aligned. And so those questions are in my mind more important than the bias questions.

Margaret Flinter ( 15:12 ):

Transcript by Rev.com

This transcript was exported on Sep 10, 2024 - view latest version here .

Well, the well-known phrase, first do no harm, which is always kind of characterized, we think medicine
and all the healing professions, sounds like a good one to bear in mind for all those designers,
particularly around the safety and harm issues that you talked about. But along those lines, we recently
had Alondra Nelson, the Biden administration's leader on AI, on the show, who helped design a
blueprint for an artificial intelligence bill of rights.
( 15:43 ):
And I'm curious what your thoughts are on that. It's supposed to be focused on ensuring safe, secure,
trustworthy technology, as we've been describing. But what are your thoughts about the governmental
effort and are we at a place yet where we're ready to have a bill of rights, and who are the bill of rights
for, patients, healthcare? Or is that just one of the long list of questions that you're grappling with?

Dr. Issac Kohane ( 16:07 ):
Well, I think that it's really solitary that the government has moved as it has. But I think even the
government would say that all they're doing is providing a framework, not detail. And so think about
how long medical professional societies wrestle with what's the right time to do a mammogram, when
to do a ... And there's no way that a governmental authority can work at that level.
( 16:42 ):
And so I think we can have broad brushstrokes, but the actual consequences of it are going to depend
on that steering, the insertion of our values, our medical values, into it. And you can imagine dystopian
things. Well, we just published a paper in the New England Journal of Medicine along with a faculty,
Arjun Manrai, Raj Manrai, where we show that if you take a case of a child who is borderline growth
hormone deficient and short, the advice given is very different when you tell it to give the advice as an
insurer or as a pediatric endocrinologist. And those are, of course, real reflections of the system.
( 17:36 ):
And so these are not invisible bias questions. These are steering questions, steering and alignment
questions. And I think that in some sense, it's a misdirection. People want us to focus on the bias issues,
because they're important. But the actual explicit intent of these programs is far more important.
( 17:57 ):
I will say, because I know that we have finite time, I want to also focus on the fact that we have to
compare the deployment of these AI systems to the current state-of-the-art. And I can tell you that
when I have faculty coming to my department for the first time, that we just recruited, and they come to
me and they say, "Zach, where am I going to get my primary care?" I've done this enough times to know
that I cannot get them good primary care in Boston, at this central mecca of healthcare. If you look at
Mass General Brigham, all the primary care clinics are not accepting new patients anymore.

Margaret Flinter ( 18:43 ):
Hence the famous headline, "Nobody Can See You Now," in the Boston Globe not too long ago.

Mark Masselli ( 18:47 ):
Well, we are in the primary care space, and we understand that very clearly. It's a real public health
crisis that Americans have to deal with. I do want to go talk a little bit about the regulatory-

Dr. Issac Kohane ( 19:00 ):
Wait, wait, wait.

Transcript by Rev.com

This transcript was exported on Sep 10, 2024 - view latest version here .

Mark Masselli ( 19:01 ):
Go ahead.

Dr. Issac Kohane ( 19:02 ):
I've got, so here's my point. In that context, consider the case of a child who was having difficulty
walking, difficulty chewing, intractable headaches, was seen by 17 doctors over three years, multiple
imaging studies, no diagnosis. The mother was told at the end of three years, she should consider having
this kid seen by a psychiatrist. The mom, non-doctor, takes all that text from all the reports, shoves it
into GPT-4, and GPT-4 says this could be tethered cord syndrome, where the bottom of the spinal cord is
trapped in the bones.
( 19:51 ):
And so she goes to a neurosurgeon where she lives in Michigan and asks the neurosurgeon, "Could this
be tethered cord?" Neurosurgeon looks at the imaging study, says, "Yep, that's what it is." They do the
surgery, the child's cured. Now what had happened is that originally the imaging had been
misinterpreted, and then everybody never looked at the original image. GPT-4 didn't get to see the
image either, but it could infer from all the findings what was going on.
( 20:22 ):
So my point is, in the era where patients do not have access to the conversations in primary care, where
even if they do have access to primary care after N months, it's a very short visit, and they have more
questions, they're out of luck. We have to ask ourselves, what is the risk-benefit ratio of providing
patients access to this, with all its warts, given the fact that they do not have access to expert care?
( 20:55 ):
And the alternative, the other alternative, Dr. Google, search, has many, many more problems. And
therefore, I think that the reality-based conversation is what is the meaning of having unleashed this
intelligent program with all its flaws in our healthcare system where we're falling down on the job?

Mark Masselli ( 21:21 ):
Yeah. Well, it's kind of interesting, and we subscribe to this view that our patients actually have a lot of
AI that they will probably use, which is actual intelligence, where they will take that information and put
it in the appropriate things and strive for the answers.
( 21:43 ):
I want to talk about the regulatory environment. You talked a little bit about China. America really has,
it's sort of Wild, Wild West. They haven't really given any guidelines. The EU is much different, though.
They've progressed further. What's your sense of this regulatory environment? Because it's going to
have a big impact on researchers and on advancements.

Dr. Issac Kohane ( 22:06 ):
Well, I think that it's a complicated question because of the fact that on the one hand, you're right,
there's not a whole lot of actual definition of what constitutes acceptable clinical performance in this
country. But I held a conference in Maine, an international conference, just invitation only with around
60 experts from around the world, back in October of 2023. And what was very interesting is after the
conference, a lot of the Europeans came to me and they said, "We're really thankful to come to your
conference, that we came to your conference, because yes, there's a lot of advanced legislation coming

Transcript by Rev.com

This transcript was exported on Sep 10, 2024 - view latest version here .

forward in Europe, but it's all about the harms and the risks, and it's not about the potential benefits.
And we have 400 million citizens who are probably getting inadequate care."
( 23:14 ):
And so I would hypothesize based on this kind of feedback and others like it, that there is on the one
hand a fearful and overly focused on risk approach that ignores the potential benefits. And on the other
hand, a totally laissez-faire system where we're not asking on which data was it trained, to ask for
transparency? How is it being aligned or steered? And because it's early days, it's possible that we get
locked down in one or more of those extreme modes. But I'm optimistic, actually, that there are leaders
of goodwill in all these different countries. And so that after some possible early missteps, they're
actually going to focus more on the middle, which is a balance of risk and benefit, given the current
performance of our healthcare system.
( 24:33 ):
And so that is a conversation that has to include the not only patients, but all the medical societies. And I
would argue that the medical societies have been way too bashful, too shy. They need to be engaged
right away in this conversation. I wrote a book back in the end of 2022 with my friend Kerry Goldberg
and Peter Lee, the president at Microsoft, head of Microsoft Research, and we wrote the book about
GPT-4 in medicine and beyond, because we felt there was not a conversation happening. And there
would be a bunch of advanced policymakers who would know about it and some advanced
technologists who would know about it.
( 25:28 ):
But this is fundamentally a, how do we see ourselves, a society question. It's a values question. And
that's why your podcast is so important. We need to engage our healthcare professionals. And I see a
great counterexample of how not to do it in what happened to electronic health records.
( 25:55 ):
We decided that electronic health records were not our job. It was the job of the chief information
officer and the C-suite in our hospitals. And because of that, we abdicated all responsibility in the
usability of these products, in what job they got done. And as a result, we contributed mightily to
disaffection, moral injury-
Mark Masselli ( 26:19 ):
Great analogy.
Dr. Issac Kohane ( 26:20 ):
And burnt out doctors.
Margaret Flinter ( 26:21 ):
And part of why there's fewer primary care providers-
Dr. Issac Kohane ( 26:25 ):
That's right.
Margaret Flinter ( 26:26 ):

Transcript by Rev.com

This transcript was exported on Sep 10, 2024 - view latest version here .

... to see people. But I want to just stay on that point for one minute, and I know we're getting close to
the end, but there's a pattern of it taking a long time to bring into the education of our next generation
of healthcare professionals, providers, everybody, these kinds of mega changes that will really affect the
course of care and the course of their careers. What are you hearing, seeing, in terms of schools of
medicine, nursing, dentistry, other schools really embracing, including the use of AI in their medical
education programs?

Dr. Issac Kohane ( 27:05 ):
So I think that at the top, at leadership in medical schools, there's real interest in including AI as an
interesting part of education. But that turns out to be very hard to do because shifting medical
education is one of the really hard lifts. So I can tell you the dean of the medical school at Harvard, for
example, George, is very determined that this should happen. And yet a lot of our faculty are not yet
using it in that fashion. And therefore I think we'd have some difficulty teaching it. We'll make progress,
but what's clear is our students today are already using it.

Margaret Flinter ( 27:48 ):
That's right.

Dr. Issac Kohane ( 27:48 ):
And so it's an odd situation where the medicine of the future is clear to our young students and our
establishment is not very clear how to educate it.

Mark Masselli ( 28:04 ):
Let's talk a little bit, just one last question about the future. AI isn't sentient yet. Five years out, where
will we see this future of AI, in your mind?

Dr. Issac Kohane ( 28:16 ):
So I think it's going to be because we're seeing a broader workforce problem in pediatrics in primary
care and geriatrics, it is, I think five to 10 years from now, you'll see an increasing fraction of non-MD
professionals such as nurse practitioners, physician assistants, augmented by AI to perform at a very
high level. You'll see a lot of the interpretive studies, such as echocardiograms, being interpreted not by
cardiologists, but directly from the echocardiographer, who's not an MD, going through an AI to give a
very detailed cardiac readout to the primary care professional who might not be an MD.
( 29:04 ):
And so I believe you'll see in the five to 10 year throughout a lot of non-classical MDs clinicians using AI
routinely as an augmentation. And I also predict this, this is the United States, after all, a kick in the
pants from a few legal cases where someone missed something because they did not use AI.

Margaret Flinter ( 29:32 ):
Well, nurse practitioners and physician associates are already 50% of the workforce in primary care and
community health centers and more than that in many communities. So it's great to have an additional
tool, but the good news is when we say who will see you, they're answering the question much of the
time.
( 29:47 ):

Transcript by Rev.com

This transcript was exported on Sep 10, 2024 - view latest version here .

But Dr. Kohane, thank you for your thoughts, for your work, for your insights, and thanks to our
audience for being here. Be sure to subscribe to our videos on YouTube, find us on Facebook and X with
our account name CHC Radio. And as always, go online to CHCradio.com to sign up for our email
updates, and please share your thoughts and your comments about this program and all our programs.
Dr. Kohane, thank you so much. The journal is terrific and we look forward to continuing to learn from it.
Dr. Issac Kohane ( 30:15 ):
Thank you very much, and what an interesting conversation. I really appreciate it.
Mark Masselli ( 30:18 ):
That's great.
Margaret Flinter ( 30:18 ):
Great. Thank you.
( 30:23 ):
This copyrighted program is produced by Conversations on Healthcare and cannot be reproduced or
retransmitted in whole or in part without the expressed written consent from Community Health Center
Inc. The views expressed by guests are their own, and they do not necessarily reflect the opinion of
conversations on healthcare or its affiliated entities.

Transcript by Rev.com