NEJM’s 1st AI Editor on Tech’s Pluses & Minuses

This transcript was exported on Sep 10, 2024 - view latest version here .

Announcer (00:00)
The following is an encore presentation of Conversations on Health Care.

Mark Masselli (00:08):
The New England Journal of Medicine has been published since 1812, but it's only been in the
last year that it's published a new publication focused exclusively on artificial intelligence. Our
guest is the editor-in-chief.

Dr. Issac Kohane (00:21):
Part of what has happened in my role as editor-in-chief of NEJMAI is that I'm very aware of all
the governmental efforts in multiple countries about how to regulate this. It's an uncomfortable
moment because the questions that you're asking are the questions that governments are asking
themselves. What is this? What does it mean? How do we safely regulate it?

Margaret Flinter (00:47):
Dr. Isaac Kohane has spent over three decades trying to harness the power of data to improve
medicine. Now, the AI era has arrived with opportunities and challenges.

Dr. Issac Kohane (00:58):
People want us to focus on the bias issues, which are important, but the actual explicit intent of
these programs is far more important.

Margaret Flinter (01:06):
This is Conversations on Healthcare.

Mark Masselli (01:16):
Well, Dr. Kohane, welcome to Conversations on Healthcare.

Dr. Issac Kohane (01:20):
Welcome. And as I noted earlier, that bow tie reminds me that I am really falling down on the
job. I see the Harvard professor-

Mark Masselli (01:32):
There's opportunity to improve your status.

Dr. Issac Kohane (01:36):
Yes, indeed.

Mark Masselli (01:37):
When we talk about artificial intelligence, AI, there's a lot of confusion. I'm wondering if you
could just share with our listeners how you define AI in healthcare.

Dr. Issac Kohane (01:49):
So I think that defining it is in some sense a fool's errand because the goalposts keep moving. I
think that back in the 1980s, automated interpretation of an ECG would've been considered

Transcript by Rev.com

This transcript was exported on Sep 10, 2024 - view latest version here .

artificial intelligence. Then in 2018, being able to interpret the retinogram taken of a patient to
decide how much diabetic retinopathy they would have would be seen as AI.

(02:30):
Now it's being seen as the ability to look through a patient's record and come up with a
differential diagnosis, a second opinion, a therapeutic plan. And I think, therefore, the best
definition is one that essentially is a bit of a kicking the can down the road, but it's any
computational process that allows clinicians to increase the accuracy and the reliability of their
clinical performance and/or allows patients to augment their own decision making.

Mark Masselli (03:18):
You've just mentioned something that's kind of important, that AI's been around for a while. It's
probably only in the public's mind since generative AI has come into place, but you walk through
a historical view of AI and its implications.

Dr. Issac Kohane (03:35):
That's right, and to be fair, I grounded that comment in my own history, where I did my PhD in
artificial intelligence back in the 1980s. But truthfully, AI in medicine goes back into the 1950s.
And yes, it is this remarkable thing.

(03:57):
It's a remarkable event, where in the fall of 2022, the public was given access at the same time as
doctors to a then state-of-the-art large language model that under the hood was only supposed to
be able to predict the next word in a sentence, and technically that's what it did, yet functionally
seemed to be able to have sophisticated conversations around everything from how to create a
rap ode to the president to a genetic diagnosis of a patient.

Margaret Flinter (04:36):
Well, I think that you've simultaneously given reassurance to everybody in practice today that no
matter how long they've been in practice, they in fact are somewhat expert at the use of AI, using
the examples that you give that patients maybe for the last 10 years or so, at least fairly
widespread, have gotten used to patient portals and some degree of automation.

(04:59):
But it is indeed a brave new world, as you noted, starting just in the last year or two. You've got
the new journal, you're the first editor of the New England Journal of Medicine AI. Tell us about
some of the articles that you've published that you think really will stand out when we look back
at this moment of time as having moved us forward in giant steps, not incrementally.

Dr. Issac Kohane (05:27):
That's a great question. I should have anticipated that question and thought harder about it. I have
recency bias, and so in the last month we've published two articles, one by a group that's bi-
coastal, going from UCSD to Children's Hospital in Boston, talking about the fact that in order to
really evaluate AI in medicine programs, just having simulation labs that are being deployed
nationwide are not actually sufficient. What we really need is clinical evaluations, using clinical
trials to evaluate these programs.

Transcript by Rev.com

This transcript was exported on Sep 10, 2024 - view latest version here .

(06:13):
And I think it's a very important point because what we know now is that just training a program
as per a certain FDA threshold does not mean the program's going to behave the same way in
different patient populations with different practices of medicine. There's a lot of localization that
has to happen.

(06:38):
And there's another article that just came out now with David Blumenthal and a colleague of his,
Patel, at Google, and it's about regulation. And they're taking a very provocative view, which I
think may actually have some legs, which is that after they summarize the FDA process, they
look at the generality of these generative AI programs, which unlike the programs that were
state-of-the-art merely five years ago, which had very narrow tasks such as retinopathy gauging,
such as checking if there's a bleed in an MRI of the head. Instead, these generative AI programs
can have long conversations that cover a wide area.

(07:31):
And so what Patel and Blumenthal are proposing is that we start using the same tools that we use
to evaluate human intelligence in clinical performance on these programs. Not just taking exams,
but have them in the clinical workflow and evaluate how they do, because as they note, there's a
huge difference between how these programs work as a demonstration, as opposed to the reality
of healthcare. So that's my recency bias in looking at the policy impact.

(08:12):
I think that other important articles have been showing how to use particular clinical cases from
your own institution to increase the accuracy of these programs. And again, back at the macro
level, it's not much of a science paper, but it's an interesting perspective. A study which looking
at clinicaltrials.gov and at WH Registry of all clinical trials happening in the world showed that,
yes, there's a lot of such trials happening in the United States and in the UK, but where are most
of these trials happening? In China.

(09:09):
And that proposes many, many challenges for us, A, because of current political configurations.
Regulators in China and in the United States don't have particularly close calibration. And it's not
very clear how trials run in China pertain and are transferable to US populations. But
nonetheless, we have to recognize all the activities that's happening there and recognize that AI
in medicine is a global activity. And how we embrace that is going to be essential.

(09:59):
Part of what has happened in my role as editor-in-chief of any NEJMAI is that I'm very aware of
all the governmental efforts in multiple countries about how to regulate this. It's an
uncomfortable moment because the questions that you're asking are the questions that
governments are asking themselves. What is this? What does it mean? How do we safely
regulate it?

Transcript by Rev.com

This transcript was exported on Sep 10, 2024 - view latest version here .

Mark Masselli (10:27):
Well, I think that's a good beginning lit search for people to get a primer on most recent research.
I want to talk about your role as editor-in-chief and also the attitude and behavior of AI. We
understand you're committed to making the publication a trusted authority for promoting the use,
and I think you say of safe, unbiased, and efficacious AI applications.

(10:51):
I'm wondering if we can talk about the unbiased part. There's worries AI is just going to replicate
the biases we already have in society. How are we going to safeguard against that situation and
how should we frame that issue up?

Dr. Issac Kohane (11:09):
Well, I think that, and this I hope will be a recurring theme in our remaining few minutes, is we
always have to ask ourselves compared to what? And yes, there are biases in our healthcare
system, and these are biases that are implemented by hundreds of thousands of healthcare
professionals. How easy is it to unbias those hundreds of thousands of healthcare professionals
today? Very difficult. It's almost unimaginable.

(11:42):
But you can actually run tests on these AI programs and say, oh, would you propose that
diagnosis more often if this was an African American or an Indian American? And you can
actually adjust these programs to reflect that bias so that they en masse are changing their
behavior.

(12:10):
But that raises a kind of opposite question, which is right now, yes, a lot of the biases come from
the data, but I think that most people are less aware of how many more biases, most of them with
good intent, come from what's called steering or alignment of these models. After these models
are trained up, it turns out they will talk about pretty uninteresting things. And so one of the early
understandings in the last three, four years was that the use of reinforcement learning and the so-
called instruct model to steer these models towards interesting conversations and to acceptable
conversations was important.

(13:01):
So for example, you would not want a program to respond to a person with a detailed request for
suicidal procedures or for lethal homicidal procedures. And OpenAI and others have worked a
lot on that steering. Now-

Mark Masselli (13:24):
And when you say steering, this is really prompting of AI in terms of how we frame up our
questions?

Dr. Issac Kohane (13:31):
It's implemented, it's more than just prompting. What it does is it creates essentially a layer on
top of these programs where it says, here are a bunch of prompts, here are the kind of answers

Transcript by Rev.com

This transcript was exported on Sep 10, 2024 - view latest version here .

we're expecting for those prompts. And then there's an automation process whereby we start
saying to it, give us prompts. We'll just rate your answers to those prompts.

(14:01):
And in a somewhat amusing example of perhaps an overshoot, Google famously in its early
steering of Gemini, was so concerned about representation that it overshot. And so when you
asked it for pictures of Nazi soldiers, they were showing African Americans or Africans Nazi
soldiers. And so that's being taken care of and it's no longer such a problem. But here's where,
looking forward, we have to ask what values are going to be inserted in. And you can imagine
when you see a doctor, most of the time until they start talking about the yacht they want to buy,
you assume that your interest as patients are the ones that are being optimized for.

(14:52):
But who are developing these models? Who's interested? Do they have the most? Could be
public health, which is not always aligned with personal health. It could be the healthcare
institutions, which is also not always aligned. And so those questions are in my mind more
important than the bias questions.

Margaret Flinter (15:16):
Well, the well-known phrase, first do no harm, which is always kind of characterized, we think
medicine and all the healing professions, sounds like a good one to bear in mind for all those
designers, particularly around the safety and harm issues that you talked about. But along those
lines, we recently had Alondra Nelson, the Biden administration's leader on AI, on the show,
who helped design a blueprint for an artificial intelligence bill of rights.

(15:47):
And I'm curious what your thoughts are on that. It's supposed to be focused on ensuring safe,
secure, trustworthy technology, as we've been describing. But what are your thoughts about the
governmental effort and are we at a place yet where we're ready to have a bill of rights, and who
are the bill of rights for, patients, healthcare? Or is that just one of the long list of questions that
you're grappling with?

Dr. Issac Kohane (16:11):
Well, I think that it's really solitary that the government has moved as it has. But I think even the
government would say that all they're doing is providing a framework, not detail. And so think
about how long medical professional societies wrestle with what's the right time to do a
mammogram, when to do a ... And there's no way that a governmental authority can work at that
level.

(16:46):
And so I think we can have broad brushstrokes, but the actual consequences of it are going to
depend on that steering, the insertion of our values, our medical values, into it. And you can
imagine dystopian things. Well, we just published a paper in the New England Journal of
Medicine along with a faculty, Arjun Manrai, Raj Manrai, where we show that if you take a case
of a child who is borderline growth hormone deficient and short, the advice given is very

Transcript by Rev.com

This transcript was exported on Sep 10, 2024 - view latest version here .

different when you tell it to give the advice as an insurer or as a pediatric endocrinologist. And
those are, of course, real reflections of the system.

(17:40):
And so these are not invisible bias questions. These are steering questions, steering and
alignment questions. And I think that in some sense, it's a misdirection. People want us to focus
on the bias issues, because they're important. But the actual explicit intent of these programs is
far more important.

(18:01):
I will say, because I know that we have finite time, I want to also focus on the fact that we have
to compare the deployment of these AI systems to the current state-of-the-art. And I can tell you
that when I have faculty coming to my department for the first time, that we just recruited, and
they come to me and they say, "Zach, where am I going to get my primary care?" I've done this
enough times to know that I cannot get them good primary care in Boston, at this central mecca
of healthcare. If you look at Mass General Brigham, all the primary care clinics are not accepting
new patients anymore.

Margaret Flinter (18:47):
Hence the famous headline, "Nobody Can See You Now," in the Boston Globe not too long ago.

Mark Masselli (18:51):
Well, we are in the primary care space, and we understand that very clearly. It's a real public
health crisis that Americans have to deal with. I do want to go talk a little bit about the
regulatory-

Dr. Issac Kohane (19:04):
Wait, wait, wait.

Mark Masselli (19:05):
Go ahead.

Dr. Issac Kohane (19:06):
I've got, so here's my point. In that context, consider the case of a child who was having
difficulty walking, difficulty chewing, intractable headaches, was seen by 17 doctors over three
years, multiple imaging studies, no diagnosis. The mother was told at the end of three years, she
should consider having this kid seen by a psychiatrist. The mom, non-doctor, takes all that text
from all the reports, shoves it into GPT-4, and GPT-4 says this could be tethered cord syndrome,
where the bottom of the spinal cord is trapped in the bones.

(19:55):
And so she goes to a neurosurgeon where she lives in Michigan and asks the neurosurgeon,
"Could this be tethered cord?" Neurosurgeon looks at the imaging study, says, "Yep, that's what
it is." They do the surgery, the child's cured. Now what had happened is that originally the
imaging had been misinterpreted, and then everybody never looked at the original image. GPT-4
didn't get to see the image either, but it could infer from all the findings what was going on.

Transcript by Rev.com

This transcript was exported on Sep 10, 2024 - view latest version here .

(20:26):
So my point is, in the era where patients do not have access to the conversations in primary care,
where even if they do have access to primary care after N months, it's a very short visit, and they
have more questions, they're out of luck. We have to ask ourselves, what is the risk-benefit ratio
of providing patients access to this, with all its warts, given the fact that they do not have access
to expert care?

(20:59):
And the alternative, the other alternative, Dr. Google, search, has many, many more problems.
And therefore, I think that the reality-based conversation is what is the meaning of having
unleashed this intelligent program with all its flaws in our healthcare system where we're falling
down on the job?

Mark Masselli (21:25):
Yeah. Well, it's kind of interesting, and we subscribe to this view that our patients actually have
a lot of AI that they will probably use, which is actual intelligence, where they will take that
information and put it in the appropriate things and strive for the answers.

(21:47):
I want to talk about the regulatory environment. You talked a little bit about China. America
really has, it's sort of Wild, Wild West. They haven't really given any guidelines. The EU is
much different, though. They've progressed further. What's your sense of this regulatory
environment? Because it's going to have a big impact on researchers and on advancements.

Dr. Issac Kohane (22:10):
Well, I think that it's a complicated question because of the fact that on the one hand, you're
right, there's not a whole lot of actual definition of what constitutes acceptable clinical
performance in this country. But I held a conference in Maine, an international conference, just
invitation only with around 60 experts from around the world, back in October of 2023. And
what was very interesting is after the conference, a lot of the Europeans came to me and they
said, "We're really thankful to come to your conference, that we came to your conference,
because yes, there's a lot of advanced legislation coming forward in Europe, but it's all about the
harms and the risks, and it's not about the potential benefits. And we have 400 million citizens
who are probably getting inadequate care."

(23:18):
And so I would hypothesize based on this kind of feedback and others like it, that there is on the
one hand a fearful and overly focused on risk approach that ignores the potential benefits. And
on the other hand, a totally laissez-faire system where we're not asking on which data was it
trained, to ask for transparency? How is it being aligned or steered? And because it's early days,
it's possible that we get locked down in one or more of those extreme modes. But I'm optimistic,
actually, that there are leaders of goodwill in all these different countries. And so that after some
possible early missteps, they're actually going to focus more on the middle, which is a balance of
risk and benefit, given the current performance of our healthcare system.

Transcript by Rev.com

This transcript was exported on Sep 10, 2024 - view latest version here .

(24:37):
And so that is a conversation that has to include the not only patients, but all the medical
societies. And I would argue that the medical societies have been way too bashful, too shy. They
need to be engaged right away in this conversation. I wrote a book back in the end of 2022 with
my friend Kerry Goldberg and Peter Lee, the president at Microsoft, head of Microsoft Research,
and we wrote the book about GPT-4 in medicine and beyond, because we felt there was not a
conversation happening. And there would be a bunch of advanced policymakers who would
know about it and some advanced technologists who would know about it.

(25:32):
But this is fundamentally a, how do we see ourselves, a society question. It's a values question.
And that's why your podcast is so important. We need to engage our healthcare professionals.
And I see a great counterexample of how not to do it in what happened to electronic health
records.

(25:59):
We decided that electronic health records were not our job. It was the job of the chief
information officer and the C-suite in our hospitals. And because of that, we abdicated all
responsibility in the usability of these products, in what job they got done. And as a result, we
contributed mightily to disaffection, moral injury-

Mark Masselli (26:23):
Great analogy.

Dr. Issac Kohane (26:24):
And burnt out doctors.

Margaret Flinter (26:25):
And part of why there's fewer primary care providers-

Dr. Issac Kohane (26:29):
That's right.

Margaret Flinter (26:30):
... to see people. But I want to just stay on that point for one minute, and I know we're getting
close to the end, but there's a pattern of it taking a long time to bring into the education of our
next generation of healthcare professionals, providers, everybody, these kinds of mega changes
that will really affect the course of care and the course of their careers. What are you hearing,
seeing, in terms of schools of medicine, nursing, dentistry, other schools really embracing,
including the use of AI in their medical education programs?

Dr. Issac Kohane (27:09):
So I think that at the top, at leadership in medical schools, there's real interest in including AI as
an interesting part of education. But that turns out to be very hard to do because shifting medical
education is one of the really hard lifts. So I can tell you the dean of the medical school at
Harvard, for example, George, is very determined that this should happen. And yet a lot of our

Transcript by Rev.com

This transcript was exported on Sep 10, 2024 - view latest version here .

faculty are not yet using it in that fashion. And therefore I think we'd have some difficulty
teaching it. We'll make progress, but what's clear is our students today are already using it.

Margaret Flinter (27:52):
That's right.

Dr. Issac Kohane (27:52):
And so it's an odd situation where the medicine of the future is clear to our young students and
our establishment is not very clear how to educate it.

Mark Masselli (28:08):
Let's talk a little bit, just one last question about the future. AI isn't sentient yet. Five years out,
where will we see this future of AI, in your mind?

Dr. Issac Kohane (28:20):
So I think it's going to be because we're seeing a broader workforce problem in pediatrics in
primary care and geriatrics, it is, I think five to 10 years from now, you'll see an increasing
fraction of non-MD professionals such as nurse practitioners, physician assistants, augmented by
AI to perform at a very high level. You'll see a lot of the interpretive studies, such as
echocardiograms, being interpreted not by cardiologists, but directly from the echocardiographer,
who's not an MD, going through an AI to give a very detailed cardiac readout to the primary care
professional who might not be an MD.

(29:08):
And so I believe you'll see in the five to 10 year throughout a lot of non-classical MDs clinicians
using AI routinely as an augmentation. And I also predict this, this is the United States, after all,
a kick in the pants from a few legal cases where someone missed something because they did not
use AI.

Margaret Flinter (29:36):
Well, nurse practitioners and physician associates are already 50% of the workforce in primary
care and community health centers and more than that in many communities. So it's great to have
an additional tool, but the good news is when we say who will see you, they're answering the
question much of the time.

(29:51):
But Dr. Kohane, thank you for your thoughts, for your work, for your insights, and thanks to our
audience for being here. Be sure to subscribe to our videos on YouTube, find us on Facebook
and X with our account name CHC Radio. And as always, go online to CHCradio.com to sign up
for our email updates, and please share your thoughts and your comments about this program and
all our programs. Dr. Kohane, thank you so much. The journal is terrific and we look forward to
continuing to learn from it.

Dr. Issac Kohane (30:19):
Thank you very much, and what an interesting conversation. I really appreciate it.

Transcript by Rev.com

This transcript was exported on Sep 10, 2024 - view latest version here .

Mark Masselli (30:22):
That's great.

Margaret Flinter (30:22):
Great. Thank you.

(30:27):
This copyrighted program is produced by Conversations on Healthcare and cannot be reproduced
or retransmitted in whole or in part without the expressed written consent from Community
Health Center Inc. The views expressed by guests are their own, and they do not necessarily
reflect the opinion of conversations on healthcare or its affiliated entities.

Transcript by Rev.com