Guardrails for Health AI: How, Why and When

Mark Masselli (00:04)
Experts say artificial intelligence could effectively disrupt healthcare by predicting illnesses and also picking the
best treatments, all while doing away with a lot of paperwork. Our guest is at the center of the promise and
worry about AI and healthcare.

Brian Anderson (00:21)
You use the term self-regulatory. That's absolutely, I think a fair, a fair term to use. These are vendors and
customers coming together, coming up with guidelines and guardrails that they see as helpful for the work
they're doing in the private sector. If state or federal officials wanna make use of those, you know, that's, that's
their call.

Margaret Flinter (00:37)
Dr. Brian Anderson is an internationally recognized author and expert in digital health, innovation, and health
standards. Now he's in charge of the Coalition for Health ai, which aims to provide guidelines for the
responsible use of ai.

Brian (00:52)
There are a lot of challenges in the ai, ecosystem right now. One of them certainly is trust in the healthcare
space. You look at survey after survey, people broadly in society don't trust ai, with really consequential
decisions like those that we face in healthcare.

Margaret (01:08)
This is Conversations on Healthcare.

Mark (01:21)
Dr. Anderson, welcome to Conversations on Healthcare.

Brian (01:25)
Thanks for having me. I'm excited to be here.

Mark (01:27)
Yeah. You know, you've been full-time in your role for almost a year, and let's start, with having you describe
the Coalition for Health ai. Who's in the coalition and what are your goals?

Brian (01:39)
Yeah, so Chai, our Coalition for Health ai, we started in the middle of the pandemic actually in March of 2021
as a coalition of the willing. At the time, many of us were working in Operation Warp speed and on the White
House Covid Task Force focused, of course, in the pandemic. But you had very non-traditional partners. You
had, you know, very competitive pharmaceutical companies. You had health systems, you had big tech
companies and small tech companies all coming together. And we appreciated the impact that we were having
in that space. And one of the questions we asked is, there are simply more that we can be doing beyond the
pandemic. We looked at ai and one of the first questions we asked was, do we have consensus at, at a
technically specific level that informs what responsible AI looks like for a developer or a health system or a
payer deploying it, and for the governance of an AI model over, over its lifetime? And the answer we quickly
came to is we don't, we have a lot of really good work happening at various organizations and companies, and
we thought that, you know, in a consequential space like health, that it's important to have consensus on what,
you know, what good, responsible, trustworthy AI looks like. And so that's really been our focus, is building a
private sector led coalition to build those technical best practice frameworks for the industry to align on, to
accelerate the kinds of innovation and development of AI building trust between vendors and their customers,
like health systems to deploy and use these AI technologies to improve patient outcomes and care. It really
quickly grew from the eight organizations that we started in March of 21. To now we have nearly 3000 different
organizations that are part of C chi. I think it's probably the largest health AI coalition on the planet. Our focus
remains the same, though. It's building that technical consensus, best practice frameworks, enabling vendors and
their customers to come together to accelerate innovation, to be able to test and validate models, to build the
kind of trust, to have common agreements around the kind of transparency that many doctors and nurses and
patients want. And doing that in a private sector led way.

Margaret (03:40)
Well, one of your coalition's accomplishments has been the release of a draft framework for Responsible Health
ai. and the framework I understand includes an assurance standards guide, which is aiming to create standards

for AI and healthcare. I'll just say right off the bat, you took on a pretty big challenge when you took on, this
work. Trying to get consensus around standards is one of the more challenging, things we try to do sometimes in
healthcare. But fill us in more on, on what the framework is all about and any specific examples of your, in your
draft framework about those assurance standards or the guide to assurance standards?

Brian (04:17)
Yeah, well, let me start with one of the, kind of the very basic principles in responsible ai, which is
transparency. The example or metaphor I like to use is when you and I go to a grocery store and we, you know,
look to buy a can of soup and we wanna know what's in it, we turn around the can of soup, we look behind and
we see the nutrition label, and we know, you know, how many carbohydrates, how much protein, how much fat
is in it, what the ingredients are. We have that kind of understanding because we have an expectation of the
transparency that we, we get to know about the things that we're gonna be buying at a grocery store. One of the
challenges in AI and in particular in responsible AI, is a common agreement on the level of transparency for the
customers of AI in partnership with vendors to understand how these models are built, what kind of data sets
were the models trained on. As a physician, myself, I might be very, concerned about knowing if a model was
trained on a particular kind of patient that I might have in front of me. Was it, was it trained on this kind of
patient? Is it appropriate for the patient that I have in front of me? Not having that kind of transparency, not
having that assurance that these models are trained on the kinds of people that I as a physician or a nurse might
be serving mm-hmm. Is, is a real challenge to building the kind of trust. And so when we did this work, we
brought together the vendor community, so big tech companies and small startups as well as their customers,
namely health systems and, and payers and health plans, and said, okay, what is the minimum threshold of
transparency from a responsible AI standpoint that the vendors and their customers can agree to in what we've
now called an AI nutrition label or our model card? That that is that base level of answering questions like,
what's in it, what are the data sets used? What are the indications? What are the limitations of using these kinds
of model? And so it's that kind of hard work building to a technical level of specificity. So getting from the
50,000 foot, you know, we all want transparency, we all want safety, but what does it actually look like? And so
that's the work that we've been focused on. That's some of the examples in the assurance standards guide.

Margaret (06:25)
We've certainly already, seen examples reported of people going down the wrong diagnostic pathway because it
turned out it had never been fed information on different genders or different ages or different backgrounds. So
that, that is really, impressive. Thanks.

Mark (06:40)
I'm, I'm, I'm just a little confused. Is this about self-regulation or is this a crosswalk over to government
regulation? And is this group, that you have, only, within the confines of the us or does it cover, is it a global
alliance that's been built up? We can all go to our app and grab something that's from, you know, the other, other
side of the globe and start using it. So where, where do you go with this framework in terms of the regulatory?
Or is this really about the industry itself self-regulating?

Brian (07:14)
Great question mark. So when we started chi, it started as a private sector led effort, and it quickly, I think, got
the attention of the previous administration. And I think many of us in CHI appreciated that in a rapidly
evolving space like AI and health AI in specific, we want our policy makers and regulators to be informed about
where the innovation community is moving, what some of these technical standards and best practice
frameworks are that, that we're developing. And so we invited them to be part of our working groups. We
invited them to be part of our community. And to this day, you know, we have active engagement from officials
from the FDA, from ONC, from NIH, from va, and they are there to learn where the private sector is moving in
this space. And, you know, certainly as they want to inform regulations and develop them, you know, they of
course are able to do that. They do that separately from CHI though. I would say, you know, where the, where
the, at a federal or state level, they want to use some of the principles or concepts that we're developing in chi.
You know, I think that we're more than happy to help them and educate them. I, I tell many of the legislators,
both at the federal level and state level, you know, to see C CHI as an educational resource.

Mark (08:29)
So It's a, it's a toolkit that people can grab onto and apply where they think it's necessary. But you don't have as
a directive really looking at this as a a, a legislative vehicle, though. Somebody might interpret it and use it as
they so chose at a state level.

Brian (08:49)

That's right. So the work, that example, the work that we're doing with Assurance Labs is purely a private
sector. What I, you might you use the term self-regulatory, that's absolutely, I think a fair, a fair term to use.
These are vendors and customers coming together, coming up with guidelines and guardrails that they see as
helpful for the work they're doing in the private sector. If state or federal officials wanna make use of those, you
know, that's, that's their call. And, you know, happy to, to educate them on the capabilities of some of the things
we're working on. But that's, that's their job, not ours.

Margaret (09:17)
Well, it's gonna be very interesting, and I, I wonder if you wanna comment on this. There's obviously the whole
business component. If you're in healthcare, you said that you're a physician, you certify your lab, you certify
the equipment that you use, you certify your electronic health record, and that that stands for something. So is
there a, is there a business model being developed around this that w would give patients and, and providers
more, confidence in the safety of the information that is being used, in their care?

Brian (09:48)
Exactly right, Margaret. I think that's the hope is if you look at any other sector of consequence in the US
economy, there are independent entities that come up with standards or best practices. There are independent
entities that do the kind of external evaluation of, you know, an airplane, a car, a consumer electrical device, a
toaster oven that we put in our home, that do that kind of evaluation independently and, and, and issue reports
that are publicly available. In this case, those kinds of evaluations and reports within the chai ecosystem, we
intend to make those available to doctors, to nurses, to hospital administrators, the, the public writ large. Mm-
hmm. You are absolutely right. I think one of the ways that we build greater trust in AI is the kind of
transparency about how these things actually perform by independent, trusted entities. And you know, I think
there are a lot of challenges in the ai, ecosystem right now. One of them certainly is trust in the healthcare space.
You look at survey after survey, people broadly in society don't trust ai, with really consequential decisions like
those that we face in healthcare.

Mark (10:52)
I wonder if it's like congress. I don't like Congress, but I like my congressman. I'm wondering on ai, is it that I
have an app that I kind of like, and I use it, but p on all of them, but I do wanna get back to the sort of issue of
cost. And, we've seen the market, upheaval happening when, new products come out online that could
potentially train AI tools at a much lower cost than had previously been predicted. I'm wondering about your
cost structure. What, what you do you see the cost for somebody adopting, your, your standards here in terms of
what it might mean for their, AI product?

Brian (11:37)
So we've tried to make the standards and, and best practice frameworks as adoptable and accessible as possible.
So an example, there are startups in our community. There are big tech companies in our community. And what
we don't want is to create playbooks that only a entrenched incumbent that has, you know, a massive army of,
you know, developers could possibly meet every, you know, item on the checklist. We want to create playbooks
that are accessible, that are deployable and usable across all of the important stakeholders that we have. And so,
you know, that's been our intention since we started, is to develop these frameworks in such a way that it's not
going to be cost prohibitive. It's not going to slow down innovation example, like, one of the reasons why we've
launched these labs is because if you ask any startup out there, particularly in the health space, what's the
number one challenge to developing a, you know, effective product? It's access to data. It's sometimes very cost
prohibitive, and it's challenging to get access to, to that kind of data. And so these labs and the ecosystem that
we're creating enables researchers to have that kind of more easy access to high quality data. Now, on the health
system side, you know, the, there are lots of questions about, you know, how much is appropriate to spend on an
AI tool? How do I measure the economic ROI once I buy it and deploy it? Is it gonna create the kind of
efficiency that I need to recoup the cost that I've, you know, given for, for the, for the purchasing of the, of the,
of the model? That is another separate part of our work is partnering with health systems and vendors to
understand like, what are the agreed upon metrics and evaluation framework to determine if a model is actually
gonna give an ROI back to the health system that, you know, these health systems all have very thin margins
and there's a lot of questions on the cost of the model, and if there's gonna be an economic ROI and we want to
work with our community to develop the frameworks to get greater clarity to that very important question.

Margaret (13:39)

Well, I'm, I'm curious, and maybe in a different, different area, but do your efforts tie in with President Trump's
Stargate announcement that aims to highlight AI's ability to transform healthcare by, having reaching the holy
grail of helping to develop a cancer vaccine?

Brian (13:56)
Well, so I can start by saying certainly the main, parties in Operation Stargate, so open ai, Oracle, are part of chi.
And so we work very closely with them as founding partners of the Coalition for Health ai. Now, Stargate, as
I'm sure probably you and your listeners are aware, is focused mostly on investment on the technical
infrastructure. So data sensors, centers, GPUs, and the infrastructure that that, that you need to, to build these
massive frontier models. Now, in the gen AI space, particularly in healthcare, there is a lot of interest in using
models, for a variety of use cases, AI scribes to help doctors and nurses and seeing patients. So they're not
staring at a computer, they're actually making contact, eye contact with their patients. Mm-hmm. Is a great
example in the generative AI space where many of the vendors work with open AI and work with an EHR, like
Oracle Health's, EHR, to do that important work. So I would say the work in CHI is definitely aligned with, and
part of what is, what I would call the end result of what Oracle OpenAI and SoftBank are looking to do, which
is to build highly capable, effective models to help address, you know, things like helping to solve cancer. There
are many, community members in CHI that are focused in the oncology or the cancer space. So we're excited to
see where that goes, where these, you know, more capable generative AI models, can help solve, you know,
some of the, the urgent questions and challenges we have in the cancer space.

Mark (15:29)
And Dr. Anderson, I'm wondering, there's been some criticism of your group, and it's mostly centered around
the idea that the coalition is, policing itself. You mentioned that you had open AI and Oracle, you also have
Microsoft and Google. I'm wondering, how do you respond to, to those, points that are being made?

Brian (15:50)
Yeah, no, I appreciate that question. When CHAI launched, it was started by a group of doctors in the private
sector, and it was clear that we had a vision and a goal that was, I think, deeply resonant with the broader
ecosystem. So yes, all of the big companies that people recognize the names of, you know, Microsoft or Google
and Amazon, some of the biggest health systems in the US are all part of chai. But at the same time, over 25%
of our members are startups with less than 50 employees. It's gonna take all of us to solve a lot of these
challenges in this space. And so, although yes, we have all of the name brands that people recognize, like Open
AI and others, we also have a lot more names of organizations that people don't recognize. And in Chi, the way
governance works in our working groups is each organization gets one vote. So it's not like any of the, you
know, recognized names get more votes than any one startup. And so, you know, candidly, the, the way
governance works in CHI is it's heavily weighted towards the startup community where they are the ones that
are able to drive the consensus work that we're doing. And it's, you know, not the Microsoft or the Amazon or
the Google that's able to do that, although those are of course, important partners. All of our community
members are equally important in that sense. It is a private sector led effort, though, mark, and so you're right,
like this is private sector innovators coming together trying to develop consensus, but I would say importantly,
it's the vendor community and the customer community which have very different equities and very different,
you know, priorities. Like the transparency example I give you as an important one of developing some of those
guidelines and guardrails. I would be remiss if I didn't mention the other really important community is the
patient community advocate community. Every one of our working groups has a patient community advocate
that is part of it, that is helping to lead it. And so their voice is equally important and central to these consensus
standards and best practice frameworks that were developed.

Mark (17:56)
That's, that's a very important point. Thanks.

Margaret (17:58)
Well, it's a very impressive list of people, and I'm, I'm so glad you added, the patient's voice, which is critical to
research. You know, the one group that you didn't mention is government. And I'm wondering what is the role
for government, at the table? Are the folks from, FDA or any other government, agencies at the table in the
discussions or, or contributing to the standards?

Brian (18:23)

Yeah, so I mean, as I was sharing earlier with the previous administration, we had active engagement in all of
our working groups from F-D-A-O-N-C-A-S-T-P-N-I-H-V-A, OCR, office of Civil Rights, and a number of
other, you know, word salad acronym, government agencies I've shared with the Trump administration and
with, legislators on the hill that the same invitation stands to the current administration. And so we welcome
participation and engagement. I think it is really important for our policymakers and our regulators, regardless
of what party they're from, to know where the private sector is going, so that if they are gonna form regulation
or policy, that it's not being done in a ham-handed misinformed way. Right? We want them to develop
regulations cognizant of where the industry is going and where innovation is going. And I think that's good for
all of us.

Mark (19:15)
I was just wondering about, the politics of it, and obviously the Biden administration is in the rear view mirror,
for the Trump administration, and, they're looking forward, certainly bypassing some of the, proposals that the
Biden administration had laid out, which were generally viewed not as onerous, more sort of guidelines. I'm
thinking about what, what we've learned from the eu, or certainly I heard the Prime Minister in England talking
about AI being one of their engines for it. What are you seeing on the international, arena in terms of the,
pathways that people are taking to address the, opportunity that AI presents?

Brian (19:57)
Yeah, so no, great question, and sorry, I didn't answer it earlier when you asked about the international,
engagement. So when CHAI started, we focused domestically. I think that was just the initial space that we
wanted to really begin building out some of these consensus frameworks. And so last year, 2024 focused solely
on the domestic front. This year, very different. Many of our partners have asked us to engage with the eu, the
organization for Economic Cooperation and Development. Mm-hmm. The OECD, which works closely with the
G 20 and the eu, has engaged with CHAI and asked us to begin developing, for, towards a summit later this
way, later this year, that we will look to engage the European Union and really take, as an example, some of the
unique policies in the EU, GDPR as an example, and, and take the frameworks that we're developing in CHI and
crosswalk that into the regulatory reality and frameworks that many of our multinational companies have to deal
with every day. So how do you as a, you know, multinational company that's both in the US and in, you know,
Denmark or Germany or Britain, develop models, deploy models, govern models in a way that is, adherent to
those regulations and the policies in those nations. And so Chai will actually be launching that work, I think in
Q1 or Q2 of this year, and pairing with, with the international community. We're also gonna be working very
closely, with Singapore, and we will have some exciting announcements to share later in March about that.

Mark (21:29)
I know the, in, in the states at least, there's the, FCEB, which are all the federal agencies that are, focused in on
this and wondering, you know, we, we have, at the present moment, some disruption between the ability to
communicate with those agencies, but the, federal government at least has a platform that has gathered all of
those agencies. I'm wondering how, how your relationship is with that body as well.

Brian (22:01)
In terms of like, you know, the pause in communication.

Mark (22:04)
Yeah.

Brian (22:04)
Is that what you’re referring to, Mark?

Mark (22:05)
Yeah.

Brian (22:05)
Yeah, I mean, so, that news just broke, I think in the last 24, 48 hours. So we're certainly wanting to be,
respectful of President Trump and the administration's desire to put things on pause. And so a lot of the
engagements that we've had this week in terms of, you know, attending community of practice meetings,
speaking to various officials within, HHS agencies, all of that has been put on pause, and so we're waiting to
hear back from them. Mm-hmm. I mean, and, and candidly, it's not a rush. I mean, I, you know, happy to speak
with the government officials when it's helpful and when it's appropriate and, and when, when they want us to.

Certainly on the private sector side, you know, our working groups are meeting daily, and the work that we're
advancing, you know, continues, that's not gonna be put on pause.

Margaret (22:51)
Well, as you, know, I think Mark and I work in the community health center space, both, with the organization,
we lead, but also the network of community health centers around the country that take care of about 31 million
people. I've, noticed in reading some of the background material, so much of your discussions are focused on
big tech and big healthcare systems. You talked about the patient voice. Now, are you trying to bring in
community health centers, safety net hospitals, and others who are really focused on the more vulnerable
populations to this work?

Brian (23:23)
Absolutely. So thank you for asking that question, Margaret. I would say, you know, certainly last year we
learned a lot, and I think one of the, if I'm being honest with myself, one of the maybe appropriate criticisms of
CHAI and the work that we did was that it was very technical and it was very detailed. We published a
responsible AI guide that's 187 pages long. And so that can be a challenge if you're a small community health
center looking to deploy AI in a responsible way. And so I think we've, we've heard that criticism, I've taken
that to heart. And so this year, we're going to be focusing on a major strategic effort working with federally
qualified health centers, community health centers across the nation to take the body of work that we've created
over the past several years, and develop playbooks that are much more accessible and usable to the lower
resourced community centers. Be them in rural Appalachia or, you know, inner city la. And you know, I think
one of the exciting kind of, you know, milestones towards that is, I can share with you, you know, for the first
time publicly here is we're sharing that, Dr. Kyu Rhee, who's the CO of the National Association for
Community Health Centers, is joining our board of directors. And so with Q'S leadership, our team will be
launching a specific, working group effort that will bring in, you know, approximately a dozen FQHCs and
community health centers to do what I just said, to take the work that we've developed and build with them the
kinds of playbooks and tooling that enables them to make use of AI tools in a responsible way. So we're really
excited about that. Great. And the hope is to bring, bring this kind of content and make it available to as many
rural clinics, you know, urban clinics, in the United States.

Mark (25:16)
Well, congratulations. That's a very exciting initiative.

Margaret (25:18)
Thanks for sharing that.

Mark (25:19)
And as a, as a physician, you, I think, one of the things you worry most about, and I think the health centers do
and, and I'm sure all of your other partners do, is what's the best advice for patients as they're dealing with ai? I
think the big question is, should they trust it? And how transparent should healthcare providers be about its use?
And again, just thinking about, from, from, wearing the shoes of a patient.

Brian (25:44)
So, I mean, I can give you my personal opinion on this. I think that health systems and doctors should be as
transparent as possible in the use of AI with their patients. AI is a tool, and as a physician, you know, there's this
principle of informed consent, right? So in terms of how we treat our patients, we inform them of the various
kinds of procedures we're doing, the risks, the benefits, the potential harm that comes to them in the use of any
tool or any procedure. And I think the same principle applies here. I mean, although it's less clear cut because as
a provider, I don't necessarily inform a patient if I'm gonna be using a stethoscope or an ophthalmoscope or an
otoscope, which are also tools that I might use. AI as a tool, I think might have more consequentiality to it than a
stethoscope or an otoscope might. And so I do think that there is a rationale that many providers share with me
that we should be informing our patients and should be having those kinds of informed conversations with our
patient if they want.

Mark (26:42)
Aren't they sitting on the other side of you with their own AI device saying, let, let me check.

Brian (26:48)

They are, yeah. Yeah. No, I mean, I think the, the reality is patients are using these tools, right? Right now every
day. Absolutely. And right. And providers need to take that information that the patients oftentimes are bringing
to them. Seriously. I, I mean, there are many stories out there of patients that, you know, went to a doctor,
doctor didn't know what to do, the patient took their, you know, medical record back to chat to your t right chat
G pt, and then suddenly, you know, they have the diagnosis, they bring it back to the provider and the
provider's, like, you know, of course that's the, that's the diagnosis. So we need to be partnering with our
patients in, in, in using these tools appropriately. I think the one, the one, caution I would, I would offer is
sharing your private medical record with any of these online tools means that you are consenting to share your
private information with, with a company. That's a great point. And just being aware of that.

Margaret (27:50)
Well, I, I have to ask you, since you, you said that you started all this work 20 20, 20 21, so, you know, we are
always living in interesting times, it seems, but that was certainly, interesting as we come up on the fifth,
anniversary of the start of Covid. but you work closely, I understand with the White House Covid Task Force
and with Operation War speed. I'm wondering if there's any lessons that you learned in these roles that you are
having to put to good use now in moving your coalition forward?

Brian (28:21)
Yeah, well, certainly lots of lessons on that. Being comfortable with ambiguity, I think is one of the biggest
lessons I learned in that time in Covid. You know, we didn't know which therapies were gonna work. We didn't,
you know, understand how people and why people were dying when this started, and yet we still worked hard
every day to try to get to the answers. I think very similarly in the AI space, we have a bunch of theories on how
to develop models, how to develop capable models. We have a hope that a GI, you know, is gonna help solve a
lot of the, you know, potential problems and challenges in health, and research. I think being comfortable that
we don't necessarily have all the answers and having a level of humility around that is really important. The
deep seek news that broke just the other day is a great example of, you know, you have a bunch of, I think, very
smart and talented developers that had initially an understanding of, you know, this is the best way to train, you
know, highly capable, generative AI frontier models. And that still may be the case, but we have a level of
uncertainty now with, you know, the R one deep seek model that came out. And so I think ambiguity and being
comfortable with that is one. I think the other challenge is, or the other lesson I've learned is building alignment
of incentives is so critically important, particularly when you have inherently competitive organizations, right?
To have a group of technology companies, all working collaboratively is not easy. Mm-hmm. Having an aligned
incentive of, hey, if we can come to consensus around an industry set of best practices mm-hmm. We can use
that as a framework to accelerate innovation, not slow it down. And so getting people aligned as an example
with that kind of incentive is how you build a coalition with inherently competitive organizations. And so those
are just two examples of, of lessons learned from the pandemic.

Margaret (30:12)
Great. And probably more to come.

Brian (30:14)
Absolutely.

Mark (30:16)
Well, thank you Dr. Anderson, for joining us and for the important work that you're doing. And thanks to our
audience for being here. Just a reminder to be sure to subscribe to our videos on YouTube and find us on
Facebook and X with our account name, CHC Radio. And as always, you can go [email protected] to sign
up for email updates, and you can share your thoughts and comments about this program. Thank you again, Dr.
Anderson and continued success will be following your work.

Margaret (30:48)
This copyrighted program is produced by conversations on healthcare and cannot be reproduced or remitted in
whole or in part without the expressed written consent from Community Health Center, Inc. The views
expressed by guests are their own, and they do not necessarily reflect the opinion of conversations on healthcare
or its affiliated entities.