Healthy AI: 2024 Year in Review

2024 Year In Review 1 Healthy AI 2024 Year In Review 2024 Year In Review 2 Introduction 2024 marked a notable year in AI and healthcare, with AI being top of mind for all healthcare players, including providers, technology companies, developers and regulators. The adoption of AI into clinical setÝngs became more common, as scribe and clinical-decision support products gained popularity and EMR vendors incorporated AI tools into their products. The federal government released guidance, established task forces and implemented the directives of the 2023 Executive Order on AI. Similarly, state regulation began to unfold with some states passing legislation around AI's use in healthcare. This publication provides an overview of important developments at the intersection of AI, healthcare and the law in 2024. We cover key actions by the federal government and states as well as other industry developments and trends. Lastly, as we begin a new year and presidential administration, we share insights into what we can expect in 2025. AUTHORS Carolyn Metnick Partner 312.499.6315 [email protected] Bio Arushi Pandya Associate 202.747.3228 [email protected] Bio Esperance Becton Associate 202.747.3230 [email protected] Bio Alexandria Foster Associate 202.747.3267 [email protected] Bio Tina Watson Law Clerk 212.896.0678 [email protected] Michael Sutton Associate 469.391.7455 [email protected] Bio Christina Nguyen Associate 212.896.0676 [email protected] Bio Table of Contents Federal Law Developments and Trends ....................................4 Federal Agency Guidance: HHS AI Strategy and Guidance...........................4 Federal Agency Guidance: AI in Healthcare Safety Program ........................5 Federal Agency Guidance: FDA Guidance.........................................................5 HHS Final Rule on Health Data, Technology, and Interoperability..............5 Enforcement Activity..............................................................................................6 Legislative Developments .....................................................................................6 Key Takeaways.........................................................................................................6 State Law Summary and Trends ..................................................7 California ..................................................................................................................7 Colorado ..................................................................................................................9 Utah ...........................................................................................................................9 Other Developments and Trends..............................................10 WHO Guidance on LMMs ..................................................................................10 EU AI Act.................................................................................................................10 CHAI Framework for Responsible Health AI ..................................................10 AI in Utilization Management ............................................................................11 Key Takeaways.......................................................................................................11 2025 Forecast................................................................................12 Sheppard Mullin Healthy AI.......................................................13 2024 Year In Review 4 Federal Law Developments and Trends While there is currently no federal law governing AI, in 2024 we saw significant advancements in the federal government's approach to regulating and promoting the use of AI in healthcare through the completion of tasks outlined in President Biden's 2023 Executive Order on Safe, Secure, and Trustworthy AI (Executive Order or EO 14110). Executive Order on Safe, Secure, and Trustworthy AI (EO 14110) On October 30, 2023, President Biden signed Executive Order 14110, laying the groundwork for the ethical and secure deployment of AI across industries, including healthcare. This comprehensive directive outlines policy goals to promote competition in the AI industry, prevent AI-enabled threats to civil liberties and national security, and ensure U.S. global competitiveness in AI. It mandates federal agencies to appoint chief AI ofÏcers (CAIOs) and develop and implement guidelines for AI use to help ensure AI systems are safe, trustworthy and aligned with public values, including in healthcare setÝngs. As a result, the U.S. Department of Health and Human Services (HHS) appointed a CAIO tasked with drafting AI policies and overseeing their integration into public health systems. The Food and Drug Administration (FDA) and the Centers for Medicare & Medicaid Services (CMS) established specialized AI task forces led by these ofÏcers. Additionally, the National Institute of Standards and Technology (NIST) updated its AI Risk Management Framework in early 2024, providing detailed guidelines for identifying, assessing and mitigating risks associated with AI technologies. Federal Agency Guidance: HHS AI Strategy and Guidance In response to the Executive Order, HHS released a detailed AI Strategy, prioritizing the application and development of AI across HHS. The strategy emphasizes leading health AI innovation, partnering with the health ecosystem and ensuring responsible AI use. Subsequently, in April 2024, HHS shared its plan for promoting responsible use of AI in automated and algorithmic systems, particularly for state, local, tribal and territorial governments administering public benefits. This plan aims to ensure that AI systems are used ethically and effectively in public health contexts. Of note, the HHS Acting Chief AI OfÏcer and Assistant Secretary for Technology Policy, Micky Tripathi, announced at the NVIDIA AI Summit in Washington, D.C., that HHS is working on its new AI strategic plan, which should arrive in January 2025. 2024 Year In Review 5 Federal Agency Guidance: AI in Healthcare Safety Program The Executive Order emphasized the need for robust regulatory mechanisms to oversee AI safety and security. The Agency for Healthcare Research and Quality (AHRQ) launched the AI in Healthcare Safety Program as part of its existing Patient Safety Organizations Program. This initiative centers on leveraging AI technologies to enhance patient safety, with a key focus on reducing medication errors and improving the accuracy of clinical decision support systems. In addition, AHRQ implemented AI safety audits for healthcare facilities adopting AI-based tools to help ensure compliance with federal standards. In October 2024, AHRQ hosted a virtual meeting to review AI's role in mitigating healthcare-associated risks and discuss the safety program's implementation. Federal Agency Guidance: FDA Guidance On December 4, 2024, the FDA issued its Final Guidance on Predetermined Change Control Plans (PCCPs) for AI-Enabled Device Software Functions. This guidance provides recommendations for manufacturers on including PCCPs in marketing submissions for AI-enabled device software functions (AI-DSFs). A PCCP allows for pre-authorized modifications to AI-DSFs, facilitating iterative improvements without the need for new marketing submissions for each change. The guidance outlines the types of modifications suitable for inclusion in a PCCP and details the necessary information to support such plans. In addition, the FDA's Center for Drug Evaluation and Research (CDER) has announced plans to issue draft guidance titled "Considerations for the Use of Artificial Intelligence to Support Regulatory Decision Making for Drug and Biological Products." This guidance aims to provide recommendations on using AI-generated data to support regulatory decisions related to the safety, effectiveness or quality of drugs and biological products. The content is informed by stakeholder feedback and the FDA's experience with numerous submissions incorporating AI components. These guidance documents reflect the FDA's commitment to providing a regulatory framework that supports the safe and effective integration of AI technologies in medical products. They offer clarity to developers and manufacturers on the FDA's expectations while promoting innovation and ensuring patient safety. HHS Final Rule on Health Data, Technology, and Interoperability HHS finalized the "Health Data, Technology, and Interoperability: Certification Program Updates, Algorithm Transparency, and Information Sharing" rule, commonly referred to as the HTI-1 Final Rule, which aims to enhance the interoperability of health information technology (Health IT), improve transparency and support the access, exchange, and use of electronic health information. While this rule involves a number of interoperability enhancements, it recognizes the increasing use of algorithms in healthcare and it establishes requirements for greater transparency. Health IT developers must now disclose information about the algorithms integrated into their certified Health IT products, including their intended use, limitations and potential biases. This measure aims to build trust and ensure that healthcare providers and patients are adequately informed about the tools influencing clinical decisions. 2024 Year In Review 6 Enforcement Activity In September 2024, the U.S. Federal Trade Commission (FTC) launched "Operation AI Comply," targeting companies that employ AI in ways deemed deceptive or unfair. This initiative led to enforcement actions against five companies, including those offering AI-powered legal services and AI-generated reviews. While not exclusively focused on healthcare, this operation underscores the FTC's commitment to scrutinizing AI applications across various sectors. These enforcement activities indicate a growing vigilance by federal agencies to ensure that AI technologies in healthcare are deployed ethically, transparently and in compliance with existing laws and regulations. Legislative Developments In February 2024, the "Healthcare Enhancement And Learning Through Harnessing Artificial Intelligence Act" or the "Health AI Act" (H.R. 7381) (HAA), was introduced in the U.S. House of Representatives. The HAA proposes a grant program to facilitate research on the use of generative AI in healthcare, among other purposes. Specifically, it directs the National Institutes of Health (NIH) to establish a grant program that will award eligible entities to perform research regarding the use of generative AI to: (i) improve the ability of health care practitioners to record comprehensive notes or ask medically relevant questions during an appointment with a patient; (ii) reduce the administrative or documentation burden on clinicians; (iii) expedite the health insurance claims process; (iv) improve the efÏciency and quality of customer service in the health care sector; or (v) otherwise improve health care, as deemed appropriate by the NIH. While there has not been much progress with this proposed legislation (which is likely in part due to the upcoming report from the House Task Force on AI providing AI guidelines and recommendations), we anticipate more legislative movement in 2025 since the release of the House Task Force on AI report. On February 20, 2024, the U.S. House of Representatives established the bipartisan House Task Force on AI to explore how Congress should pursue AI innovation while considering appropriate guardrails that safeguard against current and emerging threats. On December 17, 2024, the House Task Force on AI released its comprehensive (253-page) report, which includes guiding principles and recommendations for Congress as it develops policies on AI regulation and optimization. The task force highlighted the beneficial role AI is playing in healthcare, such as improving drug development and clinical decision-making, as well as challenges it brings, including the limited understanding of AI, regulatory uncertainty, data privacy concerns, and hesitancy over AI's role in insurance decisions and coverage. The report concluded that AI has the potential to enhance healthcare and ultimately lead to greater efÏciency, better patient care and improved outcomes. These legislative efforts continue to underscore a commitment to the advancement of AI technologies in healthcare, setÝng the stage for possible legislative action in 2025. Key Takeaways The federal government's actions in 2024 highlight a commitment to integrating AI into healthcare responsibly. By balancing innovation with rigorous oversight, these measures aim to unlock AI's potential to improve health outcomes while safeguarding patient rights and equity. The year's initiatives set the stage for continued progress in harmonizing AI technology with the complexities of healthcare delivery. 2024 Year In Review 7 State Law Summary and Trends Throughout the year, state legislators introduced bills that addressed a range of AI issues, including deepfakes, discriminatory practices, consumer protections, cybersecurity measures, disclosure requirements, and several other regulatory and policy matters. Additionally, many states, including Florida, Maryland, Massachusetts and Washington introduced AI task forces or similar advisory bodies or commissions to perform tasks such as evaluating the implementation of AI, analyzing other states' AI regulations and providing policy recommendations. Three states in particular (California, Colorado and Utah) made significant strides in passing transformative healthcare AI laws in 2024. California California enacted a series of AI laws in 2024, including AB-2013, SB-1120 and AB-3030. The AI Training Data Transparency Act (AB-2013) will require developers of generative AI systems to publish on their website documentation regarding the generative AI system or service, including a high-level summary of the datasets used to develop the AI system. The definition of "developer" is broad and includes any person that "designs, codes, produces, or substantially modifies" an AI system for public use. The law goes into effect on January 1, 2026. The Physicians Make Decisions Act (SB-1120) requires a healthcare service plan or disability insurer that uses AI for the purpose of utilization review or utilization management functions, or contracts with an entity that uses such tool to comply with certain requirements, including that the AI bases its determination on fair and equitably applied information. It further requires physician oversight in that any denial, delay, or modification to a healthcare service be based on medical necessity and made by a licensed physician. 2024 Year In Review 8 A "health care service plan" includes health plans that are licensed by the California Department of Managed Health Care (DMHC). Significantly, SB1120 incorporates a number of specific requirements that are applicable to the use of an AI tool that has utilization review or utilization management functions, including, most significantly: * The AI tool must base decisions as to medical necessity on: ᴏ The enrollee's medical or other clinical history; ᴏ The enrollee's clinical circumstances, as presented by the requesting provider; and ᴏ Other relevant clinical information contained in the enrollee's medical or other clinical record. * The AI tool cannot make a decision solely based on a group dataset. * The AI tool cannot "supplant health care provider decision making." * The AI tool may not discriminate, directly or indirectly, against enrollees in a manner which violates federal or state law. * The AI tool must be fairly and equitably applied. * The AI tool, including specifically its algorithm, must be open to inspection for audit or compliance by the DMHC. * Outcomes derived from use of an AI tool must be periodically reviewed and assessed to ensure compliance with the California Act as well as to ensure accuracy and reliability. * The AI tool must limit its use of patient data to be consistent with California's Confidentiality of Medical Information Act as well as HIPAA. * The AI tool cannot directly or indirectly cause harm to enrollees. Further, a party subject to SB-1120 must include disclosures pertaining to the use and oversight of the AI in its written policies and procedures. It must establish the process by which it reviews and approves, modifies, delays, or denies, based in whole or in part on medical necessity, requests by providers of health care services for plan enrollees. Most significantly, SB-1120 provides that a determination of medical necessity must be made only by a licensed physician or a licensed health care professional who is competent to evaluate the specific clinical issues involved in the health care services requested by the provider. In other words, the buck stops with the provider. AI cannot replace the provider's role. The law became effective on January 1, 2025. Finally, AB-3030 will require a health facility, clinic, physician's ofÏce, or group practice that uses generative AI to generate written or verbal patient communications regarding patient clinical information to provide: (i) a disclaimer that indicates to the patient that a communication was generated by generative AI; and (ii) clear instructions describing how a patient may contact a human health care provider, employee, or other appropriate person. The law exempts any communication that is read and reviewed by a human licensed or certified health care provider and became effective on January 1, 2025. 2024 Year In Review 9 Colorado On May 17, 2024, Colorado enacted the Colorado Artificial Intelligence Act (CAIA), making it the first state to enact a comprehensive law that governs the development and deployment of AI systems. The CAIA applies to any developer of a high-risk AI system and requires developers and deployers to use reasonable care to protect consumers from any known or reasonably foreseeable risks of algorithmic discrimination within high-risk systems. A "high-risk AI system" means any AI system that, when deployed, makes, or is a substantial factor in making, a consequential decision. In brief, the CAIA is designed to protect against the use of AI that results in unlawful differential treatment or impact that disfavors an individual or group based on protected classes, such as age, ethnicity, disability, genetic information, national origin and race. The CAIA goes into effect on February 1, 2026. Utah On March 13, 2024, Utah enacted the Utah Artificial Intelligence Policy Act (UAIP), making it the first state to enact a major AI-focused consumer protection law. The UAIP regulates "generative artificial intelligence," which is defined under the UAIP as an artificial system that: (i) is trained on data; (ii) interacts with a person using text, audio, or visual communication; and (iii) generates non-scripted outputs similar to outputs created by a human with limited or no human oversight. Additionally, the UAIP: (i) establishes liability for the use of AI that violates consumer protection laws if not properly disclosed; (ii) created the OfÏce of AI Policy and a regulatory AI analysis program; (iii) enabled temporary mitigation of regulatory impacts during AI pilot testing; (iv) established the AI Learning Laboratory Program to assess technologies, risks, and policy; (v) requires disclosure when an individual interacts with AI in a regulated occupation, meaning an occupation such as medicine or nursing; and (vi) grants the ofÏce rulemaking authority over AI programs and regulatory exemptions. The UAIP went into effect on May 1, 2024. The legal and policy changes surrounding AI in 2024 are indicative of how quickly governmental stakeholders are reacting to industry transformations while simultaneously balancing issues of privacy, security, transparency, bias and ethics. Further, the lack of comprehensive federal AI legislation to date creates an opportunity and a challenge for state lawmakers as they consider the development of future AI laws in 2025. 2024 Year In Review 10 Other Developments and Trends In addition to the new and updated state legislation affecting AI this year, there have been global and industry group developments in the healthcare space affecting AI. Some of these developments and trends include: * The World Health Organization's (WHO) new guidance on the ethics and governance of large multimodal models (LMMs) * The approval by the European Union of the EU Artificial Intelligence Act (EU AI Act) * The Coalition for Health AI (CHAI) draft responsible health AI framework * Mandated clinician oversight in utilization review WHO Guidance on LMMs On January 18, 2024, WHO announced that it was releasing new guidance on the ethics and governance of LMMs - a type of fast-growing generative AI technology with applications across healthcare. The guidance outlines over 40 recommendations for consideration by healthcare providers, governments and technology companies to ensure the appropriate use of LMMs. WHO's new guidance will help ensure that the LMMs will produce fewer false, inaccurate, biased, or incomplete statements, which would help people make more informed health decisions. EU AI Act On March 13, 2024, the European Parliament passed the EU Artificial Intelligence Act (EU AI Act) and the Council approved the EU AI Act in May 2024. The EU AI Act applies only to areas within EU law. The EU AI Act is a legal framework designed to regulate AI across sectors, fostering safe and transparent AI and promoting responsible AI development by employing a risk-based framework. The risk-based framework classifies AI systems into four distinct levels; the higher the risk to cause harm to society, the stricter the rules. The EU AI Act represents a groundbreaking regulatory approach to AI, setÝng global standards for ethical and accountable AI deployment. CHAI Framework for Responsible Health AI On June 26, 2024, CHAI released a draft framework for responsible health AI, which consisted of an assurance standards guide and assurance reporting checklists. The assurance standards guide provides considerations to ensure standards are met in the deployment of AI in healthcare. The assurance reporting checklists provide criteria to evaluate standards across the AI life cycle - from identifying a use case and developing a product to deployment and monitoring. To demonstrate the application of the guidelines, the guide describes six diverse examples showcasing variations in considerations and best practices in real-world scenarios. According to CHAI, the documents are meant to help healthcare organizations evaluate standards across the product life cycle while enabling artificial intelligence innovation. The release of this framework is a critical step towards ensuring the ethical and effective use of AI in healthcare. 2024 Year In Review 11 AI in Utilization Management Key Takeaways In light of a lack of a cohesive federal framework for AI regulation, industry groups such as CHAI and others have stepped in to fill the gap of providing frameworks for responsible AI use. The development of the AI Act and WHO guidance reflected a global trend towards oversight. Lastly, the focus on AI in utilization review in light of the California Act, as well as other state laws and federal actions, is likely just the tip of the spear in terms of AI-related regulation that will develop in the healthcare space. 2024 brought a wave of intensified scrutiny and oversight of AI systems used by insurers in utilization management (UM) and prior authorization processes. UM is a process whereby plans review requests for services (also known as prior authorization requests) in an effort to limit utilization of insurance benefits to services that are medically necessary and to avoid costs for unnecessary treatments. Increasingly, health plans are relying on AI to streamline internal operations, including to automate review of prior authorization requests. In particular, AI has demonstrated some promise of reducing costs as well as in addressing lag times in responding to prior authorization requests. Despite such promise, the use of AI has also raised challenges, such as concerns about AI producing results that are inaccurate, biased, or that ultimately result in wrongful denials of claims. Many of these concerns are based on questions of oversight. That is precisely what some states are targeting through new legislation. As a primary example, and detailed above, California Governor Newsom signed Senate Bill 1120 into law, which is known as the Physicians Make Decisions Act (the California Act). At a high level, the California Act aims to safeguard patient access to treatment by mandating a certain level of healthcare provider oversight when payors use AI to assess the medical necessity of requested medical services and, by extension, coverage for such medical services. Turning to other states, Georgia's House Bill 887 (which was introduced in early 2024 but has not yet been enacted) would prohibit payors from making coverage determinations based exclusively on results derived from AI tools. Similarly, New York's Assembly Bill A9149 and Oklahoma's House Bill 3577 (both of which were introduced in early 2024 but have not yet been enacted) require payors to disclose to providers and enrollees if they use AI in connection with utilization review, among other measures. In addition, Illinois' House Bill 2472 (which was passed in mid-2024 and became effective on January 1, 2024) requires utilization management programs that use AI tools in rendering adverse determinations to use "objective evidence-based criteria" in making such determinations that have been accredited by certain bodies. Further, many states have adopted the guidance of the National Association of Insurance Commissioners (NAIC) issued on December 4, 2023. The "Use of Algorithms, Predictive Models, and Artificial Intelligence Systems by Insurers" recommends that AI tools should be designed to mitigate the risk that the insurer's use of AI will result in adverse outcomes for consumers. The foregoing reflects a variety of approaches through which states hope to grapple with the increasing role of AI in utilization management. The use of AI in utilization management was also the focus of congressional inquiries and federal guidance documents. In January 2024, CMS issued the Intraoperability and Prior Authorization final rule, which addressed AI use in new timeframes for prior authorization requests. In its February FAQs, CMS reiterated that while AI could be used in coverage determinations, it could not be the sole source of medical necessity determinations. In June, certain members of Congress released a bicameral, bipartisan letter urging CMS to increase oversight of AI in coverage decisions for Medicare Advantage plans. 2024 Year In Review 12 2025 Forecast As we enter a new year and new administration, many are wondering, "what's next?" We expect to see states continuing to remain at the forefront of AI regulation and legislation, especially as the new Trump administration is expected to deregulate AI use. Federal AI healthcare legislation remains unlikely in the new administration, suggesting that the industry will continue to regulate itself. President-elect Trump has promised to repeal President Biden's Executive Order, but may prioritize AI innovation and leadership in the private sector. Certain agencies that have long-standing policies of addressing AI use, such as the FDA, may be more immune to administration changes than agencies such as HHS and CMS, which have only begun to address AI in healthcare in the last few years. At the state level, we expect that issues such as transparency, privacy and biases will remain at the forefront and the amount of AI legislation to continue to grow, even in states that may have historically shied away from regulation, such as Texas. We expect that as advancements and innovations in AI proliferate, their adoption in healthcare spaces, from hospitals to patients' homes, will grow. New AI and analytics techniques will continue to shape healthcare delivery. Stakeholders, from hospital systems to technology companies, will continue forming collaborative ventures to further advance AI in healthcare. While much remains uncertain in terms of the regulatory landscape, it is clear that not only is AI in healthcare here to stay, but it is expected to bring groundbreaking advancements in the upcoming year. 2024 Year In Review 12 2024 Year In Review 13 Sheppard Mullin Healthy AI recognizes the unique need for healthcare clients to have industry expertise around the developments relating to AI, healthcare and the law. The sensitivity of health information makes this area particularly challenging in its implication of data privacy, HIPAA, FDA issues and others. Sheppard Mullin Healthy AI is dedicated to helping clients understand and anticipate their legal and enforcement risks when incorporating AI and other emerging technologies into their operations. At a glance: * We are at the forefront of partnering with clients to ensure the development and deployment of healthcare AI technologies in compliance with law and regulations, industry standards, ethical principles and best practices. * We offer a full suite of services focused on healthcare clients, including regulatory compliance, risk management, IP protection and data strategy. * We advise clients on how HIPAA-regulated information and other sensitive information may be used with AI in compliance with law. * We assist healthcare companies in developing AI governance programs, including developing policies and procedures and providing training. * We negotiate with AI vendors to manage risk on behalf of companies contracting to use AI. * We assist providers in developing appropriate consents, notices and other documentation where needed to mitigate risk and comply with law. Sheppard Mullin Healthy AI Carolyn Metnick [email protected] Bio For more information about Sheppard Mullin Healthy AI, including our thought leadership and roundtables, please contact Carolyn Metnick. Connect and share ideas with industry leaders by joining Sheppard Mullin's Healthy AI LinkedIn Group, a hub for networking and innovation in healthcare AI. 2024 Year In Review 2024 Year In Review16