LLM Use to Identify Adolescent Patient Portal Account Access by Guardians (2024)

Research Letter

Health Informatics

April S.Liang,MD1; ShivamVedak,MD, MBA1; AlexDussaq,MD, PhD1; et al Dong-HanYao,MD1; KeithMorse,MD, MBA1,2; WuiIp,MD2; Natalie M.Pageler,MD, MEd1,2

Author Affiliations Article Information

The 21st Century Cures Act mandates electronic health record (EHR) access for patients and their legal representatives. In its balance, the Health Insurance Portability and Accountability Act (HIPAA) and state minor consent laws stipulate that adolescents can consent to specific health services and have certain privacy rights over related data.^1,2 To reconcile these legal requirements, patient portals offer differential access to the health record for adolescent vs parent and/or guardian proxy accounts. However, 64% to 76% of adolescent accounts are directly accessed by guardians,³ jeopardizing confidentiality and potentially affecting adolescents’ willingness to engage with care.⁴ Our institution developed a rules-based natural language processing (NLP) algorithm to detect direct guardian access of adolescents’ primary accounts through message content analysis³; however, low sensitivity and manual workflow limited its utility. Large language models (LLMs) have excelled in natural language-based medical tasks,⁵ and emerging EHR–LLM integrations provide opportunities for seamless workflow. In this study, a LLM’s ability to detect guardian authorship of messages originating from adolescent patient portals was tested.

This single-site diagnostic/prognostic study describes the GPT-4 (Open AI; model gpt-4-32k-0613) LLM’s performance at identifying parent- and/or guardian-authored portal messages. Messages from adolescent patient portal accounts at Stanford Children’s Health between June 1, 2014, and February 28, 2020, were sampled and manually reviewed for authorship as described in the study by Ip et al.³ Two prompts were iteratively engineered on a stratified random subset of 20 messages until perfect performance (100% sensitivity and specificity) was achieved: one focused on authorship identification (single task, eMethods in Supplement 1) and another that generated a response to the message and identified authorship (multitask, eMethods in Supplement 1). Both prompts were tested on remaining messages using our institution’s personal health information–compliant LLM (eFigure in Supplement 1) with our NLP algorithm's performance as a benchmark (eMethods and eTable in Supplement 1). To account for correlated data, performance on 1 randomly selected message per patient was analyzed (eMethods in Supplement 1). Positive predictive values (PPV) and negative predictive values (NPV) were calculated from the tested sample, then mathematically modeled on varying prevalences (eMethods in Supplement 1). The 95% CIs were calculated using the Clopper-Pearson exact method. Statistical analysis was performed with JavaScript ECMAScript 2023 from December 2023 to April 2024.

Of the 2088 test messages, 1500 (71.8%) were labeled as parent- or guardian-authored and 588 (28.2%) as patient-authored. The single-task LLM achieved a sensitivity of 98.1% (95% CI, 97.3%-98.8%), and the multitask LLM achieved a sensitivity of 98.3% (95% CI, 97.5%-98.9%). The single-task LLM achieved a specificity of 88.4% (95% CI, 85.6%-90.9%); and the multitask LLM achieved a specificity of 88.9% (95% CI, 86.1%-91.4%) (Table). This corresponded to PPV and NPV greater than 95% for multitask LLM, and the classifiers’ PPV and NPV exceeded 90% on the previously reported prevalence range³ (Figure). Single-task and multitask classifiers performed statistically identically, and removing correlated data did not significantly affect classifier performance (Table).

This study’s LLM-based classifiers accurately detected guardian authorship of messages sent from an adolescent patient portal, achieving PPV and NPV exceeding 95%. This LLM had significantly better sensitivity and NPV than our current NLP algorithm and could enhance adolescent confidentiality, identifying more instances of direct guardian access with a relatively small increase in false positives. Our head-to-head comparison of different prompts reassuringly showed no performance deterioration despite the added cognitive burden of drafting a response in the multi-task large language model classifier. Therefore, these results suggest that EHR integrations can perform both tasks in a single LLM interaction, presenting a scalable application for clinical use. Limitations included single-site data, exclusions of non-English messages, and small number of unique patients. Additionally, expert review may have misidentified the author. Challenges for implementation included the need for an HIPAA-compliant LLM instance, accounting for instances where patients permitted direct portal access by parents and/or guardians, and thoughtful communication around false-positive cases. Ultimately, reliable identification of nonpatient-authored messages has implications beyond adolescent medicine. Among adults, care partners commonly access patient portals using the patient’s credentials,⁶ especially relevant for geriatric patients or individuals with developmental differences. Our results found that this study’s LLM has potential in improving safeguards for patient confidentiality.

Accepted for Publication: April 23, 2024.

Published: June 25, 2024. doi:10.1001/jamanetworkopen.2024.18454

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2024 Liang AS et al. JAMA Network Open.

Corresponding Author: April S. Liang, MD, Division of Clinical Informatics, Stanford University School of Medicine, 453 Quarry Rd, MC 5660, Palo Alto, CA 94304 (asliang@stanford.edu).

Author Contributions: Dr Liang had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: All authors.

Acquisition, analysis, or interpretation of data: Liang, Vedak, Dussaq, Yao, Ip.

Drafting of the manuscript: Liang, Vedak, Yao, Pageler.

Critical review of the manuscript for important intellectual content: Vedak, Dussaq, Yao, Morse, Ip, Pageler.

Statistical analysis: Dussaq.

Administrative, technical, or material support: Liang, Vedak, Dussaq, Yao, Morse, Pageler.

Supervision: Ip, Pageler.

Conflict of Interest Disclosures: Dr Ip reported he is an employee of nference and has financial interest in the company. No other disclosures were reported.

Data Sharing Statement: See Supplement 2.

Additional Contributions: We thank Stanford Children’s Health data scientists Conner Brown, BSE, and William Haberkorn, BA, for executing the large language model application programming interface to generate the classifier outputs. We thank Stanford University School of Medicine clinical assistant professor of anesthesiology James Xie, MD, and former Stanford Children’s Health senior data scientist Austin Powell, MS, for their prior work developing the natural language processing algorithm. They were not compensated.