Regulatory Science Summit

The UCSF-Stanford Center of Excellence in Regulatory Science and Innovation (UCSF-Stanford CERSI) is pleased to present the inaugural Innovations in Regulatory Science Summit, a gathering of leaders in the academia, industry and regulatory sectors to discuss the role of regulatory science in medical product development. This event will take place before the 2020 J.P. Morgan Healthcare Conference, and will comprise primarily of panel discussions and short research talks by CERSI investigators on ongoing collaborative research projects with FDA investigators. Panel topics include:

  • Accelerating Clinical Trials in the Development and Approval of Innovative Medical Products
  • Real-World Evidence, Artificial Intelligence and Novel Medical Devices
  • Academia, Government and Industry in Regulatory Science: Cross-Sector Collaboration and Avoiding Conflicts of Interest
  • Advancing Discovery to First-In-Human Clinical Trials for New Medical Products



PDF iconAgenda

PDF iconSpeaker Biographies

PDF iconPoster Listing



8:00 am


Kuldev Singh, MD, MPH | Professor of Ophthalmology at Stanford University

8:05 am

Overview of UCSF-Stanford CERSI

Russ Altman, MD, PhD and Kathy Giacomini, PhD | Co-Directors of UCSF-Stanford CERSI

8:25 am

Opening Remarks from Stanford

Marc Tessier-Lavigne, PhD | President, Stanford University

8:35 am

Opening Remarks from UCSF

Sam Hawgood, MD | Chancellor, University of California San Francisco (UCSF)

8:45 am

Lightning Talk - Defining the Strength of Evidence in Therapeutic Development

Steven Goodman, MD, MHS, PhD | Associate Dean of Clinical and Translational Research at Stanford

8:55 am

Panel Discussion - Accelerating Clinical Trials in the Development and Approval of Innovative Medical Products

Laura Esserman, MD, MBA | Professor of Surgery at UCSF
Janet Woodcock, MD | Director, Center for Drug Evaluation and Research at FDA

Amy Abernethy, MD, PhD | Principal Deputy Commissioner at FDA
Steven Goodman, MD, MHS, PhD | Associate Dean of Clinical and Translational Research at Stanford
Sandra Horning, MD | Former Chief Medical Officer at Genentech/Roche
Carl Peck, MD | Chairman of NDA Partners
Alfred Sandrock, MD, PhD | Executive Vice President, Chief Medical Officer at Biogen

9:55 am


10:10 am

Lightning Talk - Patient Preferences in Upper Limb Prostheses

Leslie Wilson, PhD | Professor of Clinical Pharmacy at UCSF

10:20 am

Panel Discussion - Academia, Government and Industry in Regulatory Science: Cross-Sector Collaboration and Avoiding Conflicts of Interest

Howard Bauchner, MD | Editor-In-Chief of the Journal of the American Medical Association
Robert Califf, MD | Former FDA Commissioner, Head of Health Policy and Strategy, Google Health and Verily Life Science (Alphabet)

Mildred Cho, PhD | Professor of Pediatrics (Center for Biomedical Ethics) at Stanford
Arthur Ciociola, PhD | Vice President, Head Regulatory Affairs, Global Drug Development Ophthalmology at Novartis
Malvina Eydelman, MD | Director, Office of Health Technology 1 at FDA
Carol Linden, PhD | Director, Office of Regulatory Science and Innovation at FDA
Andrew Weber, MS | Accelerating Therapeutics for Opportunities in Medicine (ATOM), GlaxoSmithKline

11:20 am

Keynote Address

Janet Woodcock, MD | Director of Center for Drug Evaluation and Research at FDA

Moderated by Russ Altman, MD, PhD | Professor of Bioengineering at Stanford

11:50 am


12:40 pm

Lightning Talk - Safer Labeling of Pediatric Medications: Reducing Literacy-related Health Disparities among Chronically Ill Adolescents

Lee Sanders, MD | Associate Professor of Pediatrics at Stanford

12:50 pm

Panel Discussion - Real-World Evidence, Artificial Intelligence and Novel Medical Devices

Adam Gazzaley, MD, PhD | Professor of Neurology, Physiology and Psychiatry at UCSF
Anne Wojcicki | CEO of 23andMe

Russ Altman, MD, PhD | Professor of Bioengineering at Stanford
Edward Chang, MD | Professor of Neurological Surgery at UCSF
Daphne Koller, PhD | Founder & CEO of Insitro
Curtis Langlotz, MD, PhD | Professor of Radiology at Stanford

1:50 pm


2:05 pm

Lightning Talk - The Activity of Inactive Ingredients

Brian Shoichet, PhD | Professor of Pharmaceutical Chemistry at UCSF

2:15 pm

Panel Discussion - Advancing Discovery to First-In-Human Clinical Trials for New Medical Products

Jay Bradner, MD | President of Novartis Institutes for Biomedical Research
Joseph Wu, MD, PhD | Director of the Stanford Cardiovascular Institute

Hal Barron, MD | Chief Scientific Officer at GlaxoSmithKline
Tejal Desai, PhD | Chair, Bioengineering and Therapeutic Sciences at UCSF
Mathai Mammen, MD, PhD | Global Head of Research and Development at Janssen
Peter Marks, MD, PhD | Director, Center for Biologics Evaluation and Research at FDA

3:15 pm

Closing Keynote

Peter Marks, MD, PhD | Director, Center for Biologics Evaluation and Research at FDA

Moderated by George Scangos, PhD | CEO of Vir Biotechnology

3:45 pm

Closing Remarks

Russ Altman, MD, PhD and Kathy Giacomini, PhD | Co-Directors of UCSF-Stanford CERSI

4:00 pm

Reception and Poster Session (co-sponsored by the UCSF School of Pharmacy)

Hosted by B. Joseph Guglielmo, PharmD - Dean, UCSF School of Pharmacy


Please direct questions to Lawrence Lin, Director of External Relations and Outreach at UCSF-Stanford CERSI at [email protected].

Natural Language Processing

Use of Natural Language Processing to Extract Information from Clinical Text

June 15, 2017 | 8:30 am to 5:00 pm

Great Room (Building 31, Room 1503B/C)
FDA White Oak Campus
10903 New Hampshire Ave
Silver Spring, MD 20993


The objective of this workshop is to identify current and emerging natural language processing (NLP) efforts being applied to unstructured text such as clinical notes or narratives in electronic health records (EHRs). The workshop will provide insights into utility and challenges in designing and implementing NLP systems to capture relevant or missing information from clinical notes or text for monitoring postmarketing safety surveillance and informing the design and execution of clinical trials for medical products, which include drugs, biologics, and devices. The workshop will include panel discussion sessions to provide stakeholders with a forum to discuss natural language processing with experts in the field. You may view information on this workshop on the FDA's website by clicking here.

The workshop will focus on whether NLP can be applied to unstructured text in clinical notes to:

  • Identify indication or reason for medical product use, adverse outcomes or events associated with use of these products, and confounders or personal behaviors that may modify risks associated with use of these products
  • Support protocol design, feasibility, recruitment efforts and execution of clinical trials

Workshop summary

A summary of the workshop was prepared by the planning committee and may be viewed and downloaded here: PDF iconWorkshop Summary.pdf

Workshop agenda, slides, and recording

The agenda is shown below, and slides can be downloaded by clicking on the title of each talk. Speaker biographies and the PDF agenda can be downloaded here: PDF iconSpeaker Info and PDF iconNLP Workshop Agenda.

The workshop recording can be accessed here:

Time Topic Speaker

8:30 am

PDF iconWelcome

Rita Ouellet-Hellstrom, PhD, MPH

8:35 am

FDA’s interest in natural language processing of clinical texts for pharmacovigilance, pharmacoepidemiology and other uses

Robert Ball, MD, MPH

9:00 am

PDF iconCurrent trends in clinical NLP literature

In this talk, I will present a quick review of recently published papers in the area of clinical NLP, focusing on topics of particular interest and relevance to this workshop. For each paper, I will present a “mini journal club” in which I will summarize the goal, methods, results and my conclusions about the work.

Russ Altman, MD, PhD
Stanford University


Morning session


Russ Altman, MD, PhD
Stanford University

9:30 am

Lessons learned from NLP implementations at FDA

In this talk, I will first provide an overview to several ongoing FDA projects leveraging Natural Language Processing (NLP) tools for structuring and standardizing unstructured Information. For each project, I will summarize the goals, outcomes and lessons learned.

Mitra Rocca, Dipl. Inform. Med​


10:00 am

Novel NLP Methods for Medication-Related Insights from Longitudinal Patient Records

Watson for Patient Records Analytics is an initiative at IBM Research to develop novel natural language processing methods for longitudinal patient records. Our goal is to classify sentences in clinical notes, as for example that a sentence asserts a medication side effect, and extract the assertion itself. Towards this goal, we developed several NLP methods customized for clinical notes text and working on several more. We relate problems with their medication treatments, leveraging an accurate problem list that our automated method generates. Medications extracted from clinical notes are juxtaposed with orders to enable medication reconciliation. A supervised learning method achieves high accuracy in identifying sentences that discuss medication change. Plan sentence extraction has been developed. Work on related methods is underway. This talk will outline our goal, approach, what has been achieved so far, and lessons learned.

Murthy Devarakonda, PhD
IBM Research

10:30 am



10:45 am

Mining the EHR to understand disease, drugs, and adverse events

In the era of Electronic Health Records, it is possible to examine the outcomes of decisions made by doctors during clinical practice to identify patterns of care—generating evidence from the collective experience of patients. We will discuss methods that transform unstructured EHR data into a substrate to discover hidden trends, build predictive models, and drive comparative effectiveness studies in a learning health system.

Nigam Shah, MBBS, PhD
Stanford University

11:15 am

Panel Discussion - Addressing the strengths and limitations of NLP solutions

Panelists: Robert Ball, MD, MPH (FDA/CDER), Isaac Chang, PhD (FDA/CDRH), Murthy Devarakonda, PhD (IBM Research), Rita Ouellet-Hellstrom, PhD, MPH (FDA/CDER), Mitra Rocca, Dipl. Inform. Med​ (FDA/CDER), Nigam Shah, MBBS, PhD (Stanford University), Mark Walderhaug, PhD (FDA/CBER)


Russ Altman, MD, PhD
Stanford University

12:30 pm



Afternoon session


Mark Walderhaug, PhD

1:30 pm

Adapting clinical NLP methods for multi-site medical products research

Medical product clinical trials and postmarketing safety surveillance are increasingly coordinated across multiple institutional settings where secondary use of electronic health record (EHR) data makes large-scale ascertainment of outcomes more efficient. Many important outcomes are captured only in unstructured clinical narrative. Harmonizing information extracted from unstructured text in these settings entails challenges similar to those encountered when combining structured EHR data from geographically and institutionally diverse delivery systems. The adage emerging from these efforts, that “all data are local,” is at least as relevant to unstructured clinical data as it is to more widely used structured EHR data. This presentation will describe salient issues confronted when adapting and applying natural language processing (NLP) methods across multiple institutional settings. Seemingly simple tasks, such as assembling complete and representative clinical corpora, can be surprisingly challenging. Idiosyncratic characteristics of clinical documentation, including language usage, document structure, and content, makes the application of NLP methods in multi-site settings an endeavor that requires forethought and attention to detail. These and related issues will be illustrated with examples from recent NLP projects in several clinical domains, including a project now underway to extract from clinical progress notes information about patient-reported medication side effects.

David Carrell, PhD
Kaiser Permanente Washington Health Research Institute

2:00 pm

Advance Drug Safety Research with Semantic Analysis of Electronic Health Records

An adverse event is an injury to a patient and an adverse drug event is an injury to a patient resulting from a medical intervention related to pharmacotherapy. Adverse drug events complicate two million hospital stays annually, are associated with a prolonged hospital stay, account for upwards of two thirds of post-discharge complications, and are a significant contributor to escalating health care costs. The Office of Disease Prevention and Health Promotion has identified adverse drug event prevention as a patient safety priority. Electronic health records (EHRs) contain important adverse drug event-related information and manual chart review is prohibitively expensive. In contrast, biomedical natural language processing (NLP) provides automated tools that facilitate chart review and can improve patient drug safety surveillance and post-marketing pharmacovigilance through enhanced cost efficiencies and provision of real-time information. In this talk, I will first introduce an expert-annotated EHR corpus we developed. I will then describe several new deep neural network models (e.g., LSTM-CRF and memory-augmented NNs) we developed to build the state-of-the-art NLP systems for automated medication and adverse drug event detection from EHR narratives. I will also describe Item Response Theory (IRT) as a new evaluation metrics for NLP systems. Unlike the traditional evaluation metrics of recall/precision/F-score, IRT models characteristics of individual data points (called “items”) such as difficulty and discriminatory ability to estimate ability as a function of the characteristics of correctly answered items. Based on our IRT analysis, we found that deep neural network models exhibit human-like learning process and intelligence capabilities. Our work is an important step towards ADE surveillance and pharmacovigilance.

Hong Yu, PhD
University of Massachusetts Medical School


2:30 pm

Flexible NLP for varied applications and data sources, including cohort selection and adverse event coding/validation

There are a wide range of existing applications for clinical NLP. In this talk we will explore some of these, including cohort selection for clinical trials, extraction of features from EHRs to predict clinical risk, coding of data, checking of regulatory submissions, and analysis of patient feedback. We will explore some of the challenges involved in dealing with such varied data sources and how they can be addressed. In particular we will look at the use of agile text mining to quickly build applications from unannotated data.

David Milward, PhD


3:00 pm



3:15 pm

Leveraging NLP and diverse data sources to mine drug repositioning, adverse drug events, and patient-reported medication outcome information

The informatics team at Mayo Clinic has developed a battery of NLP and text mining methods and tools over the past 15 years to facilitate various clinical and biomedical research projects. Observing that individual data sources tend to have biases and inconsistent findings, we have utilized diverse data sources and various NLP techniques for mining drug-related information. In this presentation, we will start with an overview of our NLP infrastructure and resources, and then demonstrate three case studies on signal detection for drug repositioning, adverse drug events and patient-reported outcomes. We will end by sharing several learned lessons on how to leverage NLP and multiple datasets of heterogeneous nature for meaningful data-driven discovery in drug-related research.

Lixia Yao, PhD
Mayo Clinic

3:45 pm

Panel Discussion - What is the way forward for clinical NLP?

Panelists: Russ Altman, MD, PhD (Stanford University), David Carrell, PhD (Kaiser Permanente Washington Health Research Institute), Isaac Chang, PhD (FDA/CDRH), Hongfang Liu, PhD or Lixia Yao, PhD (Mayo Clinic), David Milward, PhD (Linguamatics), Rita Ouellet-Hellstrom, PhD, MPH (FDA/CDER), Hong Yu, PhD (University of Massachusetts Medical School)


Mark Walderhaug, PhD

4:45 pm

Closing Remarks

Dragutin Petkovic, PhD
San Francisco State University

Workshop Planning Committee

We thank the following members of the workshop planning committee for their time and expertise:

Donna Blum-Kemelor FDA / OCS / ORSI
Amal Mansuer FDA / OCS / ORSI
Catherine Ng FDA / OCS / ORSI
Audrey Thomas FDA / OCS / ORSI
York Tomita, PhD FDA / OCS / ORSI
Frank Weichold, MD, PhD FDA / OCS / ORSI
Rebekah Zinn, PhD FDA / OCS / ORSI
Ruth Barratt, PhD FDA / CDER
Rita Ouellet-Hellstrom, PhD, MPH FDA / CDER
Mitra Rocca, Dipl. Inform. Med​ FDA / CDER
Mark Walderhaug, PhD FDA / CBER
Isaac Chang, PhD FDA / CDRH
Russ Altman, MD, PhD UCSF-Stanford CERSI
Lawrence Lin, PhD UCSF-Stanford CERSI
Dragutin Petkovic, PhD SFSU
Anagha Kulkarni, PhD SFSU

Patient Preference

Advancing Use of Patient Preference Information as Scientific Evidence in Medical Product Evaluation

December 7, 2017 (Day 1) | 8:30 am – 5:00 pm & December 8, 2017 (Day 2) | 8:30 am – 1:00 pm

Tommy Douglas Conference Center (TDCC)
10000 New Hampshire Ave, Building 9
Silver Spring, MD 20903

Hosted by Johns Hopkins, University of Maryland, UCSF-Stanford, Yale-Mayo Clinic, and the Georgetown University Center of Excellence in Regulatory Science and Innovation and the U.S. Food and Drug Administration [Center for Devices and Radiological Health (CDRH), Center for Drug Evaluation and Research (CDER), Center for Biologics Evaluation and Research (CBER), and the Office of Chief Scientist, Office of Regulatory Science and Innovation].

About the workshop

Patients have unique perspectives about the value of the potential benefits and the impact of potential harms and burdens of their medical treatments. Scientists, clinicians, medical product developers, and regulators play critical roles in evaluating the benefits and risks of medical products. However, only patients live with their medical conditions and make choices regarding their own care. Reliable and accurate methods are needed in order to effectively incorporate patients’ values into decision-making processes.

Patient preference information is qualitative input or quantitative data elicited from patients about the desirability or acceptability of outcomes or other attributes of medical products, focusing particularly on trade-offs. The continued development of methods to elicit patient preference information and enhance its regulatory applications has the potential to contribute to a better understanding of the benefit-risk profiles of some medical products.

This workshop will:

  • Present current progress on incorporating patient preference information into medical product benefit-risk assessments
  • Provide examples of how patient preference can be collected, analyzed, and presented in a way that matters to stakeholders
  • Explore methods for appropriate measurement, interpretation, and adoption of patient preference information in a regulatory context
  • Identify future research and capacity needs in order to improve the use of patient preference information in a regulatory context

Workshop agenda, slides, and recording

The agenda is shown below, and slides can be downloaded by clicking on the title of each talk below. A PDF iconDetailed PDF Agenda and PDF iconSpeaker Biographies are also available for download.

The workshop recordings can be accessed here:

Day 1: Thursday, December 7, 2017



7:30 am


8:30 am

WELCOME SESSION: Patient Input and Regulatory Science

Carol Linden (FDA/ORSI)

Welcome Remarks
RADM Denise Hinton (FDA/OCS)

Welcome from the CERSIs
G. Caleb Alexander (Johns Hopkins University)

8:45 am

SESSION 1: Fundamental Concepts and Regulatory Context of PPI to Support Medical Product Development and Evaluation
Chair: Anindita Saha (FDA/CDRH)

PDF iconIntroduction to Session 1
Anindita Saha (FDA/CDRH)

PDF iconFDA Perspective on Patient Preference Information in Medical Product Evaluation
Anindita Saha (FDA/CDRH) and Million A. Tegenge (FDA/CBER)

PDF iconIndustry Perspective on PPI to Support Medical Product Development and Evaluation
Bennett Levitan (Janssen R&D)

PDF iconAcademic Perspective on Patient Preference Research
John F. B. Bridges (Johns Hopkins University)

PDF iconPatient Perspective on Landscape
K. Kimberly McCleary (FasterCures)

Discussion and Q&A

10:15 am


10:30 am

SESSION 2: Scientific Fundamentals of PPI Studies
Chairs: Martin Ho (FDA/CDRH) and Leslie Wilson (UCSF)

PDF iconIntroduction to Session 2
Martin Ho (FDA/CDRH)

PDF iconScientific Fundamentals of PPI Studies
Juan Marcos Gonzalez (Duke University)

PDF iconPanel Discussion and Q&A

  • Martin Ho (FDA/CDRH)
  • Telba Irony (FDA/CBER)
  • Laura Lee Johnson (FDA/CDER)
  • Leslie Wilson (UCSF-Stanford CERSI)
  • Fadia T. Shaya (University of Maryland CERSI)
  • Erica S. Spatz (Yale-Mayo Clinic CERSI)
  • Brett Hauber (University of Washington)
  • Juan Marcos Gonzalez (Duke University)
  • Becky Noel (Eli Lilly and Company)

12:15 pm


1:15 pm

SESSION 3: Discussion on In-Depth Case Studies
Chair: Telba Irony (FDA/CBER) & Michelle Campbell (FDA/CDER)

Introduction to Session 3
Million Tegenge (FDA/CBER)

1:20 pm

Case Study 1: Rare Pediatric Cancer
Moderator: Michelle Campbell (FDA/CDER)

PDF iconIntroduction to Case Study 1
Michelle Campbell (FDA/CDER)

PDF iconFDA Clinical Perspective
Gregory Reaman (FDA/OCE)

PDF iconPatient Study Perspective
Deborah A. Marshall (University of Calgary)

Patient Perspective
Nancy Goodman (Kids V Cancer)

Panel Discussion and Q&A

  • Michelle Campbell (FDA/CDER)
  • Gregory Reaman (FDA/OCE)
  • Paul G. Kluetz (FDA/OCE)
  • Leslie Wilson (UCSF-Stanford CERSI)
  • Deborah A. Marshall (University of Calgary)
  • Nancy Goodman (Kids V Cancer)
  • Tamar Krishnamurti (University of Pittsburgh)
  • Elisabeth (Liz) Piault-Louis (Genentech)

2:45 pm


3:00 pm

Case Study 2: Neurological Degenerative Disease
Moderator: Telba Irony (FDA/CBER)

PDF iconIntroduction to Case Study 2
Telba Irony (FDA/CBER)

PDF iconFDA Regulatory and Clinical Background
Heather Benz (FDA/CDRH)

PDF iconPatient Partnership Perspective
Catherine Kopil (Michael J. Fox Foundation)

PDF iconCERSI - Preference Study Perspective
Ellen M. Janssen (Johns Hopkins University) and Ira Shoulson (Georgetown University)

Panel Discussion

  • Telba Irony (FDA/CBER)
  • Heather Benz (FDA/CDRH)
  • Kerry Jo Lee (FDA/CDER)
  • Ellen M. Janssen (Johns Hopkins University)
  • Ira Shoulson (Georgetown University)
  • Catherine Kopil (Michael J. Fox Foundation)
  • Janel Hanmer (University of Pittsburgh)
  • Kara L. Haas (Johnson & Johnson)

4:25 pm

Introduction to Patient Group
Kathryn M. O’Callaghan (FDA/CDRH)

PDF iconPatient Group Success in Patient Preferences
Andrea Ferris (LUNGevity Foundation)

4:45 pm

PDF iconDay 1 Closing Remarks
Kathryn M. O’Callaghan (FDA/CDRH)

Day 2: Friday, December 8, 2017



7:30 am


8:30 am

PDF iconWelcome and Patient Preference Remarks
Theresa M. Mullin (FDA/CDER)

8:45 am

SESSION 4: CDRH Preference Sensitive Areas Discussion: Diseases and Conditions Where Patient Preference Studies Could Be Useful
Chair: Heather Benz (FDA/CDRH)
PDF iconSession 4 Handout

PDF iconIntroduction to Session 4
Heather Benz (FDA/CDRH)

FDA Perspective
Vishal Bhatnagar (FDA/CDER) and Million A. Tegenge (FDA/CBER)

PDF iconCERSI Perspective
Liana Fraenkel (Yale University)

PDF iconConsortium Perspective
Stephanie Christopher (Medical Device Innovation Consortium)

PDF iconPatient Partnership Perspective
Frank Hurst (FDA/CDRH) and Melissa West (Kidney Health Initiative)


10:00 am


10:15 am

SESSION 5: Capacity Building and Sustainability
Chair: Mimi Nguyen (FDA/CDRH), Michelle Tarver (FDA/CDRH) and Fadia Shaya (University of Maryland)

PDF iconIntroduction to Session 5
Mimi Nguyen (FDA/CDRH) and Michelle Tarver (FDA/CDRH)

PDF iconPatient Perspective
Cynthia Grossman (FasterCures)

PDF iconFDA Perspective
Ebony Dashiell-Aje (FDA/CDER)

PDF iconIndustry Perspective
Matt Reaney (Sanofi)

PDF iconProfessional Society Perspective
Shelby D. Reed (ISPOR)

PDF iconAcademic Perspective
C. Daniel Mullins (University of Maryland)

PDF iconPanel Discussion and Q&A

  • Mimi Nguyen (FDA/CDRH)
  • Cynthia Grossman (FasterCures)
  • Ebony Dashiell-Aje (FDA/CDER)
  • Matt Reaney (Sanofi)
  • Shelby D. Reed (ISPOR)
  • C. Daniel Mullins (University of Maryland)
  • Joseph S. Ross (Yale University)
  • R. Scott Braithwaite (New York University)

12:30 pm

PDF iconThe CERSI Success Story: How the CERSI Collaborations Have Helped Advance the Field of Patient Preference
Frank F. Weichold (FDA/ORSI)

12:45 pm

PDF iconSummary of the Day and Wrap-Up
Telba Irony (FDA/CBER) and Frank F. Weichold (FDA/ORSI)