PI: Prof Giuseppe Carenini, Department of Computer Science, University of British Columbia

We aim to develop tools for automated extraction from clinical documents. Unstructured documents that would be included in this proposal are pathology reports, radiology reports, etc. Most current methods for extracting information from clinical documents are based on Large Language Models. For instance, Agrawal et al. demonstrated that InstructGPT performs well at zero- and few-shot span identification, token-level sequence classification, and relation extraction on clinical text, even if it was not trained specifically in the healthcare domain [A+2022] M. Agrawal, S. Hegselmann, H. Lang, Y. Kim and D. Sontag. “Large Language Models are Few-Shot Clinical Information Extractors,” EMNLP 2022 . Even more recently, Zhou et al. showed that T5-based models can perform robust event extraction from radiology reports [ZYO2023] S. Zhou, M. Yetisgen and M. Ostendorf. “Building blocks for complex tasks: Robust generative event extraction for radiology reports under domain shifts,” 5th Clinical Natural Processing Workshop, ACL 2023 , while Miller et al. [ZM+2023] W. Zhou, T. Miller et al. “Improving the Transferability of Clinical Note Section Classification Models with BERT and Large Language Model Ensembles,” 5th Clinical Natural Processing Workshop, ACL 2023 found that synergistically combining T5-flan with BERT produces effective, transferable classifiers of clinical note sections. Building on these promising results, we will design and implement LLM solutions (e.g., based on the publicly available Llama2 released in July 2023) to populate a provincial cancer registry for real-world use for cancer control and surveillance. We will also explore other equally compelling use cases for extracting from clinical documents for various conditions (e.g., cardiology). The exact choices of the use cases will depend on the interest and expertise of the actual postdoctoral fellows and other factors, such as the availability of clinicians when the postdoctoral fellows arrive in Vancouver.

The key 12-month objective is to design and implement LLM solutions (e.g., based on the publicly available Llama2) to populate a provincial cancer registry for cancer control and surveillance.

1. One serious weakness of LLMs is that they are rather limited in the size of their input, and therefore struggle in dealing with very long documents or with large sets of documents. Expanding on the work presented by Li et al. [LJH2021] S. Li, H. Ji, J. Han. “Document-Level Event Argument Extraction by Conditional Generation,” 2021 NAACL , which frames event argument extraction as conditional generation, we will develop LLM-based techniques to extract information from highly unstructured documents that have Field of Interests (FoI) potentially located anywhere in the document (or even across documents) and can be expressed in a myriad of ways.
2. We will investigate “question-answering” approaches to extract FoI’s by incorporating the very recent impressive capabilities displayed by conversational decoder-only LLMs on medical competency exams and benchmark datasets. Remarkably, both GPT-4 [N+2023] H. Nori, E. Horvitz et al. “Capabilities of GPT-4 on Medical Challenge Problems,” arXiv preprint arXiv:2303.13375. 2023 and Med-PaLM 2 [S+2023] K. Singal, T. Tao et al. “Towards Expert-Level Medical Question Answering with Large Language Models,” arXiv preprint arXiv.2305.09617. 2023. exceeded the passing score on USMLE, a three-step examination program used to assess clinical competency and grant licensure in the United States. In practice, we will further push the performance of conversational LLMs by for instance exploring more sophisticated medical domain fine tuning and prompting strategies.

• Essential skills: Strong Natural Language Processing publication record; working knowledge & experience with Large Language Models, strong programming skills, good communication skills
• Bonus skills: Solid publication record on machine learning in general

Manual extraction of structured information from documents typically costs tremendous time and labour, motivating the need for a system for automating the process. In this proposal, we aim to develop tools for automated table extraction from documents. That is, given a collection of FoI’s, the task is to automatically extract and populate a table with the FoI’s as the column headers. For example, the BC Cancer Registry is a database of all new cancers diagnosed in BC residents. It is supposed to track the outcomes of patients treated for cancer in BC. To populate the Registry, it is critical to extract up-to-date information from streams of patient documents including physicians’ notes, pathology reports, diagnostic imaging reports, and surgical notes. However, manual review of these reports by health records analysts requires extensive human resources. Before the pandemic, the volume of work for manual extraction already exceeded capacity, and the Registry lags behind many months. With the pandemic, the lag worsens, which significantly affects patient care, and down-stream, evidence-based analyses of provincial cancer outcomes and treatment strategies.

PI: : A/Prof Mehul Motani, Department of Electrical and Computer Engineering, National University of Singapore

Generalization is the ability of a learning algorithm to perform well on unseen data. Explainability deals with providing clear and understandable reasons for the predictions of a learning algorithm. In this project, we will explore generalization and explainability from an information theoretic perspective. We propose to use information theoretic metrics based on mutual information, which are scalable and efficient to compute. We aim to establish that these metrics track generalization and memorization, as well as allow for explainability and fairness analysis. This project builds on some existing work being done in my lab which has shown that information theoretic methods show great promise for the problems being proposed.

1. Explore information theoretic metrics suitable for machine learning systems
2. Study generalization performance with respect to information theoretic metrics
3. Study explainabiity and fairness in machine learning in the context of information theory

1. Establish the properties of a suitable information theoretic metric
2. Characterize the use of the metric for predicting generalization performance
3. Characterize the use of the metric for explainability purposes

• Essential skills:
  1. Information Theory
  2. Machine Learning Theory
  3. Experience with implementing deep learning frameworks
  4. Experience with Python and other programming languages
• Bonus skills:
  1. Github and online software development

Generalization and explainability are key issues in healthcare problems because of the life and death nature of the problems being studied. Information theoretic ideas have the potential to shed light on these two issues. While they have been explored in more general scenarios, the application to healthcare has been less explored and holds significant potential.

PI: Prof Ng See-Kiong, Institute of Data Science and School of Computing, National University of Singapore

Collaborative machine learning is an appealing paradigm to build high-quality machine learning (ML) models by training on the data from many parties. In the healthcare domain, the ability to develop machine learning models over datasets distributed across data centers such as hospitals, clinical research labs, and mobile devices while preventing data leakage, will be a key enabler for AI in healthcare, as medical data are often scarce within individual institutions. However, concerns over trust and security have hindered the sharing of data as the “data silos” are inherently difficult to break at the data level, especially when personal or proprietary data are involved. We posit that the ML models can be more amenable for sharing as they are inherently more compact and self-contained with purpose-compiled knowledge from the data. Rather than requiring the learning collaborators to contribute their private data, this project will focus on enabling collaborative machine learning through allowing the collaborators to share heterogeneous black-box models, and to be appropriately incentivized based on their self-interests. Given that most current research are focused on the data level, this project will develop new model-centric collaborative machine learning methods, as well as new notions for trustable model-centric sharing and effective model management techniques for real-world model-centric platforms. We seek a postdoc who will study and develop such collaborative machine learning approaches for the healthcare domain.

1. Develop new concepts and algorithms in federated learning and artificial intelligence for trusted collaborative machine learning in healthcare;
2. Be up-to-date on state-of-the-art methodologies in related technical fields (federated learning, AI) and application domains (healthcare);
3. Develop ideas for application of research outcomes;
4. Contribute to knowledge exchange activities with external partners and collaborators;

1. Collating healthcare datasets that can be used to develop/demonstrate/evaluate federated learning approaches
2. Develop (one-shot) FL methodologies for healthcare applications
3. Explore other related approaches such as ML data/model valuation, FL incentivisation techniques, machine unlearning

• Essential skills:
  1. Specialization related to machine learning and artificial intelligence for healthcare and/or prior experience in federated learning, Bayesian optimization, or privacy/security research in data sharing;
  2. Proven ability to conduct independent research with a strong and relevant publication record;
  3. Experienced in using the latest machine learning, AI, and big data platforms;
  4. Excellent interpersonal communication and oral presentation skills in English
• Bonus skills:
  1. Github and online software development

While this project is focused on one-shot federated learning approaches, the postdoc will focus on such approaches in the healthcare setting. The ability to develop machine learning models over datasets distributed across data centers such as hospitals, clinical research labs, and mobile devices while preventing data leakage, will be a key enabler for AI in healthcare, given that medical data are often scarce within individual institutions.