Research

Automated and semi-automated 3D tumor/organ/tissue detection and segmentation

Tumor volume change is better than diameter change three weeks post therapy, in distinguishing EGFR mutant staues

Image and graph showing tumor volume change over 3 weeks

Imaging acquisition parameters affect computation of quantitative features

Graph showing imaging acquisition parameters which affect computation of quantitative features

Radiomics signature for diagnosis of hepatocellular carcinoma (HCC) in cirrhotic patients with indeterminate liver nodules

Images showing radiomics signature for diagnosis of HCC in cirrhotic patients with indeterminate liver nodules

Automated identification of optimal portal venous phase timing with CNN

Our Research

Our research has focused on the following key areas in biomarker development:

Developing radiomics and artificial intelligence (AI) models to support disease detection, diagnosis, prognosis, and treatment monitoring in oncologic and neurologic applications
Developing advanced software tools to aid in efficient and reliable obtaining of quantitative imaging biomarkers (QIBs)
Exploring reproducibility of QIBs across scanners and scanning parameters, aiming to harmonize imaging acquisition settings for robust QIBs
Creating quality control (QC) programs to identify proper image quality to study QIBs

To foster our QIB research, we developed an efficient image-based response assessment platform that integrated our semi-automated tumor segmentation algorithms and smart editing tools. Moreover, we established a radiomics analysis pipeline, which includes (but is not limited to) image QC, tumor segmentation, radiomic feature extraction, feature reproducibility test and redundancy removal, and radiomics model development and validation. The imaging platform and radiomics analysis pipeline allow us to rapidly study correlations of tumor imaging phenotypes with genotypes and/or clinical outcomes.

Using our lab-developed technologies, we have been collaborating with numerous scientists—from both academia and industry world-wide—to accelerate the development and validation of novel quantitative imaging biomarkers. We are the key contributors to the Foundation for the National Institutes of Health (FNIH) Biomarkers Consortium – Vol-PACT: Advanced Metrics and Modeling with Volumetric CT for Precision Analysis of Clinical Trial Results.

Research Highlights

Deep Learning For the Prediction of Early On-Treatment Response in Metastatic Colorectal Cancer From Serial Medical Imaging

This figure illustrates the architecture of the proposed deep learning network.

In current clinical practice, tumor response assessment is usually based on tumor size change on serial computerized tomography (CT) scan images. However, evaluation of tumor response to anti-vascular endothelial growth factor therapies in metastatic colorectal cancer (mCRC) is limited because morphological change in tumor may occur earlier than tumor size change. We analyzed the use of a deep learning (DL) network to characterize tumor morphological change for response assessment in mCRC patients. We retrospectively analyzed 1,028 mCRC patients who were prospectively included in the VELOUR trial (NCT00561470). We found that the DL network was able to predict early on-treatment response in mCRC and showed better performance than its size-based counterpart with C-Index: 0.649 (95% CI: 0.619,0.679) vs. 0.627 (95% CI: 0.567,0.638), p = 0.009, z-test. The integration of DL networks with size-based methodology could further improve the prediction performance to C-Index: 0.694 (95% CI: 0.661,0.720), which was superior to size/DL-based-only models (all p < 0.001, z-test). Our study suggests that DL network could provide a noninvasive mean for quantitative and comprehensive characterization of tumor morphological change, which may potentially benefit personalized early on-treatment decision making.

Deep learning for the prediction of early on-treatment response in metastatic colorectal cancer from serial medical imaging. Lin Lu, Laurent Dercle, Binsheng Zhao, and Lawrence H. Schwartz. Nature Communications. 12.1 (2021): 1-11.

Radiomics Pipeline for Oncology Research

This figure illustrates the radiomics pipeline by applying it to classification of lung cancer histological subtypes based on non-enhanced computed tomography. Five panels show first the CT imaging, followed by the four main parts of the pipeline: segmentation of lesion, QRF definition and extraction, dimension reduction, and model building.

Radiomics is a technique referring to the high-throughput extraction and mining of quantitative image features (also called quantitative radiomic features or QRFs) from medical images. Radiomics has gained increasing attention in oncology research in recent years, since it could provide an objective evaluation of tumor phenotypic characteristics by analyzing the distribution and relationship among pixel/voxel densities in the medical image. In our lab, a comprehensive artificial-intelligence-based (AI e.g., machine-learning and deep-learning) radiomics pipeline is established for oncology research. The radiomics pipeline mainly consisted of four parts: 1) segmentation of lesion; 2) QRF definition and extraction; 3) dimension reduction; and 4) model building. We applied the radiomics pipeline for classification of lung cancer histological subtypes based on non-enhanced computed tomography. We retrospectively collected 278 patients with pathologically confirmed lung cancer, including 88 patients with adenocarcinomas and 93 patients with squamous cell carcinomas. By using the radiomics pipeline, a machine-learning-based classification model was built and yielded classification performance with AUC (95% CI) of 0.822(0.755, 0.875). Our radiomics pipeline has been successfully applied to other oncology research studies. For examples:

Radiomics for classification of lung cancer histological subtypes based on nonenhanced computed tomography. Linning E, Lu L, Li L, Yang H, Schwartz LH, Zhao B. Academic radiology. 2019 Sep 1;26(9):1245-52.
CT-Based Radiomics Model for Predicting Brain Metastasis in Category T1 Lung Adenocarcinoma. Chen A, Lu L, Pu X, Yu T, Yang H, Schwartz LH, Zhao B. American Journal of Roentgenology. 2019 Apr 1:1-6.

Radiomic Signatures for Identification of Tumors Sensitive to Nivolumab or Docetaxel in Squamous Non-Small Cell Lung Cancer

AI-based signatures from CT imaging use early changes in radiomic features to predict tumor sensitivity to treatment using changes in tumor volume, tumor heterogeneity, and tumor infiltration along boundaries.

New patterns of disease response and progression, such as pseudoprogression and hyperprogression, have been observed in patients treated with immunotherapy, prompting the need for alternative metrics to assess response assessment and therapeutic decision-making. Radiomic signatures derived from quantitative AI-based analysis of standard-of-care CT images can act as markers of treatment efficacy and offer the potential to enhance clinical decision-making. In this study, we assessed early changes in tumor phenotype in patients with squamous non-small cell lung cancer (sqNSCLC) after treatment with Nivolumab or Docetaxel. We defined new radiomic signatures that can be used to predict tumor sensitivity related to systemic treatment. These AI-based radiomic signatures from CT imaging detected early changes in radiomic features of tumors, from baseline to first tumor assessment after treatment. They were associated with sensitivity to treatment assessed by median progression free survival (PFS) such as decrease in tumor volume, tumor heterogeneity, and tumor infiltration along the margin. Using serial radiographic measurements, we showed that treatment insensitivity and shorter overall survival (OS) were associated with an exponential increase in radiomic signature features deciphering tumor volume, invasion of tumor boundaries, and tumor spatial heterogeneity. This study demonstrates the ability of radiomic signatures to offer an approach that could guide clinical decision-making in modifying systemic therapies, and its ability to act as a prognostic biomarker for OS.

Radiomic Signatures for Identification of Tumors Sensitive to Nivolumab or Docetaxel in Squamous Non-Small Cell Lung Cancer. Laurent Dercle, Matthew Fronheiser, Lin Lu, Shuyan Du, Wendy Hayes, David K. Leung, Amit Roy, Lawrence H. Schwartz, Binsheng Zhao Poster 1910P. Presented at the European Society for Medical Oncology Congress, September 27–October 1, 2019, Barcelona, Spain.

Radiomics Machine-Learning Signature for Diagnosis of Hepatocellular Carcinoma in Cirrhotic Patients With Indeterminate Liver Nodules

The signature indicates the risk of an indeterminate liver nodule being HCC using a triphasic CT-scan (from left to right: arterial phase, portal venous phase, delta). The signature predicted a low-risk (blue, top) in a regenerative nodule and a high-risk (red, bottom) in an hepatocellular carcinoma.

Currently, indeterminate liver nodules in cirrhotic patients cannot be accurately diagnosed as hepatocellular carcinoma (HCC) on imaging using current guidelines. Definitive diagnosis with tissue biopsy is not warranted due to procedural risks. In this study, we developed a radiomics approach to non-invasively diagnose HCC in cirrhotic patients with indeterminate liver nodules. We identified a new imaging biomarker from triphasic CT scans that can be used to accurately diagnose HCC in cirrhotic patients with indeterminate liver nodules. The imaging biomarker that best characterizes indeterminate liver nodules in cirrhotic patients is the change in nodule phenotype between arterial and portal venous phases (the “washout” pattern appraised visually using European (EASL) and American (AASL) guidelines). With this imaging biomarker, a clinical decision algorithm using radiomics to diagnose HCC could be developed to reduce the rate of biopsy in cirrhotic patients with indeterminate liver nodules (current EASL guidelines) and reduce utilization of the wait-and-see strategy (current AASLD guidelines), therefore improve patients’ management and outcome.

Radiomics machine-learning signature for diagnosis of hepatocellular carcinoma in cirrhotic patients with indeterminate liver nodules. Mokrane FZ, Lu L, Vavasseur A, Otal P, Peron JM, Luk L, Yang H, Ammari S, Saenger Y, Rousseau H, Zhao B, Schwartz LH, Dercle L. Eur Radiol. 2019 Aug 23.

Automatic Liver Segmentation by Integrating Fully Convolutional Networks into Active Contour Models

This figure shows the curve evolution for (a) global region-based active contour; (b) local region-based active contour; (c) our model with the same initial circle (yellow); (d) our model with another initial circle. The final contour is in red color and intermediate contour is in green color.

Automatic and accurate three-dimensional (3D) segmentation of liver with severe diseases from computed tomography (CT) images is a challenging task. We have developed an automatic liver segmentation method based on a novel framework that integrates fully convolutional network predictions into active contour models (ACM). We use only a single network architecture to generate a pixel label map containing spatial regional information (foreground and background) as well as layered boundary information. We exploit the structured network outcome to define an external constraint force of active contour models. A unique property of the designed force is that both its strength and direction are adaptive to its position and relative distance to the object boundary. The resulting integrated active contour models have the advantages of incorporating both high-level and low-level image information simultaneously, while enforcing the smoothness of the contour. Because the external constraint force can push the evolving contour to the liver boundary and exists everywhere in the image domain, it allows us to place the initial contour far away from the liver boundary. It potentially allows us to control the evolution of the contour in order to preserve the topology of the liver.

Automatic liver segmentation by integrating fully convolutional networks into active contour models. Guo X, Schwartz LH, Zhao B. Medical physics. 2019 Jul 29.

Automated Identification of Optimal Portal Venous Phase Timing with Convolutional Neural Networks

This figure illustrates the proposed dual-input CNN architecture. Input patches are processed by convolutional and max-pooling layers. The output PVP score is processed with the concatenation of portal vein and aorta information and fully connected layers.

Most abdominal CT-scans are acquired after contrast enhancement at the “portal venous phase” (PVP). The PVP acquisition usually uses a fixed delay time after contrast injection. This parameter is set based upon the CT scanner characteristics and is not tailored for a patient’s body habitus or cardiovascular system. This leads to a variability in timing and enhancement of the PVP acquisition. Our algorithm automatically detects portal vein and aorta as the reference of contrast administration with Faster-RCNN and then outputs the probability of the CT scan being optimal contrast administration with proposed dual-input convolutional neural network (CNN). It can automatically check CT scans for their compliance within their defined imaging protocols including contrast administration. The algorithm automatically computed a PVP-timing score in 3 seconds and reached AUC of 0.837 (95% CI: 0.765, 0.890) in validation set and 0.844 (95% CI: 0.786, 0.889) in external validation set.

Our work demonstrates that a fully automatic, deep-learning derived PVP-timing recognition system can reliably and rapidly identify the optimal PVP-timing based on CT images. The rapid identification of such scans will aid in the analysis of quantitative (radiomics) features used to characterize tumors and changes in enhancement with treatment in a multitude of settings including quantitative response criteria such as Choi and MASS which rely on reproducible measurement of enhancement.

Automated Identification of Optimal Portal Venous Phase Timing with Convolutional Neural Networks. Ma J, Dercle L, Lichtenstein P, Wang D, Chen A, Zhu J, Yang H, Piessevaux H, Zhao J, Schwartz LH, Lu L,Zhao B. Acad Radiol 2019 May 28. pii: S1076-6332(19)30171-0.

Exploring Reproducibility and Variability of Quantitative Image Features Over a Wide Range of CT Imaging Acquisition Parameters

The figure on the left is a concordance correlation coefficient (CCC) heat map of 89 commonly used QIFs computed from the same-day repeat CT images reconstructed at six identical imaging settings or three different imaging settings of same slice thickness but different reconstruction kernels. The figure on the right is a CCC heat map of non-redundant QIF groups under the 15 inter-setting comparisons. Columns are arranged in descending order according to the average CCC of the inter-setting comparisons. Rows are arranged in descending order according to average CCCs of non-redundant QIF Groups.

Radiomics characterizes tumor phenotypes based on analyzing quantitative image features (QIFs) derived from radiologic imaging to improve cancer diagnosis, prognosis, prediction, and response to therapy. Although QIFs must be reproducible to qualify as biomarkers for clinical care, little is known about how routine imaging acquisition techniques or parameters affect QIF’s reproducibility. In these two studies, we explored the reproducibility and agreement of QIFs using unique, same-day repeat computed tomography (CT) data set from lung cancer patients, with each scan reconstructed at multiple imaging settings, combinations of three-slice thicknesses (1.25, 2.5, and 5mm), and two reconstruction algorithms (lung and standard). Our data suggest that QIFs are reproducible over a wide range of imaging settings. However, varying degrees of inter-setting disagreements of QIFs exist when features are computed from CT images reconstructed using different reconstruction algorithms and/or different slice thicknesses. Our findings highlight the importance of harmonizing imaging acquisition for obtaining consistent QIFs to study tumor imaging phonotypes.

Reproducibility of Radiomics for Deciphering Tumor Phenotype With Imaging. Zhao B, Tan, Y, Tsai W-Y, Qi J, Xie C, Lu L, Schwartz LH. Nat Sci Rep 6; 23428, 2016. doi:10.1038/srep23428.
Assessing Agreement between Radiomic Features Computed for Multiple Imaging Settings. Lu L, Ehmke R, Schwartz LH, Zhao B. PLoS One. 2016 Dec 29;11(12):e0166550.

Volumetric CT: A Better Imaging Biomarker for Assessing Response of EGFR Mutant Non-Small Cell Lung Cancer to Gefitinib

The figure on the left shows the detection of tumor change on CT, comparing volumetric and unidimensional measurement for an EGFR mutant tumor at baseline and at a 20-day follow-up. For this case, a significant change is detected using volume measurement (−52.4%), but not using unidimensional measurement (−4.4%). The figure on the right is a graph showing receiver operating characteristic curves for the ability of each measurement method to distinguish tumors based on presence or absence of a sensitizing mutation.

Tissue biomarker discovery is potentially limited by the conventional response assessment method based on measuring tumor diameter. Volumetric measurement of high-resolution CT imaging has the potential to more accurately capture tumor growth dynamics, allowing for more exact separation of sensitive and resistant tumors and a more accurate comparison of tissue characteristics. Our study showed, for the first time, that volumetric tumor measurement was better than unidimensional tumor measurement for distinguishing tumors with and without EGFR mutation three weeks following gefitinib therapy. Use of volume-based response assessment for the development of tissue biomarkers could reduce contamination between sensitive and resistant tumor populations, improving our ability to identify meaningful predictors of sensitivity.

A Pilot Study of Volume Measurement as a Method of Tumor Response Evaluation to Aid Biomarker Development. Zhao B, Oxnard GR, Moskowitz CS, Kris MG, Pao W, Guo P, Rusch VM, Ladanyi M, Rizvi NA, Schwartz LH. Clin Cancer Res 2010; 16:4647-53.