deep learning methods for multimodal segmentation: fusing
TRANSCRIPT
FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO
Deep learning methods for
multimodal hepatic lesion segmentation:
Fusing functional and structural medical
images
Thesis Plan
Jessica Condesso Delmoral, UP201200524
Doctoral Program in Biomedical Engineering
Supervisors: Prof. Dr. João Manuel R. S. Tavares (FEUP)
Co-supervisor: Prof. Durval C. Costa (HPP/Champalimaud Foundation)
2016/2017
Summary:
Liver cancer is one of the most lethal cancers worldwide, occurring in both primary liver cancer
and as secondary cancer form, derived from metastizations of other primary tumours. Physicians recur to
Computed Tomography (CT) for the tasks of visualization of anomalies in shape and texture of liver lesions
towards disease diagnosis, resective surgery planning and progression evaluation. Novel developments in
functional image have also validated the exploration of liver function with Positron Emission Tomography
(PET). Surgical tumour resection is considered as gold-standard for liver tumour treatment, requiring
surgical planning based on the image data, retrieved by the former imaging techniques. Medical image
analysis advances everyday towards the development of segmentation algorithms of relevant structures in
images. Emerging developments on CT image, have contributed for the appropriate analysis of liver
function in liver cancer clinical settings. In turn, developments in functional image computational analysis
of liver are very limited. The combination of both imaging modalities for the extraction of accurate liver
anatomy representations, confers a higher sensitivity and specificity in the localization of cancer lesions.
The aim of the project proposed, is to combine the anatomical and functional lesion information
towards a better visualization of the total lesion extent and anatomic location. Moreover, for resective
surgery planning the additional extraction of the liver vasculature and segmental anatomy are necessary.
State-of-the-art liver Computer Vision and image analysis techniques, include methods that require
the usage of manual or semi-manual feature engineering and fully-automatic methods. Emerging
developments in deep learning methods present great potential for the development of automatic liver lesion
analysis based on medical image. Moreover, for computer aided resection surgery planning, a subject-
specific deep learning-based liver, liver lesion and vasculature segmentation tool is proposed combined
with a final liver segments segmentation.
i
Content 1
Introduction 1
1.1. Motivation ................................................................................................................. 2 1.2. Research Goals .......................................................................................................... 3 1.3. Structure of the document.......................................................................................... 4
5
Anatomy of Liver Cancer 5
2.1. CT .............................................................................................................................. 8 2.2. Functional imaging .................................................................................................... 9 2.3. Challenges ............................................................................................................... 10
13
State of the art 13
3.1. Results ..................................................................................................................... 13 3.2. Liver and tumor segmentation in structural image .................................................. 14 3.3. Liver and tumor segmentation in functional image ................................................. 20 3.4. Liver vessel segmentation ....................................................................................... 21
23
Methods 23
4.1. Work package 1 – CT liver segmentation ............................................................... 23 4.2. Work package 2 – PET-CT tumour segmentation ................................................... 24 4.3. Work package 3 – Liver segments map segmentation............................................. 25 4.4. Work package 4 – Integration and Visualization ..................................................... 26 4.5. Overview ................................................................................................................. 26 4.6. Timetable ................................................................................................................. 27
29
Final Remarks 29
References 31
ii
iii
List of Figures
Figure 1.1. Estimated age-standardized incidence rates of liver cancer, in both sexes, worldwide in 2012
[2]. .................................................................................................................................................. 2
Figure 1.2. Cancer mortality worldwide, according to the World Health Organization and Globocan 2012
reports [2]. ...................................................................................................................................... 2
Figure 2.1. Axial abdominal cavity represented in a CT image (a), corresponding structure
representation(b), abdominal cavity location (c) and segmental division (d). 1. Right lobe of liver,
2. External oblique muscle, 3. Inferior vena cava, 4. Caudate lobe of liver, 5. Left lobe of liver, 6.
Linea alba, 7. Coeliac trunk, 8. Left gastric artery, 9. Rectus abdominis muscle, 10. Stomach, 11.
Left colic flexure, 12. Diaphragm, 13. Vertebral arch, 14. Spinalis muscle, 15. Spinous process,
16. Thoracolumbar fascia, 17. Vertebral canal and spinal cord, 18. Thoracic vertebra, 19.
Longissimus thoracis muscle, 20. Iliocostalis muscle, thoracic part, 21. Spleen, 22. Posterior
gastric artery, 23. Latissimus dorsi muscle, 24. Left lung, 25. Costodiaphragmatic recess, 26. Right
subphrenic recess, 27. Bare area of liver, 28. Intercostal lymph nodes, 28. Ostium cardiacum, 30.
Superior diaphragmatic lymph nodes, 31. Posterior mediastinum, 32. Cardiac notch, 33. Inferior
diaphragmatic lymph nodes, 34. Left paracolic gutter [92]. ........................................................... 5
Figure 2.2. The Couinaud's classification system [93] ................................................................................. 6
Figure 2.3. Hepatocellular carcinoma example, seen in a CT image [94] .................................................... 8
Figure 2.4. Examples of difficulties of liver segmentation: sample CT slices with ambiguous boundary
between liver and heart (a), stomach (b) and tumor must be included in segmentation (c). .......... 8
Figure 2.5. A 65-year-old patient with multiple liver metastases due to colon cancer. The CT image (a)
clearly shows one large confluent lesion (asterisk) and the fused PET/CT image (b) shows two
smaller hypodense lesions (arrows) that demonstrate abnormal FDG uptake [10]. ....................... 9
Figure 2.6. Examples why liver segmentation is a challenging task. In the first two images, liver tissue has
to be separated from adjacent organs stomach (a) and heart (b). The gray-values in all structures
are highly similar, which makes boundary detection difficult without a-priori information about
the expected shape in these regions. In the third image (c), the tumor should be segmented as part
of the liver. However, there is a considerable intensity difference between both structures, which
often leads to misclassification of the tumor as non-liver tissue. ................................................. 10
Figure 4.1. Examples architectures of input feature maps integration from multimodal images [95]. ....... 25
iv
v
List of Tables
Table I. Average values of performance measures and scores of different method results evaluated on the MICCAI
liver segmentation grand challenge. Measures reported as mean and standard deviation over all test images.
Maximum overlap error, average volume different, average distance, root mean squared distance, max
volume distance, and the final average score.
Table II. Average values of performance measures and scores of different method results evaluated on the MICCAI
2008 liver lesion challenge. Measures reported as mean and standard deviation over all test images. Maximum
overlap error, average volume different, average distance, root mean squared distance, max volume distance,
and the final average score.
Table III. Average values of performance measures and scores of different method results evaluated on the image
dataset, using CNN-based methods. Measures reported with different segmentation accuracies.
vi
vii
Abbreviations and Symbols
2D Two-dimensional
3D Three-dimensional
AAM Active Appearance Model
ASM Active Shape Model
CAD Conputer Aided Diagnosis
CNN Convolutional Neural Network
CT Computed Tomography
DBN Deep Belief Networks
DICOM Digital Imaging and Communications in Medicine
DL Deep Learning
DSC Dice Similarity Coefficient
FOV Field of View
GC Graph cut
GM Genetic Algorithm
HCC Hepatocellular carcinoma
HU Hounsfield Units
IVC Inferior Vena Cava
ML Machine Learning
MRI Magnetic Resonance Imaging
RBM Restricted Boltzmann Machine
ROI Region of Interest
PET Positron Emission Tomography
SAE Sparse Autoencoder
SPECT Single Positron Emission Tomography
SSM Statistical Shape Model
SVM Support Vector Machine
WHO World Health Organization
viii
1
Introduction
The liver is essential for human survival and is a vital organ in the execution of key functions such
as human metabolism, detoxification, protein synthesis, glycogen storage, hormone production, and
production of biochemicals necessary for digestion. It is the largest organ in the human body weighting
approximately 1.5 Kg, accounting for nearly 2% to 3% of average body weight. The functions that this
organ performs are necessary for the proper homeostasis of the body, whereas functional alterations
suffered upon disease of the liver tissues can become life threatening. Alterations of liver cells can be caused
by inherited/genetic disorders (such as hemochromatosis or Wilson’s disease); immune system -related
pathologies (such as autoimmune hepatitis, primary biliary cirrhosis, primary sclerosing cholangitis);
infections (such as viral infections e.g. hepatitis A, B, or C); cancer (such as hepatocellular carcinoma or
cholangiocarcinoma); chronic alcohol abuse, obesity (causes non-alcoholic fatty liver disease) or
intoxication (e.g. with toxic drugs). Pathological alterations in the liver, evolves firstly into tissue scarring
(cirrhosis), which can further progress to liver failure and become a life-threatening illness. Among the
abovementioned diseases, liver cancer presents an alarming prevalence in a global scale and is one of the
most lethal cancers worldwide [1].
Liver cancer is characterized by the development of abnormal cell accumulations, that will appear
represented differently in structural images, such as Computed Tomography (CT), and in turn, appear with
differentiated functional behaviour in functional images, such as Positron Emission Tomography (PET).
Physicians recur to different medical imaging techniques for the tasks of visualization of anomalies in shape
and texture of liver towards disease diagnosis, resective surgery planning and progression evaluation. One
of the most characteristic properties of the liver is its exceptional regenerative capability, derived from its
modular/segmental structure which makes it possible to consider the organ as a set of functionally
independent units. The anatomy of the liver has been standardized by the known Couinaud system, which
divides the liver into eight functional lobes, which are independently supplied by its artery, veins, and bile
duct. To preserve the adequate liver function, the resection of lesion masses is made with respect to the
segments that contain the liver lesion. Moreover, the liver’s high regenerative power, can be useful and
improve the success of specified treatment courses. Hence, in the presence of liver cancer, physicians need
to eliminate the malignant mass or masses present in the tissues, based on an informed decision of the liver,
liver lesions, vasculature and segments locations.
Software tools that extract accurate anatomic representations in three-dimensions (3D), for
different patients and in a convenient amount of time, are the solution that bioengineering, computer vision
and medical image analysis can solve.
Computerized medical image analysis, can assist in tasks of disease detection, visualization,
diagnosis, staging, monitoring and therapy planning. State-of-the-art liver Computer Vision techniques, are
continuously developing, presenting novel techniques for the proper analysis of liver lesions in medical
images, for tumour quantification. For this purpose, as in many other medical conditions, the need for
software tools that would aid the clinical analysis of the information present in medical images, is of high
importance. In the current study, we intend to present an overview of the state of the art techniques of
visualization, segmentation and data analysis targeting the liver, cancer diagnosis and surgery planning.
This chapter describes the clinical motivations of the presented work, the research goals that are expected
from liver medical image analysis, followed by a summarized list of the objectives aimed with this project,
and finally, an outline of the contents organization in this document, by chapter.
2
1.1. Motivation
Cancer is a major clinical issue in healthcare worldwide, being a current open subject tackled by
researchers, from different areas. Liver cancer occurs as both primary liver cancer, whose genesis occurred
within the liver, and as secondary cancer form, derived from metastizations of other organ’s primary
tumours. Hepatocellular carcinoma (HCC) is the most common type of primary liver cancer. It usually
develops after either a viral Hepatitis B/C infection or a liver cirrhosis. Cirrhosis is a liver condition
characterized by replacement of healthy liver tissue with fibrosis, scar tissue, and nodules. Approximately
50% of the deaths caused by HCC occurred in China. In Western countries, the most frequent liver tumors
are metastasis. The disease’s world distribution in 2012 is presented in Figure 1.1.
The problem resides in the increasing incidence of the disease. According to the World Health
Organization (WHO), liver cancer is the fifth most common, and the second most lethal type of cancer,
accounting for 7.5 million deaths in 2012 (Figure 1.2).
Understanding the anatomy of a target organ is fundamental in medical diagnosis. The task of
lesion characterization and quantification with respect to type and size of cancer tissue, and localization
with respect to segments, vessel structures, or surrounding organs, are the primary factors to investigate.
The lack of symptoms in the early stage of the disease makes it very difficult to diagnose liver cancer before
Figure 1.1. Estimated age-standardized incidence rates of liver cancer, in both sexes, worldwide in 2012 [2].
Figure 1.2. Cancer mortality worldwide, according to the World Health Organization and
Globocan 2012 reports [2].
3
it is advanced. Hence, for the majority of patients, surgical tumour resection is the gold-standard treatment
as the reoccurrence rate is the lowest and the long-term survival rate is the highest for this type of procedure.
The following task of treatment planning depends highly on 1) disease staging and 2) lesion location
relatively to vessels and critical structures, meaning that it is highly dependant on the previous task. Course
of treatment may include radiotherapy, where gross target tumor delineation is required, chemotherapy,
specially used when tumor is divided in several outbreaks, which requires knowledge of all the tumor
locations and size, and surgical resection, where precise location of the tumor with respect to the whole
liver and individual segments and vessels is required.
The development of efficient and highly accurate computational tools that can execute the
abovementioned tasks would constitute a major contribution in the clinical context of the disease. In
computational science, an emerging class of algorithms, called deep learning, that have already
revolutionized a wide number of data analysis problems, has been in the past decade introduced in image
analysis and later in medical image analysis. Deep learning methods derive from the broader field of
machine learning, which is based the concept of giving systems the ability to acquire their own knowledge
based on example, by extracting patterns from raw data, instead of relying on hard-coded knowledge and
rules engineered by humans. These algorithms could extract statistical discrimination rules from data
representations of the given problems, and so forth researchers developed a wide set of algorithms that
solve higher and higher complex problems. Deep learning methods, derived from the need of more complex
computational tools when the degree of complexity of the data increased. They are inspired on the structural
arrangement of human brains, and have presented an unparalleled performance, relatively to previous
algorithms. They have moreover, in several cases even rival and exceeded the accuracies achieved by
humans. For these reasons, the usage of these algorithms to solve the problems proposed in this project
represent a very promising step.
1.2. Research Goals
The aim of this PhD project is to develop an image-based, patient-specific, automatic liver and
lesion segmentation tool to assist the physician in successfully visualize the livers entire anatomy and in
performing an informed decision of the anatomic distribution of liver tissue, hepatic lesions, vasculature
tree and liver segments. A proper Computer Aided Diagnosis (CAD) and surgery planning system requires
the collection of all this information.
The aim of this project consists in the development of a segmentation framework of liver lesions
from CT and functional abdominal images, using deep learning methodologies, namely Convolutional
Neural Networks. The goal is to combine the anatomical and biological image data towards a better
visualization and volume estimation of both the liver and total lesion extent, to be included in a three-
dimensional visualization tool, for resective surgery planning.
Specifically, the following objectives will be set:
➢ An algorithm for the segmentation of the whole liver in CT images;
➢ An algorithm for the segmentation of the liver lesions in CT images;
➢ An algorithm for the joint segmentation of liver lesions using both CT and PET images;
➢ An algorithm for the segmentation of the liver vessels;
➢ Building an anatomic model of the 8 segments that constitute the liver;
➢ An algorithm to segment the extracted liver anatomies into the eight segments that constitute it;
➢ A computational framework for 3D visualization of all the previously segmented liver structures.
The present document intends gather a comprehensive understanding in the state of the art
techniques used for computational analysis of liver cancer in medical images. Firstly, many research
sources have been tuned into accurate segmentation of the liver in CT images, being extensively validated
in the literature using a vast range of medical image processing algorithms. Secondly, the usage of
functional images for liver cancer computational analysis is quite limited. This, it firstly due to the lack of
access to functional imaging in the clinical setting, also hampering the access to viable image datasets, and
secondly, because only in the past decade the usage of functional imaging in oncological diagnosis has
validated the development of computational tools for these purposes.
Hence, computational tools for the segmentation of liver and liver lesions, using both functional
and structural images has not been reported previously in the literature, to the best of our knowledge.
4
Moreover, the integration of liver, liver lesion, and segment partition into one patient specific analysis tool
is the main contribution of the works proposed in this document.
Finally, the usage the most state of the art deep learning algorithms in the medical field is very
promising. Namely, Convolutional Neural Networks have presented the potential to enable the development
of the complex computational system proposed.
1.3. Structure of the document
This document is organized into five chapters. Chapter 2 presents an overview regarding the main
considerations of liver anatomy. Chapter 3 provides a comprehensive list of the most cited methods focused
on liver segmentation, liver lesion segmentation, CAD systems and liver vasculature extraction, in the area
of computer vision, namely in classification and recognition of objects represented in images. Chapter 4
features a detailed explanation about the methods that we intend to develop during the PhD project. Finally,
the fifth chapter presents the final remarks of this work, indicating the expected results and advantages of
this novel framework.
5
Anatomy of Liver Cancer
The liver is located in the right upper quadrant of the abdominal cavity, between the right
hemidiaphragm, protected by the thoracic cage, in right and anterior direction, and in close vicinity with
the right kidney (distally), stomach, adrenal gland (spleen), the small intestine and colon (left), and pancreas
(left). The anatomic positioning of the liver and organ arrangement within the abdominal cavity is presented
in Figure 2.1. The anatomic position of the liver in the circulatory system is optimal for two of its main
functions: 1) gathering, processing, and storage of metabolites and 2) neutralization and elimination of toxic
substances.
The healthy liver tissue is called parenchyma which is constituted by hepatocytes, the liver cells,
specialized in blood filtering to remove toxins. The remaining connective tissue is called the stroma and
serves as support, maintaining the liver fixated the abdominal cavity through ligamentous extensions.
) )
) )
Figure 2.1. Axial abdominal cavity represented in a CT image (a), corresponding structure representation(b), abdominal
cavity location (c) and segmental division (d). 1. Right lobe of liver, 2. External oblique muscle, 3. Inferior vena cava,
4. Caudate lobe of liver, 5. Left lobe of liver, 6. Linea alba, 7. Coeliac trunk, 8. Left gastric artery, 9. Rectus abdominis
muscle, 10. Stomach, 11. Left colic flexure, 12. Diaphragm, 13. Vertebral arch, 14. Spinalis muscle, 15. Spinous
process, 16. Thoracolumbar fascia, 17. Vertebral canal and spinal cord, 18. Thoracic vertebra, 19. Longissimus thoracis
muscle, 20. Iliocostalis muscle, thoracic part, 21. Spleen, 22. Posterior gastric artery, 23. Latissimus dorsi muscle, 24.
Left lung, 25. Costodiaphragmatic recess, 26. Right subphrenic recess, 27. Bare area of liver, 28. Intercostal lymph
nodes, 28. Ostium cardiacum, 30. Superior diaphragmatic lymph nodes, 31. Posterior mediastinum, 32. Cardiac notch,
33. Inferior diaphragmatic lymph nodes, 34. Left paracolic gutter [92].
6
The liver is the most vascularized organ in the human body receiving approximately a total of 25%
of the total cardiac arterial output. It is supplied by four vessel structures: the hepatic artery and sub-
branches supply 25% to 30% of the liver blood input with oxygen; the portal vein, supplies the remaining
70% to 75% of blood input, carries nutrient-rich blood from the intestines and the colon; the processed
blood leaves the organ through the hepatic vein; and the bile produced by liver cells drains through the
biliary tree into the gallbladder [2]. The special arrangement of vessels allows division of the liver into
eight segments. The livers segmental structure is able to regenerate after a serious disease that affects only
a part of the liver. The Couinaud’s segment distribution is represented in Figure 2.2. Since segments are
functionally independent, the one(s) affected by the pathology can be completely removed. In such cases
the other segments can grow significantly, which can compensate the loss of liver volume due to the
operation or intervention. This way, the disabled segments are not regenerated rather their function is
restored by the other parts.
The liver’s job is vital, unlike for instance that of the spleen, and cannot be substituted by an
artificial organ or medical device, as is the case of the heart or the kidneys. Hence, alterations to the normal
function of liver cells and tissues can potentially become life threatening.
Cancer is characterized of abnormal cells that start to rapidly divide (forming lumps), aggregate
inside an organ (primary tumour), and can subsequently spread to other parts of the body, by the circulatory
tract, to start growing in other organs (metastasis). When these alterations occur to the liver, a wide variety
of clinical scenarios can occur in the anatomical and functional sense. There are several different
pathologies that can occur in the liver, whereas, the oncological medical scenarios can be derived from
benign or malignant alterations of the normal tissues of the liver. Alterations to the liver morphology can
occur as benign lesions such as hepatocellular adenomas, hemangiomas, focal nodular hyperplasias, or
nodular regenerative hyperplasia, or as malignant lesions such as hepatocellular carcinoma or secondary
liver cancer.
Benign Tumors
Hepatocellular Adenoma: hepatic adenomas are more frequently observed in young women,
whose pathogenesis is suspected to have a hormonal influence. The condition is associated in several studies
with exposure to estrogens (i.e. oral contraceptives, OCPs), anabolic androgens, and genetic disorders.
These adenomas are cell aggregations, present normally in the right lobe, which can grow into large
dimensions ≥ 5 cm in diameter, and be multiple. These two characteristics make these adenomas highly
Figure 2.2. The Couinaud's classification system [93]
7
susceptible to becoming malignant, and susceptible to vascular invasion and subsequently sudden
haemorrhage [3].
Focal Nodular Hyperplasia: is the second most common benign tumor and are also more frequent
in women. These tumors are more solid masses with fibrous core and stellate projections containing atypical
hepatocytes, biliary epithelium, Kupffer cells, and inflammatory cells. These lesions have decreased portal
vein supply and are associated with the development of portal hypertension. Detection can be made via
helical CT, angiography and Magnetic Resonance Imaging (MRI) but difficultly with ultrasound.
Hemangioma: similarly to the previous two tumors, it is more frequent in women and the most
common benign liver tumor among the three. Hemangiomas are asymptomatic and therefore, are usually
detected by accident. Result of an abnormal development of blood vessel endothelial cells. The risk of
malignant evolution, from these structure is null.
Nodular Regenerative Hyperplasia: Nodular Regenerative Hyperplasia consists of multiple hepatic
nodules formed as a result of periportal hepatocyte regeneration accompanied with atrophy of surrounding
tissue. Its pathogenesis is unknown but studies show that it may be related with abnormalities in the hepatic
blood, being formed as the response to atrophy of the liver, caused by vascular obstruction of small portal
veins or hepatic arteries. The most common symptom in non-cirrhotic liver is portal hypertension and, in
some cases, anomalous liver function with progressive symptoms are observed [4].
Malignant Tumors
Hepatocellular Carcinoma (primary tumor): in the context of liver cancer, previous
studies refer in the vast majority to HCC given that it is the most common malignant condition of the liver.
Epidemiology shows that HCC is more common in men and most frequently arises in a cirrhotic liver.
However, cirrhosis may be an important contributor to HCC although not being an essential precondition.
In turn, the mortality due to the disease is very high in countries where there is high prevalence of viral
infection by hepatitis B and C. Other causes may be hereditary hemochromatosis and food contamination.
Hence, it arises from hepatocytes in a liver which is almost always chronically diseased, and often cirrhotic.
Geographically, in countries where alcohol abuse is very high, and where this abuse subsequently
becomes the most common cause of cirrhosis, most likely are countries with high incidence of
hepatocellular carcinomas. This is the case of the United States, where data shows that 80% of
hepatocellular carcinomas are derived from cirrhotic livers, with alcohol as the most common underlying
cause, followed by hepatitis B infection. Moreover, an early diagnosis becomes more conditioned due to
the similarity of symptoms caused by HCC which may be interpreted as a progression of the cirrhosis
disease.
The disease evolution presents a very characteristic behaviour. Upon the tumor hepatocellular
carcinoma mass growth, the irrigation supply becomes more demanding and mechanisms of angiogenesis
are promoted locally. Hence, new branches from the hepatic artery are formed invading many times the
liver masses, in order to provide more nutrients and oxygen to the cancerous tissue.
Screening is primarily done via ultrasound imaging since it is less expensive and is able to identify
tumors greater than 3 cm. When masses are identified in more sophisticated imaging modalities such as CT
(Figure 2.3) or MRI, and malignancy is suspected, a percutaneous liver biopsy of a part of the region
detected can confirm the diagnosis. Moreover, liver cancers can be classified according to tumor severity
and corresponding life expectancy by two standards: the CLIP (Cancer of the Liver Italian Program) and
Okuda staging systems [5]. It is nonetheless difficult to diagnose liver tumor at a stage where resectability
is still viable, and it may be carefully studied though patient imaging, taking into account the liver segments
and the vascularized tumor mass, which add risk to this type of surgery. In some particular cases liver
transplantation is also a course of treatment, whereas survival after transplantation for patients with a single
lesion of no more than 5 cm, or 3 or less lesions with maximum of 3 cm, is viable [4].
Metastic liver tumor (secondary tumor): travelling of cancerigenous cells into the liver and
metastization is very common, deriving very commonly from lung, stomach of colon cancer. This may be
due to high intake of blood supply. Depending on the stage of detection, the strategies of resection have
evolved and increased patients life expectancy [6].
8
2.1. CT
Clinical diagnosis of liver disease is firstly accessed through CT imaging. The clinical
protocols of identification of liver disease include several variations upon the stage of clinical intervention.
Diagnosis is performed via biopsy but previous lesion location is done via morphological imaging.
Computed tomography imaging works based on similar principles of traditional X-Rays, consisting in the
computerized image reconstruction of the radiation image function into tomographic image sections. A
beam of X-rays is wheeled around the target body section, from which the transmitted beam intensity is
reconstructed according to the acquisition angle.
The result is a stack of multiple 2D grayscale image slices, representing the screened
result of a given body thickness, usually between 1 to 4 cm, hence representing a 3D volumetric
representation of the soft tissues. Image resolution varies from 256x256 to 512x512 pixels.
The key properties to retain from CT imaging of abdominal soft tissues is the reliability
in the representations extracted, due to knowledge that the intensity values represent X-ray absorption
coefficients. This factor, makes CT imaging the golden-standard modality of liver imaging, as opposed to
MRI. There are some studies that still try to validate MRI imaging of liver. However, for structural analysis
of soft tissues intensity values in MRI cannot be associated, with reliability, with any material or tissue
type, due to the physics of MR image acquisition. This implies that the liver present intensity
inhomogeneities in MRI, as opposed to CT, where the liver parenchyma is nearly homogeneous.
Finally, given that the liver is a soft organ, and the abdominal cavity is highly populated, its shape
is dependent on adjacent organs within the abdomen. Hence, the imaging conditions are never ideal, since
Figure 2.3. Hepatocellular carcinoma example, seen
in a CT image [94]
) ) ) a c b
Figure 2.4. Examples of difficulties of liver segmentation: sample CT slices with ambiguous boundary between liver
and heart (a), stomach (b) and tumor must be included in segmentation (c).
9
the appearance and shape of the liver does not entirely present very clearly defined edges, which may in
times be not visible on many sides of the liver. In figure 2.3 this is observable as well as the close intensity
differences between the liver and the heart, spleen or stomach. This may be aggravated by other pathologies
that can affect the appearance of the liver.
2.2. Functional imaging
Functional or molecular image consists in Positron Emission Tomography (PET) and Single
Positron Emission Tomography (SPECT). PET, such as all molecular imaging methods, is considered a
diagnostic complementary method, minimally invasive. This imaging modality requires the intravenous
administration of a radiopharm. Specifically, it measures the dynamic metabolic uptake of specific proteins
in target human tissues. This imaging methodology is carried out by the usage of relevant high affinity
molecules that are known to bind to specific receptors and tissues in the human body. Thus, pertinent
metabolic studies use key functional molecules metabolized by the target tissue to be analysed, which are
labelled with a radio-isotope, that will start to decay. The decay radiation, that takes place upon intra-venous
injection of the radiolabelled molecules will allow us to map the binding of these molecules into the target
tissues. Functional studies based on this imaging process are spread over thirty years of history.
Functional imaging plays an important role in cancer treatment. By revealing chemical activity of
specially designed radiotracers. Functional imaging is capable of detecting a wide range of abnormal organ
events in the human body, including tumorous activity. Normally, radiological imaging techniques depend
on structural and morphological alterations of organs, in order to detect the presence of a pathological
alteration. However, molecular image provides functional information about the metabolic processes
carried out by tissues, and subsequently, of alterations these may suffer. Cancer is a disease that is
primordially characterized by the alteration of tissue cells, which may have suffered some metabolic
alterations, causing them to abnormally replicate, becoming malignant, accumulating into build-ups of
abnormal cells. Thus, the advantage resides in the earlier detection of abnormal metabolic behaviour of
tissues, that precede morphological alterations of those tissues. In this setting, functional imaging plays an
important role in earlier detection of disease, as well as in proper spatial delineation of abnormal tissue.
It has been extensively validated, for more than 50 years now, that most tumors have a higher
utilization and metabolism rate of glucose. Moreover, PET imaging has been widely applied to cancer
clinics as a tool for tumor detection and diagnosis. However, the produced images have low spatial
resolution in its nature, making it very difficult for precise localization of tumor. Precise localization is
important for radiation therapy, a mainstream tumor treatment method, for treatment efficiency and
avoiding side-effect. Therefore, PET needs to be combined with CT. Improvements performed to functional
imaging equipments allowed for the scientific validation of the resulting images, and wide
commercialization for clinical usage. In the last decade, specially designed scanners combining PET and
CT were developed, and SPECT-CT has become a mainstream imaging modality ever since.
Figure 2.5. A 65-year-old patient with multiple liver metastases due to colon cancer. The CT image
(a) clearly shows one large confluent lesion (asterisk) and the fused PET/CT image (b) shows two
smaller hypodense lesions (arrows) that demonstrate abnormal FDG uptake [10].
10
For the specific diagnosis of liver cancer, the literature provides studies over the last five years
that validate the added value of using functional imaging, whereas an example of the added information
that can be obtained from the combination of both modalities is presented in Figure 2.5.
2.2.1. Quantitative evaluation of radiotracer uptake in molecular image
A qualitative assessment of the functional image done by experts is often sufficient for tumor
assessment and detection. However, accurate tumor diagnosis and assessment require quantitative
evaluation of the PET scanner records, since such records vary with time and dose of radiotracer. Popular
semi-quantitative and quantitative methods are standard uptake value (SUV), tumor-to background ratio
(TBR), nonlinear regression techniques, total lesion evaluation (TLE), and the Patlak-derived methods.
Among them, SUV is the most popular technique, quantifying the physiological activity of cellular
metabolism at a pixel level or at a voxel level. SUV represents the tissue concentration of the radiotracer at
a given time divided by a few normalization factors. These factors include injected dose, patient size in
terms of weight (𝐵𝑊, and a decay factor related to the radiotracer. It is defined as:
𝑆𝑈𝑉 =𝐶𝑖𝑚𝑔(𝑡)
𝐼𝐷/𝐵𝑊
where 𝐶𝑖𝑚𝑔(𝑡) stands for total radiotracer concentration in time t, 𝐼𝐷 represents injected dose (in Bq units)
and 𝐵𝑊 is body weight (in g or kg). SUV is used in localization for normalization purposes.
The SUV measurement provides an index of tracer uptake in a ROI or voxel of interest, normalised
to the injected dose and to a normalisation factor (NF) based on the subject’s anthropometric characteristics.
The simplicity and versatility of the SUV make it suitable for clinical routine, as it can be used with a
variety of PET tracers and does not require any arterial cannulation. [18F]-FDG is the most widely used
tracer applied in oncology for diagnosis and tumour staging. For details of the available different
quantitative evaluation methods, the reader can refer to the review work [7].
2.3. Challenges
Many surgeons have developed strategies to improve resectability techniques applied to large
tumors combining it with other complementary treatments such as chemotherapy, which require a close
monitoring of the treatment effect and disease evolution. The literature states that despite the increase pf
successful surgical resection procedures, statistics regarding long-term survival remained somewhat
similar, ranging between 25 % and 40 %. These studies showed the high number of patients developing
recurrent intra- or extrahepatic tumor manifestations mainly due to the presence of undetected tumor sites
at the time of surgery [8]. Hence, the increase in accuracy of the extensive preoperative study of the patient’s
anatomy before liver resection is highly necessary, to accurately define the hepatic lesions, intra- and extra-
hepatically.
Liver segmentation is the basis for computer-based surgery planning of interventions as tumor
resection, living donor transplantations, or minimal invasive surgery [9].
a b c
Figure 2.6. Examples why liver segmentation is a challenging task. In the first two images, liver tissue has to be
separated from adjacent organs stomach (a) and heart (b). The gray-values in all structures are highly similar, which
makes boundary detection difficult without a-priori information about the expected shape in these regions. In the third
image (c), the tumor should be segmented as part of the liver. However, there is a considerable intensity difference
between both structures, which often leads to misclassification of the tumor as non-liver tissue.
11
The liver shares a similar intensity distributions with its surrounding organs (e.g., the heart, the
right kidney and the spleen). This may difficult the analysis of computational tools, especially for automatic
liver detection. Furthermore, other structures may appear with intensity values, and/or adjacent to the liver
walls, making the process of discarding of these structures, by computational algorithms, highly
challenging. In turn, the shape and appearance of the liver vary largely across subjects. Finally, the presence
of tumors or other abnormalities may result in serious intensity inhomogeneity.
The added value of FDG PET to conventional anatomical imaging such as computed tomography
(CT) for assessment of patients with liver metastases has been well documented [8], [10]. However,
previous record of computational algorithms for multi-modal liver cancer analysis are sparse, and the
presence of hybrid PET/CT image acquisitions equipment has been made widely available in clinical
centres very recently.
12
13
State of the art
In this section, we report the criteria used to identify the relevant works for this review. An
overview of the main topics addressed by the selected articles is presented.
A search was performed on PubMed, Scopus and ISI Web of Sciences databases between 10 of
April of 2017 and 10 of June of 2017. Keywords used: “liver”, “segmentation”, “vessel”, “vasculature”,
“lesion”, “hepatocellular carcinoma”. As a result, a total of 721 articles were obtained and, posteriorly
evaluated. In all 721 articles, we analyse the title and the abstract. In this step, the follow criteria were
established, namely: 1) the study must be written in English; 2) patents were excluded; 3) studies with main
focus different of the proposed study were excluded. Tables I, II and III presents an overview of the included
works. As such, a total of 78 articles were selected for the current review.
3.1. Results
Liver and liver tumor computational analysis is a theme that has deserved the attention of
researchers for several years. The challenge however, became highly popularized in the past decade,
motivating the organization of three conference challenges within this period.
The area has evolved from 2D analyses to 3D advanced computational simulations, 3D models of
varied nature from the fields of Computer Vision, image analysis and machine learning. The nature of
different algorithms allows the classification of these algorithms based on the amount, and type of data they
use.
The primary segmentation algorithms were based on image analysis, human engineered rules, and
solution-specific methods, which we will henceforth call in this document as non-learning methods.
Thresholding is the simplest of these algorithms, to separate image regions based on intensity limits. More
advanced methods were included in this group such as deformable, graph-based, and level-set based
models. Deformable models have proven to be very effective tools for complex anatomic motion
reconstructions, or in segmentation of broadly varying shape objects. In previous work, the usage of simple
intensity based segmentation algorithms were used to iteratively segment and adapt a model into the
structures boundaries, called active contours. Other methods derived by different approaches were
extensively validated in the literature such as level-set representations whereas a contour iteratively evolves
dependant on a specific mathematical problem formulation and in turn, graph representations of images
have also been very successful at partitioning dissimilar regions.
Novel segmentation algorithms are continuously emerging, since the accurate modelling of liver
structures still requires further development. Machine learning algorithm came to revolutionize the field of
image analysis. They are in their majority based in advanced statistical analysis of the image data, via the
extraction of different, and simpler representations of the original image data.
Tumor segmentation using both medical images under study can be subdivided into two parts:
initial tumor recognition and its sequential delineation. Recognition determines the tumor location from
other similar regions in the image. In the CT image, this consists in the identification of the liver by manual
or computational methods. In functional images, this consists in the discrimination from other normal
tissues, of regions with high uptake. In many modern algorithms, a rough area of tumor region defined by
clinicians is still needed, hence working as semi-automatic methods. A comprehensive outline of the
literature studies relevant in the field under analysis will be presented in the following sub-sections.
14
3.2. Liver and tumor segmentation in
structural image
Liver detection and segmentation in medical images has been reported using CT, MRI and
Ultrasound. In a liver medical diagnosis system, fully automatic methods are desirable, however, is a
challenging task suffering several stages of development. Methods in the literature have focused on tumor
segmentation in abdominal Dynamic Contrast-Enhanced MRI [11]. However, the most widely used method
for liver diagnosis is CT, and hence, it is also the imaging modality with most prevalence the literature.
As an overview, the study of CT-based liver segmentation has been validated until the last decade
using mainly statistical and atlas-based segmentation models, and were in the past decade overcome by the
higher performance presented by machine learning methods. Moreover, in the past decade, three liver and
liver lesion segmentation workshop challenges were organized, boosting the interest of the scientific
community towards this task.
Non-learning and Image analysis methods
In 2007 a CT liver segmentation challenge was organized for MICCAI 2007 conference. Until the
past decade, the greatest contributions made in the field of whole liver segmentation were reported upon
the MICCAI Sliver 2007 challenge, making available a dataset of 40 contrast-enhanced CT images. It
should be noted that the image dataset was composed of livers containing liver lesions, testing the
robustness of the segmentation algorithms to identify liver lesions correctly as belonging to the object of
interest. In this setting, the top ten methods proposed in the automatic segmentation category, included
deformable models [12]–[16], Statistical shape models (SSMs) [17]–[19], level-set methods [20], [21],
atlas-based methods [22]. The semi-automatic methods proposed by the top six authors in this category
included graph-cuts segmentation [23], flood-filling segmentation [24], a level-set based deformable
segmentation [20], [25], radial basis function guided level-sets segmentation [26] and atlas-based
segmentation [27], further refined by manual interaction.
As could be expected, in the vast majority, the semi-automatic segmentation methods proposed at
the time surpassed the performance of the automatic methods. The challenge evaluated the results by
comparison to expert-generated references and using in a combined scoring method which evaluated a set
deviation metrics to ground truths. The performance results of the MICCAI Sliver challenge are presented
in Table I. It should be noted that the top three best performing automatic segmentation methods in the
challenge were all based on Statistical shape models. SSMs, primarily presented by Cootes et al., consisted
in statically model prior shape data as a parametric set of equations that focus rather on the boundary of the
region to be segmented instead of its internal voxels [28]. These types of algorithms became very popular
in medical image segmentation, despite presenting lack of flexibility. To overcome this, Kainmuller et al.
presented a combined SSM-constrained segmentation followed by a deformable mesh guided by a heuristic
tissue classifier. The method included a thresholding initialization step and allies the free-form intensity
dependant flexibility conferred by the deformable step, being the best automatic method presented in the
challenge [18]. Semi-automatic methods based on user interactive refined graph-cut (GC) segmentation
won the competition [23]. Graph partitioning methods consider the image to be segmented as a graph,
composed of nodes representing image voxels, and edges connecting the neighbouring nodes, weighted by
a given dissimilarity rule. This graph representation can be partitioned into connected components
according to criteria describing the properties of the expected segments resulting in the segmentation of the
image. The dissimilarity measure can incorporate gradient, intensity, texture or any other images features.
GC methods use a minimum cost function between all possible cuts of a graph representation of the images,
requiring background and object manual initialization.
In subsequent years, other methods have continuously been proposed in the literature and
tested on the same MICCAI grand challenge dataset, allowing the direct performance comparison. Later,
automatic segmentation methods were submitted to the challenge outperforming the abovementioned
studies. SSM methods were continuously proposed with different modifications, such as combinations with
free-form constraining [29], level-set [30], graph-cut [31] methods, among others. Graph-based methods
such as GC was proposed by many authors in semi-automatic segmentation methods due to its ability to
interactively edit the segmentation [32].
15
The level-set algorithm is another mathematical formulation of an iteratively evolving surface or
contour. According to this technique the contour is represented using a signed function, i.e. the level-set
function, where the zeros valued locations corresponds to the actual contour. The level-set function requires
Table I. Average values of performance measures and scores of different method results evaluated on the MICCAI
liver segmentation grand challenge. Measures reported as mean and standard deviation over all test images.
Maximum overlap error, average volume different, average distance, root mean squared distance, max volume
distance, and the final average score.
Method Runtime
(min)
Overlap
error (%)
Volume
difference (%)
Avg. distance
(mm)
RMS
(mm)
Max distance
(mm)
Final
Score
MICCAI grand challenge automatic methods
Kainmuller
et al [18] 15 6.1 ± 2.1 -2.9 ± 2.9 0.9 ± 0.3 1.9 ± 0.8 18.7 ± 8.5 77
Heimann et
al. [17] 7 7.7 ± 1.9 1.7 ± 3.2 1.4 ± 0.4 3.2 ± 1.3 30.1 ± 10.2 67
Saddi et al.
[15] 5.5 8.9 ± 1.8 1.2 ± 4.4 1.5 ± 0.4 3.4 ± 0.8 29.3 ± 8.4 64
Schmidt et
al. [13] 6-20 10.4 ± 1.9 -4.9 ± 3.0 1.7 ± 0.4 3.1 ± 1.1 24.0 ± 8.0 63
Chi et al.
[12] 34 9.1 ± 2.8 2.6 ± 6.3 1.7 ± 0.6 3.3 ± 1.2 30.8 ± 9.2 62
Ruskó et al.
[14] 0.5 10.1 ± 4.5 -3.8 ± 6.4 1.7 ± 0.6 3.5 ± 2.3 26.7 ± 11.7 61
Seghers et
al. [19] 30 10.7 ± 2.5 -6.8 ± 2.3 1.8 ± 0.4 3.2 ± 1.1 25.2 ± 10.1 60
Furukawa et
al. [21] 36 10.8 ± 3.7 -7.3 ± 4.7 1.9 ± 1.1 3.7 ± 1.9 31.6 ± 12.7 56
Rikxoort et
al. [22] 45 12.5 ± 1.8 1.8 ± 4.2 2.4 ± 0.3 4.4 ± 1.5 32.4 ± 13.7 53
Susomboom
et al. [16] 25 26.4 ± 24 -11.5 ± 30 10.2 ± 13 17.1 ± 18 74.0 ± 41.5 24
MICCAI grand challenge semi-automatic methods
Beichel et
al. [23] 36 5.2 ± 0.9 1.0 ± 1.7 0.8 ± 0.2 1.4 ± 0.4 15.7 ± 3.5 82
Beck and
Aurich [24] 7 6.6 ± 1.6 1.8 ± 2.5 1.0 ± 0.3 1.9 ± 0.4 18.5 ± 4.1 77
Dawant et
al. [25] 20 7.2 ± 1.2 2.5 ± 2.3 1.1 ± 0.2 1.9 ± 0.5 17.1 ± 5.4 76
Lee et al.
[20] 7 6.9 ± 1.4 1.3 ± 2.9 1.1 ± 0.3 2.1 ± 0.5 21.3 ± 4.0 75
Wimmer et
al. [26] 4-7 8.1 ± 1.1 6.1 ± 2.6 1.3 ± 0.2 2.2 ± 0.4 18.7 ± 4.6 69
Slagmolen
et al. [27] 60 10.4 ± 3.1 3.7 ± 6.2 2.0 ± 0.7 5.0 ± 2.4 40.5 ± 18.2 52
Other methods validated in the MICCAI grand challenge dataset
Chen et al.
[31] 6 6.5 ± 1.8 -2.1 0 ± 2.3 1.0 ± 0.4 1.8 ± 1.0 20 0 ± 9.3 -
Moghbel et
al. [33] - 7.19 ± 1.78 3.41 ± 2.7 0.83 ± 0.27 1.7 ± 0.57 17.05 ± 5.97 77.52
Platero and
Tobar [34] - 5.2 ± 1.2 0.9 ± 1.2 1.0 ± 0.4 2.2 ± 1.1 26.9 ± 10.7 76,3
Hu et al.
[35] - 5.35 0.17 0.84 1.78 19.58 80.3
Lu et al.
[36] - 5.90 2.70 0.91 1.88 18.94 77.8
Dou et al.
[37] 1.5 5.42 1.75 0.79 1.64 33.55 -
16
a contour initialization which then is computed, incorporating the contour propagation speed that is defined
at each voxel. The advantage of this approach is that it can handle topological changes of the contour, but
these methods can be time-consuming and it is difficult to handle over-segmentation. Level-set have been
successfully applied to whole liver segmentation. Wimmer et al. (2009) latter proposed a level-set based
ASM, that used a Parzen density estimation, to boost classifiers to analyse appearance information [30].
Image-based methods, besides level-set algorithms were proposed in semi-automatic methods.
Ruskó et al. proposed an improved histogram-guided Region Growing (RG) method, obtaining competitive
results in multiphase CT images [38].
More sophisticated SSMs such as Active Appearance Models (AAMs) have also been proposed
in the literature. Transversal to SSMs and image and contour-based algorithms, initialization always stood
as the biggest issue in the performance of these methods. Chen et al. proposed a novel combined AAM with
a free-form segmentation based on Livewire and GC algorithms, for abdominal multi-organ pose
estimation, initialization and segmentation [31]. The method was validated and compared with the dataset
and results of the MICCAI Sliver challenge presenting similar performance and demonstrating less
computational cost. Tomoshige et al. proposed a relaxed conditional SSM with conditional features error
model, with a subsequent free deformation step demonstrating very positive results for automatic liver
segmentation, on a private dataset of 144 non-contrast CT images, expressing performance as a Jaccard
index valued 0.86 [39].
The MICCAI Sliver dataset was also used for the development of liver lesions segmentation.
Another pertinent graph-based algorithm that became popular in medical image analysis is the Random
Walker (RW) algorithm for segmentation, proposed by Grady (2006) [40]. Moghbel et al. tested the
performance RW for liver lesion segmentation, developed on an image dataset composed of healthy and
unhealthy livers, combined with a rib-cage removal using B-spline contouring, achieving a promising
average performance expressed as a Dice similarity index (DSC) valued 0.94, and 0.91 for the MICCAI
Sliver challenge [33].
In turn, another popular type of segmentation algorithms were atlas-based methods. Probabilistic
atlases (PA) use shape priors and spatial relationship information. Atlas-based methods were also presented
in the MICCAI Sliver challenge, however their initial formulation did not outperform the previous model-
and image- based methods, not being able to handle properly the very high variation that characterized the
liver shape among different patients. Nonetheless, Okada et al. proposed a novel method combining the
properties of both probabilistic atlases for initialization followed by an optimized SSMs fitting to the image
intensities [41]. Lastly, Xu et al. proposed a registration based on dense 3D-scale invariant feature transform
(SIFT) features to find correspondences between source images and target atlas. The labels of the source
image are later used to segment the target image, and the method was validated on the MICCAI Sliver
challenge, with performances of 96% dice overlap [42]. In turn, Platero and Tobar developed a method
combining the spatial normalization with the segmentation method based on standard CRF models to guide
19 atlases for liver segmentation, presenting high DSC valued 95%–97% for the MICCAI 2007 Sliver
dataset [34].
In the following year, MICCAI 2008 conference launched the MICCAI 2008 Workshop on 3D
Liver Tumor Segmentation Challenge, making available CT data from 30 liver tumors. At this time,
research in medical image started to incorporate machine learning methods. Hence the proposed methods
are composed by a mixed variety of algorithms. Similarly to the previous challenge, semi-automatic
methods outperformed automatic ones. In this competition, the top ranked method presented, consisted in
a semi-automatic method combining graph-cuts with a watershed low-level segmentation [43]. Moreover,
region-growing algorithms [44], thresholding only [45] and combined with filtering techniques [46],
intensity-based analysis [47], [48], deformable models [49], [50].
Machine Learning based methods
Apart from the mentioned methods, novel segmentation techniques incorporating
machine learning algorithms have gained attention by researchers. Machine learning, as in many other fields
of image analysis, has grown preponderantly, presenting superior results in a varied number of
computational tasks. More recently, a sub-field of machine learning algorithms, denominated as deep
learning, is the current state of the art set of algorithms that are being validated in the literature for signal
and image processing tasks. Medical image analysis research field has also followed the deep learning trend
17
and is currently applying these algorithms to a wide number of segmentation tasks. An overview of the key
first machine learning incorporations with standard methods, pure machine learning algorithms and more
advanced deep learning methods applied to liver organ segmentation follow. Among these, data-driven
methods for pixel labelling were widely used for target object segmentation.
In the context of the MICCAI 2008 workshop challenge, learning-based algorithms were proposed
based on Adaboost models [51], Bayesian probabilistic methods and Support Vector Machines (SVM)
voxel classification.
SVM belongs to the supervised learning methods that combine linear algorithms with linear or
non-linear kernel functions SVM by finding the best generalizing hyperplane with maximal margin
separating the two classes. The second best method proposed in the MICCAI 2008 competition was a semi-
automatic method using supervised SVM voxel classification strategy in 2D slices, further propagated
adjacent slices for tumor segmentation [52]. Other Bayesian learning algorithms [44] were also proposed.
The results of the segmentation algorithms presented in the competition for liver tumor segmentation are
outlined in Table II.
In subsequent years also, many authors used the challenge dataset for method validation. Zhang
et al. proposed a watershed liver initialization followed by SVM-based semi-automatic regional tumor
classification algorithm [53]. The authors validate their method on the MICCAI liver tumor challenge
dataset, and the method outperforms the results distance to ground truths, than the previous methods.
Machine learning algorithms were also applied successfully to whole segmentation. Selver et al.
developed a fully automated liver segmentations method that employs pre-processing for exclusion of
neighbouring structures, k-means clustering, and multilayer perceptron (MLP) for feature based boundary
recognition, and post-processing for removing miss-segmented objects and smoothing liver contours [56].
Table II. Average values of performance measures and scores of different method results evaluated on the MICCAI
2008 liver lesion challenge. Measures reported as mean and standard deviation over all test images. Maximum
overlap error, average volume different, average distance, root mean squared distance, max volume distance, and
the final average score.
Method Overlap
error (%)
Volume
difference (%)
Avg. distance
(mm) RMS (mm)
Max distance
(mm) Final Score
MICCAI grand challenge automatic methods
Stawiaski [43] 29,49 23,87 1,50 2,07 8,29 73
Zhou et al. [52] 30.02 19.31 1.52 2.18 10.52 72
Nugroho et al.
[48] 31.21 15.76 1.75 2.56 11.73 71
Choudhary et al.
[46] 32.14 22.58 1.77 2.40 9.29 70
Wong et al. [44] 32.14 22.58 1.77 2.40 9.29 70
Moltz et al. [45] 30.55 25.32 1..55 2.20 9.13 72
Smeets et al.
[54] 34.58 17.79 2.01 2.67 10.09 69
Shimizu et al.
[51] 28.98 18.29 1.81 2.35 7.78 65
Ben-Dan et al.
[55] 49.12 39.52 3.04 3.73 11.38 56
Schmidt et al.
[49] 52.95 80.02 3.91 5.19 16.28 48
Hame et al. [50] 47.33 111.11 5.40 7.87 25.07 48
Kubota et al.
[47] 53.72 44.75 3.73 4.60 13.81 38
Other methods validated in the MICCAI 2008 grand challenge dataset
Zhang et al. [53] 31.14 19.57 1.56 2.09 7.9 -
18
The method is validated in CT images with and without contrast media. Goryawala et al. propose a 3D
whole liver volume segmentation method, with semi-automatic k-means clustering initialization followed
by 3D RG algorithm, ensuring a parallel computational processing framework [57]. Wu et al. propose a
supervoxels analysis, generated by the 2D simple linear iterative clustering, of images segmented with
graph-cut algorithm [58]. Supervoxels are generated from a set of clustering algorithm types that divide the
images according to texture distributions. Another feature-based segmentation using SVM classification of
textural features with combined morphological operations was proposed by Luo et al. [59]. He et al.
proposed a three-level AdaBoost-guided ASM to segment the liver in CT images, including an Adaboost
voxel labelling initialization, a profile classifier, refined by an ASM mesh model [60]. Zheng et al. propose
a feature-learning-based random walk method for liver segmentation using CT images. A learning step
consisting on Haar, histogram of oriented gradient (HOG), local binary pattern (LBP), and gray level co-
ocurrence matrix (GLCM) features are learned by an Adaboost guided Support Vector Machines (SVM)
model, which generates automatic seed points on the original test image, to carry the automatic
segmentation step via Random Walks algorithm [61].
Markov Random Fields (MRF) theory is another ML algorithm widely used in the literature. This
algorithm considers that hidden node representing a label (e.g. object of interest, background, etc.) is
assigned to each observation node (e.g. set of features extracted form voxel or a connected set of voxels).
The method computes the hidden node configuration with the highest probability given the observation
nodes and the built-in model. Alomari et al, proposed an MRF contour initialization followed by an active
contour segmentation refinement [62].
Finally, a particular interesting method was proposed by Vijayalakshmil and Subiah [63]. The
authors method performs a classification with local binary pattern images for categorising normal and
abnormal liver images. They use texture feature extraction with LBP and Legendre moments, and validate
the classification method with a dataset of 197 CT (77 normal, 125 HCC), achieving a very positive
accuracy of 96.17%.
Deep learning based methods
From the machine learning field, emerged the sub-field of deep learning algorithms, which have
presented a previously unseen performance by computational methods, in a variety of tasks. The field of
deep learning methods has evolved from the first applications of these methods to signal processing and
later to image analysis. Hence, deep architectures consist in specific kinds of neural networks, and range
from Deep Belief Neural Networks (DBNs), Sparse Autoencoders (SAEs) and finally Convolutional Neural
Networks (CNNs). From these basic types of architectures, based on the known Neural Networks
architectures, each author’s method is characterised by different neural network design. These design
details range from the number of hidden layers, type of layer, layer distribution, type of activation among
others. Hence, the methods technical characteristics are described briefly for each approach in the literature.
The first applications of DL approaches to liver segmentation were proposed using SAEs. The first
method using deep neural networks for liver segmentation was proposed by Shin et al. in a multi-organ
segmentation pipeline applied to Dynamic Contrast Enhanced Magnetic Resonance Imaging (DCE-MRI)
[61]. The authors base their knowledge in previous applications of deep architectures to object recognition
in natural images and propose an SAE neural network to separately learn 256 visual and temporal features
in an unsupervised and automatic manner, and detect multiple organs in a time series. The authors seek to
propose the first deep architecture procedure for medical image segmentation tasks, tackling the substantial
shape changes that characterize the abdominal organs such as kidneys, liver, heart among others, hence
using a data sets from two studies of liver metastases and one study of kidney metastases. The SAE was
characterized by sparsity Kullback-Leibler -based constraint application to logistic sigmoid activations,
interleaved with max-pooling operations after each layer. A first step of tissue type labelling was performed,
followed by separation of each organ recurring to a parameter optimization using to context-specific feature
manipulation, to identify each organ in a supervised manner, obtaining a classification accuracy of
approximately 64.8% for the liver.
Lu et al. proposed a segmentation method combining a 3D CNN to obtain liver probability maps
which would be further incorporated in the image information term of the energy functional of a GC
algorithm. The authors validated the method on forty CT volumes, 20 taken from the MICCAI Sliver07
challenge and 20 from the 3Dircadb datasets. The authors use stacked convolutional and average pooling
layers, trained by gradient-based backpropagation. The authors take advantage of the capability of GC
19
method of handling loose boundaries between tissues with similar intensity distributions. The proposed
method is the first 3D CNN application to liver segmentation, and obtained a score of 77,8 by the Sliver
challenge evaluation method [36].
Hu et al. proposed an automatic 3D CNN, trained to output liver probability maps complemented
with a globally optimized surface segmentation. The authors used 42 CT images from MICCAI Sliver
challenge train dataset and a local database, and validated the method on the challenge’s test dataset
obtaining an overall score of 80.3, surpassing any other methods previously proposed [35].
Christ et al. proposed a cascaded CNN in 2D with a 3D dense conditional random field (CRF)
approach as a post-processing step, to achieve higher segmentation accuracy while preserving low
computational cost and memory consumption [64].
Dou et al. presented a novel 3D CNN equipped with fully convolutional architecture and a 3D
deep supervision mechanism to comprehensively address the challenges of volumetric medical image
segmentation [37]. The deep supervision method, to which the authors name DSN, uses an objective
function that guided the training of the upper and lower layer, propagating more efficiently the
representation of the features, as well as speeding up the training process, by patch-based methods and
volume-to-volume learning and inference. The authors CNN consists in 6 convolutional layers, 2 max-
pooling layers, and one softmax output layer layers finalised with 2 deconvolutional layers, trained via
stochastic gradient descent (SGD), in a fully convolutional architecture with 3D kernels, outputting a
spatially arranged classification across the whole input pixels. The resulting predictions are rougher and
obtained in low-dimensional layers, which is thus solved by the deconvolutional final layers. Furthermore,
the authors emphasise the gradient instability that has been reported in previous works, where propagated
‘fading’, exploding or vanishing of gradient magnitudes occurs. A DSN mechanism is proposed in the
network architecture by softmax extraction of outputs predictions from intermediate layers convolutional
results, connecting them directly to the final output layer, and whose weighting is incorporated in the
backpropagation objective function. Finally, the authors propose the incorporation of a fully connected
CRF for segmentation refinement, which has advantage in capturing complicated shaped object such as
those with holes or thin structures. The results were evaluated on the MICCAI Sliver challenge dataset
presenting superior Volume Overlap Error and computational cost.
From this point, the ISBI 2017 conference, released a challenge focusing on liver lesion
segmentation, this time making available a total of 200 abdominal contrast-enhanced CT images of liver
tumors. The dataset was obtained from from several different clinical sites with different scanners and
protocols, thus having largely varying spatial resolution and fields-of-view. The in-plane resolution ranges
from 0.60 mm to 0.98 mm, and the slice spacing from 0.45 mm to 5.0 mm. The axial slices of all scans
have an identical size of 512 × 512, but the number of slices in each scan differs greatly and varies between
42 and 1026. The following three works presented, were developed under this scope of this challenge and
will be reviewed in detail.
Chlebus et al. proposed a two-step segmentation method, initiated by a liver volume segmentation
followed by a liver lesion candidate detection and classification [65]. In the first step, a coarse segmentation
was obtained from an ensemble of three CNNs trained with each of the three orthogonal interpolated
volumes, followed by a 3D CNN liver refinement. The axial, sagittal and coronal volumes trained three 2D
U-net models which whose softmax outputs were used to train a final 3D U-net that originated a full mask
volume of original resolution. Based on the liver mask ROI extraction the second step consists in a two-
step tumor segmentation using a 2D CNN for tumor segmentation, tumor candidate detection via 3D
connected component analysis of the mask volumes, followed by a Random Forest tumor candidate
filtering. Similar to the first step, a 2D U-net was used for the tumor segmentation and were trained with
class balanced boundary patches of the axial dataset, however, the authors hypothesize that the patching
step used for training penalized the specificity of these segmentations, requiring thus, the incorporation of
a 46 feature extraction and classification step of these candidate tumor mask. The method scored a Dice
coefficient of 0.65, and second place in the challenge, including a tumor candidate classification accuracy
with the random forest approach of 90%.
Bi et al. propose a cascaded deep residual networks (ResNet) approach to segment the liver lesions
[66]. As pre-processing these authors apply data argumentation strategies including random scaling, crops
and flips. ResNet uses shortcut connections to avoid training degradation though deeper layers, whereas
the optimal results are calculated by an averaging output of the different networks. The key feature of the
architecture proposed is the cascading of parameter learning, made from learning the training data and from
the previous iteration result outputs. The testing segmentations are further incorporated with a multi-scale
20
rescaling strategies whose outputs averaging produce the final predictions. Given the architecture depth of
20 hidden layers training the entire volumes would become computationally non-viable, so the authors
randomly pick a balanced dataset of axial slices, of a total of 8802 images to train the neural network via
stochastic gradient descent. The network is pretrained firstly on the ImageNet dataset for parameter fine-
tuning, which is then further fine-tuned with the liver dataset.
Hoogi et al. focus their works on level-set parameter optimization via a CNN. The rough liver
lesion probability maps outputted from a CNN are used to compute the contour initialization and level-set
curve parameters [67]. Finally, Ben-Cohen et al. proposed a fully convolutional CNN, presenting a detailed
architecture evaluation. The authors obtain a maximum 0.89 DSC for the whole liver segmentation
validated on the Sliver dataset, and a true positive rate of 0.86 for liver lesion segmentation.
3.3. Liver and tumor segmentation in
functional image
Liver and/or liver tumor segmentation methods are very sparse in the literature is
considered a more challenging task as a wide range of tumor tissue appearances makes for a difficult
segmentation. Moreover, this segmentation is usually done on a noisy background as the low contrast of
liver tumors mandates the use of contrast enhanced CT scans, increasing the amount of noise in images.
Segmentation of hepatic tumors has only been explored in mainly, the literature records, to the
best of our knowledge.
Due to the specific intensity distribution these images have, thresholding methods
intuitively present fairly good results. Magnander et al. presents a method to extract liver foci from SPECT
images acquired with 111In-octreotide. The authors propose segmentation with a thresholding method based
on connected component analysis [70].
Hsu et al. propose an ASM method initialized by a Genetic Algorithm (GM) -generated
initial contour to segment the whole liver in PET images [71].
Finally, a robust watershed method was also experimented for the task of liver
segmentation. Shonket et al. experiment the performance of 2D and 3D Watershed (WS) method [72].
According to the watershed approach the gradient magnitude of a grayscale image is considered as a
topographic surface. Voxels located on segment boundaries, where the gradient magnitude has local
Table III. Average values of performance measures and scores of different method results evaluated on the image
dataset, using CNN-based methods. Measures reported with different segmentation accuracies.
Author Method Dataset Accuracy
Lu et al. [36] Liver 3D CNN, combined with GC refinement Sliver challenge: 30 CT VOE: 5.70
Hu et al. [35] Liver 3D CNN combined with globally
optimized surface Sliver challenge: 30 CT
VOE: 5.35;
DSC: 97.25
Dou et al. [37] Fully convolutional CNN with deep supervision Sliver challenge: 30 CT VOE: 5.42
Christ et al. [64] Liver and tumor CT CNN U-net, cascaded
fCNN and dense 3D CRF CT: 5 healthy, 15 tumor 0.94 DSC
Han X. [68] Deep Convolutional Neural Network LiTs challenge: 70
healthy, 130 tumor CT 0.67 DSC
Chlebus et al.
[65]
CNN followed by RF-based lesion candidate
filtering
LiTs challenge: 70
healthy, 130 tumor CT 0.65 DSC
Bi et al. [66] ResNet based CNN with cascaded optimization LiTs challenge: 70
healthy, 130 tumor CT 0.5 DSC
Ben-Cohen et al.
[69] Liver and tumor 2D CNN 22 CT tumor 0.89 DSC
Hoogi et al. [67] Lesion CT CNN 2D CNN obtained probabilities
are used to drive active contour model 112 CT, 164 MRI tumor 0.71 DSC
21
maximum, correspond to watershed lines. The method virtually places water drops to each voxel position.
From each voxel the water flows downhill to a local minimum. Voxels draining into the same local
minimum form one basin and represent one segment of in the image. The simplest implementation of this
approach uses priority queue to encounter all voxels of the gradient magnitude image starting from those
located in a local maximum. Watershed segmentation usually divides images into large number of
partitions. There are various strategies to merge smaller basins into larger ones based on different similarity
criteria. Shonket et al. conclude that 2D WS presented higher volumetric accuracy than 3D WS, in the
phantom studies performed. Phantoms model a range of tumor shapes, and structures, whose known
dimensions allow the accuracy measurement of imaging equipments, and subsequently of segmentation
algorithms.
3.4. Liver vessel segmentation
Liver vessels are structures of interest when it comes to liver cancer surgery planning. Specifically,
vessels are used to approximate liver segments.
Region growing methods are one of the most common in the literature, because of their simplicity.
Region growing segmentation of liver vessels has been reported as a pre-processing step in the literature
[73]. Oliveira et al., for example, calculated a Gaussian mixture model of three Gaussians for liver vessels,
nodules and parenchyma. The resulting mixture model was then used to derive thresholds for a region
growing algorithm. The authors further propose a method that classifies branches to belong to the hepatic
vein if they were longer than 15% of the liver height. Therefore, the biggest component in the first slice
containing vessels is searched and merged with the biggest overlapping component on the next slice until
no overlapping vessels are found anymore. This is repeated until three main branches are found. The portal
vein is identified as the biggest connected component when the previously found hepatic veins are
subtracted from the vessel segmentation [74].
Level set methods have also been proposed to segment liver vessels in their original formulation
[75] and in a modified energy minimization problem, where the level set function is set to the output of a
quadrature vessel enhancement filter. The resulting energy formula is solved using the calculus of variation
and Euler-Lagrange method [76].
Methods using graph-cuts also showed potential in this task [74]. Pamulapati et al. registered
contrast enhanced and non-contrast enhanced CT data to define a regional term derived from three energy
functions. A data energy computes penalties on a 4D histogram of object and background voxels. Another
energy penalizes voxels that do not rapidly enhance between both phases, and the third energy is derived
from a “vesselness” function [74].
Drechsler et al. proposed a novel multiscale Hessian-based analysis for vessel enhancement and a
and the proposed thresholded region growing method [73].
Only recently, machine learning methods were applied to this segmentation. Zeng et al. proposed
the extreme learning machine method for liver vessel segmentation [77]. At first, the image noise is
removed by anisotropic filter and then vessel filters like sato, frangi, offset medialness filter and strain
energy filter are used to extract vessel structure features. Then extreme learning machine (ELM) is applied
to segment liver vessels from the background. Shi et al. proposed an automatic liver segmentation method
based on multilevel local region-based sparse shape composition with initialization based on the shape of
blood vessels. The idea is that the liver vessels can be used to acquire patient specific prior information
such as shape and size of the liver, providing a more accurate initialization [78]. Finally, very recently,
Kitrungrotsakul et al. proposed a CNN method, composed of three stacked 2D CNN, in a deeply
convolutional architecture, combining liver vasculature segmentations of sagittal, coronal and axial views.
The authors validate their method on 12 CT volumes, obtaining an overall DSC performance valued as 0.83
[79].
Very few of the proposed methods in the literature make a performance analysis, aside from visual
inspection. This is due to the time-consuming task that would represent manual annotation of liver vessels.
22
23
Methods
The PhD project consists of four main work packages (WPs) executed in two stages. In the first
stage (WP1) an extensive study of segmentation strategies with different Convolutional Neural Networks
architectures will be evaluated to segment abdominal CT images. Next the further integration of the
architectures developed will be adapted to a multi-modal segmentation, from PET and CT images, in WP2.
In a WP3 stage the incorporation of key aspects of the liver for the target purpose of Liver resective surgery
planning will be studied, integrating a vasculature segmentation and liver lobule map extraction. A final
WP4 of integration the former three data sources, of liver, lesion, and lobule map division in an interactive
visualization tool will be carried.
4.1. Work package 1 – CT liver segmentation
The study of liver and liver lesions has been tackled very recently in ISBI challenge, the Lits liver
lesions challenge. The top scoring algorithms consisted in three different CNN architectures, obtaining a
maximal accuracy of 0.65 DSC. There is still much, to be improved regarding CNN methods used for the
liver and liver lesion segmentation based solely on CT images.
The dataset of the Lits challenge is composed of 200 contrast -enhanced abdominal CT volumes,
obtained from different scanner equipments (retrieved from different clinical imaging centres), and of
images with different number of slices and resolution. The development of segmentation algorithms capable
of extracting volumetric segmentations of these structures, accurately, regardless of the equipment and
conditions of acquisition is very relevant in the clinical treatment planning of the patients. Accurate liver
segmentation, isolating it from the surrounding anatomical structure is a critical component of a cancer
CAD system.
Data available:
• ISBI 2017 Lits challenge for liver lesion segmentation – 200 contrast-enhanced CT.
• MICCAI 2007 Sliver challenge for liver segmentation – 35 CT.
• Partner medical institutions providing 30 CT, at the moment. This number of studies can
possibly increasing over time.
Task 4.1.1 - Pre-processing: Image noise is present in CT volume images, which is one the
fundamental issues with this type of images. One of the most successful algorithms is non-local means
denoising, which has proven, in previous works, to be effective in boundary preservation and enhancement.
Hounsfield unit conversion of the images has been reported as an advantageous pre-processing of the
images, eliminating non-liver tissues and facilitating the segmentation task.
Task 4.1.2 – Convolutional Neural Networks building: CNNs are models (networks) composed of
many layers that transform input data (images) to outputs (object location probability maps) while learning
increasingly higher-level features. A CNN will be used as model, and trained end-to-end for parameter
optimization of the hidden convolutional operations towards liver and lesion segmentation.
Different architectures can be evaluated, including post-processing methods for result refinement:
• 3D multi-scale convolutional neural networks (CNN) [80].
• Deep convolutional neural networks (DCNN) present key representation robustness in
their learning process to object locations, which is useful for object recognition tasks. Ben-Cohen et al.
validated this approach in a reduced dataset [69].
24
• Research has shown that combining DCNNs and Conditional Random Fields (CRFs) can
significantly improve scene parsing accuracy.
• For learning-based methods including CNNs, the learned predictions are usually refined
by traditional methods for further accuracy improvement. Further fine tuning of the derived contours.
Level-sets are a class of active contours that present very robust segmentations, by mathematical modelling
of a signed distance function of an initial contour (zero level-set). The robustness of level-sets can thus be
added to the final refinement of the segmentations derived from the probability maps generated from CNNs
[81].
Implementations are expected to be developed using Google Tensorflow, Keras, Theano and Caffe
python tools and adequate GPU processors.
Task 4.2.3 - Validation of the segmentation approach: The methods performance will be evaluated
with expert-generated references, considered the golden standard, and rated accordingly to detected
deviations. Commonly used metrics based on volumetric overlap and surface distances, such as Volumetric
Overlap Error, Dice similarity coefficient, Relative Volume Difference, Average Symmetric Surface
Distance and Maximum Symmetric Surface Distance. The segmentations also will be reviewed by an expert
collaborator. Manual segmentations, whereas metrics of accuracy between the segmented point clouds will
be computed.
4.2. Work package 2 – PET-CT tumour
segmentation
Very simple methods have attempted the present task of liver lesion segmentation using
multimodal imaging. Specifically, only two research records were found. The first, consists in a semi-
automatic model-guided segmentation, using a primary Statistical Region Merging algorithm followed by
a model-to-image registration. The method was tested on both CT-only and PET/CT image volumes [82].
Secondly, a learning-based segmentation was performed using a binary tree quantization algorithm for
clustering of image structures [83]. Both methods were applied to the fused PET/CT image.
Task 4.2.1 – Tumor segmentation: The goal of using FDG-PET for tumor volume delineation is
complementing the morphological information provided by CT, with the biological tumour data. Many
segmentation algorithms have been applied in PET-only segmentations such as thresholding, region
growing, classifiers, clustering, edge detection, Markov random field models, deformable models among
others. It is intended to take advantage of the natural co-registration of the PET/CT images to integrate the
previous ROI volume segmentation of the whole liver.
Data available:
• Partner medical institutions providing 30 PET/CT, at the moment. The number of studies
can possibly increasing over time.
Multi-modal segmentation using CNN methods, based on both structural and functional image has
been reported by several authors in segmentation tasks targeting other body structures. For instance, very
recent works are being published targeting the field of brain structures and tumors segmentation, using
multimodal images. The usage of the multimodal image data can be performed by different approaches. In
classical CNN approaches, image data information is used, retrieved as high-level patch features, to which
the subsequent layers hierarchically apply further transformations, generating in a final layer, final
segmentation maps as outputs. These architectures can integrate multimodality.
It is not of great value and does not allow the proper evaluation of the hierarchical feature learning
from each individual modality, to use the originally fused PET/CT image data as input in CNN
architectures. Multimodal image data, as has been used in previous studies, can be integrated by two
essential strategies. Feature maps extracted from the input images can be fused into one feature map that is
inputted into the CNN. Other architectures include an initial layered architecture of individually trained
CNNs that act as feature extractors from each image type, which are in more deep layers shared and used
for the final probability maps output. Illustrations of both examples are represented in Figure 4.1.
25
Based on the literature, the CNN methods can be proposed using different approaches such as:
• Fused feature maps from different MRI contrast modalities have been used for brain tissue
segmentations, by convolutional feature extraction, and subsequent deep convolutional layers,
that enforce competitions between features at the same spatial location across the different
feature maps [84].
• Deep CNN architectures, can be composed of different feature extractor constituting
individual CNNs, that can in deeper layers be joint by local response normalization layers [85].
• Lesion candidates pre-selection in each modality, using strategies of PET thresholding,
and CT pre-CNN liver mask and lesion detection, can be combined in a final CNN lesion
candidate refinement [86].
The choice of an adequate architecture for this task will evaluated. The inclusion of multi-modality
in this segmentation task is the main contribution of the proposed work plan, and has not been reported in
the literature using CNN methods.
Task 4.2.2: Experimental validation – The developed automatic segmentation method requires the
careful validation by an expert collaborator. Manual segmentation will be considered the gold standard,
whereas metrics of accuracy between the segmented point clouds will be computed.
4.3. Work package 3 – Liver segments map
segmentation
Liver resections are performed by physicians based on CT volume data, which mentally visualize
the liver segments, lesions and vasculature positioning to plan the segments to resect. Analysis of the 3D
vasculature structure of the liver is important to modulate and improve surgical and resection planning, as
well as radiotherapy puncture location decisions. Moreover, it delimits the liver segments, which determine
the surgery strategy adopted and aids in predicting postoperative residual liver capacity. Liver segments are
divided by the Couinaud’s segmentation standard into nine regions.
Task 4.3.1 – Liver vasculature segmentation - Recent advances in machine learning methods,
deformable models are becoming very popular for liver vessel segmentation. Usually, an initial
segmentation is achieved using machine learning algorithms such as pixel clustering methods and then
refined by deformable models such as level sets. Validating vessel segmentation results are very difficult
since these validations are generally obtained using corrosion casts that require expensive equipment and
Figure 4.1. Examples architectures of input feature maps integration from
multimodal images [95].
26
are very tedious to prepare and evaluate. Only recently, a CNN method was applied to this task [79]. CNNs
have thus the potential to extract the tubular structures present within the liver.
Task 4.3.2 – The livers segmental structure can be organized in a liver atlas, built from the
available image dataset. Probabilistic atlas can be derived from a previously known segment distribution
within different liver examples. These atlases will capture the average boundary of an organ or structure of
the body, and the confidence or the complete spatial distribution of probabilities that a voxel belongs to a
given structure in the atlas. An atlas-based segmentation can subsequently be derived by registration. The
main idea, is that the segmentations remains rigid for vessel segmentation and the segments segmentation
becomes non-rigid to the whole livers structure.
Task 4.3.3 –The validation of the vasculature segmentations and quantifications of the 3D models
will be validated by a medical image expert collaborator.
4.4. Work package 4 – Integration and
Visualization
Task 4.4.1 - Surgical planning of liver tumor resections requires detailed three-dimensional
understanding of the complex arrangement of vasculature, liver segments and tumors. Hence, the final step
of this framework includes the development of a computational visualization tool to assembly all the
segmented structures.
4.5. Overview
There are several factors that take a role in CNN performance, and that will be taken into
consideration. When training CNNs, the efficient implementation on modern powerful GPUs to train large
networks with huge number of parameters is necessary, the proposal of useful tricks like Rectified Linear
Unit (ReLU), batch normalization and dropout strategies, are recent performance improvement approaches
that will be addressed. Other CNN optimization strategies will be searched and studied.
The network computational optimization is a big challenge when it comes to CNNs, especially in
the case of deep CNNs. research strategies that intend to speed up the training an open issue among
researchers. Recent work from Google, implements layer module, thought to preceed the convolutional
layer, that makes affine transformations to the input image in order to help models become more invariant
to translation, scale, and rotation [87].
The main contributions of our approach are the following: (1) structured output delineation for the
region of interest (ROI) liver using a fully end-to-end trained CNN, (2) structured delineation of the liver
lesion, (3) integration of both PET and CT information for liver and tumor detection and visualization,
outputting data representations adequate for proper surgery planning and radiotherapy target volume
definition.
The final purpose of this work, is the design of a modular software platform suitable for the
visualization, analysis of the anatomy of the liver and liver lesions in 2D, 3D, and volumetric quantification.
The importance of volumetric quantification lies on the specific aid in surgery planning, whereas the
necessity is not only in anatomy visualization, but also in the definition of resective volume and evaluation
of the viability of the remaining liver volume for patient survival.
Whenever possible, to assure some of the functions desired for the platform, computational
libraries that are available in an open access based should be considered. Few examples of those
computational libraries are: Newmat (Algebric and Matrix manipulations -
http://www.robertnz.net/nm_intro.htm), OpenCV (Image processing and analysis -
http://sourceforge.net/projects/opencv/), VTK (Computer Graphics - http://www.vtk.org/), Free DICOM
(DICOM files - http://www.idoimaging.com/index.shtml) and ezDICOM (DICOM files -
http://sourceforge.net/projects/ezdicom/). The platform proposed will have a modular and open
architecture. The set of works described here will have a final period of thesis writing.
27
4.6. Timetable
2017 2018 2019 2020
Task Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb
4.1.1
4.1.2
4.1.3
4.2.1
4.2.2
4.3.1
4.3.2
4.3.3
4.4.1
Thesis writing
28
29
Final Remarks
In the literature, the top performing recently published automatic liver segmentation methods,
report as good results as DSC values of 0.96 [42], [60]. Conference challenges were the biggest booster for
the development of research in the field, and provided a platform for testing and comparing different
approaches for the topic.
The inclusion of PET image as a screening tool of the disease was also equationated in the literature
in clinical studies. However, the development of computational tools that incorporate data from both
medical images are at fault. Hence, liver lesion delineation and proper surgery planning may benefit from
the inclusion of PET image as a secondary data source.
As an overview, it is possible to conclude that the structures of interest are not only whole liver
and liver lesion, but also liver vasculature and anatomic segment partition. Hence for computer aided
diagnosis (CAD) purposes and surgery planning, the adequate system would provide accurate information
regarding these four anatomic structures. Furthermore, this information is only pertinent for physicians
when integrating their 3D distribution adequately, facilitating and enhancing their visualization capability.
Different sources of image data were identified in the literature:
• MICCAI 2007 liver segmentation challenge – 35 CT images [88].
• MICCAI 2008 liver lesion segmentation challenge – 30 contrast-enhanced CT images
[89].
• 3D-ircadb01 dataset – 20 contrast-enhanced CT (15 HCC and 5 healthy subjects) [89].
• ISBI 2017/MICCAI 2017 liver lesion challenge – 200 contrast-enhanced HCC subjects
[89].
• Cancer Imaging Archive TCGA-LIHC – 75 CT subjects and 1 PET subjects [90].
• MIDAS dataset – 4 CT images (4 HCC patients) [91].
The segmentation of liver is difficult due to the fact that the CT image includes other
organs like spleen, pancreas, kidney. Hence the work plan proposed is based on three main aspects
identified from the literature analysis:
• CT is the most widely used technique to analyse the liver anatomy, in the particular tasks of cancer
screening and treatment planning.
• Multi-modal techniques have presented improved results in other organ and cancer-related
segmentation tasks;
• The availability of big datasets make possible the development of new computational methods,
based on deep neural network architectures.
Some problems in the segmentation results were transversal to several studies such as:
• High variability of liver shape may compromise the accuracy;
• Close vicinity with other organs provoked lack of robustness against leakage to nearby organs
with similar intensities, in many methods;
• Liver lesions present nearby the liver limits also withdrawn the robustness of liver boundary
delineations, of several methods;
• In CNN methods the choice of input patches needs to capture adequate context information,
otherwise compromising the proper segmentation of lesion internal pixels, specifically in lesions
with higher dimension.
30
From the literature reviewed, research in the future is intuitively based in CNN methods. However,
there is still room for advanced improvement using these types of algorithms, which are continuously being
updated by new proposed architectures. The range of performance metrics reported in the literature,
positively indicates that there is still space for significant improvement of these algorithms. Hence, the main
objective in the present, is the development of CNN methods sufficiently robust tackle the proposed
segmentation tasks in images retrieved from different acquisition machines and image resolutions.
Regarding liver functional analysis via PET imaging, the literature is highly sparse, proving that
the development of novel computational multi-modal tools for liver analysis are at fault, and can potentially
aid the full detection of liver lesions, and subsequently, the job of physicians.
31
References
[1] L. Rahib, B. D. Smith, R. Aizenberg, A. B. Rosenzweig, J. M. Fleshman, and L. M. Matrisian,
“Projecting Cancer Incidence and Deaths to 2030: The Unexpected Burden of Thyroid, Liver, and
Pancreas Cancers in the United States,” Cancer Res., vol. 74, no. 11, 2014.
[2] WHO, “Globocan 2012,” International Angency for Research onCancer, 2012. [Online].
Available: https://gco.iarc.fr/today/home.
[3] S. Vilarinho, E. Z. Erson-Omay, K. Mitchell-Richards, C. Cha, C. Nelson-Williams, A. S.
Harmanci, K. Yasuno, M. Gunel, and T. H. Taddei, “Exome analysis of the evolutionary path of
hepatocellular adenoma-carcinoma transition, vascular invasion and brain dissemination,” J.
Hepatol., vol. 67, no. 1, pp. 186–191, 2017.
[4] T. Nakashima and M. Kojiro, Hepatocellular carcinoma : an atlas of its pathology. Tokyo:
Springer-Verlag, 1987.
[5] I. Levy and M. Sherman, “hepatocellular carcinoma: assessment of the CLIP, Okuda, and Child-
Pugh staging systems in a cohort of 257 patients in Toronto,” Gut, vol. 50, no. 6, pp. 881–885,
2002.
[6] A. H. Mir, M. Hanmandlu, and S. N. Tandon, “Texture Analysis of CT Images for Early Detection
of Liver Malignancy,” Biomed Sci Instrum, vol. 31, pp. 213–217, 1996.
[7] B. Foster, U. Bagci, A. Mansoor, Z. Xu, and D. J. Mollura, “A review on segmentation of positron
emission tomography images,” Comput. Biol. Med., vol. 50, pp. 76–96, 2014.
[8] M. Selzner, T. F. Hany, P. Wildbrett, L. McCormack, Z. Kadry, and P.-A. Clavien, “Does the novel
PET/CT imaging modality impact on the treatment of patients with metastatic colorectal cancer of
the liver?,” Ann. Surg., vol. 240, no. 6, pp. 1027-34–6, 2004.
[9] T. Heimann, B. van Ginneken, M. A. Styner, Y. Arzhaeva, V. Aurich, C. Bauer, A. Beck, C. Becker,
R. Beichel, G. Bekes, F. Bello, G. Binnig, H. Bischof, A. Bornik, P. Cashman, Ying Chi, A.
Cordova, B. M. Dawant, M. Fidrich, J. D. Furst, D. Furukawa, L. Grenacher, J. Hornegger, D.
Kainmuller, R. I. Kitney, H. Kobatake, H. Lamecker, T. Lange, Jeongjin Lee, B. Lennon, Rui Li,
Senhu Li, H.-P. Meinzer, G. Nemeth, D. S. Raicu, A.-M. Rau, E. M. van Rikxoort, M. Rousson, L.
Rusko, K. A. Saddi, G. Schmidt, D. Seghers, A. Shimizu, P. Slagmolen, E. Sorantin, G. Soza, R.
Susomboon, J. M. Waite, A. Wimmer, and I. Wolf, “Comparison and Evaluation of Methods for
Liver Segmentation From CT Datasets,” IEEE Trans. Med. Imaging, vol. 28, no. 8, pp. 1251–1265,
2009.
[10] C. S. Reiner, P. Stolzmann, L. Husmann, I. A. Burger, M. W. H?llner, N. G. Schaefer, P. M.
Schneider, G. K. von Schulthess, and P. Veit-Haibach, “Protocol requirements and diagnostic value
of PET/MR imaging for liver metastasis detection,” Eur. J. Nucl. Med. Mol. Imaging, vol. 41, no.
4, pp. 649–658, 2014.
[11] S. Urbán, L. Ruskó, and A. Nagy, “A Self-learning Tumor Segmentation Method on DCE-MRI
Images,” in Lecture Notes in Computer Science. Image Analysis and Recognition. ICIAR 2016, A.
Campilho and F. Karray, Eds. vol 9730, Springer, Cham, 2016, pp. 591–598.
[12] Y. Chi, Y. Chi, P. M. M. Cashman, O. Bello, and R. I. Kitney, “A Discussion on the Evaluation of
A New Automatic Liver Volume Segmentation Method for Specified CT Image Datasets,” in
MICCAI - Workshop on 3D Segmentation in the Clinic: a Gand Challenge, 2007, pp. 167–175.
[13] G. Schmidt, G. Schmidt, M. Athelogou, R. Schönmeyer, R. Korn, and G. Binnig, “Cognition
Network Technology for a Fully Automated 3D Segmentation of Liver,” in MICCAI - Workshop
on 3D Segmentation in the Clinic: a Gand Challenge, 2007, pp. 125–133.
[14] L. Ruskó, G. Bekes, G. Németh, and M. Fidrich, “Fully automatic liver segmentation for contrast-
enhanced CT images,” in MICCAI - Workshop on 3D Segmentation in the Clinic: a Gand
Challenge, 2007, pp. 143–150.
[15] K. A. Saddi, K. A. Saddi, M. Rousson, and F. Cheriet, “Global-to-Local Shape Matching for Liver
Segmentation in CT Imaging,” in MICCAI - Workshop on 3D Segmentation in the Clinic: a Gand
Challenge, 2007, pp. 207–214.
[16] R. Susomboon, D. S. Raicu, and J. Furst, “A Hybrid Approach for Liver Segmentation,” in MICCAI
- Workshop on 3D Segmentation in the Clinic: a Gand Challenge, 2007, pp. 151–160.
[17] I. W. Tobias Heimann, Hans-Peter Meinzer, “A Statistical Deformable Model for the Segmentation
of Liver CT Volumes - Semantic Scholar,” in MICCAI - Workshop on 3D Segmentation in the
32
Clinic: a Gand Challenge, 2007, pp. 161–166.
[18] D. Kainmüller, T. Lange, and H. Lamecker, “Shape Constrained Automatic Segmentation of the
Liver based on a Heuristic Intensity Model,” in Proc. MICCAI Workshop 3D Segmentation in the
Clinic: A Grand Challenge, 2007, pp. 109–116.
[19] D. Seghers, P. Slagmolen, Y. Lambelin, J. Hermans, D. Loeckx, F. Maes, and P. Suetens,
“Landmark based liver segmentation using local shape and local intensity models,” in MICCAI -
Workshop on 3D Segmentation in the Clinic: a Gand Challenge, 2007, pp. 135–142.
[20] J. Lee, N. Kim, H. Lee, J. B. Seo, H. J. Won, Y. M. Shin, and Y. G. Shin, “Efficient liver
segmentation exploiting level-set speed images with 2.5D shape propagation,” in MICCAI -
Workshop on 3D Segmentation in the Clinic: a Gand Challenge, 2007, pp. 189–196.
[21] D. Furukawa, A. Shimizu, and H. Kobatake, “Automatic Liver Segmentation Method based on
Maximum A Posterior Probability Estimation and Level Set Method,” in MICCAI - Workshop on
3D Segmentation in the Clinic: a Gand Challenge, 2007, pp. 117–124.
[22] E. van Rikxoort, Y. Arzhaeva, and B. van Ginneken, “Automatic segmentation of the liver in
computed tomography scans with voxel classification and atlas matching,” in MICCAI - Workshop
on 3D Segmentation in the Clinic: a Gand Challenge, 2007, pp. 101–108.
[23] R. Beichel, R. Beichel, C. Bauer, E. Bornik, E. Sorantin, and H. Bischof, “Liver Segmentation in
CT Data: A Segmentation Refinement Approach,” in MICCAI - Workshop on 3D Segmentation in
the Clinic: a Gand Challenge, 2007, pp. 235–245.
[24] A. Beck and V. Aurich, “HepaTux – A Semiautomatic Liver Segmentation System - Semantic
Scholar,” in MICCAI - Workshop on 3D Segmentation in the Clinic: a Gand Challenge, 2007, pp.
225–234.
[25] B. M. Dawant, L. Brian, and L. Senhu, “Semi-automatic segmentation of the liver and its evaluation
on the MICCAI 2007 grand challenge data set,” in MICCAI - Workshop on 3D Segmentation in the
Clinic: a Gand Challenge, 2007, pp. 215–221.
[26] A. Wimmer, A. Wimmer, G. Soza, and J. Hornegger, “Two-stage Semi-automatic Organ
Segmentation Framework using Radial Basis Functions and Level Sets,” in MICCAI - Workshop
on 3D Segmentation in the Clinic: a Gand Challenge, 2007, pp. 179–188.
[27] P. Slagmolen, A. Elen, D. Seghers, D. Loeckx, F. Maes, and K. Haustermans, “Atlas based liver
segmentation using nonrigid registration with a B-spline transformation model,” in MICCAI -
Workshop on 3D Segmentation in the Clinic: a Gand Challenge, 2007, pp. 197–206.
[28] T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham, “Active Shape Models-Their Training and
Application,” Comput. Vis. Image Underst., vol. 61, no. 1, pp. 38–59, 1995.
[29] X. Zhang, J. Tian, K. Deng, Y. Wu, and X. Li, “Automatic Liver Segmentation Using a Statistical
Shape Model With Optimal Surface Detection,” IEEE Trans. Biomed. Eng., vol. 57, no. 10, pp.
2622–2626, 2010.
[30] A. Wimmer, G. Soza, and J. Hornegger, “A generic probabilistic active shape model for organ
segmentation.,” MICCAI 2009. Lect. Notes Comput. Sci., vol. 12, no. Pt 2, pp. 26–33, 2009.
[31] X. Chen, J. K. Udupa, U. Bagci, Y. Ying Zhuge, and J. Jianhua Yao, “Medical Image Segmentation
by Combining Graph Cuts and Oriented Active Appearance Models,” IEEE Trans. Image Process.,
vol. 21, no. 4, pp. 2035–2046, 2012.
[32] L. Massoptier and S. Casciaro, “A new fully automatic and robust algorithm for fast segmentation
of liver tissue and tumors from CT scans,” Eur. Radiol., vol. 18, no. 8, pp. 1658–1665, 2008.
[33] M. Moghbel, S. Mashohor, R. Mahmud, and M. I. Bin Saripan, “Automatic liver tumor
segmentation on computed tomography for patient treatment planning and monitoring,” EXCLI,
no. 15, pp. 406–423, 2016.
[34] C. Platero and M. C. Tobar, “A multiatlas segmentation using graph cuts with applications to liver
segmentation in CT scans.,” Comput. Math. Methods Med., vol. 2014, p. 182909, 2014.
[35] P. Hu, F. Wu, J. Peng, P. Liang, and D. Kong, “Automatic 3D liver segmentation based on deep
learning and globally optimized surface evolution,” Phys. Med. Biol., vol. 61, no. 24, pp. 8676–
8698, 2016.
[36] F. Lu, F. Wu, P. Hu, Z. Peng, and D. Kong, “Automatic 3D liver location and segmentation via
convolutional neural network and graph cut,” Int. J. Comput. Assist. Radiol. Surg., vol. 12, no. 2,
pp. 171–182, 2016.
[37] Q. Dou, H. Chen, Y. Jin, L. Yu, J. Qin, and P.-A. Heng, “3D Deeply Supervised Network for
Automatic Liver Segmentation from CT Volumes,” MICCAI 2016. Lect. Notes Comput. Sci., vol.
9901, 2016.
[38] L. Ruskó, G. Bekes, and M. Fidrích, “Automatic segmentation of the liver from multi- and single-
phase contrast-enhanced CT images,” Med. Image Anal., vol. 13, no. 6, pp. 871–882, 2009.
[39] S. Tomoshige, E. Oost, A. Shimizu, H. Watanabe, and S. Nawano, “A conditional statistical shape
33
model with integrated error estimation of the conditions; Application to liver segmentation in non-
contrast CT images,” Med. Image Anal., vol. 18, no. 1, pp. 130–143, 2014.
[40] L. Grady, “Random walks for image segmentation.,” IEEE Trans. Pattern Anal. Mach. Intell., vol.
28, no. 11, pp. 1768–83, 2006.
[41] T. Okada, R. Shimada, Y. Sato, M. Hori, K. Yokota, M. Nakamoto, Y.-W. Chen, H. Nakamura,
and S. Tamura, Automated segmentation of the liver from 3D CT images using probabilistic atlas
and multi-level statistical shape model, Lecture No., vol. 4791 LNCS, no. PART 1. Springer,
Berlin, Heidelberg, 2007.
[42] Y. Xu, C. Xu, X. Kuang, H. Wang, E. I.-C. Chang, W. Huang, and Y. Fan, “3D-SIFT-Flow for
atlas-based CT liver image segmentation,” Med. Phys., vol. 43, no. 5, pp. 2229–2241, 2016.
[43] J. Stawiaski, E. Decenciere, and B. F., “Interactive Liver Tumor Segmentation Using Graph-cuts
and Watershed,” in Grand Challenge Liver Tumor Segmentation (2008 MICCAI Workshop), 2008.
[44] D. Wong, J. Liu, F. Yin, Q. Tian, W. Xiong, J. Zhou, Q. Yingyi, T. Han, S. Venkatesh, and S.
Wang, “A semi-automated method for liver tumor segmentation based on 2D region growing with
knowledge-based constraints,” in Grand Challenge Liver Tumor Segmentation (2008 MICCAI
Workshop), 2008.
[45] P. H. Moltz J.H., Bornemann L., Dicken V., “Segmentation of Liver Metastases in CT Scans by
Adaptive Thresholding and Morphological Processing,” in Grand Challenge Liver Tumor
Segmentation (2008 MICCAI Workshop), 2008.
[46] Z. G. A. Choudhary A., Moretto N., Pizzorni Ferrarese F., “An entropy based multi-thresholding
method for semi-automatic segmentation of liver tumors,” in Grand Challenge Liver Tumor
Segmentation (2008 MICCAI Workshop), 2008.
[47] T. Kubota, “Efficient Automated Detection and Segmentation of Medium and Large Liver Tumors:
CAD Approach,” in Grand Challenge Liver Tumor Segmentation (2008 MICCAI Workshop), 2008.
[48] H. A. Nugroho, D. Ihtatho, and H. Nugroho, “Contrast Enhancement for Liver Tumor
Identification,” in Grand Challenge Liver Tumor Segmentation (2008 MICCAI Workshop), 2008.
[49] K. J. Schmidt G., Binnig G., Kietzmann M., “Cognition Network Technology for a Fully
Automated 3D Segmentation of Liver Tumors,” in Grand Challenge Liver Tumor Segmentation
(2008 MICCAI Workshop), 2008.
[50] Y. Hame, “Liver Tumor Segmentation Using Implicit Surface Evolution,” in Grand Challenge
Liver Tumor Segmentation (2008 MICCAI Workshop), 2008.
[51] A. Shimizu, T. Narihira, D. Furakawa, H. Kobatake, S. Nawano, and K. Shinozaki, “Ensemble
segmentation using AdaBoost with application to liver lesion extraction from a CT volume,” in
Grand Challenge Liver Tumor Segmentation (2008 MICCAI Workshop), 2008.
[52] J. Zhou, W. Xiong, Q. Tian, Y. Qi, J. Liu, W. K. Leow, T. Han, S. Venkatesh, and S. Wang, “Semi-
automatic Segmentation of 3D Liver Tumors from CT Scans Using Voxel Classification and
Propagational Learning,” in Grand Challenge Liver Tumor Segmentation (2008 MICCAI
Workshop), 2008.
[53] Xing Zhang, Jie Tian, Dehui Xiang, Xiuli Li, and Kexin Deng, “Interactive liver tumor
segmentation from ct scans using support vector classification with watershed,” in Annual
International Conference of the IEEE Engineering in Medicine and Biology Society, 2011, pp.
6005–6008.
[54] D. Smeets, D. Stijnen, B. Loeckx, D. De Dobbelaer, and P. Suetens, “Segmentation of liver
metastases using a level set method with spiral-scanning technique and supervised fuzzy pixel
classification,” in Grand Challenge Liver Tumor Segmentation (2008 MICCAI Workshop), 2008.
[55] I. Ben-Dan and E. Shenhav, “Liver Tumor segmentation in CT images using probabilistic
methods,” in Grand Challenge Liver Tumor Segmentation (2008 MICCAI Workshop), 2008.
[56] M. A. Selver, A. Kocaoglu, G. K. Demir, H. Dogan, O. Dicle, and C. Guzelis, “Patient oriented and
robust automatic liver segmentation for pre-evaluation of liver transplantation,” Comput. Biol.
Med., vol. 38, no. 7, 2008.
[57] M. Goryawala, S. Gulec, R. Bhatt, A. J. McGoron, and M. Adjouadi, “A low-interaction automatic
3D liver segmentation method using computed tomography for selective internal radiation
therapy.,” Biomed Res. Int., vol. 2014, p. 198015, 2014.
[58] W. Wu, Z. Zhou, S. Wu, and Y. Zhang, “Automatic Liver Segmentation on Volumetric CT Images
Using Supervoxel-Based Graph Cuts,” Comput. Math. Methods Med., vol. 2016, pp. 1–14, 2016.
[59] Suhuai Luo, Qingmao Hu, Xiangjian He, Jiaming Li, J. S. Jin, and M. Park, “Automatic liver
parenchyma segmentation from abdominal CT images using support vector machines,” in 2009
ICME International Conference on Complex Medical Engineering, 2009, pp. 1–5.
[60] B. He, C. Huang, G. Sharp, S. Zhou, Q. Hu, C. Fang, Y. Fan, and F. Jia, “Fast automatic 3D liver
segmentation based on a three-level AdaBoost-guided active shape model,” Med. Phys., vol. 43,
34
no. 5, pp. 2421–2434, 2016.
[61] Y. Zheng, D. Ai, P. Zhang, Y. Gao, L. Xia, S. Du, X. Sang, and J. Yang, “Feature Learning Based
Random Walk for Liver Segmentation,” PLoS One, vol. 11, no. 11, p. e0164098, Nov. 2016.
[62] R. S. Alomari, S. Kompalli, and V. Chaudhary, “Segmentation of the liver from abdominal CT
using markov random field model and GVF snakes,” in Proceedings - CISIS 2008: 2nd
International Conference on Complex, Intelligent and Software Intensive Systems, 2008.
[63] B. Vijayalakshmi and V. Subbiah, “Classification of CT liver images using local binary pattern
with Legendre moments,” Curr. Sci., vol. 110, no. 4, pp. 687–691, 2016.
[64] P. Ferdinand Christ, M. A. Ezzeldin Elshaer, F. Ettlinger, S. Tatavarty, M. Bickel, P. Bilic, M.
Rempfler, M. Armbruster, F. Hofmann, W. H. Sommer, S.-A. Ahmadi, and B. H. Menze,
“Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural
Networks and 3D Conditional Random Fields,” Int. Conf. Med. Image Comput. Comput. Interv.,
vol. Springer I, pp. 415–423, Oct. 2016.
[65] G. Chlebus, H. Meine, J. H. Moltz, and A. Schenk, “Neural Network-Based Automatic Liver Tumor
Segmentation With Random Forest-Based Candidate Filtering,” arxiv reprint: arXiv:1706.00842,
2017. .
[66] L. Bi, J. Kim, A. Kumar, and D. Feng, “Automatic Liver Lesion Detection using Cascaded Deep
Residual Networks,” arxiv Repr. arXiv1704.02703, 2017.
[67] A. Hoogi, C. F. Beaulieu, G. M. Cunha, E. Heba, C. B. Sirlin, S. Napel, and D. L. Rubin, “Adaptive
local window for level set segmentation of CT and MRI liver lesions,” Med. Image Anal., vol. 37,
2017.
[68] X. Han, “Automatic Liver Lesion Segmentation Using A Deep Convolutional Neural Network
Method,” arXiv:1704.07239, 2017.
[69] A. Ben-Cohen, I. Diamant, E. Klang, M. Amitai, and H. Greenspan, “Fully Convolutional Network
for Liver Segmentation and Lesions Detection,” in Deep Learning and Data Labeling for Medical
Applications. LABELS, DLMIA 2016., Lecture No., C. Springer, Ed. 2016.
[70] T. Magnander, E. Wikberg, J. Svensson, P. Gjertsson, B. Wängberg, M. Båth, and P. Bernhardt,
“A novel statistical analysis method to improve the detection of hepatic foci of 111In-octreotide in
SPECT/CT imaging,” EJNMMI Phys., vol. 3, no. 1, 2016.
[71] C.-Y. Hsu, C.-Y. Liu, and C.-M. Chen, “Automatic segmentation of liver PET images,” Comput.
Med. Imaging Graph., vol. 32, no. 7, 2008.
[72] S. Ray, R. Hagge, M. Gillen, M. Cerejo, S. Shakeri, L. Beckett, T. Greasby, and R. D. Badawi,
“Comparison of two-dimensional and three-dimensional iterative watershed segmentation methods
in hepatic tumor volumetrics,” Med. Phys., vol. 35, no. 12, pp. 5869–5881, 2008.
[73] A. S. Maklad, M. Matsuhiro, H. Suzuki, Y. Kawata, N. Niki, T. Utsunomiya, and M. Shimada,
“Extraction of liver volumetry based on blood vessel from the portal phase CT dataset,” in
Proceedings of SPIE, 2012, p. 83142O.
[74] D. A. Oliveira, R. Q. Feitosa, and M. M. Correia, “Segmentation of liver, its vessels and lesions
from CT images for surgical planning,” Biomed. Eng. Online, vol. 10, no. 1, p. 30, 2011.
[75] V. Selvalakshmi and D. Nirmala, “A novel region based segmentation of hepatic tumors and hepatic
vein in low contrast CTA images using Bernstein polynomials,” Biomed. Res., 2017.
[76] G. Lathen, J. Jonasson, and M. Borga, “Blood vessel segmentation using multi-scale quadrature
filtering,” Pattern Recognit. Lett., vol. 31, no. 8, pp. 762–767, 2010.
[77] Y. Z. Zeng, Y. Q. Zhao, M. Liao, B. J. Zou, X. F. Wang, and W. Wang, “Liver vessel segmentation
based on extreme learning machine,” Phys. Medica, vol. 32, no. 5, pp. 709–716, 2016.
[78] C. Shi, Y. Cheng, F. Liu, Y. Wang, J. Bai, and S. Tamura, “A hierarchical local region-based sparse
shape composition for liver segmentation in CT scans,” Patern Recognit., vol. 50, pp. 88–106,
2015.
[79] T. Kitrungrotsakul, X.-H. Han, Y. Iwamoto, A. H. Foruzan, L. Lin, and Y.-W. Chen, “Robust
hepatic vessel segmentation using multi deep convolution network,” SPIE Med. Imaging, p.
1013711, 2017.
[80] K. Kamnitsas, C. Ledig, V. F. J. Newcombe, J. P. Simpson, A. D. Kane, D. K. Menon, D. Rueckert,
and B. Glocker, “Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain
Lesion Segmentation,” Mar. 2016.
[81] K. H. Cha, L. Hadjiiski, R. K. Samala, H.-P. Chan, E. M. Caoili, and R. H. Cohan, “Urinary bladder
segmentation in CT urography using deep-learning convolutional neural network and level sets,”
Med. Phys., vol. 43, no. 4, pp. 1882–1896, Mar. 2016.
[82] J. Sedlar, M. Bajger, M. Caon, and G. Lee, “Model-Guided Segmentation of Liver in CT and PET-
CT Images of Child Patients Based on Statistical Region Merging,” in 2016 International
Conference on Digital Image Computing: Techniques and Applications (DICTA), 2016, pp. 1–8.
35
[83] V. G. Sekhar, S. N. Kumar, L. Fred, and S. Varghese, “An improved color segmentation algorithm
for the analysis of liver anomalies in CT/PET images,” in 2016 IEEE International Conference on
Engineering and Technology (ICETECH), 2016, pp. 1151–1154.
[84] W. Zhanga, R. Lia, H. Dengb, L. Wangc, W. Lind, S. Jia, and D. Shen, “Deep Convolutional Neural
Networks for Multi-Modality Isointense Infant Brain Image Segmentation,” Neuroimage, vol. 108,
pp. 214–224.
[85] H.-I. Suk, S.-W. Lee, D. Shen, and Alzheimer’s Disease Neuroimaging Initiative, “Hierarchical
feature representation and multimodal fusion with deep learning for AD/MCI diagnosis,”
Neuroimage, vol. 101, pp. 569–582, 2014.
[86] A. Teramoto, H. Fujita, O. Yamamuro, and T. Tamaki, “Automated detection of pulmonary nodules
in PET/CT images: Ensemble false-positive reduction using a convolutional neural network
technique,” Med. Phys., vol. 43, no. 6, pp. 2821–2827, 2016.
[87] M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu, “Spatial Transformer Networks,”
05-Jun-2015. [Online]. Available: http://arxiv.org/abs/1506.02025. [Accessed: 15-Jul-2017].
[88] MICCAI, “Sliver07 dataset,” 2007. .
[89] MICCAI, “Grand Challenge Liver Tumor Segmentation (2008 MICCAI Workshop),” 2008.
[Online]. Available: http://www.midasjournal.org/browse/journal/45.
[90] C. imaging Archive, “Cancer Genome Atlas Liver Hepatocellular Carcinoma,” 2017. [Online].
Available: https://wiki.cancerimagingarchive.net/display/Public/TCGA-LIHC.
[91] “MIDAS dataset,” 2016. [Online]. Available: www.insight-journal.org/midas/collection/view/38.
[92] T. Moeller and E. Reif, Pocket Atlas of Sectional Anatomy, Vol. II: Thorax, Heart, Abdomen and
Pelvis. Thieme.
[93] Claude Couinaud, Le foie: études anatomiques et chirurgicales. Paris: Masson & Cie, 1957.
[94] M. Moghbel, S. Mashohor, R. Mahmud, and M. I. Bin Saripan, “Review of liver segmentation and
computer assisted detection/diagnosis methods in computed tomography,” Artif. Intell. Rev., pp. 1–
41, 2017.
[95] X. Cheng, L. Zhang, and Y. Zheng, “Deep similarity learning for multimodal medical images,”
Comput. Methods Biomech. Biomed. Eng. Imaging Vis., pp. 1–5, Apr. 2016.