Invited Paper:Semantic knowledge for histopathological image analysis:from ontologies to processing portals and deep learning

Yannick L. Kergosiena,b and Daniel Racoceanuc,d

aSorbonne Universites, UPMC Univ Paris 06, INSERM, Univ Paris 13, LIMICS, ParisbUniversite de Cergy-Pontoise, Cergy-Pontoise, France

cPontifical Catholic University of Peru, Av. Universitaria 1801, San Miguel Lima 32, PerudSorbonne Universites, UPMC Univ Paris 06, CNRS, INSERM, (LIB), 75005, Paris, France

ABSTRACT

This article presents our vision about the next generation of challenges in computational/digital pathology.The key role of the domain ontology, developed in a sustainable manner (i.e. using reference checklists andprotocols, as the living semantic repositories), opens the way to effective/sustainable traceability and relevancefeedback concerning the use of existing machine learning algorithms, proven to be very performant in the latestdigital pathology challenges (i.e. convolutional neural networks). Being able to work in an accessible web-serviceenvironment, with strictly controlled issues regarding intellectual property (image and data processing/analysisalgorithms) and medical data/image confidentiality is essential for the future. Among the web-services involved inthe proposed approach, the living yellow pages in the area of computational pathology seems to be very importantin order to reach an operational awareness, validation, and feasibility. This represents a very promising way togo to the next generation of tools, able to bring more guidance to the computer scientists and confidence to thepathologists, towards an effective/efficient daily use. Besides, a consistent feedback and insights will be morelikely to emerge in the near future – from these sophisticated machine learning tools – back to the pathologists–, strengthening, therefore, the interaction between the different actors of a sustainable biomedical ecosystem(patients, clinicians, biologists, engineers, scientists etc.).

Beside going digital/computational – with virtual slide technology demanding new workflows –, Pathologymust prepare for another coming revolution: semantic web technologies now enable the knowledge of experts tobe stored in databases, shared through the Internet, and accessible by machines. Traceability, disambiguationof reports, quality monitoring, interoperability between health centers are some of the associated benefits thatpathologists were seeking. However, major changes are also to be expected for the relation of human diagnosisto machine based procedures. Improving on a former imaging platform which used a local knowledge base anda reasoning engine to combine image processing modules into higher level tasks, we propose a framework wheredifferent actors of the histopathology imaging world can cooperate using web services – exchanging knowledgeas well as imaging services – and where the results of such collaborations on diagnostic related tasks can beevaluated in international challenges such as those recently organized for mitosis detection, nuclear atypia, ortissue architecture in the context of cancer grading. This framework is likely to offer an effective context-guidanceand traceability to Deep Learning approaches, with an interesting promising perspective given by the multi-tasklearning (MTL) paradigm, distinguished by its applicability to several different learning algorithms, its non-reliance on specialized architectures and the promising results demonstrated, in particular towards the problemof weak supervision –, an issue found when direct links from pathology terms in reports to corresponding regionswithin images are missing.

Keywords: Semantics, Ontology, Web-Services, Deep Learning, Multitask learning, Computational/DigitalPathology, Challenges

Further author information: (Send correspondence to D.R. or Y.L.K.)D.R.: E-mail: daniraco@gmail.com ; Y.L.K.: E-mail: y.l.kergosien@gmail.com

1. TRACEABILITY AND MEDICAL FEEDBACK IN COMPUTATIONALPATHOLOGY

In the last decade, the virtual slide technology generated a real revolution in the use of image processing fordiagnostic pathology. However, automation in Medicine has to be carefully validated by tests and clinical trialsand further requires certification and official agreements not unlike requirements for drugs to go from developmentto market. Only partial task automation can find numerous enough homogeneous case samples to be practicallyvalidated, and even then, multicenter studies are often necessary, raising major interoperability issues. Westarted identifying such candidate tasks from the terms used in the standards for pathology reporting, restrictingto cancer grading, and addressed the problems of semantic interoperability by using semantic web tools. In,1

we used web services to map pathology terms used in cancer grading to UMLS semantic types and inferred thepossibility of associating a quantitative entity to these terms, the computation of such quantitative traits fromimages being taken as an a priori candidate for image analysis algorithms. To refine that scheme, a more detailedknowledge of available image analysis algorithms and their performances was suitable. We thus addressed in2

the semantics of the imaging tasks recently proposed as challenges at histopathologic imaging internationalconferences, where the question to be answered and the image database are well defined. Besides, the methodsused by the winners are summarized in papers accessible to semantic annotation and further semantic modelling.The most relevant imaging and learning tools constitute, therefore, a bridge from the anatomopathologicalsemantics (mainly related to quantification and grading tasks based on slide image analysis) to the imaging andlearning semantics - specific to these specialties.

Going beyond these initial efforts, we now concentrate on two prospects for development towards the effectiveuse of available semantic knowledge in assisting pathology. First, we address the problem of combining availableelementary imaging tasks into workflows applicable to higher level questions of every day pathology. Thatproblem has been addressed by the MICO∗ project where some logic inference engine and a knowledge base wereused to combine elementary imaging tasks. What we now see as desirable improvements on MICO’s approachis the use of web collaboration and standards for the building and maintenance of the knowledge base - whereMICO used local collaboration between pathologists and imaging scientists - together with newer reasoningtechnologies such as the engines provided for description logics and designed to fit the format in which semanticknowledge is available on the web. Second, as we observed the success of neural networks solutions (especiallythe convolutional/deep learning sort) at the histopathologic challenges, and since these approaches need not berestricted to lower level tasks, we study how a neural net addressing higher level questions could profit fromthe available semantic knowledge. As opposed to the explicit reasoning approach – where available knowledgecan pre-exist as a set of inference rules or can be converted to something usable by the reasoning engine –,neural nets look more like ”black boxes” between their input and output layers, with knowledge fed to themthrough examples and supervision and neural net cooperation more problematic. In addition, creating a semanticguidance for these methods, enables generating and consolidating some feedback insights for the Pathologists,essential in reaching a clinical understanding and validation of the results, as and effective adoption of theassociated tools in future protocols.

This article presents our vision about the next generation of challenges in computational/digital pathology.The key role of the domain ontology, developed in a sustainable manner (i.e. using reference checklists andprotocols, as the living semantic repositories), opens the way to effective/sustainable traceability and relevancefeedback concerning the use of existing machine learning algorithms, proven to be very performant in the latestdigital pathology challenges (i.e. convolutional neural networks). Being able to work in an accessible web-serviceenvironment, with strictly controlled issues regarding intellectual property (image and data processing/analysisalgorithms) and medical data/image confidentiality is essential for the future. Among the web-services involved inthe proposed approach, the living yellow pages in the area of computational pathology seems to be very importantin order to reach an operational awareness, validation, and feasibility. This represents a very promising way togo to the next generation of tools, able to bring more guidance to the computer scientists and confidence to thepathologists, towards an effective/efficient daily use. Besides, a consistent feedback and insights will be morelikely to emerge in the near future – from these sophisticated machine learning tools – back to the pathologists

∗MICO project (COgnitive MIcroscopy) - ANR TecSan: http://daniraco.free.fr/projects.htm

–, strengthening, therefore, the interaction between the different actors of a sustainable biomedical ecosystem(patients, clinicians, biologists, engineers, scientists etc.).

2. HARNESSING THE SEMANTIC WEB TECHNOLOGIES TO COMBINEIMAGING AND MACHINE LEARNING ALGORITHMS

Knowledge modeling is as old as artificial intelligence, if not computer science, and benefited from the developmentof mathematical logic which took place in the first half of the twentieth century. First using LISP, then Prolog,many projects - among which medical applications such as the celebrated MYCIN3 but also histopathologymotivated studies4- translated human knowledge into so-called expert systems mostly consisting of a base offacts and rules (sentences from first order logic with some restrictions) with an inference engine operating onit to answer queries from a user. Most problematic in these approaches, however, were a non satisfactoryhandling of negation, and the possibility of non terminating computations. With the advent of the Internet,some standards for the exchange of rules where adopted and are still in use, but the main stream of what isnow known as the Semantic Web came from the development of specialized logics, called Description Logics(DL).5 Description Logics are tailored to provide different compromises between the inescapably antagonisticproperties of expressivity (the amount of reality that can be modelled) and complexity (the time and memoryresources needed to answer queries). Still within first order logic (FOL) and forbidding some constructions of it(in fact providing a limited set of permitted syntactic constructions), they thus explicitly restrict expressivity toguarantee at least decidability and further some tractability with guaranteed bounds. The knowledge for DLs isstored as unary and binary predicates (or relations) rather than rules, and Deduction in DL often uses tableauxmethods (as opposed to resolution and unification in Prolog).

With collaborative semantic modeling now possible, each contributor can focus on the formalization of alimited part of reality that has not been addressed before and reuse existing models as much as possible. Such apartial semantic model is called an ontology, which can be defined as ”a formal, explicit specification of a sharedconceptualization”.6–8 Ontologies are usually shared on the web through portals such as Bioportal (. . . ). TheW3C promotes OWL (the World wide web Ontology Language) in several flavors such as OWL Light, OWL DLand OWL full (not a DL) from less to more expressive. Knowledge is formalized in relations written, e.g., in theRDF standard (the Resource Description Framework, a layer upon XML), as sets of triples (subject, predicate,object) in so-called triple stores, but only if it uses triples permitted by OWL is it guaranteed to be processedefficiently by reasoners. Ontology visual editors such as Protege (. . . ) enable medical experts with moderatelogical training to build an ontology of a limited domain, to test its logical consistency using the embeddedreasoner, and to make it publicly available to humans and machines in OWL format on a web portal.

Pathology diagnosis is a major element for the management of severe and chronic disease. Despite theintegration of digital modalities in teaching and research, their daily clinical use is still to be done. Futureanatomopathological services need to use these digital technologies in valid routine pathological diagnosis andhealthcare protocols, by integrating the Whole Slide Images (WSI) observation for diagnosis purposes in a wholelarge specific Digital Pathology (DP) case record. This is able to generate an operational DP process in whichthe innovation consists in linking the microscopic examination of WSI to specific or generic annotations definedas micro-semiology semantic references. The generation of a structured and standardized image-related reportis, therefore, possible, by providing an excellent preliminary traceable document, to be confirmed and refined bythe pathologist, allowing her to save precious time and focus on possible critical issues / cases.

Through Digital Pathology, the future of pathology is on the way to reinforce its ethical and dynamicalstrengths. With the emergence of omics and integrative approaches, a traceable, semantically indexed secondopinion will thus become essential for patients and healthcare professionals in personalized medicine.

Continuing the path of our previous semantic cognitive virtual microscopy initiatives9–11∗†, we proposed asustainable way to bridge the content, features, performance and usability gaps12,13 between anatomo-pathologyand WSI analysis. In this sense, a preliminarily work has been recently published by our team1 proposing theuse of the College of American Pathologists (CAP) organ-specific Cancer Checklists and associated Protocols(CC&P)‡ to develop a sustainable approach to locating available relevant knowledge using existing semantic

†FlexMIm project (Collaborative Pathology): http://www.systematic-paris-region.org/en/projets/flexmim‡Cancer Checklists and associated Protocols (CC&P): http://www.cap.org/web/oracle/webcenter/portalapp/

repositories (NCBO Bioportal, UMLS§ semantic types). We continued this work by building an imaging ontol-ogy consolidated from Matlab, ITK and Image J functionalities. This imaging semantics was linked with thepathology ontology by analyzing the results issued from the recent challenges organized in this area.

Indeed, standing as important milestone on the way to routine Digital Pathology, a series of pioneer interna-tional benchmarking initiatives have been launched for mitosis detection at MITOS 2012¶ (continued by AMIDA2013, MITOS 2014‖ and TUPAC 2016∗∗), nuclear atypia grading at ATYPIA 2014 and glandular structures de-tection GlaS 2015†† some of the fundamental grading components in diagnosis / prognosis. These initiativesallow envisaging a consolidated validation referential-database for Digital Pathology in the near future.

In a recent publication2 we have been introducing this approach in a form of a tryptic between CC&P-generated semantics, challenge-based effective medical imaging tools selection and the imaging semantics struc-tured from the most used imaging software, as specified above.

Going beyond these studies, the MICO framework suggests elaborating a web-service based framework, inwhich all major pattern recognition tools (including most of the effective machine and deep learning structures)will be triggered using a very precise and structured semantic frame. Service composition on the flight is thenpossible, by composing different component services, according to the priorities of the pathologist (quality, fastpreliminary evaluation, precise annotation etc.).

This could constitute a new generation of challenges (see Fig. 1) in which each machine-learning, each medicalimaging tool provider, as well as each database holder will keep its intellectual property and confidentiality, byopening - through web-service interfaces and consensual standardized APIs (Application Programming Inter-faces), the use of the tools and the semantic information extracted for two major use cases:

1. Daily work (second opinion, annotation and quantification assistance, pre-filled report generation);

2. Clinical trials and medical research (knowledge consolidation, state of the art, query-based research, simi-larities and causalities exploration, meta-reporting).

3. HISTOPATHOLOGY KNOWLEDGE INPUT FOR DEEP NEURAL NETLEARNING GUIDANCE

Among the winners of recent histology image processing challenges, neural networks based methods appearedmore and more frequently, mainly through the paradigm of deep learning. Neural network approaches need notbe restricted to lower level tasks, and it is natural to ask how a neural net addressing higher level questions couldprofit from the available semantic knowledge. In contrast to the explicit reasoning approach where availableknowledge can pre-exist as a set of inference rules or can be converted to something usable by the reasoningengine, neural nets look more like black boxes between their input and output layers, with knowledge fed tothem through examples and supervision. In this context, any feedback to the clinicians, any traceability and anycooperation within human operators and deep neural net structures seems more problematic and will probablybecome a key issue in any clinical applications. Going beyond trying to describe special architectures, e.g., todirectly interact with hidden layers, and only using inputs and outputs of standard - possibly convolutional -neural networks, we propose two ways of taking advantage of available formal histopathologic knowledge forlearning.

§NCBO bioportal: http://bioportal.bioontology.org/; UMLS: https://www.nlm.nih.gov/research/umls/¶MITOS - Int. Conf. Pattern Recognition (ICPR), Tsukuba, Japan, 2012: http://mitos_2012/‖MITOS & ATYPIA 2014 - ICPR 2014: http://mitos-atypia-14.grand-challenge.org/

∗∗TUPAC 2016: http://tupac.tue-image.nl/; AMIDA 2013: http://amida13.isi.uu.nl/††GlaS 2015 - MICCAI 2015: http://www2.warwick.ac.uk/fac/sci/dcs/research/combi/research/bic/glascontest/

Figure 1. A new generation of web portal and services, based on sustainable ontology, indexing (yellow pages) of the mostpertinent Machine Learning and Medical Image Analysis tools for Computational Pathology

3.1 Selection of samples and learning tasks

The questions addressed by image challenges proposed at international conferences need to be carefully formulatedby humans (the organizers) using their expertise (a non-formal knowledge) to build a proposed couple (thequestion to be answered and the provided database annotated with correct answers) for which current algorithmsare likely to reach interesting performance. With the availability of massive medical databases which includeimages (from PACS‡‡) and clinical data (from HIS§§) and also the availability of high throughput processingpower, background search for such algorithms using neural networks or other learning algorithms - a form ofmedical data mining - could be attempted for such couples if they were available. Identifying and buildingsuch couples is a costly operation for humans and a probable bottleneck for such projects. However, the resultscited in the previous sections show that pathology reports can be annotated by semantic servers, thus enablingthe selection of cases which associate the presence in the image of known components of an entity. In thatway, the database of cases to be provided for learning can be built automatically from a base of reports anda semantic annotator. Ideally, the supervision provided for learning would include precise image annotationscertified by pathology experts supporting the assertions in their reports, such as the micro-semiological featuresalready mentioned. However, such enriched records are very scarce and - in general - the available cases do notinclude pointers to the corresponding digital images. For example, if the presence of some nucleus abnormalityis mentioned in the pathology report, the corresponding nuclei (or at least a representative sample of them) arenot pointed in the image (i.e., their coordinates within the image are not provided) and segmentations of thesame nuclei are even less likely to be given (we shall refer to this as “weak supervision” in the next paragraph).How then could one hope successfully learning from such case bases? As a first answer, let us notice that inthe framework of a cooperative set of processing units, and keeping the same example, the segmentations of allthe nuclei of the slides could be readily available, e.g., systematically computed in parallel, using (1) alreadyknown imaging algorithms and (2) the knowledge that such traits are semantically relevant. As a consequence,

‡‡PACS - Picture Archiving and Communication Systems§§HIS - Health Information Systems

the presence of a certain abnormality in one of the nuclei of the slide - an important but incomplete informationfrom the report - can be mapped to a tractable list of processed sub image candidates, possibly augmented withcomputed standard shape of texture descriptors, and becomes usable by a neural network for learning potentialimage features associated to a certain term. In a way, that approach is still close to the task decomposition andthe semantically guided combination of available imaging operators used by MICO. It also applies to similaroperational use cases aimed at directly helping the pathologist in its day to day tasks. We now address anotheruse case where the aim is the development of new diagnostic procedures until the stage where they can beevaluated by diagnostic clinical trials.

3.2 Multitask learning and the guidance of neural nets

A second option for feeding knowledge in neural networks and learning algorithms corresponds to the use of themechanism of “hints”14 for back-propagation networks, later known as Multi Task Learning (MTL),15 which wasapplied, among many other cases, to knowledge transfer in medical diagnosis16 and more recently adapted todeep learning.17 The principle of MTL is that in order to learn a task A from a set SA of examples and answers,it is often beneficial to simultaneously learn a second task or even several tasks. The answers to these other tasksare fed to the network as extra outputs of the last layer (see Fig. 2) during the learning phase (of course theseanswers have to be known: the extended supervision has to be available). The inner layers are thus constrainedto be useful to several tasks at the same time and performance as well as learning time have been observed to beimproved by this device. In the context of histopathologic knowledge, a natural choice for concurrent tasks whena first term has to be learned is the set of terms which occur as neighbors in a conceptual graph or as childrenin a semantic tree. As a partial but convincing answer to the question asked in the former section, the workof18 shows the effectiveness of using multi-task learning to achieve “weakly supervised learning”, i.e., learningfrom the incomplete supervision mentioned there, where images were associated to terms referring to the mereexistence of pixels or regions within them with no location given. The authors used decision forests but MTLis much more general and has been used in particular for the convolutional of neural nets which were found tobe effective by the recent histopathology challenges. In this second approach, semantic information is fed in thelearning unit even in the absence of available handles on learning subunits corresponding to terms.

Figure 2. Multi Task Learning for a generic neural network. If, instead of training only output 1 with proper supervisionto perform the main task, one adds outputs 2 to n with proper supervisions to be trained in parallel to perform tasks2 to n, performance and learning time for task 1 can be improved for clever choices of tasks 2 to n which can thus beconsidered to provide ”hints” to the system for learning task 1.

4. CONCLUSION

This article presents our vision about the next generation of challenges in computational/digital pathology.The key role of the domain ontology, developed in a sustainable manner (i.e. using reference checklists andprotocols, as the living semantic repositories), opens the way to effective traceability and relevance feedbackconcerning the use of existing machine learning algorithms, proven to be very performant by the latest digitalpathology challenges. Working in a web-service environment, in which the issues of intellectual property anddata confidentiality can be strictly controlled, is essential for the future. Among the web-services involved in theproposed approach, the living yellow pages in the area of computational pathology seems to be very importantto reach an operational awareness, validation, and feasibility. We strongly believe that this represents a verypromising way to go to the next generation of tools, able to bring more confidence to the pathologists, a keyissue towards their daily clinical and research use. The results recorded during the challenges (mostly on publicdatabase), are also able to give a reference quality guideline to the potential users, worldwide. Finally, multitasklearning offers a great potential to actual deep learning techniques towards a more adaptable and versatile useof this very important family of technologies.

REFERENCES

[1] Traore, L. and al., “A sustainable visual representation of available histopathological digital knowledge forbreast cancer grading,” Diagnostic Pathology 2(1) (2016).

[2] Traore, L., Kergosien, Y. L., and Racoceanu, D., “Bridging the semantic gap between diagnostic histopathol-ogy and image analysis.,” Stud Health Technol Inform. 235, 436–440 (2017).

[3] Buchanan, B. G. et al., [Rule-based expert systems ], vol. 3, Addison-Wesley Reading, MA (1984).[4] Bartels, P. and Hiessl, H., “Expert systems in histopathology. ii. knowledge representation and rule-based

systems.,” Analytical and quantitative cytology and histology 11(3), 147–153 (1989).[5] Baader, F., [The description logic handbook: Theory, implementation and applications ], Cambridge univ.

press (2003).[6] Gruber, T. R., “A translation approach to portable ontology specifications,” Knowledge acquisition 5(2),

199–220 (1993).[7] Guarino, N. and al., “What is an ontology?,” in [Handbook on ontologies ], 1–17, Springer (2009).[8] Studer, R., Benjamins, V. R., and Fensel, D., “Knowledge engineering: principles and methods,” Data &

knowledge engineering 25(1-2), 161–197 (1998).[9] Racoceanu, D. and Capron, F., “Towards semantic-driven high-content image analysis. an operational in-

stantiation for mitosis detection,” Computerized Medical Imaging and Graphics 2, 2–15 (2015).[10] Racoceanu, D. and al., “Towards efficient collaborative digital pathology: A pioneer initiative of the flexmim

project,” Diagnostic Pathology 1(8) (2016).[11] Racoceanu, D. and Capron, F., “Semantic integrative digital pathology: Insights into microsemiological

semantics and image analysis scalability,” Pathobiology 83(2-3), 148–55 (2016).[12] Deserno, T. M., Antani, S., and Long, R., “Ontology of gaps in content-based image retrieval,” Journal of

Digital Imaging 22(2), 202–215 (2009).[13] Tutac, A. and de Timisoara, U. P., [Formal representation and reasoning for microscopic medical image-

based prognosis ] (2010).[14] Suddarth, S. C. and Kergosien, Y. L., [Rule-injection hints as a means of improving network performance

and learning time ], 120–129, Springer Berlin Heidelberg, Berlin, Heidelberg (1990).[15] Caruana, R., [Multitask Learning ], 95–133, Springer US, Boston, MA (1998).[16] Silver, D. L., Mercer, R. E., and Hurwitz, G. A., “The functional transfer of knowledge for coronary artery

disease diagnosis,” tech. rep., Comput. Sci (1997).[17] Collobert, R. and Weston, J., “A unified architecture for natural language processing: Deep neural networks

with multitask learning,” in [Proceedings of the 25th International Conference on Machine Learning ], ICML’08, 160–167, ACM, New York, NY, USA (2008).

[18] Vezhnevets, A. and Buhmann, J. M., “Towards weakly supervised semantic segmentation by means ofmultiple instance and multitask learning.,” in [2010 IEEE Conference on Computer Vision and PatternRecognition (CVPR) ], 3249–3256, IEEE Computer Society (2010).