Sentiment analysis in healthcare

Download Sentiment analysis in healthcare

Post on 11-Aug-2014

1.121 views

Category:

Data & Analytics

81 download

DESCRIPTION

This presentation compares four tools for analysing the sentiment in the content of free-text survey responses concerning a healthcare information website. It was completed by Despo Georgiou as part of her internship at UXLabs (http://uxlabs.co.uk)

TRANSCRIPT

Sentiment Analysis in Healthcare A case study using survey responses Outline 1) Sentiment analysis & healthcare 2) Existing tools 3) Conclusions & Recommendations Focus on Healthcare 1) Difficult field biomedical text 2) Potential improvements Relevant Research: NLP procedure: FHF prediction (Roy et. al., 2013) TPA: Who is sick, Google Flu Trends (Maged et. al., 2010) BioTeKS: analyse biomedical text (Mack et. al., 2004) Sentiment Analysis Opinions Thoughts Feelings Used to extract information from raw data Sentiment Analysis Examples Surveys: analyse open-ended questions Business & Governments: assist in the decision-making process & monitor negative communication Consumer feedback: analyse reviews Health: analyse biomedical text Aims & Objectives Can existing Sentiment Analysis tools respond to the needs of any healthcare- related matter? Is it possible to accurate replicate human language using machines? The case study details 8 survey questions (open & close-ended) Analysed 137 responses based on the question: What is your feedback? Commercial tools: Semantria & TheySay Non-commercial tools: Google Predication API & WEKA Survey Overview 0 20 40 60 80 100 1 2 3 4 5 NumberofResponses Score Q.1: navigation Q.2: finding information Q.3: website's appeal Q.6: satisfaction Q.8: recommend website Semantria Collection Analysis Categories Classification Analysis Entity Recognition TheySay Document Sentiment Sentence Sentiment POS Comparison Detection Humour Detection Speculation Analysis Risk Analysis Intent Analysis Commercial Tools Results 39 51 47 Semantria Positive Neutral Negative 45 8 84 TheySay Positive Neutral Negative Introducing a Baseline 0 20 40 60 80 100 1 2 3 4 5 NumberofResponses Score Q.1 Q.2 Q.3 Q.6 Q.8 Neutral Classification Guidelines Equally positive & negative Factual statements Irrelevant statements Class Score Range Positive 1 2.7 Neutral 2.8 4.2 Negative 4.3 - 5 Introducing a Baseline Example Polarity Class CG 102 not available Hence: Negative Neutral Classification But Factual Statement Positive or negative? Final label: Neutral Q.1 Q.2 Q.3 Q.6 Q.8 Avg. 3 5 4 5 5 4.4 Introducing a Baseline 24 18 95 Manually Classified Responses Positive Neutral Negative Google Prediction API 1) Pre-process the data: punctuation & capital removal, account for negation 2) Separate into training and testing sets 3) Insert pre-labelled data 4) Train model 5) Test model 6) Cross validation: 4-fold 7) Compare with baseline Google Prediction API Results 5 122 10 Classification Results Neutral Negative Positive WEKA 1) Separate into training and testing sets 2) Choose graphical user interface: The Explorer 3) Insert pre-labelled data 4) Pre-process the data: punctuation, capital & stopwords removal and alphabetically tokenize WEKA 5) Consider resampling: whether a balanced dataset is preferred 6) Choose classifier: Nave Bayes 7) Classify using cross validation: 4-fold WEKA Results Resampling: 10% increase in precision 6% increase in accuracy Overall, 82% correctly classified The tools Semantria: range between -2 and 2 TheySay: three percentages for negative, positive & neutral Google Prediction API: three values for negative, positive & neutral WEKA: percentage of correctly classified Evaluation Tool Accuracy Commercial Tools Semantria 51.09% TheySay 68.61% Non-Commercial Tools Google Prediction API 72.25% WEKA 82.35% Evaluation Tool Kappa statistic F-measure Semantria 0.2692 0.550 TheySay 0.3886 0.678 Google Prediction API 0.2199 0.628 WEKA 0.5735 0.809 Evaluation Evaluation 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Negative Neutral Positive PrecisionValue Class Comparison of Precision Semantria TheySay Google API WEKA Evaluation 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Negative Neutral Positive RecallValue Class Comparison of Recall Semantria TheySay Google API WEKA Evaluation: Single-sentence responses Tool Accuracy based on correct classification All responses Single- sentence Responses Commercial Tools Semantria 51.09% 53.49% TheySay 68.61% 72.09% Non-Commercial Tools Google Prediction API 72.25% 54% WEKA 82.35% 70% Conclusions Semantria: business use TheySay: prepare for competition & academic research Google Prediction API: classification WEKA: extraction & classification in healthcare Conclusions Commercial tools: easy to use and provide results quickly Non-commercial tools: time-consuming but more reliable Conclusions Is it possible to accurate replicate human language using machines? Approx. 70% accuracy for all tools (except Semantria) WEKA: most powerful tool Conclusions Can existing SA tools respond to the needs of any healthcare-related matter? Commercial tools can not respond Non-commercial can be trained Limitations Only four tools Small dataset Potential errors in manual classification Detailed analysis of single-sentence responses was omitted Recommendations Examine reliability of other commercial tools Investigate other non-commercial tools, especially NLTK and GATE Examine other classifiers (SVM & MaxEnt) Investigate all WEKAs GUI Recommendations Verify labels using more people Label sentence as well as the whole response Negativity associated with long reviews Questions