Predicting Stock Price MovementUsing Social Media Analysis

Derek Tsui

Stanford University

Abstract

In this project, we aim to predict stock prices by using machine learning techniques on data from StockTwits,a social media platform for investors. We demonstrate the results, and compare the prediction error, of severalclassification and regression techniques using aggregated StockTwits messages as a source of input.

I. Introduction

Social media platforms such as StockTwitscan provide a wealth of information onreal-world patterns and behaviors. We offeran analysis on a specific application of socialmedia, pertaining to finance: using aggregatedStockTwits message data to make statisticallysignificant price predictions. Our underlyingassumption is that there exists a correlationbetween market price action and the metricsthat we extract from this aggregate sentiment,indicating that it can provide a meaningful,actionable slice of real market conditions andpsychology.

Existing works looking into this topichave found correlations between bullishsentiment on Twitter and short-term priceanomalies of stocks [1], as well as messagevolume peaks and abnormal price action [2].However, a critical issue commonly foundin previous research is statistical inaccuracyand over-fitting, including the selection ofequities for which the results are demonstrablyfavorable. Given that firms increasingly usedata mining, machine learning and other auto-mated statistical techniques in various stagesof trading, including decision-making andexecution [3], further rigorous development ofthis kind of large-scale social media analysishas the potential to provide or augment an

additional source of investment alpha.

II. Data

We performed analysis on the componentstocks of the Dow Jones Industrial Average.Data was collected for the period December2013 to December 2016, totaling 756 tradingdays. Two main datasets were used:

i. StockTwits Data

StockTwits data was collected and downloadedin raw JSON format, totaling over 540,000 mes-sages. Significant pre-processing was necessaryto generate "bag-of-words" feature vectors, in-cluding:

• removing stop-words and company names• removing posts mentioning/tagging mul-

tiple stocks (i.e. "$AAPL $FB $GOOG")• aggregation of posts by date

Sentiment polarity was also extracted fromuser-generated "bullish"/"bearish" tags, forwhich a rolling mean of the ratio was calcu-lated.

ii. Price Data

Daily split-adjusted stock price data was col-lected via the Yahoo Finance API. We focusonly on the closing price data for the purposes

1

of this project. The main piece of data that weextract from this is the forward 3-day return,calculated as a percentage change for the futureprice movement three days ahead of today’strading day’s price; the formula for this is:

return =pt+3 − pt

pt(1)

where pt+3 is the price at time period t + 3(in this case, 3 days ahead). This data was usedas the prediction target to model the short-termcorrelation with social media activity. A shorterperiod was not selected, in order to smooth outthe effects of daily market noise. Althoughwe initially sought to use binary classificationonly, regression models tended to outperformclassification with respect to accuracy and per-formance.

iii. Training

70% of the data was used for feature selection,of which a third was reserved exclusively forfeature selection and cross validation. The re-maining 30% was used as the test set; this wasthe StockTwits and price data from January2016 to December 2016, a time period sepa-rated off from that of the training set, to avoidany inclusion of look-ahead bias.

iv. Features

Two feature sets were used for the differentmethods: a pure bag of words model, and amodel using a combination of word frequencyfeatures and sentiment metrics.

Pure Bag of Words. After pre-processing,the frequencies of all 6,839 words occurring atleast 25 times in the data are calculated using1) the tf-idf metric, a statistical metric for wordimportance in a document that takes the prod-uct of term frequency (TF) and inverse docu-ment frequency (IDF), and 2) Laplace smooth-ing. These metrics are then used as features inour multinomial model.

TF(t) =# of times t occurs in the document

total # of terms in the document

Figure 1: Cross-validation accuracy in recursive featureelimination.

buy earnings new todayeps good price long

short pt support bearish

Table 1: Sample of words selected by the mixed featuremodel. NB: "pt" shorthand for "price target"

IDF(t) = logtotal # of documents

# of docs with the term t in it

Mixed Feature Model. The number of wordfrequency features was reduced to 1000 firstby using uni-variate chi-squared score ranking,and then finally to 506 by using recursive fea-ture elimination with 5-fold cross validation.Other features from the social media data, in-cluding message volume change, and polaritymetrics were later included in this model:

• Message volume, 1-day change (percent-age)

• Message volume vs. 10-day average mes-sage volume (percentage)

• Message polarity, calculated as the differ-ence of bullish to bearish messages, di-vided by the total number of sentiment-tagged messages

III. Methods

We consider several methods for analyzing ourdata, with the primary goal of obtaining a pre-diction success rate above 50% (and ideally,

2

above 60% to be considered significant). Sincethis particular problem is classification-basedin nature (returning a ’bullish’ or ’bearish’ sig-nal for the desired time period, we will firsttest out the efficacy of a classification model.All models were tested using scikit-learn.

i. Naive Bayes

We use the multinomial model for this genera-tive learning algorithm, which assumes featureindependence and seeks to maximize:

p(y) =n

∏i=1

p(xi|y)

For the input, Naive Bayes was trained usingjust a pure bag of words model, and assignedthe probabilities to either a positive y = 1 ornegative y− 0 class, from which a long/shortprediction can be generated.

ii. Support Vector Regression

We use this good "off-the-shelf" learning al-gorithm for regression, which, similar to howSVC finds a decision boundary that maximizesthe margin, aims to minimize the ε-insensitiveloss function created by Vapnik [4]:{

0 |y− f (x, ω)| ≤ ε|y− f (x, ω)| − ε else

This model is trained on the mixed featuremodel. For our regression cases we simply mapthe predicted y-values beyond a reasonablethreshold (0.5%) to valid positive and negativesignals.

iii. K-Nearest Neighbors Regression

We tested a non-parametric method, k-nearestneighbors clustering on the same features andprediction testing schema as SVR, using the Eu-clidean distance metric. We hypothesized thatthis method could potentially outperform theparametric models, which may overfit on all ofthe noise in our financial data. The parameterk was optimized with 5-fold cross validation.

Figure 2: Cross-validation accuracy in recursive featureelimination on k-NN regression.

IV. Results

The test accuracy for each of the learning mod-els is shown below:

Table 2: Test accuracy, compared

Ticker Test Accuracy

Naive Bayes 0.5099SVR 0.5682k-NN Regression 0.5448

Average 0.5410

Table 3: Best and worst stocks, Naive Bayes example

Ticker Test Accuracy Trades

UTX 60.0% 80JNJ 59.3% 86INTC 57.9% 126

MRK 43.6% 78BA 42.2% 166MMM 37.5% 16

For practical trading purposes, however, themore important metric is profitability. Themodel predictions for the positive and negativeclasses were used to generate long/short sig-nals on the test data, and a simulated portfolioallocated 33% equity to daily equal-weightedlong/short positions each held for 3 days.The resulting profit-and-loss (PnL) curves are

3

Figure 3: Comparative PnL curve. Dow index movement in blue.

graphed on the next page in comparison tothe DJIA index performance for the same timeperiod.

This graph shows the results on the test pe-riod, from Jan 2016 to Dec 2016. AlthoughNaive Bayes is consistently profitable, it signifi-cantly underperforms the other methods andthe broader market. SVR returns a total of 25%within this 11-month period, after simulatedcommission costs.

Overall, the regression models proved to bemore accurate and more actionable as tradingsignals compared to binary classification. Oneplausible reason for this result is that binaryclassification (especially the generative modelused in Naive Bayes) will attempt to fit to thenoise inherent in stock market price movement,and lumps small, statistically insignificant up-ward movements indiscriminately with largeones. The test error rates were below whatthis project initially aimed to achieve; however,the signal’s positive performance indicates thatthe selected features are in fact meaningful,and capture some insight into short-term mar-ket movements. Notably, the SVR and KNNmodels are still very correlated to the market,although the SVR model does deviate from themarkets in certain time-frames. Despite thebroad selection of stocks used in this analysiscompared to previous works, the training set

and test set are biased towards positive classesfor this time period (the 2013-2016 period wasoverall a positive time for markets). They arealso all large-cap, blue chip companies thatlargely move in tandem with the broader mar-ket as a whole. Thus, further research couldalso analyze a wider of basket of stocks, ide-ally not highly correlated to broader markets,to verify and improve on our findings. Futurework on this topic could also involve testingrecurrent neural networks on the data, whichwould be particularly suitable for time seriesprediction problems like this one.

References

[1] Huina Mao, Scott Counts and JohanBollen. Quantifying the effects of onlinebullishness on international financial mar-kets. European Central Bank, 2015.

[2] Gabriele Ranco, Darko Aleksovski, GuidoCaldarelli, Miha Grc̆ar, and Igor Mozetic̆.The Effects of Twitter Sentiment on StockPrice Returns. PLoS ONE, 2015.

[3] Cade Metz. The Rise of the ArtificiallyIntelligent Hedge Fund. Wired, 2016.

[4] Corinna Cortes and Vladimir Vapnik. Sup-port vector networks. Machine Learning,20:273–297, 1995.

4