stock price prediction using bert and gan

Stock price prediction using BERT and GANPriyank Sonkiya∗[email protected]

The LNM Institute of InformationTechnology

Jaipur, Rajasthan, India

Vikas [email protected]

The LNM Institute of InformationTechnologyJaipur, India

Anukriti [email protected]

The LNM Institute of InformationTechnologyJaipur, India

ABSTRACTThe stock market has been a popular topic of interest in the recentpast. The growth in the inflation rate has compelled people to in-vest in the stock and commodity markets and other areas ratherthan saving. Further, the ability of Deep Learning models to makepredictions on the time series data has been proven time and again.Technical analysis on the stock market with the help of technicalindicators has been the most common practice among traders andinvestors. One more aspect is the sentiment analysis - the emotionof the investors that shows the willingness to invest. A variety oftechniques have been used by people around the globe involvingbasic Machine Learning and Neural Networks. Ranging from thebasic linear regression to the advanced neural networks peoplehave experimented with all possible techniques to predict the stockmarket. It’s evident from recent events how news and headlinesaffect the stock markets and cryptocurrencies. This paper proposesan ensemble of state-of-the-art methods for predicting stock prices.Firstly sentiment analysis of the news and the headlines for thecompany Apple Inc, listed on the NASDAQ is performed using a ver-sion of BERT, which is a pre-trained transformer model by Googlefor Natural Language Processing (NLP). Afterward, a GenerativeAdversarial Network (GAN) predicts the stock price for Apple Incusing the technical indicators, stock indexes of various countries,some commodities, and historical prices along with the sentimentscores. Comparison is done with baseline models like - Long ShortTerm Memory (LSTM), Gated Recurrent Units (GRU), vanilla GAN,and Auto-Regressive Integrated Moving Average (ARIMA) model.

KEYWORDSStock-market, Deep-Learning, Artificial-Intelligence, Sentiment-Analysis, NLP, GAN, BERT

ACM Reference Format:Priyank Sonkiya, Vikas Bajpai, and Anukriti Bansal. 2021. Stock price pre-diction using BERT and GAN. In Proceedings of ACM Conference (Con-ference’17). ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from [email protected]’17, July 2017, Washington, DC, USA© 2021 Association for Computing Machinery.ACM ISBN 978-x-xxxx-xxxx-x/YY/MM. . . $15.00https://doi.org/10.1145/nnnnnnn.nnnnnnn

1 INTRODUCTIONThe stock market, because of its high volatility, is a new field forresearchers, scholars, traders, investors and corporate. The amountof computational power and techniques that have been developedhave made it possible to predict the market to an extent. RecurrentNeural Networks (RNN) have been proven to predict the time se-ries data efficiently, but the problem of vanishing and explodinggradients gave rise to the LSTM [4]. LSTM having a good numberof parameters have a greater training time in learning the short andlong term dependencies. This problem created the need for what isnow called the Gated Recurrent Units (GRUs). These GRU have agate less than the LSTM (only the reset and the update gate); hencethe training time is comparatively less, and so is the efficiency better[40].

The sentiment analysis plays a vital role in the markets regard-ing the investor’s emotions and their belief in the market and ona specific stock. Different newspapers and article headlines werescrapped for the sentiment analysis for Apple Inc’s news using fin-BERT which is a fine-tuned version of BERT on a financial corpus.BERT, known to achieve state of the art in NLP tasks, is furthertrained on the financial corpus to create the finBERT model avail-able in the transformers library [3]. The output from the sentimentanalysis along with technical indicators like 7-day moving average(7-DMA), 21-day moving average (21-DMA), MACD (moving aver-age convergence divergence), bollinger bands (upper and lower),EMA (exponential moving averages), log momentum, RSI (RelativeStrength Index), closing indexes of stock market’s of various coun-tries is given for prediction to a Generative Adversarial Network(GAN). The effect of technical indicators has been described byBarboza et al [32] which depicts how important these indicatorsare.

GANs were initially put forward by Goodfellow et al [16] forgenerating synthetic images, but the model proposed in this paperhas been modified to generate sequences instead of 2-dimensionaldata. The GAN uses a GRU as a generator and a 1-dimensionalConvolutional Neural Network (CNN) as a discriminator. As theword adversarial suggests, the generator and the discriminatorplay a min-max game against each other. The generator keeps ongenerating synthetic data, and the discriminator evaluates it tobe fake or real. Usually, the generator starts generating data fromnoise N(0,1) passed as a latent input vector. In contrast, the proposedmodel takes the sentiment scores as the input vector, which hasproven in the early convergence of the model. The model predictsstock prices on 5-day, 15-day and 30-day horizons and is evaluatedbased on the RMSE (root-mean-squared-error).

The organization of this paper is as follows: Section 2 presentsthe work done in past, related to stock prediction using artificialintelligence. Section 3 talks about theoretical concepts of the paper

arX

iv:2

107.

0905

5v1

[q-

fin.

ST]

18

Jul 2

021

https://doi.org/10.1145/nnnnnnn.nnnnnnn



Conference’17, July 2017, Washington, DC, USA Priyank and Vikas, et al.

which includes the technical indicators and the models that areused for prediction and comparison. Section 4 involves the prepa-ration of data from various sources and the feature engineeringperformed. Details of the proposed model is elaborated in section5. Experimental evaluations, model training and results of priceprediction are discussed in Section 5. Finally, concluding remarksand avenues of future research work are made in Section 7.

2 RELATEDWORKStock price prediction has been done with a variety of techniquesranging from empirical, numerical, statistical to machine learning.Starting with the data itself, Chen et al[9] and Long et al [24] usedhistorical prices only for predicting stock prices. On the other hand,Singh et al [37], Patel et al [33] have added various technical indica-tors to the historical data as well. Gonadliya et al [15] have scrapeddata from sources like - twitter, rss feeds and news portals and thenused Bag of Words (BoW), Tf-idf, N-grams to create different fea-ture sets. They finally used different algorithms like - Naive Bayes,Decision Trees, Logistic Regression, Random Forest and SupportVectors Machines (SVM) to compare the classification accuracy ofthe stock market movement. Bharadwaj et al [6] described the needto incur sentiment analysis while predicting stock prices.

Moghaddam et al [28] uniquely proposed a model that usesthe day of the week as a feature for two different timesteps, fourdays and nine days. They have compared Multilayer Perceptron(MLP) and Recurrent Neural Networks ( RNN ) and concluded withthe comparison in transfer functions OSS and tansgig. The formerperformed better with four-day timestep and later with nine days.Kesavan et al [20] extracted data from twitter and news events inaddition to the financial data of INFOSYS from moneycontrol webportal. Taking google glove dictionary as a reference, they usedLSTM and lexicon-based NLP techniques to calculate the polarity ofsentiments. They finally compared the regression results of LSTMwith the polarity of sentiments.

Nabipour et al [30] used various technical indicators like n-daymoving average, RSI, CCI of sectors, diversified financials, basicmetals, non-metallic minerals and petroleum. They have used ANN,RNN, LSTM and TREE-BASED (AdaBoost, Random Forest, Deci-sion Trees and Xgboost ) models. Taking root -mean squared erroras the metric they have concluded that LSTM performs the bestbut takes a lot of time for training. This gave the insight of us-ing GRU in the proposed model of this paper. Mustafa et al [19]concluded in their research paper that the commodity prices (Soft,Hard, Energy) have no relation with the stock prices, especially inthe south Asian countries. Maqsood et al [26] have experimentedon a massive range of models like linear regression, support vectorregression, neural networks, twitter sentiment analysis, the 2012terrorist attack and Brexit. They concluded that neural networkshad the best performance. Sousa et al [38] used CNN and pretrainedgoogle model - BERT in their proposed paper and found with BERTperforms better than CNN for sentiment analysis. Similarly, Li etal [21] have also used BERT, LSTM’s with attention and SVM toanalyze the investor sentiment in the stock market. BERT withattention was shown to perform better than LSTM with attentionand SVM based on accuracy score. Kostadin et al [27] comparedmore than 50 approaches of NLP tasks ranging from lexicon-based

approaches to transformers. They concluded that BERT model -distilled-RoBERTa achieves the state of the art results and is bestfit for classification and sentiment analysis. Artha et al [2] in theirpaper proposed a different than usual methodology to create a 2dimage of the stock prices. The 2d images were given as input to a2-d CNN for classification. Their proposed model works with 96percent accuracy on stock price movement.

Liu et al [23] used a hierarchical capsule-based network forextracting attentive features from news and tweets. They havesucceeded in increasing the accuracy by 2.45 percent from theprevious baseline HCAN and StockNet models. While Liu et al[23] used simple GRU after the embedding layer, Mousa et al [29]proposed a similar model with bi-directional GRU ( bi-GRU) afterthe word embedding to analyze news and tweets. In addition tothis, they have given candlestick and bollinger band image inputto the final capsule. They comparison with LSTM, GRU, bi-GRUis done in which their proposed model outperformed. Gandhmalet al [14] have put forward a review paper covering almost everyalgorithm ranging from 2013 - 2018. They have concluded that onlyNeural Network-based techniques are capable of predicting thestock market efficiently.

Patel et al [34] have used simple LSTM to forecast the stockprices. In the feature set, they have added an extra attribute of thepolarity of news and headlines. Eachword is assigned a polarity, andthe average polarity for a day is calculated to be added as a featureto the LSTM.They have also compared the results of includingsentiments v/s excluding them. It was found by them, that whensentiments are considered a feature, the model trains faster andperforms efficiently. Bhanja et al [5] used a mixture of a layer ofstacked LSTM and 1-d CNN for predicting stock prices and havecompared with stacked LSTM and 1-d CNN. The mixture modelof stacked LSTM and 1-d CNN layers performed better than theindividuals.

Zhou et al [43] used simple Generative Adversarial Networkswith LSTM as generator and CNN as a discriminator. They haveused 13 technical indicators as well. For different timestep and out-put horizons, they have compared GAN with LSTM, ANN, SVMand ARIMA models. The GAN outperforms the rest of the models.K. Zhang et al [42] have also proposed a Generative AdversarialNetwork (GAN) but with LSTM as the generator and Multi-layer-perceptron (MLP) as the discriminator. Further, they have comparedthe results with LSTM, ANN (Artificial Neural Network) and SVR(Support Vector Regression). They have found that the generativeadversarial network performs better in terms of the root meansquare error (RMSE) and mean absolute error (MAE) values. Theabove described models of the Generative Adversarial networksdon’t incorporate the sentiment analysis on the stock market data.From this literature survey we can observe that many researchershave found and proved that neural networks are capable of predict-ing the stock market. Among the Neural Networks, the GenerativeAdversarial Network performs the best, which was initially put for-ward by Goodfellow et al [16]. Sentiment analysis is as vital as doingthe regression analysis with the technical indicators. Hence ourproposed method ensembles the two methodologies of predictingthe stock market by doing sentiment analysis on the data and thengiving the sentiment scores a latent vector input to a regressionmodel as explained in the upcoming sections.

Stock price prediction using BERT and GAN Conference’17, July 2017, Washington, DC, USA

(a)

(b)

(c)

Figure 1: Apple’s stock performance: (a) Apple stock price (b) Technical Indicators (c) Fourier transforms

3 THEORETICAL BACKGROUNDThe American stock market is one of the most popular and attrac-tive stock markets in the world. With a market capitalization ofmore than 46,994,123 million dollars, the NYSE (New York Stock Ex-change) rose to 34 percent during the last financial year. NYSE wasestablished in 1792 as the primary dematerialised (DMAT) institu-tion for regulating stocks. Predicting stock market involves the useof technical indicators which are discussed in the next subsection.

3.1 Technical IndicatorsThe technical indicators are huge in number but we have discussedthose indicators which are used in our experimental setup (Figure 1).

• Moving Averages: A simple moving average (SMA) 7-dayor 21-day is the mean of the last 7 days and 21 days re-spectively. It gives a good representation of the historicalprices of previous 7 and 21 days. The simple moving averagedoesn’t consider the importance of recent dates. The expo-nential moving average (EMA) on the other hand gives more

importance to recent dates with the help of an exponentialterm. Simple moving average (SMA) = (P1+P2+P3. . . ..PN)/N;where P1 , P2 , P3 are the stock prices for previous N days .

• Bollinger Bands (BB): There are three lines that determinethe bollinger band, the simple moving average line, the up-per and the lower line. The upper and lower lines are +20standard deviations and -20 standard deviations away fromthe moving average line respectively. Bollinger Band = SMA(simple moving average) ± 20SD (standard deviation)

• Moving Average Convergence Divergence (MACD): Itgives a threshold of whether the market is overbought oroversold by crossovers. MACD = 26 days EMA (exponentialmoving average) - 12 days EMA. LogMomentum - Simply thelogarithm of the closing stock price of a daywhen aggregatedgives an overall picture of the stock trend.

• RSI (Relative Strength Index): RSI generates signals onbullish and bearish momentum of the market by specifyingthresholds. An asset is overbought if RSI is greater than 70percent and under boght if less than 30 percent.


• Fourier transforms: Breaking the closing price into trigono-metric components formaximumpattern recognition. Fouriertransforms were calculated for 3, 6, and 9 components of theclosing price .

3.2 Sentiment AnalysisSentiment analysis refers to finding the intuition and emotion of alanguage [22]. This natural language processing results in the po-larity of sentences classifying them as positive, negative or neutral.Sentiment analysis is highly used by business professionals and ser-vice oriented corporations that rely heavily on customer’s feedback.Sentiment analysis has equal importance to evaluate a company’sstock price in addition to the technical analysis. The models toperform sentiment analysis range from lexicon based approachesto sequence-to-sequence models to transformers in the currentera. The proposed model in this paper uses a modified version ofBERT(Bi-Directional Encoder Representations from Transformers),the state of the art model for NLP related tasks [13, 31].

3.3 Bi-Directional Encoder Representationsfrom Transformers (BERT)

BERT uses transformers[39] which are attentive models and arecapable of establishing relationships between the words via anencoder for the input and decoder for output. The traditional NLPmodels take input as one word at a time whereas transformersbased BERT takes the entire sentence at once to learn the pragmaticmeaning hidden between the words.

Our proposed model doesn’t use BERT but a fine-tuned ver-sion of it, which is finBERT, trained on TRC2-financial data andon Financial PhraseBank [25]. The fine tuned model finBERT wasintroduced by Araci [3] and is trained on a huge corpus of finan-cial data making it fit for use in financial sentiment analysis. ThefinBERT model is available in transformers library and can be usedby creating a transformer pipeline.

3.4 Long Short Term Memory (LSTM)LSTM models have improved the drawback of regular RNN’s hav-ing vanishing and exploding gradients introduced by Hochreitar etal [18]. LSTM’s perform exceptionally well with time series datafor generating sequences. The internal structure of LSTM has 3gates - Forget, Input and Output. Forget gate regulates the informa-tion that has to be retained or discarded. The input gate based oncell states learns the dependencies and conditions which assists inrecognizing and remembering sequences. The output gate decideswhich information will be propagated forward as an input to thenext cell.

3.5 Gated Recurrent Units (GRU)GRUs are similar to LSTM but they have a gate less than LSTM.GRU’s were introduced by Kyunghyun Cho et al [11]. GRUs lackthe output gate as found in the LSTM. They only consist of anupdate and reset gate. Just like LSTM’s, GRU is also capable ofcapturing sequence patterns in the data. GRU’s not having the cellstate have to directly propagate information by hidden states. Sincethey consist of fewer parameters their training time is less and it is

observed that while LSTM performs well on large datasets, GRUperforms better on small datasets.

3.6 Auto Regressive Integrated Moving Average(ARIMA)

ARIMA models were first used by George Box et al [8] as a mathe-matical approach to study changes on the time series data. ARIMAis a univariate model that tries to find patterns on its own past datavia a parameter. The ARIMA model primarily focuses on makingthe sequence stationary since it uses lags as predictors. The predic-tors need to be independent of each other and not correlated. Themodel consists of 3 parameters (p, d, q) where ‘p’ refers to the orderof the Auto-Regressive (AR) part where it inherently means thenumber of lags that will be used as predictors. ‘q‘ refers to the orderof the Moving-Average (MA) term, the number of lags that shouldgo in the model. Finally ‘d’ refers to the degree of differencing thatneeds to be done in order to make the model stationary.

3.7 GANGoodfellow et al[16] developed GAN for generating images. Sincethen GAN’s are being modified according to the use. On the basisof regularization parameters or on the basis of type of generatorand discriminator losses different versions of GAN’s have come uplike - InfoGAN [10], time series GAN [41]. Initially the generator isfed a random noise (seed) having normal distribution and it givessome output for the discriminator to analyze. The discriminatordecides as to which category the data belongs or is the distributionsimilar to that of the real data. In such a fashion the generator anddiscriminator keep fighting against each other till the point thatthe discriminator is confused to decide whether the data gener-ated by the generator is fake or real. Having such functionalitiesthe Generator is usually chosen with good regressive capabilitieslike LSTM’s or GRU’s whereas the discriminator is chosen withgood differentiating capabilities like CNN’s. Adel et al [36] haveconcluded their paper with generative models performing betterthan the discriminative models.

Figure 2: Data Preparation

4 DATA PREPARATIONFollowing paragraphs explain the steps involved in the data prepa-ration for our experiments. The same is also depicted in Figure 2.The historical data for APPLE Inc. was downloaded from yahoo


finance from July 2010 till mid July 2020. The technical indicatorsdefined in this paper (Section 3.1) were added to the feature setalong with American stock indexes like - NYSE, NASDAQ and S&P500. Apart from these, stock exchange indexes of London, India,Tokyo, Hong Kong, Shanghai and Chicago were also used in thefeature set. Finally the historical prices of commodities like Crudeoil, Gold, US-dollar and historical prices of tech giants like Google,Amazon and Microsoft were considered in the data-set. The his-torical data of Apple from yahoo finance contains open price, lowprice, high price, close price, adjusted close price and volume of theshares traded on that day. The final dataset contains all the abovementioned features - the historical stock data of APPLE Inc., thetechnical indicators mentioned in section 3.1, the popular stockindexes of various countries, commodities like gold and oil. Finallyin addition to passing the sentiment vector as a latent space tothe generator the sentiment values were also used as a feature inthe dataset. The final dataset was normalized using MinMax scalerimported from sci-kit learn. Normalization maps the range of thedata from -1 to 1 and hence reduces the complexity of the data. Thenormalized value after scaling is calculated using the formulae:

𝑥𝑠𝑐𝑎𝑙𝑒𝑑 =𝑥 − 𝑥𝑚𝑖𝑛

𝑥𝑚𝑎𝑥 − 𝑥𝑚𝑖𝑛(1)

Here 𝑥𝑠𝑐𝑎𝑙𝑒𝑑 refers to the normalized value , 𝑥𝑚𝑎𝑥 represents themaximum attribute value and 𝑥𝑚𝑖𝑛 refers to the minimum attributevalue.

4.1 Data ScrappingTo perform sentiment analysis news and headlines were scrapedfrom Seeking Alpha website using Beautiful Soup [35] from July2010 to July 2020. The scraped articleswere cleaned and pre-processedusing NLTK (Natural Language Toolkit) library [7] and were fed tothe finBERT transformer pipeline for sentiment analysis.

4.2 Creating TimestepsTime series analysis requires creating time steps for creating look-back data for which any model can learn the long term sequencesof the data. Since we already have provided technical indicatorswith lookback data, the time series stamping was performed for3 days only. Previous 3 day data was taken to predict the price ofthe 4th day data. 70 percent of the data was sent for training andthe rest 30 percent for testing. The original shape of the data was(2517, 37) where 2517 is the total number of days and 37 stands forthe number of features. This database after creating the time-stepbased look back data got converted to (2494, 3, 37) where 3 standsfor the time-step parameter. After the train-test split the shape oftraining data was (1746, 3, 37) and the shape of testing data was(748, 3, 37).

5 PROPOSED MODELAll the experimental programs are coded using Keras [12] APIof TensorFlow framework [1] [17]. The hardware setup includesNvidia DGX-1 server with Dual 20-core intel Xeon e5-2698 v4 2.2Ghz processor, supported by 512 GB DDR4 RAM and 256 GB (8x32)GPU memory.

Figure 3: Proposed Model

The proposed model takes GRU as the generator . The generatoris represented by equation -

𝑔\ : 𝑍 → 𝑋 (2)

where, \ stands for the weights of the Neural Network. In anormal GAN, Z would represent the latent input vector which isN (0,1). In the proposed model, the traditional Z is replaced withthe outputs of the sentiment analysis for early convergence. Thegenerator outputs 𝑔\ (Z) which is represented by 𝑃\ . The final aimof the generator is to select \ in such a fashion that the output of thegenerator 𝑔\ (Z) is estimated similarly to the original distribution X.In the first iteration the generator creates synthetic data from thelatent space vector and propagates that to the discriminator. Thediscriminator classifies the data and back propagates the error backto the generator for updating the weights. So in each iteration thegenerator keeps on getting better in generating the data. We useGRU as the generator for our proposed model. The model beginswith1024 units in the first layer, 512 in the second and 256 in thethird layer. To prevent over-fitting of the model a dropout of 20percent was also introduced in every layer. In the end 3 denselayers were added each with units 128, 64 and 1 respectively. Thegenerator model takes the input dimension, output dimensions andthe feature size as input. The generated output is the next day’sstock prices which is passed on to the discriminator.

The proposedmodel takes 1-dimensional Convolutional Networkas the Discriminator. The discriminator is represented by

𝐷𝑤 : 𝑋 → [0, 1] (3)


In this equation parameter w represents the weights of the discrim-inator. For the real distribution 𝑃𝑥 the discriminator assigns 1 tothe samples and for the distribution generated by the generator𝑃\it assigns 0. In the proposed GAN model, 1-d CNN was used as thediscriminator. 32, 64 and 128 units were taken in the first three lay-ers of the CNN respectively with LeakyRelu as activation function.Alpha value of LeakyRelu was taken to be 0.01 in all three layers.The kernel size of 3, 5 and 5 was chosen in the first three layersrespectively. Next a flatten layer was deployed to convert the datainto 1-dimension followed by a dense layer with 220 units. Furthera layer of LeakyRelu was used with default alpha parameter ( 0.3 )followed by another dense layer again with 220 units and activationfunction as Relu. Finally a dense layer with 1 unit and sigmoidactivation function was used in the end to estimate whether thedata it receives belongs to the real data or the generated data by thegenerator. After classification of the data the discriminator calcu-lates the loss between the two distributions - real and generated andback propagates to the generator as a signal to update its weightsand create a better distribution which is similar to the real one. Theloss function of a GAN is based on the KL-JS divergence. In thetraining step the GAN uses cross-entropy to differentiate betweenthe two distributions - the real distribution and the generated dis-tribution. Minimizing this cross-entropy results in minimizing theKL-JS divergence resulting in similar distributions.

The GAN model is represented by (\,𝑤). Both the individualneural networks work against each other. The generator is trying tominimize the difference between real and generated distributionswhereas the discriminator is trying to maximize it. This can berepresented in form of a min-max equation:

min\

max𝑤

𝐸 [𝑙𝑜𝑔(𝐷𝑤 (𝑋 )) + 𝑙𝑜𝑔(1 − 𝐷𝑤 (𝑔\ (𝑍 )))] (4)

In equation (4) the objective of the discriminator is to correctlydifferentiate between the real distribution of stock prices v/s thegenerated distribution. For achieving the same, the discriminatortries to maximize the probabilities in the equation so that the gen-erator output samples are classified as generated ones 𝐷𝑤 (𝑔\ (𝑍 ))and the original distribution samples as real one 𝐷𝑤 (𝑋 ). The gen-erator wants the discriminator to classify the generated sample𝐷𝑤 (𝑔\ (𝑍 )) as belonging to the real distribution. Hence the gen-erator tries to minimize equation (4) for fooling the discriminator.Finally, the expectation (E) of the min-max equation makes surethat the back propagation of error not only takes place for a samplebut for the entire data .

Taking KL-JS divergence into consideration the min-max prob-lem boils down to - min\ JS(𝑃𝑥 || 𝑃\ )

TRAINING - Using binary cross entropy as a differentiatingmeasure the weights w of the discriminator move in the positivedirection in contrast to \ of the generator which move in the oppo-site direction to minimize the difference between two distributions.For each iteration samples from the original distribution is takenas {𝑋1, ..., 𝑋𝑁 } and the corresponding generator outputs are calcu-lated as {𝑔\ (𝑍1), ....𝑔\ (𝑍𝑁 )} . The same process is repeated for theremaining iterations and the parameters are updated according tothe loss function:

Table 1: Comparison of proposed model with traditional ap-proaches and the work of Zhang et al

Models RMSE (Training) RMSE (Testing)

ARMIA 10.366 18.2469LSTM 1.66 2.939GRU 0.70025 2.96GAN 0.583 2.369GAN (Zhang et al)[42]

Not Available 4.1026

Proposed Model (S-GAN)

0.5606 1.827

𝑤 → 𝑤 + [𝑛∑︁1

∇𝑤 [𝑙𝑜𝑔(𝐷𝑤 (𝑋𝑖 )) + 𝑙𝑜𝑔(1 − 𝐷𝑤 (𝑔\ (𝑍𝑖 )))] (5)

\ → \ − [

𝑛∑︁1

∇\ [𝑙𝑜𝑔(1 − 𝐷𝑤 (𝑔\ (𝑍𝑖 )))] (6)

where, [ denotes the learning rate. The final proposed model ispresented in the Figure 3.

6 EXPERIMENTAL RESULTSAlong with the proposed model ARIMA, LSTM, GRU and plainvanilla GAN with noise as latent input were also evaluated forcomparison. For evaluating the model and for comparison RootMean Square Error (RMSE) metric was chosen. Finally the modelswere compared on 5-day, 15 day, and 30 day predictions.

𝑅𝑀𝑆𝐸 =

√︄∑𝑛1 (𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑𝑖 −𝐴𝑐𝑡𝑢𝑎𝑙𝑖 )2

𝑁(7)

Here predicted refers to the predicted stock price and actual isthe real stock price. N denotes the number of data points (numberof days).

Auto regressive integrated moving average (ARIMA)model takes3 parameters p,d and q as discussed in the theoretical background.Being a univariate model ARIMA was run only on the closing priceand to maintain the synchronization of data 3 time steps were usedto generate the time sequenced data like in other models. Thoughincreasing the timesteps could have resulted in better results forARIMA but for comparison purposes we have considered only 3timesteps. So, ARIMA (4,1,0) was used for training and ended upwith a high RMSE of 10.366 . The testing data also as expectedhad high RMSE 18.2469. Again, this was done just for comparisonpurposes because every other model was run on 3 time step dataand used technical indicators having historical prices but ARIMAbeing univariate only close price was considered.

Namini et al [36] have proven the efficiency of bi-directionalLong Short Term Memory (LSTM) over the uni-directional one. Forcomparison we have used bidirectional LSTM with 128 cells withtwo dense layers having 64 and 1 unit respectively. With the lossfunction as simple mean square error (MSE) the model was trainedfor 150 epochs, with a batch size of 64 and learning rate of 0.0012.


(a)

(b)

(c)

(d)

(e)

Figure 4: Comparison of various models and proposed S-GAN on Test Data. (a)ARIMA (b) LSTM (c) GRU (d) GAN (e)Proposed S-GAN

Adam optimizer was used to update the weights and on the trainset a RMSE of 1.66 was observed from the year 2010 to 2017. Thetesting RMSE was 2.939 on the data after 2017 which was used forevaluating the model.

Gated Recurrent Units (GRU), the third model for comparisonwas built with twoGRU cell layers with 128 and 64 units respectivelyalong with two dense layers of 32 and 1 units. Mean squared error(MSE) was used as a loss function and like LSTM, Adam optimizerwas used to update the network weights. With learning rate 0.0005,batch size as 128 and number of epochs as 50 RMSE of 0.70025 wasobserved on the train set. On the test data the calculated RMSEwas 2.96. As expected the GRU works similar to LSTM with lessparameters and less training time.

The plain Generative Adversarial Network (GAN) with normaldistribution as noise was trained with the same hyper-parametersas shown in figure 3, without sentiment vector as latent noise.With learning rate 0.00016, batch size 128 and epochs as 165 Adamoptimizer was used to update the weights. The training RMSE wasobserved to be 0.583 on training data 2010-2017. The testing RMSEcalculated was 2.369 on data from 2017 till end.

Our proposed model (named S-GAN) uses sentiment analysisoutput as a latent vector and has the same hyper-parameters asthe vanilla model discussed in the above subsection for direct com-parison of results. So with learning rate as 0.00016, batch size 128,epochs being 165, the training RMSE was observed to be 0.5606.The testing RMSE as expected turned out to be the lower which is1.827. As expected when given the sentiment vector as latent inputthe model has proved to be efficient in two ways. Firstly the modelshowed early convergence since the sentiment vector already con-tains the buying and selling trends of the market. In addition tothe early convergence, the model performed better than the vanillamodel by having almost 35% less RMSE value and 50% lesser RMSEthan the model Zhang et al [42] proposed. Figure 4 and Table 1and shows the prediction results of S-GAN and other traditionalapproaches on the Apple Inc stock.

Apart from the testing data in totality, predictions were alsoperformed in an interval of 5,15 and 30 days for assessing thepracticality of the model. The graphs and the respective RMSEvalues are shown in Figure 5 and Table 2.

Table 2: Comparison of S-GANwith other traditionalmodelson 5, 15 and 30 day horizons

Models 5 Days 15 Days 30 Days

ARMIA 0.121 0.3658 0.7317LSTM 0.0196 0.0585 0.117GRU 0.01975 0.05925 01185GAN 0.0158 0.0474 0.0948ProposedModel (S-GAN)

0.0122 0.0366 0.0732

7 CONCLUSION AND FUTUREWORKOur proposed model S-GAN outperformed the traditional time se-ries forecasting models like GAN, GRU, LSTM and ARIMA. It’s alsoevident from our proposed model that in addition to the technicalanalysis on the stock market, sentimental analysis also impactsthe market situation by numerically calculating the emotions andsentiment of the investors. Our model is an advanced version of


(a) (b) (c)

Figure 5: Performance of S-GAN and other traditional models (a) 5 days (b) 15 days (c) 30 days

what Zhang et al [42] proposed that does not involve sentimentanalysis and uses LSTM as generator and MLP as discriminator.Hence we observe better results in RMSE including sentiment anal-ysis as latent space while using GRU and CNN as generator anddiscriminator respectively. The proposed model successfully cap-tures the trends in the stock market data with the historical stockmarket as input data besides analyzing the sentiments. The gener-ator is able to confuse the discriminator whether the data is realor is generated. Further the 5, 15 and 30 days predictions show theapplied use case of this model in daily life stock market investments.Having all the attribute values of a particular day, the model is ableto predict the next day price. It’s also fruitful for inter-day traderswho already have the next day analysis of closing prices by thetime stock market ends for a particular day. Concluding, with thehelp of this model an intuitive decision can be made the momentthe stock market closes for a day and closing prices are declared.The investors and traders can decide based on the predicted pricefor the next day whether to buy a particular stock or sell it.

The advancements in artificial intelligence have made it possibleto predict anything involving data, in such a scenario our next goalis to focus on reward based learning and prediction. With the helpof reinforcement learning the next milestone is to create a bot andregulate it with a reward function for getting returns on the stockmarket. Further another aspect of future work is to achieve a realtime stock market prediction for intraday trading with the help ofassessing real time news and real time stock price movements.

REFERENCES[1] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey

Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al.2016. Tensorflow: A system for large-scale machine learning. In 12th {USENIX}symposium on operating systems design and implementation ({OSDI} 16). 265–283.

[2] ArthaAndriyanto, AntoniWibowo, andNorhaslinda Zainal Abidin. 2020. SectoralStock Prediction Using Convolutional Neural Networks with Candlestick Patternsas input Images. International Journal 8, 6 (2020).

[3] Dogu Araci. 2019. Finbert: Financial sentiment analysis with pre-trained languagemodels. arXiv preprint arXiv:1908.10063 (2019).

[4] Martin Arjovsky, Amar Shah, and Yoshua Bengio. 2016. Unitary evolution recur-rent neural networks. In International Conference on Machine Learning. PMLR,1120–1128.

[5] Samit Bhanja and Abhishek Das. 2019. Deep learning-based integrated stackedmodel for the stock market prediction. Int. J. Eng. Adv. Technol 9, 1 (2019),5167–5174.

[6] Aditya Bhardwaj, Yogendra Narayan, Maitreyee Dutta, et al. 2015. Sentimentanalysis for Indian stock market prediction using Sensex and nifty. ProcediaComputer Science 70 (2015), 85–91.

[7] Steven Bird. 2006. NLTK: the natural language toolkit. In Proceedings of theCOLING/ACL 2006 Interactive Presentation Sessions. 69–72.

[8] George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. 2015.Time series analysis: forecasting and control. John Wiley & Sons.

[9] Lu Chen, Yonggang Chi, Yingying Guan, and Jialin Fan. 2019. A hybrid attention-based EMD-LSTM model for financial time series prediction. In 2019 2nd Interna-tional Conference on Artificial Intelligence and Big Data (ICAIBD). IEEE, 113–118.

[10] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and PieterAbbeel. 2016. Infogan: Interpretable representation learning by informationmaximizing generative adversarial nets. In Proceedings of the 30th InternationalConference on Neural Information Processing Systems. 2180–2188.

[11] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau,Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phraserepresentations using RNN encoder-decoder for statistical machine translation.arXiv preprint arXiv:1406.1078 (2014).

[12] F Chollet. 2020. et al. 2015. Keras.[13] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert:

Pre-training of deep bidirectional transformers for language understanding. arXivpreprint arXiv:1810.04805 (2018).

[14] Dattatray P Gandhmal and K Kumar. 2019. Systematic analysis and review ofstock market prediction techniques. Computer Science Review 34 (2019), 100190.

[15] Chetan Gondaliya, Ajay Patel, and Tirthank Shah. 2021. Sentiment analysis andprediction of Indian stock market amid Covid-19 pandemic. In IOP ConferenceSeries: Materials Science and Engineering, Vol. 1020. IOP Publishing, 012023.

[16] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarialnets. Advances in neural information processing systems 27 (2014).

[17] Antonio Gulli and Sujit Pal. 2017. Deep learning with Keras. Packt PublishingLtd.

[18] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-termmemory. Neuralcomputation 9, 8 (1997), 1735–1780.

[19] Mustafa Ildırar and Erhan İşcan. 2015. The Interaction between Stock Pricesand Commodity Prices: East Europe and Central Asia Countries. Kazan, Russia,September (2015), 9–11.

[20] M Kesavan, J Karthiraman, Rajadurai T Ebenezer, and S Adhithyan. 2020. Stockmarket prediction with historical time series data and sentimental analysis ofsocial media data. In 2020 4th International Conference on Intelligent Computingand Control Systems (ICICCS). IEEE, 477–482.

[21] Menggang Li, Wenrui Li, Fang Wang, Xiaojun Jia, and Guangwei Rui. 2021.Applying BERT to analyze investor sentiment in stock market. Neural Computingand Applications 33, 10 (2021), 4663–4676.

[22] Bing Liu et al. 2010. Sentiment analysis and subjectivity. Handbook of naturallanguage processing 2, 2010 (2010), 627–666.

[23] Jintao Liu, Hongfei Lin, Liang Yang, Bo Xu, and Dongzhen Wen. 2020. Multi-element hierarchical attention capsule network for stock prediction. IEEE Access8 (2020), 143114–143123.

[24] Wen Long, Zhichen Lu, and Lingxiao Cui. 2019. Deep learning-based featureengineering for stock price movement prediction. Knowledge-Based Systems 164(2019), 163–173.

[25] Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala.2014. Good debt or bad debt: Detecting semantic orientations in economic texts.Journal of the Association for Information Science and Technology 65, 4 (2014),782–796.

[26] Haider Maqsood, Irfan Mehmood, Muazzam Maqsood, Muhammad Yasir, SitaraAfzal, Farhan Aadil, Mahmoud Mohamed Selim, and Khan Muhammad. 2020.A local and global event sentiment based efficient stock exchange forecastingusing deep learning. International Journal of Information Management 50 (2020),432–451.

[27] Kostadin Mishev, Ana Gjorgjevikj, Irena Vodenska, Lubomir T Chitkushev, andDimitar Trajanov. 2020. Evaluation of sentiment analysis in finance: from lexiconsto transformers. IEEE Access 8 (2020), 131662–131682.


[28] Amin Hedayati Moghaddam, Moein Hedayati Moghaddam, and Morteza Esfand-yari. 2016. Stock market index prediction using artificial neural network. Journalof Economics, Finance and Administrative Science 21, 41 (2016), 89–93.

[29] Ramin Mousa, Sara Nazari, Ali Karhe Abadi, Reza Shoukhcheshm, Moham-mad Niknam Pirzadeh, and Leila Safari. 2021. TI-Capsule: Capsule Networkfor Stock Exchange Prediction. arXiv preprint arXiv:2102.07718 (2021).

[30] Mojtaba Nabipour, Pooyan Nayyeri, Hamed Jabani, Amir Mosavi, Ely Salwana,et al. 2020. Deep learning for stock market prediction. Entropy 22, 8 (2020), 840.

[31] László Nemes and Attila Kiss. 2021. Prediction of stock values changes usingsentiment analysis of stock news headlines. Journal of Information and Telecom-munication (2021), 1–20.

[32] Felipe Barboza Oriani and Guilherme P Coelho. 2016. Evaluating the impactof technical indicators on stock forecasting. In 2016 IEEE Symposium Series onComputational Intelligence (SSCI). IEEE, 1–8.

[33] Jigar Patel, Sahil Shah, Priyank Thakkar, and Ketan Kotecha. 2015. Predictingstock market index using fusion of machine learning techniques. Expert Systemswith Applications 42, 4 (2015), 2162–2172.

[34] Jay Pareshkumar Patel, Nikunj Dilipkumar Gondha, Jai Prakash Verma, andZdzislaw Polkowski. 2021. Event-Triggered Share Price Prediction. In EvolvingTechnologies for Computing, Communication and Smart World. Springer, 83–96.

[35] Leonard Richardson. 2007. Beautiful soup documentation. Dosegljivo: https://www.crummy. com/software/BeautifulSoup/bs4/doc/.[Dostopano: 7. 7. 2018] (2007).

[36] Sima Siami-Namini, Neda Tavakoli, and Akbar Siami Namin. 2019. The perfor-mance of LSTM and BiLSTM in forecasting time series. In 2019 IEEE International

Conference on Big Data (Big Data). IEEE, 3285–3292.[37] Ritika Singh and Shashi Srivastava. 2017. Stock prediction using deep learning.

Multimedia Tools and Applications 76, 18 (2017), 18569–18584.[38] Matheus Gomes Sousa, Kenzo Sakiyama, Lucas de Souza Rodrigues, Pedro Hen-

rique Moraes, Eraldo Rezende Fernandes, and Edson Takashi Matsubara. 2019.BERT for stock market sentiment analysis. In 2019 IEEE 31st International Confer-ence on Tools with Artificial Intelligence (ICTAI). IEEE, 1597–1601.

[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is allyou need. In Advances in neural information processing systems. 5998–6008.

[40] Peter T Yamak, Li Yujian, and Pius K Gadosey. 2019. A comparison betweenarima, lstm, and gru for time series forecasting. In Proceedings of the 2019 2ndInternational Conference on Algorithms, Computing and Artificial Intelligence.49–55.

[41] Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar. 2019. Time-seriesgenerative adversarial networks.

[42] Kang Zhang, Guoqiang Zhong, Junyu Dong, Shengke Wang, and Yong Wang.2019. Stock market prediction based on generative adversarial network. Procediacomputer science 147 (2019), 400–406.

[43] Xingyu Zhou, Zhisong Pan, Guyu Hu, Siqi Tang, and Cheng Zhao. 2018. Stockmarket prediction on high-frequency data using generative adversarial nets.Mathematical Problems in Engineering 2018 (2018).

stock price prediction using bert and gan

Documents