Web Intelligence

For many years, a lot of research has been conducted on the stock price prediction [10]. Early research focuses on efficient market hypothesis and random walk theory [8] which indicates that the stock prices are driven by new information such as news rather than previous prices. Nowadays many different kinds of model have be developed to forecast the stock price [17,29,36]. And some recent research indicates that although news are unpredictable however according to the behavioral finance, the early indicators could be detected through different approaches. Many researchers suggest to use online social media (blogs, micro-blogs, facebook etc.) to obtain these indicators and improve the forecasting performance [3,24].

The number of social media website users around the world has a high rapid growth over the past 10 years. The exponentially increased user-generated content in social media websites such as Facebook and Twitter has provided a potential data source for opinion mining as users share their opinions in these websites by text posts, pictures and even videos. In China, Weibo is one of the largest microblog social media websites and the platform is similar as Twitter. The distinctive nature of social media websites makes it a valuable source for mining public opinions or emotions. Firstly, a lot of users post their opinions about different topics in the website which represent their emotions in some extent. Secondly, these posts are highly time sensitive. Active users in these websites post their opinions every day, even every hour. Thirdly, some social media websites such as Weibo have an approval mechanism for registration which makes sure each user in these systems is a real person. Fourthly, some organizations and companies use social media websites such as Facebook, Twitter and Weibo as their formal channel to publish their opinions to the public. As a result the data collected from these social media websites can help researchers to understand the public mood states or emotions.

Recently significant progress has been achieved in extracting states of public mood directly from social media websites such Twitter and use them to predict the stock price movement [14]. However, the number of research on Chinese social media websites is limited and the number of research on how to use the textual sentiment analysis result of social media websites towards Chinese stock market is even lower. In recent years, China has already been the second largest economies and the stock market in China has a great impact to the global stock markets. The intraday trading volume on Shanghai Stock Exchange (SSE) has been more than 1 thousand billion RMB on 20th April 2015 which is more than any stock exchange intraday trading volume in the history. Instead of Shanghai Stock Exchange, there are also Shenzhen Stock Exchange and Hong Kong Stock Exchange in China. How the stock market price in China be affected by the public mood change is important for economists, traders and socialist.

This paper is interested in discussing whether the public mood states extracted from Chinese social media websites such as Weibo can represent the real opinion of publics towards the stock market and how the public mood states impact the stock market. Although researches on behavioral economics and finance have already demonstrate the correlation between emotions and the stock price, this paper will discuss whether Weibo is a valid source to monitor the public mood states about the stock market and demonstrate the causality between the Weibo public mood states and the stock price. This paper first presents a topic-based approach to extract public mood states from Weibo. After that a Granger causality analysis is used to investigate the hypothesis that the extracted public mood states are predictive of the stock price movement in China. The experimental results indicate that some public mood states such as the “Happiness” and “Disgust” states are highly correlated with the movement of stock price. Finally, the nonlinear correlation between the stock price movement and the public mood in Weibo is discussed. A topic-based nonlinear autoregressive model with exogenous Weibo mood input is then proposed to predict the future stock price movement. The prediction accuracy of the price movement direction is competitive and high. The model could be used as a reference for the investment decision making process.

In recent years, social media websites, such as Facebook, Twitter and Weibo become more and more popular. The user number is enormous and there are different user communities with diversified natures in these websites. A lot of data is published by the online users including comments and news in these websites. The data, to some extent, represents the public emotions or opinions. An increasing number of research has attempted to integrate sentiment analysis result of the online data into different models.

Some research has already been conducted on using the overall public mood from Twitter to predict the stock market price [3]. However a lot of unrelated posts are in these social media websites. If researchers want to discuss the emotion impact on a stock, they need to find out the text which is highly related to the stock. For example, even most users have bad comments on a movie in Twitter that should not have a big impact on the stock price of an oil company. As a result, to filter the unnecessary noise, this paper uses a topic-based approach to analyze the public mood, and discuss how users’ comments or emotions about these topics affect the stock market. Because of the characteristic of social media websites, discussions and news in these websites such as Twitter and Weibo are topic based. For example, a lot of posts are published to discuss the impact of “Shenzhen-Hong Kong Stock Connect (SZ-HK connect)” in Weibo. In Twitter, users use hashtags to identify different topics for discussion. Extracting the public mood states or opinions based on topics has two advantages. First, it can filter out unrelated information and only focus on opinions about the specific topic. Second, the public mood states about some special topics are easily be reused to discuss the relationship between different topics and data. One of the data source discussed in this paper is the text content from a Chinese social media website called Weibo. It was launched in August 2009. As of mid 2014, there are 167 million monthly active users, more than 25 million posts each day.

The database used in this paper is a collection of public Weibo posts that was recorded from Jan 2015 to Feb 2016. For each post the database provides an identifier, the datetime of the submission and the text content. All these posts are crawled for Weibo.com. After that the posts are classified by date and stop-words and punctuation are removed for each post. As this paper is discussing the public mood related to Chinese stock market, 3 topics for discussion are selected which are: “Shanghai-Hong Kong Stock Connect (SH-HK connect) and Shenzhen-Hong Kong Stock Connect (SZ-HK connect)”, “Interest rate and reserve rate cut in China”, “Greece Government-Debt Crisis”. The reason why these 3 topics are selected is that from 1st Jan 2015 to 31th Dec 2015, they are mostly discussed topics related to the stock market in the website. SH-HK connect has been launched on November 2014 and SZ-HK connect is planned to launched in 2016. It allows the individual and institutional traders in Shanghai or Shenzhen to trade the stock market in Hong Kong through the Connect. And the traders in Hong Kong also can use it to trade the stock market in Shanghai or Shenzhen. It provides different choices and an arbitrage chance for the traders in Hong Kong and Mainland China. It is considered as a great milestone in the development of China stock market and people believe that it will have a positive impact on China stock market. Greece Government-Debt Crisis is the debt crisis that happens in European Union (EU) in 2015, people worry that if Greece cannot pay the debt on time, it will cause the financial crisis in EU and the international crisis will be triggered again. Interest rate and reserve rate cut means the central bank in China plan to cut the bank lending interest rate and the amount of cash banks must keep in reserve. It will increase the flow of money in the economy and reduce the cost of financing to support the investment and developments of the real economy. As it will help to boost the growth of economy, it has a positive impact on the stock market. The second source of data is the daily end price of stocks in Hong Kong and Shanghai stock exchanges which could be downloaded from public available websites such as Yahoo!Finance and http://finance.sina.com.cn/.

The system framework can be summarized as follows. In the first phase, it will filter the posts and only extract the posts containing the keywords related to the 3 topics. Posts are grouped under the same topic. In the second phase, it will use two tools to measure the sentiment score for different emotions in the posts. The first tool, Jieba, will help to analyze the Chinese text and segment the text into meaningful Chinese words. It is an useful tool to segment user-generated content in forums [33]. The second tool, Chinese Emotion Word Ontology (CEWO), is a Chinese sentiment word dictionary which classifies the Chinese emotion words into 7 different categories [6]. A novel method to generate the sentiment score or mood states for each post based on the CEWO is applied to the dataset. After that it combines the sentiment score for all posts on the same day under the same topic. As a result, the daily sentiment score for different emotions are generated for each topic. The sentiment score across the observation time can be regarded as time series. In the third phase, it investigates the hypothesis that public mood measured by our mechanism is predictive of future stock price using Granger causality analysis. As the public mood generated is related with different stocks, the impact on different kinds of stocks are discussed. At the end, a nonlinear autoregressive model with exogenous Weibo mood inputs is proposed to predict the price and moving directions of a stock.

As mentioned above, the raw text data is restored in database first. And then the Chinese segmentation tool Jieba is used to segment the text into different words. The Jieba tool use the dynamic programming to find out the most probable combination based on the word frequency. The segmentation result will be restored in the database as well. The next step is to generate the sentiment score or mood states from each post. For this part, the model will use the Chinese Emotion Word Ontology constructed by the IR lab in Dalian University of Technology. This ontology is constructed based on Ekman theory of 6 basic emotions (Happiness, Sadness, Surprise, Fear, Disgust and Anger). And one more emotion “good” is added to make it more comfortable for Chinese language analysis [6]. Figure 1 shows the mapping from English to Chinese for the 7 categories of emotion words. These categories are also known as different dimensions of public mood. Each emotion word in the Ontology has its own category tag such as “Happiness”, intensity value and polarity value.

The proposed method to generate the sentiment score or public mood states including 4 steps. First, Jieba is used to segment the post and translate the post to a list of meaningful words including different lexical class such noun, verb, adv and adj. Secondly, it will process the word list and search each word of the list in the Chinese Emotion Word Ontology based on its lexical class. If a word i is found in the Ontology, the intensity value of the word in the Ontology will be recorded as ${\mathit{s}}_{\mathit{i}}$. Thirdly, the sentiment score of the post will be calculated as follows:

Where I(j) is a set including all words found in jth emotion category of the Ontology for a post, I(1) means the words found in the first emotion category of the Ontology which is the category of “Happiness”. The algorithm assumes that each post only presents one kind of emotion which is the one that has the overall highest intensity value or sentiment score. As a result each post in the database will be attached an tag which identifies the category it belongs to such as “Happiness”. In the fourth step, the frequency of the posts belong to the same emotion category is calculated for each day. The value will be represented as ${\mathit{x}}_{\mathit{i}\mathrm{,}\mathit{t}}$ where i belongs to one of the 7 emotion categories and t means date t in the observation period. Then the time series for emotion i can be represented as ${\mathit{X}}_{\mathit{i}}$.

After processing the raw text data of Weibo posts, for each topic discussed, the model generates 7 time series. Each time series represent the movement of one dimension of the public mood or emotion such as “Happiness”. Another data source is the time series of the stock price. The time series of the stock close price is downloaded from Yahoo!Finance and other public websites. As the emotion time series are associated with different stocks. the close prices from 1st Jan 2015 to 31th Mar 2016 for 4 different stocks including the Hang Seng Index (HSI), Shanghai Stock Exchange Composite Index (SSECI), Hong Kong Exchanges and Clearing Limited corporation stock (stock code is 388) and Shanghai Connect Index (stock code is 000159) are downloaded.

To validate the ability of the proposed method to capture various aspects of public moods, the algorithm is applied to the Weibo posts published from 1st Jan 2015 to 31th Mar 2016. The distribution of the posts indicates that the main part of the positive topic posts such as “interest rate cute” are the posts with the positive emotion such as “Happiness” and “Good” and vice versa. The distribution of the posts is shown in Fig. 2.

As mentioned, the interest rate and reserve rate cut will booth the economy and the stock market. Actually, the China government has cut the interest rate or reserve rate three times during 1st Jan to 31 July 2015 (on 28th Feb, 10th May, 27th June). As a result, most posts about the interest rate and reserve rate cut locate in the positive emotions such as “Happiness” and “Good” which is 85% of all related posts. The distribution of the posts related to SH-HK and SZ-HK connect is similar. However the distribution of the posts related to Greece government-debt crisis is different. As a lot of people is worried about the future of the European Union financial system and whether the Greece crisis will cause an international government debt crisis, there is almost 27% posts is classified as “Disgust” emotion. There are also a lot of posts locating in the “Good” category as the Greece government-debt problem is solved in July. As the posts are mainly distributed in the three categories:“Happiness”, “Good”, “Disgust”, this paper will discuss the correlation between the stock market and these three emotions.

Figure 3 shows the two public mood time series after the data processing which are “Happiness” time series and “Good” time series for the topic “interest rate and reserve rate cut”. On 10th May, China central bank announced a new round of interest rate and reserve rate cut which is an event that may have a unique, significant and complex effect on the public mood. The result demonstrates that the time series generated through the approach successfully identifies the public emotional response to the announcement of interest rate cut. It has a significant but short-lived up tick in positive sentiment “Happiness” and “Good” on that day. Actually after the announcement, the sentiment score increases a lot sharply. In addition, before the announcement the sentiment score starts to climb up as there are some rumors appearing in the market which mentioned that the government will cut the interest rate soon and the expectation of the new round interest rate cut becomes higher.

To visualize the correlation between the public mood time series and the stock price movement, the data set for the topic “Shanghai-Hong Kong Stock Connect (SH-HK connect) and Shenzhen-Hong Kong Stock Connect (SZ-HK connect)” for the period 1st April to 31th May is used. The reason for using this time period is that on 8th April, Chinese investors, for the first time, used the entire 10.5 billion RMB daily quota in a cross-border programme buying Hong Kong stocks, boosting turnover under the Shanghai-Hong Kong Stock Connect to a record. The buying makes the Hang Seng Index increase 3.8 percent to its highest level in nearly seven years.

Two time series are plotted in Fig. 4. First one is the daily close price of Hang Seng Index. And the Second one is the time series for the public mood state “Happiness”. To normalize and compare the value in these two time series, the z-score of their delta value is calculated. As shown in Fig. 4, both time series frequently overlap or on the same direction. The movement of the two time series is similar according to their peaks and bottoms. On 8th April, both time series climb to the peak with almost the same z-score which is more than 1.5 standard deviations comparing to the average (Hang Seng Index: 1.74 and Happy Mood: 1.68). These behaviors of the time series demonstrate that the “Happiness” time series generated by the approach correctly indicate the public mood change to some positive events and is related with the Hang Seng Index price. The correlation between these two time series will be discussed in more details in next section.

Many economic time series are unit root process [25]. Therefore, the Augmented Dickey-Fuller (ADF) test is applied to each time series for identifying the unit root process. As the generated time series are proved to be unit root process, first differencing method is used to minimize the chance of spurious relationships. The public mood time series that is not covariance stationary is converted to ${\mathit{D}}_{\mathit{i}}$,

To avoid the effect of various magnitudes, the public mood time series ${\mathit{D}}_{\mathit{i}}$ is normalized by computing the z-scores:

Where $\overline{{\mathit{D}}_{\mathit{i}}}$ is the average of ${\mathit{D}}_{\mathit{i}}$ and ${\mathit{\sigma }}_{\mathit{i}}$ is the standard deviation of ${\mathit{D}}_{\mathit{i}}$ over time period n. In the experiment, as it will discuss the moving average of stock price, n is defined as 10 and $\overline{{\mathit{D}}_{\mathit{i}}}$ is the 10 days moving average of the sentiment score change.

The time series of stock prices is defined to reflect the daily changes in stock market value, i.e, the delta value between the close price of a stock in day t and day $\mathit{t}-1$. Z-score computing is also applied to the delta value which helps to standardize the stock price change. ${\mathit{P}}_{\mathit{i}\mathrm{,}\mathit{t}}$ equals to the z-score value of the delta. After establishing the public mood time series, to discuss whether variations of the public mood states affect the price in the stock market, this paper will apply the Granger causality analysis to the time series generated by the new Chinese sentiment analysis method, ${\mathit{Z}}_{\mathit{i}}$, and the stock close price time series, ${\mathit{P}}_{\mathit{i}}$. This paper performs the Granger causality analysis according to the linear model as shown in Eq. (4) for the period of time between 1st Jan to 31st July 2015. As there are three topics to discuss and these topics are related to different stocks, different linear regression experiments between specific stock price time series and the public mood time series will be conducted.

Where ${\mathit{P}}_{\mathit{t}}$ is the z-score of the stock day close price difference between day t and $\mathit{t}-1$, ${\mathit{Z}}_{\mathit{t}}$ is the z-score of the sentiment score change of a specific public mood dimension or emotion such as “Happiness”. The value of n is set to 7, as previous research [3] has demonstrated that public mood on day $\mathit{t}-3$ has the highest correlation with stock price on day t.

First the topic “SH-HK and SZ-HK connect” is discussed. Table 1 presents the linear regression result between the time series for the close price of Shanghai Connect Index (stock code: 000159) and the time series of the emotion “Happiness” related to the topic “SH-HK and SZ-HK connect”. Shanghai Connect Index is the composite index including all stocks that can be traded through the SH-HK connect by international traders in Hong Kong. As a result, the price movement of the Shanghai Connect Index is directly affected by traders’ opinions or emotions about SH-HK connect. The result shows that the lag value of ${\mathit{Z}}_{\mathit{t}-3}$ has a P-value equal to 0.028515 which is lower than the significance level 0.05. Based on the result of the Granger causality, the null hypothesis that the mood series do not predict the price movement of Shanghai Connect Index can be rejected, i.e. ${\mathit{\gamma }}_{1\mathrm{,}2\mathrm{,}3\mathrm{,}\dots \mathrm{,}7}\ne 0$. However only the lag with 3 days has significant causal relations with price movement of the Index which is similar as Bollen’s research [3] on Twitter post. In his research, he mentioned changes of public mood match shifts in the stock price that occur 3–4 day later. This behavior can be explain as that it may take some time for the public mood in social media websites to gain a momentum that will affect investors decision in the real market. Other 6 emotions time series are not significantly correlated with the time series of Shanghai Connect Index according to the p-value.

To further demonstrate the forecast ability of the “Happiness” time series to other stocks, the p-value of the Granger causality correlation with 2 more stocks are calculated as well. The 2 stocks are HSI and the Hong Kong Exchanges and Clearing Limited corporation stock (stock code is 388). They are highly related to the topic “SH-HK and SZ-HK connect”. HSI is the composite Index of Hong Kong stocks which will be affected by the SH-HK connect as more capitals can be invested on Hong Kong market. Hong Kong Exchanges and Clearing Limited Company is the company that provides the service of SH-HK connect. More trader are investigating through the SH-HK connect, more commissions it can receive. As it can gain more profits, the valuation of the stock is changed and the price will be increased as more investors will be attracted. The results are shown in Table 2. According to the p-value 0.045231, the lag value of ${\mathit{Z}}_{\mathit{t}-3}$ has significant correlation with the price movement of HSI. Similar as HSI, the price movement of 388 is significantly correlated with the lag value of ${\mathit{Z}}_{\mathit{t}-4}$. The p-value is 0.010188 which is much lower than the significance level 0.05. The null hypothesis can be rejected and the “Happiness” time series is predictive of the stock price movement including Shanghai Connect Index, HSI and the stock of Hong Kong Exchanges and Clearing Limited.

To demonstrate the predictive ability of public mood states with different topics, the Granger causality analysis is also applied to other topics. For the topic “Greece Government-Debt Crisis”, the public mood time series other than “Disgust” is not significant correlated with the price movement of HSI. Only the lag value ${\mathit{Z}}_{\mathit{t}-2}$ of the time series of “Disgust” has significant causal relation with the price movement of HSI considering the significance level 0.1. The experiment result is shown in Table 3.

For the topic “interest rate and reserve rate cut”, as it is a method that China government used to booth the China stock market, the paper will discuss the correlation between the public mood time series and the close price of Shanghai Stock Exchange Composite Index (SSECI). The results are shown in Table 4. According to the p-value (0.007864), the lag value of ${\mathit{Z}}_{\mathit{t}-3}$ for emotion “Happiness” is highly correlated with the price movement of SSECI considering the significance level 0.05. And for the “Good” time series, considering the significance level 0.1 and p-value 0.0676, the lag value of ${\mathit{Z}}_{\mathit{t}-3}$ is also correlated. In addition, the lag value of $\mathit{t}-2$ and $\mathit{t}-3$ of the “Disgust” time series have significant causal relationship with the SSECI value as well as the p-value is 0.041898 and 0.008356. The experiments are not testing the actual causation but whether one time series has predictive information about the price movement or not. The result shows that the null hypothesis that the stock price movement is not correlated with the change of public mood extracted from Weibo can be rejected with high level confidence and the time series of “Happiness”, “Good” and “Disgust” could be used to predict the price movement of SSECI. The reason why only “Happiness”, “Good”, “Disgust” are correlated with the stock price is because of the characteristics of Weibo. Most of the post are related and classified to these 3 dimensions through the proposed new Chinese sentiment analysis model. Another reason is that investors’ decision is more likely be affected by strong emotions which has high polarity such as “Happiness” and “Disgust”. In addition, the spread of these strong emotions such as “Happiness” in the social media websites is more quickly as it will attract more attentions than other kinds of emotions. As a result, there will be more posts with the same strong emotions.

Although Granger causality analysis result has proved that certain public mood dimensions are correlated with the specific stock price movement in China, the analysis is based on linear regression. However the relation between public mood and the stock market is more likely to be non-linear. The ability of artificial neural network in solving the non-linear time series modeling issue has already been proved [16,18]. Some researchers have already applied the neural network to the topic of stock price prediction [17]. Zhu et al. indicate that neural network models augmented with trading volumes leads to improvements in forecasting performance under different terms of forecasting horizon [36]. In addition, some research has been done to forecast the exchange rate using some specific type of neural network such as self-organizing dynamic neural network [19]. In previous literatures [36] trading volume is considered as an augmentation to improve the performance of neural networks. In this paper, whether the public mood states from Weibo could reinforce the forecasting ability of neural network will be discussed.

As the stock market in China has already been one of the largest market in the world, it is important to understand its behavior according to the public mood states. This paper investigates whether public mood states derived from large-scale collection of Weibo posts on weibo.com is correlated or predictive of the price of different stocks in China market. The results demonstrate the public mood states can be indeed tracked using the proposed new Chinese sentiment analysis method on the large-scale Weibo posts. In addition, the Granger causality analysis results indicate that the null hypothesis that the public mood is not correlated with the stock price can be rejected and the time series of public mood states is predictive of the price movement of different stocks. Among the 7 observed emotions, only some are Granger causative of the stock price movement. The emotions “Happiness” and “Disgust” have significant causative relationships with different stock prices. Movement of these two emotions is correlated with the price movement which occurs in 3–4 days. These 3 days time lag is also discussed in the research of using public mood in Twitter to predict stock price [3]. A nonlinear autoregressive model with exogenous sentiment inputs furthermore demonstrates the possibility of using public mood states to predict the stock price movement in China. The public mood states help to improve the prediction accuracy even using the basic neural network model. Given the performance, if more related topics for a stock are found out, the accuracy could be higher considering more completed topic-based sentiment inputs. Regarding the model, if more sophisticated and complicated model is implemented according to the topic-based mood, it is possible that the prediction result would be even better.

In the future, more experiments will be conducted to understand why the time lag exists and how to avoid and use it for real time algorithm trading. Other important factors will be examined in future research such as the geo-location effect and spam detection. In addition, in this paper, it only discusses the relationship between the close price of stocks and the daily movement of public mood. How the intraday public mood change affects the real-time stock price movement will be discussed in future research as well.