Exploring Dow Index Variance using Data Mining Technology

Exploring Dow Index Variance using Data Mining Technology

Abstract. Trends in the Dow Jones Industrial Average (DJIA) are often used as an indicator of future economic tendency.The DJIA data complexity cannot be effectively analyzed using the common statistical tools such as Fast Fourier Transform (FFT) to study the time period hidden in complex datasets. Thirty-three years’ DJIA data, from 1986 to 2018, were downloaded in Yahoo Finance. Applying “Peak Analysis”, the primitive period analysis method, a quasi-ten-year cycle was revealed. Downward trend of the stock and economic markets can be predicted after ten years of steady development. This conclusion coincides with other well-known experts.    

Modern data mining technology offers the possibility to achieve accurate and reliable predictions. The Google open source, TensorFlow, will be experimentally adopted in our future stock prediction project. Data ingesting, pro-processing, training TensorFlow, stock predicting, post-processing and decision making are the schemas of the stock trading system currently under development.  

                Key words:  DOW, Fast Fourier Transform, Peak Analysis, Data Mining, Deep Learning

 

1  Introduction

The Dow Jones Industrial Average (DJIA) index, also known as the Dow, is a stock market index  that reflects on how the thirty largest public companies traded in the market behaves. The DJIA average is not a weighted mean, nor is it representative of the market capitalization of each individual company included within the index. It is simply used to indicate the average stock price, per share, for each company. The DJIA is the sum of the prices of all 30 stocks divided by the Dow Divisor.  The divisor is adjusted in case of stock splits, spinoffs or similar structural changes, to ensure that such events do not in themselves alter the numerical value of the DJIA. As of the end of June 2018, the Dow divisor is 0.14748071991788. It means that for every $1 of change in price for any given stock within the index, the average is equal to a 6.781-point movement in the market. The timely Dow index is critical and essential for the traders to make decision when to buy or sell stocks.  

Get Help With Your Essay
If you need assistance with writing your essay, our professional essay writing service is here to help!
Essay Writing Service

    The stock variance seems erratic, volatile. On Christmas Eve of 2018, the DJIA finished down 653 points, or 2.9%, representing its worst decline on the session prior to Christmas in the 122-year-old blue-chip gauge’s history.  Despite this occurrence, the next day the market rallied strongly in a clear display of large volatility .It rallied in volatile trade on Wednesday, the first full trading session after the worst Christmas Eve trading session in history. The index ended 1,086 points higher in the last hour of trade, an increase of 5 percent over Monday’s abbreviated session. It is the biggest ever point jump in U.S. stock history. The irrational stock market brings huge risk and challenge in its investment to the trader while they pursue hard to profit. On the flip side, a non-efficient market presents great opportunity as well. Through more than one hundred years’ Dow index data recorded, the data mining technologies including commonly used statistic methodologies can be attempted to predict the future stock price movements. This trend analysis is based on the idea that what has happened in the past gives the traders the ideas what will happen in the future. The traders can perform decision making from trend analysis results to have maximum profits.      

    The Dow index is driven by a handful of public companies.  Currently, the biggest components of the Dow are Boeing (NYSE:BA); McDonald’s (NYSE: MCD); Goldman Sachs (NYSE:GS); United Health (NYSE:UNH); Exxon Mobile (NYSE: XOM); Intel (NASDAQ: INTC); and Apple (NASDAQ: AAPL) etc.  All the attention given to the Dow hitting new record highs, or falling to new lows, can certainly be regarded as reflecting the performance of these 30 companies. Even if those 30 public companies are only a fraction of the overall U.S. economy, the Dow index is still the reflectance generated from the whole economic, political and human social activities. Following lead of the natural science, the social science has been categorized in a state of chaos [1]. With the new discoveries, the uncertainty, nonlinearity, and unpredictability in the natural realm had piqued the interest of social scientists. Measuring chaos and discovering the nonlinear dynamics become the new research subjects.  With the development of advanced computing technologies including large hard disk storage, fast Central Process Unit (CPU), and superior Graphics Processing Unit (GPU), the deep learning has been rapidly applied in the recent tens of years.  The deep learning and data mining technologies had been used for stock market returns [2]. 

    In this short paper, we downloaded 33 years DJIA data from 1986 to 2018 in Yahoo Finance and analyzed the primary period cycles from the view of statistics. The future plan to utilize the Google’s open source TensorFlow to develop a stock market predicting and decision system is added.

2  Data

 

The 33 years’ Dow index, DJIA, were downloaded in Yahoo Finance. The trade dataset covers “open”, “high”, “low”, “close”, “Adj close” and “volume”.  “open” is the data value at the starting each day. “high” and “low” represent the highest and lowest record traded on the same day. “close” is the close price adjusted for splits. “Adj close” stands for “adjusted close price adjusted for both dividends and splits”. “volume” is the total amount of trading happened in the whole day.  Table 1 is example trading record on Dec. 31, 2018. The 33 years’ daily data are adopted from Jan. 1, 1968 to Dec. 31, 2018.

Table 1. DJIA trade on Dec. 31, 2018

           Date

Open

High

Low

Close

Adj Close

Volume

Dec 31, 2018

23153.94

23329.49

23118.30

23248.77

23248.77

185202246

 

3  Dow index period exploring with statistics technology

 

Many factors could impact the stock market and brings it up and down. Among all these factors, economy is the main element. The business cycle is the pattern of expansion, contraction and recovery. It is mainly measured by the Gross Domestic Product (GDP) and the unemployment. GDP rising and unemployment shrinking means in the expansion phases, while reversing in periods of recession. The economy thus can be observed to go through four periods – expansion, peak, contraction and trough. The intrinsic relationship between stock market and the economic period leads the existing of the period of stock. The period could be investigated, analyzed through the trade data peaks and troughs. Usually, a period is measured through the two continuous peaks. Discovering the stock period turns to count the peaks of the long series data.

     Fig.1 is the daily DJIA from 1986 to 2018 with 8315 values during 33 years. The Dow index showed upside down significantly. Obviously, no period can be easily drawn from the first impression. The regular statistics tools like Fast Fourier Transform (FFT), single and multiple regressions cannot be effectively applied to analyze periods from those long DJIA datasets. The only left choice to find the period from those intricate data is from data peaks. A local peak is defined as a stock data which is either larger than the two neighboring samples. Counting a peak value is only through comparing its adjacencies, it has nothing to do with other remote data.  Xn is a peak data if Xn>Xn-1 and Xn>Xn+1, n can be the second to the second last value in the dataset. Fig. 2 is the peaks plotted from the original DJIA data performed the first step. 2122 peaks were selected. The periods are still not easily perceived. Fig. 3 is the peaks selected using the peak data from the first step. 510 peaks were selected, greatly deceased from last step. Fig. 4 plots the 118 peaks performed the third step. Fig. 5 displays 20 peaks the fourth step. Fig. 6 only has three peaks left while processing till the last fifth step. The periods do not exist among the three points. Therefore, the 10 year period is explored using the 20 peaks in Fig. 5 from the step four. 

   As of January, 2019, the stock was right after the biggest peak within 33 years, and the stock became even more bumpy. Volatility was driven by signs of a global economic slowdown, concerns about monetary policy, political dysfunction, inflation fears, trade wars, and worries about increased regulation of the technology sector. Depending where we look from Fig. 1, the stock is currently in the stage of down tendency, or luckily and hopefully, it approaches the stage before rolling over to the next upward cycle.

Fig. 1.  Daily DJIA from 1986 to 2018

Fig. 2. DJIA Peaks during 33 years (Step 1)

 Fig. 3. DJIA Peaks during 33 years (Step 2)

Fig. 4.  DJIA Peaks during 33 years (Step 3)

Fig. 5. DJIA Peaks during 33 years (Step 4)

Fig. 6. DJIA Peaks during 33 years (Step 5)

4  Dow index and stock prediction with deep learning technology

The stock pattern with certain period is meaningful from a long time perspective. Pattern lets us know some future stock variance, but it is not very helpful to the stock traders who need the timely product. Predicting stock and being aware of the actual stock value in advance has practical significance for the traders to make decision to buy or sell stocks for profiting most. However, the movement in the stock exchange depends mainly on capital gains and losses, most people consider the erratic stock market unpredictable.

    The rapidly growing artificial intelligence (AI) technique is a computer based system to interpret external data, to learn from such data, and to use those learnings to achieve specific goals and tasks through flexible adaptation. AI has been successfully used in understanding human speech, competing at the highest level in strategic game systems, autonomously operating cars, cancer research, and intelligent routing in content delivery networks and military simulations. AI has been applied to a diverse range of topics: cancer research, self-driving cars, and image recognition.

AI has been successfully applied in a diverse multitude of fields. The stock market can also be viewed as a particular artificial intelligence problem. It can be considered as an intelligent treatment of past and present financial data in order to predict the stock market future behavior [3,4,5]. Data mining techniques are used to evaluate past stock prices and acquire useful knowledge through the calculation of some financial indicators. Data mining techniques use past data points, and gather useful information to help predict future behavior. Attempts to forecast the stock price movement that is generally subject to many forces, GDP growth, employment rate, interest rate, monetary policies such as weakening or strong currency, high or low tax rate, corporate earnings, business and consumer spending, new technology appearance and development, political and social upheavals etc. [6]. All these factors could be used as the inputs to feed a deep learning system.

     Google TensorFlow Application Programming Interface (API) is used as the machining learning model to predict the future stock price and provide decision making suggestion [7]. TensorFlow is an open source software library for data flowing programming. It is a flexible interface for expressing, implementing and executing machine learning algorithm. It was borne from real world experience in conducting research and Google products and services, and this system has been used for conducting research and for deploying machine learning systems into production across more than a dozen areas of computer sciences and other fields. Google TensorFlow programs and source codes are available at www.tensorflow.org.  We ran it on a Linux platform.

     From the past tons of data to find the reliable variables and parameters during the deep learning to mostly achieve the best possible goals, training is a critical and indispensable step. Training plays more important roles in selecting the algorithm and determining more significant variables in prediction through looking at the past performance. Pre-processing the historical data is important before the data is ingested into the machine learning system. Data-gathering is often loosely controlled, resulting in out-of-range values, impossible data combinations, missing values, etc. Analyzing data that has not been carefully screened can produce misleading results. Thus, making quality control (QC) data is first and foremost before running training the learning system. Any good learning system is a concept hypotheses model trained from the post backgrounds. It can only better represents the past events. The generated data as the future stock price should be evaluated, interpreted, incorporated into other possible evaluation system to check for potential conflicts with previously induced stock variance. Post-processing cannot be left out to avoid any possible risk. Decision making is the last step of a stock trading system. Trading is not similar to find or predict any pattern from the long time datasets. Facing the instant stock change, purely relying on any computer based AI system is to be prudent. Qian [5] investigated the predictability of DJIA index with some inductive machine-learning classifiers, pointed out the prediction accuracy up to 65 percent. The role from the human being cannot be disregarded. The traders should make the final choice to buy or sell based their experiences or another independent decision making system. Table 2. is the schema of our future stock deep learning and trading system.        

Table 2. Schema of future stock deep learning and trading system

5 Conclusions and discussions

The DJIA is an important index to describe the economy variance and its future trend. The conventional and widely used statistical analysis tools such as FFT, Exponential smoothing etc. cannot be effectively used to explore the trend from the complicate DJIA dataset. “Peak analysis”, the very primitive method through measuring two continuous peaks, is applied on analyzing 33 years DJIA data, from 1986 to 2018, downloaded from Yahoo Finance.  A quasi-ten-year period was revealed existing in stock market. This quasi-ten-year period is an approximate value instead of a precise number. Over about a decade steady growth, the stock market faces downward. 

     Several famous economists cited by Klebnikov [8] made coincident statements similar to ours. Ray Dalio, hedge fund manager, said “the probability of a recession prior to the next presidential election would be relatively high, 70%”. JPMorgan’s real time recession forecast model suggests the chance of a market downturn at 70% by 2020. The Duke University/CFO Global Business Outlook survey released 80% of U.S. chief financial officers a recession will hit the economy by 2020.    

    As of May, 2019, the stock had just climbed near the top within 33 years, and the DOW has been over 26000. Not everyone trusted the economy will fall in any recession depending on the strong economy performance, such as the lowest unemployment since 1950. Larry Kudlow [8], National Economic Council director, kept optimistic, “I know there’s a lot of permission out there. I do not share that permission.”  However, the economic development follows some certain regular pertain as we pointed a quasi-ten-year period exists in the DOW variance. Volatility was been driven by signs of a global economic activities including concerns about monetary policy, political dysfunction, inflation fears, trade wars etc.. The stock is in the stage of downward tendency. Facing this tough situation, investment should be wary.

     Stock analysis deals with the study of these patterns. It uses different techniques and strategies, mostly automatic that trigger buying and selling orders depending on different decision making algorithms. Therefore it can be viewed as an artificial intelligence problem in the data mining field. The analysis have techniques from artificial intelligence applied to it , due to the automated nature of modern stock trading. Deeping learning has been revolutionizing virtually every aspect of financial and investment decision making. It has been employed world widely to tackle difficult tasks involving intuitive judgement or requiring the detection of data patterns which elude conventional analytic techniques. Neural networks are already being used to trade the securities markets, to forecast the economy and to analyze credit risk.

Currently, machine learning is being applied to finance in areas including, but not limited to economic forecasts and credit risks. The Google open source, TensorFlow, has been downloaded and installed on our Linux platform, it will be trained and applied in our future deep learning project to predict the stock movement including the Dow index as the first experimental development. The input data has to be QCed in the pro-pressing stage to remove any erratic value. Post-processing the forecasted stock value cannot be overlooked even if many observers believe neural networks will eventually outperform even the best traders and investors. Date ingesting, pro-processing, training the deep learning system and making prediction, post-processing and decision making are the schema or our future stock trading system. After system processing, the decision can be made to buy or sell the stock.

 

References:

Kiel, D., Elliott E.: Chaos theory in the social sciences: foundations and applications. The University of Michigan    Press (2004) 

Enke, D., Thawornwong, S.: The use of data mining and neural networks for forecasting stock market return. Expert          Systems with Applications, 29, pp. 927-940 (2005)

Wang, F.: Mining stock price using fuzzy rough set system, Expert Systems with Applications, 24. Pp. 13-23 (2003)

Trippi, R., Turban E.: Neural Networks in Finance and Investing: Using Artificial Intelligence to Improve Real  World Performance. McGraw-Hill, Inc., New York, NY, USA (1992) 

Qian B., Rasheed K.: Stock market prediction with multiple classifiers. Applied Intelligence. Vol. 26. Issue 1, Pages. 25-33 (2007)

O’Sullivan, A., Sheffrin, S.: Economics: Principles in Action. Upper Saddle River, New Jersey 07458: Pearson Prentice Hall. p. 290. ISBN 0-13-063085-3 (2003)

Abadi M.,Agarwal A., Barham P., et al.:  TensorFlow: Large-scale machine learning on heterogeneous systems (PDF). TensorFlow.org. Google Research. Retrieved (2015)

Klebnikov, S.: Recession 2020? These Experts Say It’s Possible. Money, March, pp. 17 -19 (2019).