Exploiting market state data for forecasting stock prices: an analysis using predictive algorithms for the Dow Jones and Nasdaq indexes

Abstract

Abstract Predicting asset price movements is relevant for investment analysis, risk monitoring, and economic forecasting. This study evaluates whether a cross-sectional representa‐ tion of the market state can improve short-horizon index forecasting relative to a uni‐ variate historical-price benchmark. The paper is framed as a forecasting-comparison study, while market-efficiency testing, asset-pricing restrictions, portfolio efficiency, and trading profitability are treated as complementary evaluation layers. We com‐ pare market state data and historical price data for the Dow Jones Industrial Average and the Nasdaq-100. The dataset spans 2005 to 2024, and the empirical evaluation focuses on two single-year experiments: 2019, as the base pre-COVID implementation period, and 2024, as a post-COVID robustness check in a bullish market environment. We introduce a Kalman-inspired linear forecasting framework that uses deterministic matrix estimation and dimensionality reduction. Three market state estimation variants are compared with a univariate time series Kalman benchmark. The findings show that the relative usefulness of historical price data and market state data depends on the forecasting target and the market context, consistent with the no free lunch intuition that performance depends on the match between method, target, and data context.