Time Series Analysis

A time series \(\{X_t\}\) is weakly stationary if:

1. Constant mean: \(\mathbb{E}[X_t]=\mu\) for all \(t\)
2. Constant variance: \(\mathrm{Var}(X_t)=\sigma^2\) for all \(t\)
3. Autocovariance depends only on lag:

\[\gamma(h)=\mathrm{Cov}(X_t,X_{t+h})=\mathbb{E}[(X_t-\mu)(X_{t+h}-\mu)]\]

It does not depend on calendar time \(t\).

Example: \(\mathrm{Cov}(X_5,X_6)=\mathrm{Cov}(X_{100},X_{101})\) because both have lag \(h=1\).

Intuition: only the lag matters, not when it occurs.

Many time series models assume stable moments and dependence over time. Without stationarity, parameters can drift, inference can break down, and forecasts can be unreliable.

Autocorrelation measures the total correlation between a series and its lagged values:

\[\rho(h)=\mathrm{Corr}(X_t,X_{t-h})=\frac{\gamma(h)}{\gamma(0)}\]

It includes both direct effects and indirect effects transmitted through intermediate lags.

Partial autocorrelation measures the direct relationship between \(X_t\) and \(X_{t-h}\) after removing the linear effects of intermediate lags \(X_{t-1},\dots,X_{t-(h-1)}\).

Formally, it is \(\phi_{hh}\), the coefficient on \(X_{t-h}\) in:

\[X_t=\phi_{h1}X_{t-1}+\cdots+\phi_{hh}X_{t-h}+\varepsilon_t\]

ACF: total correlation (what propagates).

PACF: direct correlation (what actually causes it).

Patterns:

• AR(p): PACF cuts off after lag \(p\), ACF decays.
• MA(q): ACF cuts off after lag \(q\), PACF decays.
• ARMA: both decay gradually.

A random walk is:

\[X_t=X_{t-1}+\varepsilon_t\]

Shocks persist, variance grows with time, and the process is non-stationary.

Stationary processes mean-revert and shocks decay over time. Random walks do not mean-revert; shocks have permanent effects and variance increases with time.

Mean reversion implies deviations from a long-run mean tend to shrink over time. Stationary AR processes are mean-reverting; random walks are not.

An autoregressive model is:

\[X_t=\phi_1X_{t-1}+\cdots+\phi_pX_{t-p}+\varepsilon_t\]

The current value depends on past values.

When the roots of the characteristic polynomial lie outside the unit circle, so shocks decay rather than persist/explode.

A moving average model is:

\[X_t=\varepsilon_t+\theta_1\varepsilon_{t-1}+\cdots+\theta_q\varepsilon_{t-q}\]

The series depends on past shocks.

ARMA combines AR and MA components and is used for stationary time series with both persistence (AR) and shock dynamics (MA).

Differencing helps remove trends/unit roots to achieve stationarity:

\[\Delta X_t=X_t-X_{t-1}\]

An ARIMA model is \(\text{ARIMA}(p,d,q)\), where \(d\) is the number of differences needed for stationarity.

How to choose \(p\) and \(q\):

• Choose \(d\) first (difference until approximately stationary; avoid over-differencing).
• Use PACF to suggest \(p\) (direct lag effects).
• Use ACF to suggest \(q\) (shock persistence).
• Compare candidate models using AIC/BIC and prefer a parsimonious model with stable diagnostics.

Parsimony: if two models fit similarly, prefer the simpler one (fewer parameters).

Over-differencing can remove meaningful long-run information, increase noise, and induce artificial negative autocorrelation.

To assess whether a series is non-stationary due to a unit root versus stationary around a mean (and possibly a trend).

Unit root tests often have low power. Failure to reject the null suggests strong persistence and cautions against assuming stationarity.

Non-stationary series are cointegrated if a linear combination of them is stationary, implying a stable long-run equilibrium relationship.

Correlation measures short-run co-movement; cointegration captures a long-run equilibrium relationship even if short-run deviations occur.

Step 1 (long-run equilibrium):

\[y_t=\beta x_t+u_t\]

where \(u_t\) is the equilibrium error.

Step 2 (short-run dynamics + correction):

\[\Delta y_t=\alpha\,(y_{t-1}-\beta x_{t-1})+\gamma_1\Delta y_{t-1}+\gamma_2\Delta x_{t-1}+\varepsilon_t\]

Short-run changes respond to the lagged equilibrium error.

The term \((y_{t-1}-\beta x_{t-1})\) measures deviation from long-run equilibrium. The coefficient \(\alpha\) is the speed/direction of adjustment. Typically \(\alpha<0\) implies deviations are corrected over time.

Structural breaks are changes in the data-generating process (e.g., shifts in mean, variance, or relationships) over time.

They violate stationarity assumptions, bias parameter estimates, and degrade forecast performance if unaccounted for.

Preserve time order using expanding or rolling windows. Random shuffling can cause leakage and invalid evaluation.

Look-ahead bias occurs when future information leaks into model training/feature construction, producing overly optimistic performance.

MAE measures average absolute error and is more robust to outliers. RMSE penalizes large errors more heavily (squares errors), emphasizing tail mistakes.