Machine Learning in Finance: Applications & Getting Started 2026

What Is Machine Learning in Finance?

Machine learning in finance is the use of algorithms that learn patterns from data to make predictions or decisions - without being explicitly programmed with rules for every scenario. Instead of a human analyst writing "if the P/E ratio is below 15 and earnings growth exceeds 10%, buy", a machine learning model ingests thousands of data points, finds statistical relationships on its own, and produces a prediction: the probability that a stock will go up, the likelihood that a loan will default, or the expected volatility of an asset over the next month.

The financial industry has used quantitative models for decades. What's changed is scale and flexibility. Traditional quantitative finance relies on models with clear mathematical foundations - the Black-Scholes option pricing model, mean-variance portfolio optimisation, linear factor models. These are powerful but rigid. They assume specific functional forms and distributions that real markets don't always follow. Machine learning relaxes those assumptions. A gradient boosting model or a neural network can capture nonlinear relationships, interaction effects, and regime-dependent behaviour that a linear regression would miss entirely.

That said, machine learning isn't magic. Financial data is noisy, non-stationary, and adversarial - other market participants adapt to any signal that becomes known. The signal-to-noise ratio in financial prediction is far worse than in computer vision or natural language processing, where ML has had its most dramatic successes. A model that achieves 51% accuracy on next-day stock direction can be enormously profitable at scale, but building one that reliably exceeds 50% after transaction costs is genuinely difficult.

In 2026, machine learning is used across virtually every area of finance: trading, risk management, fraud detection, credit scoring, portfolio construction, compliance, and client analytics. The firms that use it best tend to treat it as one tool in a broader quantitative toolkit rather than a replacement for domain expertise.

---

How ML Is Used in Finance

Machine learning applications in finance extend well beyond trading. Here are the main areas where ML models are deployed at scale in 2026.

Trading and Alpha Generation

This is the application that gets the most attention. Quantitative hedge funds and prop trading firms use ML models to predict short-term price movements, identify mispricings, and generate trading signals. The models typically combine price data, fundamental data, and alternative data sources to produce alpha scores - predictions of expected excess returns.

The reality is more nuanced than "AI picks stocks." Most successful implementations use ML for specific components of a larger system: feature selection, signal combination, or execution optimisation. A firm might use gradient boosting to combine 200 weak signals into a single composite alpha, while the portfolio construction and risk management layers remain traditional optimisation. For a practical overview of algorithmic approaches, see the AI trading bot guide.

Risk Management

ML models excel at modelling tail risk and complex dependencies. Traditional Value at Risk (VaR) models assume normal distributions and linear correlations - assumptions that break down during market crises. ML approaches, particularly tree-based models and neural networks, can capture nonlinear dependencies and fat-tailed distributions that better reflect real market behaviour. Banks and asset managers use ML for stress testing, counterparty credit risk estimation, and real-time risk monitoring.

Fraud Detection

This is one of the most mature ML applications in finance. Payment processors and banks deploy ensemble models (typically random forests and gradient boosting) that score every transaction in real time. The models flag unusual patterns - a credit card used in two countries within an hour, a sudden change in spending behaviour, transaction amounts that deviate from historical norms. False positive rates have dropped substantially as models have improved, saving billions in fraud losses while reducing friction for legitimate customers.

Credit Scoring and Lending

Traditional credit scoring uses a handful of variables - income, employment history, existing debt, payment history. ML models can incorporate hundreds of features, capturing subtle patterns that improve default prediction. Online lenders like Zopa and Funding Circle use ML-based credit models that consider non-traditional data alongside conventional metrics. The challenge is explainability: regulators require that lending decisions can be explained to applicants, and a complex neural network's reasoning is harder to articulate than a simple scorecard.

Portfolio Construction and Optimisation

ML methods are increasingly used to estimate covariance matrices, forecast expected returns, and determine optimal portfolio weights. Techniques like hierarchical risk parity (which uses clustering algorithms to group correlated assets) and ML-enhanced factor models offer improvements over classical mean-variance optimisation, particularly when the number of assets is large relative to the available historical data.

NLP and Sentiment Analysis

Natural language processing models analyse news articles, earnings call transcripts, social media posts, central bank communications, and regulatory filings to extract sentiment signals. Transformer-based language models fine-tuned on financial text can classify sentiment with reasonable accuracy. These signals are typically combined with price and fundamental data in a multi-signal alpha model rather than traded in isolation.

Alternative Data Processing

Satellite imagery of car parks, shipping container movements, web traffic data, app download statistics, credit card transaction aggregates - the universe of alternative data has exploded. ML models, particularly computer vision models for satellite data and NLP models for text data, are essential for turning these raw data sources into structured features that can feed a trading or investment model.

---

ML for Trading: What Actually Works

There's a lot of hype around ML for trading, so an honest assessment is useful. Here's what practitioners have learned after a decade of applying modern ML to financial markets.

Signal is weak. In image classification, a model might achieve 95%+ accuracy. In next-day stock return prediction, anything above 51-52% accuracy is exceptional. Financial signal-to-noise ratios are extremely low. This means that even successful ML trading models are wrong nearly half the time on any individual prediction. The edge comes from making many bets where you're slightly more right than wrong, then compounding that over thousands of trades.

Overfitting is the main enemy. With enough features and a flexible enough model, you can fit any historical dataset near-perfectly. The problem is that most of the patterns you find are noise - artefacts of the specific historical sample that won't repeat. The single most important skill in ML for finance is controlling overfitting: using proper cross-validation, limiting model complexity, and maintaining rigorous out-of-sample testing.

Feature engineering matters more than algorithm choice. The difference between a random forest and gradient boosting on the same feature set is usually small. The difference between a model with well-engineered features and one with raw data is often enormous. Practitioners spend 70-80% of their time on data preparation and feature engineering, and 20-30% on model selection and tuning.

Ensemble methods dominate. Gradient boosting (XGBoost, LightGBM, CatBoost) is the workhorse algorithm at most quantitative firms. It handles mixed feature types well, is relatively resistant to overfitting with proper regularisation, and trains quickly. Random forests serve as a solid baseline. Simple models like regularised linear regression remain competitive for many tasks and offer better interpretability.

Deep learning has a narrower niche. Neural networks and LSTMs have been successful for specific applications - high-frequency order book data, NLP for text analysis, processing alternative data like satellite imagery - but they haven't broadly displaced tree-based methods for tabular financial data. Deep learning requires more data and more careful tuning, and overfitting risk is higher with small financial datasets.

Stationarity is a fundamental challenge. Financial markets change over time. A pattern that worked from 2015 to 2020 may not work from 2021 to 2026 because market microstructure has changed, participants have adapted, or the macroeconomic regime has shifted. Models need to be retrained regularly and monitored for decay.

---

Key ML Algorithms for Finance

Not all algorithms suit every financial problem. The table below summarises the most commonly used ML methods in finance, with honest assessments of their strengths and limitations.

For most practitioners starting out, the recommended path is: start with regularised linear regression as a baseline, move to gradient boosting for improved accuracy, and only explore deep learning when you have a specific problem that requires it (sequential data, unstructured data, or very large datasets).

---

The ML Pipeline for Finance

Building a machine learning model for finance follows a structured pipeline. Each stage has finance-specific considerations that differ from generic ML practice.

1. Data Collection and Cleaning

Financial data comes from multiple sources: exchange feeds (prices, volumes), fundamental databases (earnings, balance sheets), alternative data vendors (satellite, web scraping, NLP-processed news), and macroeconomic databases (central bank rates, GDP, inflation).

Cleaning financial data is more involved than in most ML domains. You need to handle corporate actions (stock splits, dividends, mergers), account for survivorship bias (only including companies that exist today biases your dataset), ensure point-in-time accuracy (using data that was actually available at each historical date, not revised figures), and deal with missing data (especially for alternative data sources that may not have full history).

2. Feature Engineering

This is where domain expertise meets data science. Raw data rarely feeds directly into a model. You transform it into features that capture economically meaningful relationships. Common financial features include lagged returns at various horizons, technical indicators (moving averages, RSI, Bollinger Bands), volume-based features (relative volume, volume-price trends), cross-sectional ranks (how a stock compares to its sector peers), and macroeconomic indicators (yield curve slope, VIX, credit spreads).

A dedicated section on feature engineering follows below.

3. Train/Validation/Test Split with Temporal Awareness

This is where financial ML diverges most sharply from standard ML practice. You cannot use random cross-validation with time-series data because it introduces look-ahead bias - the model trains on future data and "predicts" the past.

Instead, financial ML uses walk-forward validation (also called time-series cross-validation). The training set always comes before the validation set chronologically. A typical setup:

• Training window: 3-5 years of historical data.
• Validation window: The next 6-12 months (used for hyperparameter tuning).
• Test window: The following 6-12 months (held out entirely until final evaluation).
• Walk forward: Slide all windows forward and repeat.

This simulates how the model would have actually been used in real time.

4. Model Selection and Training

Start simple. A regularised linear regression establishes a baseline. If it shows no predictive power, a more complex model is unlikely to help - the problem is probably in the features or the data, not the algorithm. If the baseline shows some signal, try gradient boosting to capture nonlinear relationships. Compare models using out-of-sample metrics relevant to your task: Sharpe ratio for return predictions, AUC for classification tasks, or calibrated probability scores for risk estimation.

5. Backtesting

The model's predictions feed into a simulated trading strategy. The backtest must account for transaction costs (bid-ask spreads, commissions, market impact), execution delays (you can't trade at the closing price if your signal is computed after the close), position limits (realistic constraints on portfolio concentration), and borrowing costs for short positions. An overly optimistic backtest is worse than no backtest at all. For detailed guidance on backtesting approaches, the quantitative trading strategies guide covers the full workflow.

6. Live Monitoring and Model Maintenance

Models decay. Financial markets are non-stationary, and a model's predictive power diminishes over time as the relationships it learned shift. Production ML systems need monitoring dashboards that track prediction accuracy, feature importance stability, and realised versus predicted returns. Retraining schedules - weekly, monthly, or triggered by performance degradation - keep the model current.

---

Common Pitfalls in Financial ML

Machine learning projects in finance fail more often than they succeed. Understanding the common failure modes can save months of wasted effort.

Look-Ahead Bias

The most dangerous and often the most subtle error. Look-ahead bias occurs when your model uses information that wouldn't have been available at the time of prediction. Examples: using revised GDP figures instead of the initial release, training on data that includes the full sample period (including the "future"), or computing features using the entire time series rather than only data available up to each prediction point. Even small amounts of look-ahead bias can make a worthless model appear profitable.

Overfitting

A model that memorises the training data instead of learning generalisable patterns. With financial data - where signal-to-noise ratios are low - overfitting is the default outcome if you're not actively preventing it. Signs of overfitting: spectacular backtest results that don't replicate out of sample, model performance that degrades sharply on new data, and high sensitivity to small changes in the training period.

Prevention: regularisation (L1, L2, or tree-depth limits), proper cross-validation (walk-forward, not random), limiting the number of features, and using ensemble methods.

Data Snooping

Also called p-hacking or multiple testing bias. If you test 100 different feature combinations and pick the one that works best, there's a high probability that the best result is just noise. The more variations you test, the more likely you are to find something that looks good by chance. Corrections for multiple testing (Bonferroni, Benjamini-Hochberg) are essential but rarely applied in practice.

Survivorship Bias

If your dataset only includes companies that currently exist, you're missing all the companies that went bankrupt, were delisted, or were acquired. This systematically biases your data toward winners and makes models appear more predictive than they are. Always use survivorship-bias-free datasets that include delisted securities.

Non-Stationarity and Regime Changes

Financial time series are not stationary - their statistical properties change over time. A model trained on data from a low-interest-rate, low-volatility regime may fail completely when rates rise and volatility spikes. Techniques to address this include using shorter training windows (so the model reflects recent behaviour), including regime indicators as features, and testing across multiple market environments.

Transaction Costs and Market Impact

A model that predicts returns with high accuracy but requires frequent trading in illiquid securities may be net unprofitable after costs. Always include realistic transaction cost estimates in backtesting. Market impact - the price movement caused by your own trading - is particularly important for larger position sizes and less liquid instruments.

---

A Simple ML Trading Example in Python

Here's a practical example: predicting next-day stock returns using scikit-learn with proper time-series cross-validation. This example uses real-world methodology - temporal splits, no look-ahead bias, and realistic evaluation.

import numpy as np
import pandas as pd
import yfinance as yf
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import accuracy_score, classification_report
from sklearn.model_selection import TimeSeriesSplit

# Download historical data
ticker = "AAPL"
df = yf.download(ticker, start="2018-01-01", end="2026-01-01")

# --- Feature Engineering ---
df["return_1d"] = df["Close"].pct_change()
df["return_5d"] = df["Close"].pct_change(5)
df["return_20d"] = df["Close"].pct_change(20)

df["volatility_20d"] = df["return_1d"].rolling(20).std()
df["volume_ratio"] = df["Volume"] / df["Volume"].rolling(20).mean()

df["sma_5"] = df["Close"].rolling(5).mean()
df["sma_20"] = df["Close"].rolling(20).mean()
df["sma_ratio"] = df["sma_5"] / df["sma_20"]

# RSI (14-day)
delta = df["Close"].diff()
gain = delta.clip(lower=0).rolling(14).mean()
loss = (-delta.clip(upper=0)).rolling(14).mean()
df["rsi_14"] = 100 - (100 / (1 + gain / loss))

# Target: next-day return direction (1 = up, 0 = down)
df["target"] = (df["return_1d"].shift(-1) > 0).astype(int)

df.dropna(inplace=True)

feature_cols = [
    "return_1d", "return_5d", "return_20d",
    "volatility_20d", "volume_ratio", "sma_ratio", "rsi_14"
]

X = df[feature_cols]
y = df["target"]

# --- Walk-Forward Validation ---
tscv = TimeSeriesSplit(n_splits=5)
results = []

for fold, (train_idx, test_idx) in enumerate(tscv.split(X)):
    X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]
    y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]

    model = GradientBoostingClassifier(
        n_estimators=100,
        max_depth=3,
        learning_rate=0.05,
        subsample=0.8,
        random_state=42
    )
    model.fit(X_train, y_train)

    preds = model.predict(X_test)
    acc = accuracy_score(y_test, preds)
    results.append(acc)

    print(f"Fold {fold + 1}: Accuracy = {acc:.4f}")

print(f"\nMean accuracy: {np.mean(results):.4f}")
print(f"Std: {np.std(results):.4f}")

# --- Feature Importance ---
model.fit(X, y)  # final model on all data for feature importance
importance = pd.Series(
    model.feature_importances_, index=feature_cols
).sort_values(ascending=False)

print("\nFeature importance:")
print(importance)

A few things to note about this example:

• Temporal splits only. TimeSeriesSplit ensures training data always precedes test data. No random shuffling, no look-ahead.
• Simple features. Lagged returns, volatility, volume ratio, moving average ratio, and RSI. These are standard technical features - a starting point, not a finished model.
• Realistic expectations. If this model achieves 52-53% accuracy out of sample, that's a reasonable result. Anything above 55% on daily data should be treated with suspicion - it likely indicates a bug or data leakage.
• No transaction costs yet. This only measures directional accuracy. Converting this into a profitable trading strategy requires adding position sizing, transaction cost modelling, and risk management.

Extending the Example with a Simple Backtest

# Simple backtest: go long when model predicts up, flat otherwise
df_test = df.iloc[test_idx].copy()
df_test["prediction"] = model.predict(X_test)
df_test["strategy_return"] = df_test["prediction"] * df_test["return_1d"].shift(-1)
df_test["cumulative_strategy"] = (1 + df_test["strategy_return"]).cumprod()
df_test["cumulative_buyhold"] = (1 + df_test["return_1d"].shift(-1)).cumprod()

total_strategy = df_test["cumulative_strategy"].iloc[-1] - 1
total_buyhold = df_test["cumulative_buyhold"].iloc[-1] - 1

print(f"Strategy return: {total_strategy:.2%}")
print(f"Buy & hold return: {total_buyhold:.2%}")

This gives a rough sense of whether the model's predictions translate into actual returns. In practice, you'd add transaction costs of 5-10 basis points per trade and measure the Sharpe ratio rather than raw returns.

---

Feature Engineering for Financial ML

Feature engineering is where most of the value is created in financial ML. Raw price and volume data are inputs; the features you construct from them are what the model actually learns from. Good feature engineering encodes financial intuition into a form the model can use.

Lagged Returns

The simplest features: returns over various lookback periods. 1-day, 5-day, 20-day, and 60-day returns capture momentum and mean-reversion effects at different horizons. Cross-sectional ranks of returns (how a stock's 5-day return compares to all other stocks in the universe) often work better than raw returns because they're more stationary.

Technical Indicators

Moving average ratios, RSI, Bollinger Band width, MACD - these are standard quantitative trading features. On their own, most technical indicators have weak predictive power. But as features in an ensemble model that combines many signals, they contribute incremental information. The key is to avoid redundancy: many technical indicators are highly correlated with each other.

Volume Features

Volume carries information about conviction and liquidity. Useful features include relative volume (today's volume divided by the 20-day average), volume-price correlation over a lookback window, on-balance volume, and abnormal volume around events like earnings announcements.

Cross-Sectional Features

Rather than modelling each stock independently, cross-sectional features capture how a stock compares to its peers. Examples: a stock's return rank within its sector, the distance between a stock's valuation ratio and the sector median, or the stock's beta relative to its industry group. These features are more stationary than raw values and encode relative value information that's central to many factor investing approaches.

Fundamental Features

Earnings yield, book-to-market ratio, debt-to-equity, revenue growth, analyst earnings revision momentum - these are the building blocks of fundamental factor models. When used as ML features, they're typically standardised (z-scored) within sectors to remove level differences between industries.

Macro and Market-Wide Features

Features that capture the broad market environment: the VIX level and its recent change, the yield curve slope, credit spreads (the difference between corporate and government bond yields), and market-wide momentum or breadth indicators. These provide context - a model might learn that momentum signals work differently in high-volatility regimes versus low-volatility ones.

Alternative Data Features

This is the frontier. Satellite-derived features (car park occupancy as a proxy for retail sales), NLP-derived sentiment scores from news and social media, web traffic trends, app download counts, and patent filing activity. These data sources require substantial preprocessing - often using ML models themselves - before they become features for a trading model.

def engineer_features(df: pd.DataFrame) -> pd.DataFrame:
    """Build a standard feature set from OHLCV data."""
    feat = pd.DataFrame(index=df.index)

    # Lagged returns
    for lag in [1, 5, 10, 20, 60]:
        feat[f"return_{lag}d"] = df["Close"].pct_change(lag)

    # Volatility at multiple horizons
    daily_ret = df["Close"].pct_change()
    for window in [10, 20, 60]:
        feat[f"vol_{window}d"] = daily_ret.rolling(window).std()

    # Volume features
    feat["volume_ratio_20d"] = df["Volume"] / df["Volume"].rolling(20).mean()
    feat["volume_ratio_5d"] = df["Volume"] / df["Volume"].rolling(5).mean()

    # Moving average ratios
    feat["sma_5_20"] = df["Close"].rolling(5).mean() / df["Close"].rolling(20).mean()
    feat["sma_20_60"] = df["Close"].rolling(20).mean() / df["Close"].rolling(60).mean()

    # RSI
    delta = df["Close"].diff()
    gain = delta.clip(lower=0).rolling(14).mean()
    loss = (-delta.clip(upper=0)).rolling(14).mean()
    feat["rsi_14"] = 100 - (100 / (1 + gain / loss))

    # Bollinger Band width
    sma20 = df["Close"].rolling(20).mean()
    std20 = df["Close"].rolling(20).std()
    feat["bb_width"] = (2 * std20) / sma20

    # High-low range normalised by close
    feat["range_norm"] = (df["High"] - df["Low"]) / df["Close"]

    return feat

The function above creates roughly 20 features from OHLCV data. In production, a quant firm might use 200-500 features per stock, including cross-sectional and fundamental data, all carefully tested for predictive power before inclusion.

---

How Top Firms Use ML

The most successful quantitative firms in 2026 treat machine learning as infrastructure rather than a standalone strategy. Here's what's publicly known about how the leading firms approach ML.

Two Sigma

Two Sigma, founded in 2001, has been one of the most vocal proponents of data science in investing. The firm employs over 1,800 people, a significant portion of whom are engineers and data scientists. Two Sigma's approach centres on ingesting and processing massive volumes of data - traditional market data, fundamental data, and a wide range of alternative data sources - and using ML models to find patterns that predict returns. The firm has invested heavily in distributed computing infrastructure to support ML training at scale. Their research process is highly systematic: hypotheses are tested rigorously, and only signals that survive out-of-sample validation enter production.

Renaissance Technologies

Renaissance Technologies, founded by mathematician Jim Simons in 1982, is widely considered the most successful quantitative firm in history. The Medallion Fund has generated annualised returns above 60% before fees since 1988. Renaissance hires almost exclusively from mathematics, physics, and computer science - rarely from finance. While the firm is famously secretive, former employees have described an approach that combines signal processing, pattern recognition, and statistical learning across multiple asset classes and timeframes. Renaissance's edge is thought to come less from any single algorithm and more from decades of accumulated infrastructure, data, and institutional knowledge about what works and what doesn't.

Citadel

Citadel's quantitative strategies division uses ML across the investment pipeline - from alpha generation to execution optimisation. The firm allocates capital dynamically across strategies and has invested heavily in technology infrastructure, including co-located servers and proprietary execution systems. Citadel's approach is multi-strategy: ML models inform systematic equity, fixed income, commodities, and macro trading. The firm's scale allows it to invest in data acquisition and infrastructure that smaller firms can't match.

What They Have in Common

Despite different cultures and approaches, the leading ML-driven firms share several characteristics:

• Data infrastructure first. They invest as much in data pipelines, storage, and quality control as in model development.
• Ensemble approaches. They combine many weak signals rather than relying on a single model.
• Rigorous testing. Multiple layers of out-of-sample testing, walk-forward validation, and live paper trading before real capital is deployed.
• Continuous retraining. Models are updated regularly as market conditions change.
• Domain expertise. ML engineers work alongside experienced traders and researchers who understand market structure, transaction costs, and risk management.
• Proprietary data. Much of the edge comes from data sources and processing pipelines that competitors don't have access to.

---

Getting Started with ML in Finance

If you're looking to build skills in machine learning for finance, here's a practical learning path based on what the industry actually values.

Step 1: Build a Strong Statistical Foundation

Before touching ML libraries, make sure you're comfortable with probability, statistics, hypothesis testing, regression analysis, and time-series concepts like stationarity and autocorrelation. ML models are statistical estimators - if you don't understand bias-variance tradeoffs, p-values, or confidence intervals, you'll struggle to evaluate whether a model is genuinely predictive or just overfitting.

Recommended starting points: "An Introduction to Statistical Learning" by James, Witten, Hastie, and Tibshirani (free online) is an excellent foundation. For the finance-specific statistics, "Statistics and Data Analysis for Financial Engineering" by Ruppert and Matteson covers time-series and financial applications well.

Step 2: Learn Python for Data Science

Python is the standard language for ML in finance. You need proficiency with:

• pandas for data manipulation and time-series handling.
• numpy for numerical computation.
• scikit-learn for core ML algorithms (regression, classification, cross-validation, preprocessing).
• matplotlib/seaborn for data visualisation.
• statsmodels for classical statistical models and time-series analysis.

Step 3: Study Core ML Algorithms

Understand how the main algorithms work, not just how to call them. For each algorithm, know: what objective function it optimises, what assumptions it makes, how it handles overfitting, and what its hyperparameters control. Focus on linear regression (with regularisation), decision trees, random forests, gradient boosting, and basic neural networks.

Step 4: Apply ML to Financial Data

Start with a project: download historical stock data using yfinance, engineer features, train a model to predict returns, and backtest the results. You'll immediately encounter the challenges discussed above - overfitting, non-stationarity, low signal-to-noise ratio. Working through these problems hands-on is far more educational than reading about them.

Step 5: Learn Finance-Specific ML Techniques

Once you have the basics, focus on techniques specific to financial applications: walk-forward cross-validation, purged cross-validation (to handle overlapping labels), feature importance in the presence of multicollinearity, and portfolio-level evaluation (Sharpe ratio, maximum drawdown) rather than pure prediction accuracy. Marcos Lopez de Prado's "Advances in Financial Machine Learning" is the standard reference for these topics.

Step 6: Explore Advanced Topics

Depending on your interests: deep learning for sequential data (LSTMs, transformers for time-series), reinforcement learning for execution and portfolio optimisation, NLP for financial text analysis, or Bayesian methods for uncertainty quantification. These are specialisations - master the fundamentals first.

# A minimal template for walk-forward ML research in finance

import pandas as pd
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_squared_error

def walk_forward_backtest(
    df: pd.DataFrame,
    feature_cols: list[str],
    target_col: str,
    train_years: int = 3,
    test_months: int = 6,
) -> pd.DataFrame:
    """
    Walk-forward backtest: train on a rolling window, predict the
    next period, slide forward, and repeat.
    """
    results = []
    dates = df.index.to_series()
    min_date = dates.min()
    max_date = dates.max()

    train_start = min_date
    while True:
        train_end = train_start + pd.DateOffset(years=train_years)
        test_end = train_end + pd.DateOffset(months=test_months)

        if test_end > max_date:
            break

        train_mask = (dates >= train_start) & (dates < train_end)
        test_mask = (dates >= train_end) & (dates < test_end)

        X_train = df.loc[train_mask, feature_cols]
        y_train = df.loc[train_mask, target_col]
        X_test = df.loc[test_mask, feature_cols]
        y_test = df.loc[test_mask, target_col]

        if len(X_train) < 100 or len(X_test) < 20:
            train_start += pd.DateOffset(months=test_months)
            continue

        model = GradientBoostingRegressor(
            n_estimators=100, max_depth=3,
            learning_rate=0.05, subsample=0.8
        )
        model.fit(X_train, y_train)

        preds = model.predict(X_test)
        fold_results = pd.DataFrame({
            "date": df.loc[test_mask].index,
            "actual": y_test.values,
            "predicted": preds,
        })
        results.append(fold_results)

        train_start += pd.DateOffset(months=test_months)

    return pd.concat(results, ignore_index=True)

---

Frequently Asked Questions

Can machine learning actually predict stock prices?

ML models can predict stock returns with slightly better than random accuracy - and in quantitative finance, "slightly better than random" is valuable. A model that correctly predicts the direction of next-day returns 52% of the time, applied across hundreds of stocks with proper position sizing, can generate a Sharpe ratio above 1.0. But no ML model can reliably predict whether a specific stock will go up or down tomorrow with high confidence. The edge is statistical and emerges over many predictions, not from any single forecast. Models that claim very high accuracy on financial prediction tasks almost invariably suffer from look-ahead bias, overfitting, or unrealistic assumptions about execution.

What programming language should I learn for ML in finance?

Python is the clear standard. The scikit-learn, pandas, numpy, and PyTorch/TensorFlow ecosystem covers nearly everything you need for research. For production systems at quant firms, performance-critical components (execution engines, real-time data processing) are often written in C++ or Rust, but the research and model development is almost universally done in Python. R is used at some firms for statistical analysis, but Python's dominance has grown steadily. SQL is also essential for working with large financial databases.

Is deep learning better than traditional ML for finance?

Not in general. For structured, tabular financial data (price histories, fundamental ratios, technical indicators), gradient boosting methods like XGBoost and LightGBM consistently match or outperform deep learning approaches while being faster to train and easier to interpret. Deep learning excels in specific niches: processing unstructured data (text via NLP models, satellite images via CNNs), modelling high-frequency order book dynamics (where sequential patterns in tick data suit recurrent architectures), and handling very large datasets where the volume of data justifies the model's complexity. The practical advice is to start with gradient boosting for tabular problems and only move to deep learning when you have a clear reason.

How do I avoid overfitting in financial ML?

Overfitting is the central challenge. The most important practices are: (1) always use walk-forward or time-series cross-validation, never random splits; (2) apply regularisation - limit tree depth, use L1/L2 penalties, and set minimum sample sizes for leaf nodes; (3) limit the number of features, removing those with low importance or high correlation with other features; (4) hold out a final test set that you don't touch until your model is completely finalised; (5) be sceptical of exceptional backtest results - if your model shows a Sharpe ratio above 3.0 on daily equity data, there's almost certainly a bug; and (6) track multiple testing adjustments, because the more model configurations you try, the more likely you are to find one that fits noise.

What maths do I need for ML in finance?

A solid grounding in linear algebra (vectors, matrices, eigenvalues), calculus (partial derivatives, gradient descent), probability theory (distributions, Bayes' theorem, conditional probability), and statistics (hypothesis testing, regression, maximum likelihood estimation) covers the core requirements. For time-series applications, you'll also need familiarity with stationarity, autocorrelation, and cointegration. Stochastic calculus is important for derivatives pricing but less central to ML-specific work. You don't need to be a mathematician, but you do need to understand what your models are doing under the surface - otherwise you can't diagnose problems when they inevitably arise.

What is the difference between ML and traditional quant finance?

Traditional quantitative finance builds models from first principles - deriving pricing formulas from assumptions about how markets work (efficient markets, no arbitrage, specific return distributions). Machine learning takes a data-driven approach - letting algorithms find patterns directly from data without specifying the functional form in advance. In practice, the best quant firms combine both. A traditional factor model might provide the framework (value, momentum, and quality factors drive returns), while ML improves the implementation (gradient boosting combines dozens of factor signals with nonlinear weighting). Think of traditional quant finance as theory-driven and ML as data-driven - the most effective approach uses both.