Predictive Analytics Guide: What to Predict, Baselines, Backtesting and Drift

Start with the decision, not the data

A prediction is only useful if someone does something different because of it. Before any data work, write down the decision, who makes it, how often and what a wrong prediction costs in each direction. That one page decides the prediction target, the time horizon, the accuracy you need and whether the project is worth doing.

If the decision is made once a quarter by gut feel in a meeting, a better forecast may not change anything. If it is made thousands of times a day by a system, such as ranking products or routing tickets, a small improvement compounds.

Data readiness checklist

• History: the outcome you want to predict has happened many times, across at least one full seasonal cycle (a year for retail and events).
• Timestamps you can trust: you know when each fact became known, not just when the row was last updated. Without this you cannot rebuild what the system knew on a past date.
• Stable definitions: "active customer", "churn" and "order" mean the same thing across the whole history, or the changes are documented.
• Joined sources: the outcome table can be linked to the drivers (prices, promotions, marketing, support tickets) by reliable keys.
• Access: read access to the production database, warehouse or exports, agreed with whoever owns the data and the privacy policy.
• Known gaps: outages, migrations and test data are marked so they can be excluded.

If several of these fail, the first project is data engineering, not modelling. That is not wasted work: the same pipelines feed reporting and every later model. Our hire data engineers page covers that role.

Build a baseline before a model

A baseline is the simplest prediction that someone could defend: last week's sales, the same week last year, the average of the last four weeks, "customers who have not logged in for 30 days will churn", or "show the most popular items". It costs almost nothing, ships in days and sets the bar. Google's Rules of Machine Learning puts it bluntly: "Don't be afraid to launch a product without machine learning" and "Keep the first model simple and get the infrastructure right."

Report every model result next to the baseline. A model that is 4 percent more accurate than a rule may not be worth its running cost and complexity; a model that is 30 percent better usually is.

Evaluation: backtest as if it were live

The most common way predictive projects mislead their owners is a test that lets the model see the future. Random train and test splits do this with time-ordered data: the model learns from next month's rows and is then tested on last month's. The fix is backtesting. Pick a series of past dates; for each, train only on data available before that date and predict the period after it; then average the errors. The forecasting textbook Forecasting: Principles and Practice calls this evaluation on a rolling forecasting origin.

• Leakage: remove any feature that would not have been known at prediction time, such as a "cancelled_at" field in a churn model or a refund flag in a fraud model.
• Metrics that match the decision: mean absolute error in units for stock; precision in the top 100 customers for a retention team that calls 100 a week; hit rate or diversity for recommendations.
• Beware MAPE on small numbers: percentage error explodes when actuals are near zero, which is common for slow-moving products.
• Segment results: check accuracy by region, product line and customer type. An overall average can hide a segment where the model is worse than the baseline.
• Read the output as well as the score: look at what the model actually recommends or predicts for real cases before trusting a metric.

Worked example: Paint Nite's recommendation replay harness

Paint Nite runs ticketed live painting events across the US and Canada. As part of rebuilding the platform, we built a painting recommendation engine and tuned it over four measured iterations using an offline replay harness. A replay harness feeds historical situations through a candidate algorithm and records what it would have chosen, so each version can be compared on the same past data before any customer sees it.

Early on, the harness caught the recommendation algorithm selecting the same artwork 187 times out of 187 before that version reached production. An aggregate accuracy score could look fine in that situation; looking at the distribution of what the algorithm actually chose exposed it at once. The lesson generalises: for any ranking or recommendation model, measure diversity and coverage of the outputs alongside accuracy, and read a sample of real outputs every iteration.

Deployment: batch first, then real time

1. Batch: score everything nightly or weekly and write results to a table the app and reports read. This covers most forecasting, churn and scoring use cases and is easy to debug.
2. Shadow mode: run the model live beside the current process without acting on it, and compare. This tests the production data path, which often differs from the training data in small, important ways.
3. Controlled rollout: act on the predictions for part of the traffic or one region, and compare against a holdout that keeps the old process.
4. Real time: serve predictions on request only when the decision is made in the moment, such as ranking a page, and only once the batch version has proved its value.

Keep the feature calculations in one shared code path for training and serving. Differences between the two are a classic source of models that score well in testing and badly live.

Drift, monitoring and retraining

Every predictive model gets worse over time because the world changes: prices change, a competitor launches, a pandemic shuts events, a new marketing channel brings different customers. Data drift is when the inputs change; concept drift is when the relationship between inputs and outcome changes. You need to notice both.

Retrain on a schedule that matches how fast your world changes (weekly for fast retail, quarterly for slow B2B), and treat each retrained model like a code release: backtest it, compare it with the current one, and keep the option to roll back.

What it costs and who you need

A first predictive feature in an existing product falls in our $8k to $60k AI feature band, and we usually start with discovery ($3k to $8k) to check the data and build the baseline. The breakdown of data work, model work and running costs is in machine learning app development cost. For the service, see machine learning and predictive analytics development and our AI and data services. If the data is images or video rather than tables, read computer vision use cases.