London NO2 Forecasting: Why ARIMA and Prophet Failed Me

Air quality data is notoriously messy, and trying to predict nitrogen dioxide levels in London proved that standard tutorials don't prepare you for real-world noise. My initial attempt at this project was a disaster because I treated the time series as a clean mathematical sequence rather than a chaotic set of sensor readings.

The first wall I hit was data cleaning. Real-world sensor data has gaps, outliers, and "flatlines" where the hardware just hangs. If you feed that directly into a model, your residuals will be a mess.

I started with ARIMA, but the stationarity requirements were a nightmare. I spent way too much time differencing the data just to get a stable p-value, and the results were still lagging behind the actual trends. Then I tried Prophet, thinking the additive model would handle the seasonality better, but it over-smoothed the peaks. I was getting a "pretty" curve that completely missed the actual pollution spikes I was trying to predict.

The turning point was realizing my validation strategy was broken. I was using random splits, which is a cardinal sin in time series. I had to implement time-based cross-validation (rolling window) to see how the model actually performed on "future" data it hadn't seen.

Here is the basic logic I used for the rolling window validation to stop the data leakage:

from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5)
for train_index, test_index in tscv.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    # Fit and evaluate model here

By switching to this AI workflow and focusing on the specific seasonality of urban traffic patterns rather than relying on "out-of-the-box" hyperparameters, the error rates finally dropped. It turns out that for environmental data, the preprocessing and the validation strategy matter more than the specific LLM agent or forecasting library you choose.