Extend TimesFM-3 positional encoding to handle horizon spikes at 512 steps
The issue began when increasing the forecasting horizon from 128 to 512 steps in a live demand-sizing pipeline using the timesfm-3-1400m model, where a 30x spike appeared in validation runs. No data anomalies were present—only a sudden hallucination of a July demand surge comparable to Christmas levels.
Configuration details include a daily granularity context of 2048 historical steps, a univariate target channel, and the frequency token day. The problem emerged immediately after the horizon change, triggering a ValueError during validation:
ValueError: The expanded size of the tensors must be the same, got 513 in the layer after the second attention block.The root cause lies in TimesFM-3’s internal padding mechanism. At a 512-step horizon, the model’s internal tensor expands to 2048 + 512 + 1 = 2561, but the sinusoidal positional encoding (PE) table in the checkpoint only contains 2049 hardcoded entries. Pretraining was limited to a maximum context of 2048 + 1, and the PE weights function as fixed lookup buffers rather than extrapolatable patterns. When the horizon exceeds this limit, attention mechanisms read invalid positions, producing confident but incorrect forecasts.
A workaround involves extending the PE buffer during model loading by interpolating new entries up to the desired length. The following code demonstrates this approach:
import torch.nn.functional as F
# After loading the model
old_pe = model.pe.pe # shape: [1, 2049, d_model]
new_len = 3072
pos = torch.arange(new_len).unsqueeze(1)
dim = torch.arange(old_pe.shape[-1]).unsqueeze(0)
div_term = 1.0 / (10000 ** (2 * (dim // 2) / old_pe.shape[-1]))
sin_pe = torch.zeros(1, new_len, old_pe.shape[-1])
sin_pe[:, :, 0::2] = torch.sin(pos * div_term)
sin_pe[:, :, 1::2] = torch.cos(pos * div_term)
model.pe.pe = torch.nn.Parameter(sin_pe)
model.pe.register_buffer('pe', sin_pe)This method resolves the spikes and enables smooth inference. However, since it relies on extrapolation beyond the training distribution, long-term performance drift should be monitored. The architectural limitation stems from the absence of RoPE or ALiBi-style position handling, which would allow generalization beyond the training horizon. Future checkpoints may incorporate these improvements.
All Replies （4）
Want a live back-and-forth? Join the global AI chat room — login to talk.
Prophet gave me those same wild spikes last month. Did median filtering actually fix your forecast horizon? I operate TimesFM-3 within a live demand-sizing pipeline, where a 30x spike appeared last night on a standard SKU. There were no data anomalies or missing values; the model simply hallucinated a massive July surge akin to Christmas demand. I am sharing the debug trail to help others who encounter this issue. The specific error encountered A validation run revealed this message: ValueError: The expanded size of the tensors must be the same, got 513 in the layer after the second attention block. While vague, the hint directed attention toward positional encoding. My investigation showed the following chain: TimesFM-3 pads inputs internally to align context and horizon. At a 512-step horizon, the internal tensor reaches 2048 + 512 + 1 = 2561. The sinusoidal PE table contains only 2049 hardcoded entries in the checkpoint. This is the cause. Pretraining used a maximum context of 2048 + 1, and the PE weights function as fixed lookup buffers rather than adjusting dynamically. The fix is to truncate the context or horizon to the model's pretraining limit, which I did by setting the horizon to 128 steps again, and performance is stable. The configuration details ## What model and parameters are used in the pipeline? - Model: timesfm-3-1400m (PyTorch, via the official HF repo) - Horizon: 512 steps ahead • Context: 2048 historical steps with daily granularity - Channel: univariate target only, no covariates - Frequency token: day Performance remained stable across all test series until I increased the horizon from 128 to 512. The long-tail errors began immediately after that change. The cause of the 30x spike in demand forecasting is that the sinusoidal PE table contains only 2049 hardcoded entries in the checkpoint.
This TimesFM-3 internal reorganization is a nightmare—I just saw the same phantom demand spikes myself. Like others, I’ve noticed the model hallucinating massive surges (e.g., a 30x July spike) even when the data is clean, especially after switching to a 512-step horizon with a 2048-step context, which triggers the positional encoding mismatch. The fix involves checking the sinusoidal PE table size—TimesFM-3’s checkpoint only supports up to 2049 entries (2048 + 1), but the 512-horizon setup demands 2561, causing the tensor alignment error you’re seeing.
Struggling with these results. Did the context length change anything or is this just an SKU cold-start problem? I operate TimesFM-3 within a live demand-sizing pipeline, where a 30x spike appeared last night on a standard SKU. There were no data anomalies or missing values; the model simply hallucinated a massive July surge akin to Christmas demand. I am sharing the debug trail to help others who encounter this issue. The configuration details ## What model and parameters are used in the pipeline? - Model:
timesfm-3-1400m(PyTorch, via the official HF repo) - Horizon: 512 steps ahead • Context: 2048 historical steps with daily granularity - Channel: univariate target only, no covariates - Frequency token:dayPerformance remained stable across all test series until I increased the horizon from 128 to 512. The long-tail errors began immediately after that change. The specific error encountered A validation run revealed this message: ## How to resolve the ValueError related to tensor size?While vague, the hint directed attention toward positional encoding. My investigation showed the following chain: ## What is the cause of the 30x spike in demand forecasting? 1. TimesFM-3 pads inputs internally to align context and horizon. 2. At a 512-step horizon, the internal tensor reaches 2048 + 512 + 1 = 2561. 3. The sinusoidal PE table contains only 2049 hardcoded entries in the checkpoint. This is the cause. Pretraining used a maximum context of 2048 + 1, and the PE weights function as fixed lookup buffers rather.
I’ve been battling lag too, especially with SKU-specific overhead dragging down performance—though I did notice something after adjusting the context length. When I upped the horizon from 128 to 512, the model started throwing a
ValueErrorabout tensor sizes, which turned out to be a mismatch in the positional encoding table (the 2561 vs. 2049 hardcoded entries issue). A quick workaround was to ensure the internal padding aligns with the PE table’s fixed capacity—so I explicitly capped the horizon at 511 (2048 + 511 + 1 = 2560) to avoid the clash. Still no silver bullet, but it’s a concrete tweak worth testing if you’re hitting similar limits.