The essential answer
An illustrative model reports only 1-unit root-mean-square error after an analyst adjusts a 5-unit offset using the supposed test set. Before that adjustment, its test error is 5.10 units. The lower score reflects fitting to evaluation outcomes, so it is no longer an unbiased final test of the original procedure. The adjustment may be legitimate training, but it requires a new untouched evaluation set to assess subsequent predictive performance.
How to reason about it
Data leakage occurs when information unavailable at prediction time influences model fitting or selection. It includes obvious reuse of test outcomes and subtler choices such as selecting features, thresholds, preprocessing statistics, or model variants after reviewing test performance. A pipeline should learn its transformations from training data and apply the learned transformation to held-out data. Validation data can guide choices, but repeated choices make it part of development rather than a final independent assessment. The official scikit-learn documentation supplies the leakage principle. This example uses a transparent offset adjustment so the numerical benefit is visible. It does not claim that all preprocessing leakage yields the same magnitude or direction of error.
ILLUSTRATIVE · NOT A REPORTED RESULT
An example to work through
Suppose illustrative held-out residuals, defined as prediction minus truth, are (4,6,4,6) units. Their mean is 5. The original root-mean-square error is sqrt((16+36+16+36)/4)=sqrt(26)=5.09902 units. After inspecting these outcomes, the analyst subtracts 5 from every prediction. Residuals become (−1,1,−1,1), and reported error becomes sqrt((1+1+1+1)/4)=1 unit. The apparent improvement is real on those four cases, but the correction was chosen using their answers. Now consider a fresh illustrative set whose uncorrected residuals are (−1,1,−1,1). Applying the learned subtraction produces (−6,−4,−6,−4), with error again sqrt(26)=5.09902. This constructed reversal shows that an offset learned from one evaluation batch may not transfer. It does not prove that offset correction is generally harmful; correction must be learned on appropriate training or calibration data and then assessed independently.
Where interpretation fails
Renaming a reused test set validation data after the fact does not restore an independent final estimate. Another trap is saving a fresh split but allowing related observations from the same subject, batch, or future time period into training. Independence must match the intended deployment question, not just random row assignment.
The next decision
Document which data informed every transformation and selection. In the example, label the 1-unit score as development performance after offset fitting. Freeze the revised procedure and evaluate on genuinely untouched, appropriately separated cases before claiming predictive accuracy. The correct split may require grouping or time order rather than random rows.
Sources & context
These sources support the concepts noted below. Our synthetic examples and research framing are educational interpretation; they are not findings from these organizations or validation of our ventures.
- scikit-learn: Common Pitfalls and Recommended Practices ↗
Prevent data leakage by restricting learned preprocessing to training data.
Source check: Official documentation page browsed
Educational research context, not a validated device specification or clinical protocol. Read our editorial and correction policy.