By now you have four problems and a natural response to all of them. You have seen wicks that nobody meant to make, bars that stand in for hours of nothing, securities that are not what the search box implied, and markers that could not have been placed when they appear to have been. So you write a cleaning stage: strip the obvious rubbish out before anything else runs. The first rule you write is this — drop any bar whose high is more than twenty per cent above the previous close and which the price never revisited in the following five sessions. It removes the freak trades from six years of history in one pass, the backtest improves noticeably, and everything about it feels like housekeeping. It is the single most expensive line in the system, and the reason is in the clause you added to make it careful.
An accountant preparing your accounts removes the entries that "obviously" should not be there — the duplicated bill, the payment that was reversed, the transfer that was really a transfer. Every removal is defensible one at a time. What nobody notices is that he made each judgement in March, knowing how the year ended, and the tidied accounts are now a description of a year that could only be written afterwards. Anybody using them to forecast next year is forecasting from a document that had the answer in it.
A cleaning rule is an accountant. If it decides what to remove using anything that happened after the bar, the cleaned history is a history that could only have been written afterwards — and a backtest run on it is being run on a series that had the answers in it.
Where exactly the rule reads the future
Split the rule into its two clauses and check each against one question: could this have been evaluated at the moment the bar closed?
- The previous close is known before the session opens
- The high is known when the bar closes
- Evaluable at the bar. This clause is causal and legitimate
- On its own it is a blunt filter, but it is an honest one
- Requires five sessions that have not happened yet
- At 11.04 you cannot know whether ₹340 is a fold or the first rung of a re-rating
- Not evaluable at the bar. This clause reads the future
- And it selects on the outcome — it deletes exactly the spikes that failed
And there is a second failure underneath the first. The cleaned series is not reproducible live. At 11.04 on Tuesday you have to decide whether to act, and the rule that built your backtest cannot be evaluated until Monday. Whatever the live system does at 11.04, it is not what the tested system did — so the test was not a test of it. This is [[Look-ahead bias]] arriving through the back door, in the data pipeline rather than the strategy, where reviewers rarely look because the strategy file contains no future references at all.
Four cleaning rules that could have been run at the bar
The remedy is not to stop cleaning. It is to accept a weaker rule that only uses what existed at the time, and there are several good ones. All four below are computable at the close of the bar they act on, and all four can therefore be run identically in the backtest and in the live feed — which is the whole test of a cleaning rule.
| Rule | What it uses | What it costs you |
|---|---|---|
| Cap the bar at its price band | The previous close and the published band, both known before the session | Nothing, and it catches almost nothing either — the band is wide. Worth having as a sanity floor rather than as the main filter |
| Cap the true range at a multiple of its recent median | The last twenty or fifty true ranges, all complete | A genuine one-day crash is capped along with the folds. That is the honest price of a causal rule, and it is usually the right trade because the alternative distorts both stops and sizing |
| Require a minimum number of trades in the bar | The trade count or volume of that bar | Excellent on intraday data, where a fold is a handful of trades and a real move is thousands. Needs a feed that carries trade counts, which not all do |
| Compute the indicator on closes only | Nothing at all — you simply stop reading the extremes | The cleanest option and the most underrated. It edits no data, so nothing can go wrong in the pipeline; you give up the indicators that need the high and the low, and for many systems that is no loss |
Cap rather than delete
When you do act on a bar, there is a choice between removing the row and limiting the value, and it is not a close call. Deleting a bar shortens the series, so every rolling window after it spans a different set of bars and every date-indexed comparison shifts by one. Winsorising — the statistical name for replacing an extreme value with the most extreme value you are prepared to accept — leaves the row in place, keeps the length of every window, and touches only the field that was wrong.
- Keep the raw series untouched, always. Cleaning is a stage in a pipeline, not an edit to a file. If your cleaned data is the only copy you have, you can never re-examine a decision, never change the rule, and never establish whether a result was caused by the strategy or by the filter.
- Write the rule down in the same place as the strategy rules. The threshold, the window, the field it modifies. A cleaning rule is a parameter of the system, and it is subject to the same discipline — if the results are sensitive to whether the multiple is four or six, you have found a fitted parameter rather than a fix.
- Log every bar the rule touches. Not the count — the actual list, with dates and names. A rule you believed would fire twice a year and is firing on three per cent of bars is not cleaning data; it is reshaping the series, and you want to discover that by reading the log rather than by wondering why live results diverge.
- Run the identical rule on the live feed. A cleaner that exists only in the backtest means you have tested one system and are trading another. This is the most common way the whole exercise fails, and it fails silently.
- Re-run the test with cleaning switched off. If the strategy is profitable both ways, the cleaning is housekeeping. If it is only profitable with the cleaner on, then whatever you have found is a property of the filter, and the honest description of your result is a description of the filter.
The filter that made the system work
Your breakout system returns 4 per cent a year with the cleaner off and 19 per cent with it on. The cleaner drops bars whose high is more than 20 per cent above the previous close and which price did not revisit within five sessions. It fires on about 2 per cent of bars across the universe. You have a fortnight before you were planning to go live.
Module checkpoint: when the numbers are wrong
5 questions. Answers are revealed once you submit all of them.
1.A stock with an ATR(14) of ₹4.50 opens against a previous close of ₹301, makes a high of ₹303, prints a four-second low of ₹268 and closes at ₹299. Your system sets stops at 2×ATR and risks a fixed ₹5,000. What changes?
2.Two stocks show RSI(14) = 50. In the thin one, ten of the last fourteen bars had no trades and were printed flat, and the other four were two up ₹1 and two down ₹1. In the liquid one, average gain and average loss are both ₹2. Both now rise ₹6 in a bar. What do the oscillators read?
3.You buy a breakout at 10.40 intending to square off the same afternoon. The exchange has that security in trade-to-trade settlement. What happens when you sell at 2.50?
4.Your chart marks a swing low at ₹412 on 14 March under a definition requiring three subsequent higher lows, and your backtest buys the next open at ₹415. What is the defect, and what test proves it?
5.A cleaning rule drops any bar whose high is more than 20 per cent above the previous close and which price did not revisit within five sessions. The backtest goes from 4 per cent a year to 19 per cent. What have you learnt?
Bachche ka saal ka average 62% hai. Woh paanch paper hata deta hai — "un dinon tabiyat kharab thi", "us din bijli chali gayi thi" — aur average 84% ho jaata hai. Har ek bahaana sach hai. Gadbad yeh hai ki har faisla number dekhne ke baad liya gaya, aur agle paper se pehle woh yeh nahi keh sakta ki "yeh wala hatega". 84% kisi kaam ka nahi, kyunki agla paper 62% waala bachcha hi de raha hai. Data cleaning ka rule bilkul yahi cheez hai. "Aisa bar hata do jiska high pichhle close se 20% upar ho aur jispe bhaav agle paanch session mein wapas na gaya ho" — pehla hissa theek hai (pichhla close aur us bar ka high, dono usi waqt maujood hain), doosra hissa paanch din aage jhaank raha hai. Aur woh sirf optimism nahi jodta: woh theek wahi spike mitata hai jo fail ho gaye, aur jo chal gaye unhe rakh leta hai — isiliye breakout system ka result 4% se 19% ho jaata hai. Live mein 11.04 baje aapko faisla lena hai aur woh rule agle Somvaar se pehle chalta hi nahi. Seedha upaay: aisa rule lo jo usi bar pe nikal jaaye — jaise true range ko pichhle bees bar ke median ke chaar guna pe cap kar do — bar mitao mat, cap karo (row rahegi toh har rolling window ki lambai bhi rahegi), aur ek baar cleaning band karke bhi test chala lo. Agar munafa sirf filter ke saath aata hai, toh jo aapne dhoondha hai woh strategy nahi, filter hai.
- A cleaning rule is part of the system, and it must be evaluable at the bar it acts on or it is look-ahead bias.
- A rule that asks what happened afterwards selects on outcome — it deletes the failures and keeps the successes.
- Cap rather than delete: winsorising leaves the row in place, so every rolling window keeps its length.
- Keep the raw series, write the rule down, log every bar it touches, and run the identical rule on the live feed.
- Re-run with cleaning off — if the result only exists with the filter on, the result is a description of the filter.
Mark it done to track your progress through the curriculum.