Which page should an editor open first?

01The decision this supports

Introduction

An editor opens Monday morning with capacity for about fifty pages and a portfolio of tens of thousands. Nothing in that situation is improved by a list of everything that looks unhealthy; what is missing is an order. That is the decision this work supports, and it is the reason the metric here is precision@K at the team's real capacity rather than accuracy.

The unit is one pseudonymous content item in one snapshot. The output is a ranked queue where every row carries a risk score, the reasons behind it, a suggested action and a confidence label. The action a human takes is to review — check business value, check whether a sibling page absorbed the demand, check seasonality — and then refresh, fix metadata, protect or simply watch.

The costs of being wrong are asymmetric. A false alarm burns an editor's hour and disturbs a page that was stable. A miss lets a page with real demand decay quietly until recovery is expensive. Since the top of the list is what gets acted on, the top of the list is what gets measured.

A threshold rule can select a group of pages. It cannot put them in order — and the order is the entire decision.

02The data, and the split that made it a prediction problem

Data

Everything here runs on data/raw/content_refresh_anonymized.csv, the public anonymized starter slice of the FlyRank ML Internship dataset: 30,000 content items across 32 pseudonymous clients, with Google Search Console and GA4 metrics aggregated over one rolling 90-day window. The full ~79M-row warehouse release is gated and requires a token this run did not have; the feature contract is source-agnostic, but the published numbers come from the public slice. That is a limitation, stated rather than hidden.

The snapshot is cross-sectional — one row per page, no time axis — which normally rules out an honest forward-looking task. It does not, because the 90-day totals decompose exactly into three 30-day sub-windows (verified: maximum reconstruction error is zero). That decomposition is what turns a snapshot into a decision point.

What was excluded, and why

• Label-derived: trend_pct and trend_direction. The label is computed from them; using them is circular.
• Window-spanning: every *_90d total and rate — impressions, clicks, sessions, CTR, average position, engagement rate, scroll rate, AI traffic share, days with impressions — plus every last-30-day column.
• Product decisions: impression_tier, position_tier, freshness_tier. These encode a judgement an existing system already made; they may be a baseline to beat, never an input.
• Identifiers: content_id and client_id are pseudonyms used for grouping and splitting only.

Population

Pages with fewer than 100 impressions in the prior 30 days are excluded: below that floor a “decline” is noise, and no editor should spend an hour on it. That drops 11,990 pages and leaves 18,010 pages across 30 clients. The filter reads only pre-decision columns, so it carries no outcome information — but it does narrow what these results can speak about.

Public safety. No client names, domains, URLs, page titles or search queries appear anywhere in this work — only pseudonymous IDs and aggregates.

03Method: 24 features that existed before the outcome

Methodology

Label. A page is labelled declined when its impressions over the most recent 30 days fell more than 20% against the previous 30 days — the product's own definition of trend_direction == "down". Base rate: 61.6%.

Features. Twenty-four columns, all knowable at the decision point: demand level in the two visible windows (log impressions, clicks, sessions), the momentum already visible between them, prior-window click-through and how it shifted, content age at the decision point, article length with a measured/not-measured flag, and keyword context (search volume, competition, CPC) with its own flag. Missingness in this dataset follows content_type, so a blind fillna(0) would quietly encode content type into the model — flags first, then fill.

Baselines. Two, both frozen before modelling started. The Week-4 rule, kept exactly as written (stale × visible × impressions, reading two contaminated columns), and a legal rewrite of the same idea using only decision-time columns: a page is worth reviewing first if it already lost ground between the two visible windows and still carries demand, with a small penalty for below-median click-through. No fitted weights in either.

Validation. GroupKFold(5) grouped by client_id, with every metric computed out-of-fold. Pages from one client share hidden character — same site, same editorial team — so a random split lets a model memorise the client and call it skill. Measured here, that memorisation is worth +0.0421 PR-AUC: 0.7604 on a random row split versus 0.7183 grouped by client.

Leakage tests. Three, all in the audit notebook. Deliberately adding trend_pct pushes ROC-AUC to 0.9997 — proof the harness can see leakage when it exists. Train-with versus train-without on the window-spanning group is reported below as its own result. A scan of the final feature set for product flags and IDs comes back clean. (My first version of that scan used substring matching and produced false positives on the legal prior_*_trend columns — the check was fixed to match exact names.)

04Results: a modest, real win — and one loud failure

Results

All metrics out-of-fold under GroupKFold(5) grouped by client. Base rate 0.6155 — 11,086 of 18,010 pages declined. The leaky row is shown to be disowned, not claimed.

The failure worth more than the win

The rule I built in Week 4 — old content with high impressions — scores 12 of 18,010 pages above zero. Eleven of those twelve did decline, so it is precise on the sliver it can see. For the other 17,998 pages every score ties at zero, which means their order is arbitrary. That is what ROC-AUC 0.500 and precision@200 falling back to the base rate actually describe. The rule is a serviceable filter and not a ranker, and a review queue needs the second.

What leakage would have bought

Put the window-spanning columns back and the same random forest reports precision@50 of 1.000 and precision@200 of 0.995. Nothing improved except the appearance of the result: impressions_90d minus the two visible windows is the label window. This number is on the page to be disowned.

How steady is the win?

Across five different client holdouts, PR-AUC moves between 0.63 and 0.7021 (mean 0.6678 ± 0.0258) and precision@50 sits at 0.816 ± 0.048. With thirty clients, that range is the honest result — not the single 0.7183. Simplicity also won on merit: the logistic regression edged the random forest (0.7183 vs 0.7108), so the readable model is the one that ships.

Is the win bigger than the noise?

Thirty clients, one of which holds 31.6% of the pages, is a small sample, and a point estimate on a small sample is a claim, not evidence. So the headline was attacked six ways, and each answer was committed as a receipt before this paragraph was written.

1000 resamples of the 30 clients with replacement; percentile 95% intervals; every model is scored on the same resample, so gaps are paired. Every row is a receipt in work/outputs/evidence_audit.json, written by work/scripts/evidence_audit.py.

Can the score be read as a probability?

Mostly, and not at the top. The Brier score beats always predicting the base rate (skill +0.052), the expected calibration error is 0.028, and the middle deciles sit on the diagonal. But the calibration slope is 0.78: the riskiest tenth averages a predicted 0.84 while 0.76 actually declined. The queue therefore prints a confidence label rather than a percentage, and the labels behave as ordered — high 77.7% declined, medium 68.7%, low 48.5%.

Decile 10 is the riskiest. Out-of-fold scores.

What the model leans on

Importance = drop in PR-AUC when the column is shuffled, measured on held-out clients. These traffic columns are correlated with one another, so the coefficients read as direction, not as isolated effects. Article length sits far below this table at ≈ 0.0009.

Holding clicks fixed, a recent impression spike is associated with higher decline risk. The clearest early warning in this data is exposure that did not convert.

Three signals, checked one at a time

The model is not the only way to read this data, and simple grouped comparisons are easier for an editor to sanity-check. Three, each with the verdict the numbers actually supported:

• Prior momentum — confirmed, with a fold. Pages that had already lost more than half their demand between the two visible windows declined at 70.6%, falling to 52.4% for pages up 20–50%. But pages that spiked more than +50% turn back up to 58.7%: a jump in impressions is not, by itself, good news.
• Low click-through at high exposure — confirmed, and stronger than expected. Among pages with ≥500 impressions, decline runs monotonically from 73.6% in the lowest click-through quartile to 48.9% in the highest — a 24.7-point spread on roughly 2,780 pages per quartile.
• Longer articles hold up better — mixed, and it does not survive. Grouped, the 1,000–2,000-word band does decline more (79.4%, n = 1,444). But the two label groups average 3,483 against 3,432 words, the <1,000-word bucket holds only 15 pages and must not be read, and once demand and click signals are present, length adds essentially nothing (importance ≈ 0.0009).

05What this work cannot claim

Limitations & honest framing

1. One snapshot, not a time series

   I reconstruct one decision point from one 90-day window. Nothing here tests whether the ranking holds across months.

2. Decline, seasonality, consolidation and noise look identical here

   A page marked “declining” may simply be handing demand to a sibling page, or following an annual cycle.

3. The results speak for 18,010 pages, not 30,000

   The ≥100-impression demand floor drops 11,990 low-volume pages. For those, this system is silent — not reassuring.

4. Thirty clients is a small sample

   PR-AUC moves between 0.63 and 0.7021 depending on which clients are held out. That range is the result, not the single 0.7183. The client-bootstrap interval on the precision@50 gap over the rule, -0.06 to +0.24, crosses zero.

5. Search position could not be used at all

   The only position column is a 90-day mean that spans the outcome window. A central SEO signal had to be dropped; the warehouse's daily table would restore it.

6. Content freshness could not be used either

   68.3% of pages were updated inside the outcome window, so days_since_last_update knows the future.

7. Public slice, not the full warehouse

   The gated ~79M-row release needs a token this run did not have. The feature contract is source-agnostic; the published numbers are from the public slice.

8. The label is a product definition

   “Down” means a >20% impression drop because that is how the product defines it. Move the threshold and the population moves with it.

9. No causal evidence exists in this work

   No experiment, no control group, no causal design. This work cannot say that refreshing a page recovers its traffic, and says nothing whatsoever about Google's algorithm.

06What to do first

Ranked recommendations

1. 1Work the model queue instead of an age or freshness sort — start with the top 50.

   EvidencePrecision@50 0.88 against 0.74 for the transparent rule and a 0.616 base rate. In the first 50 hours of review, roughly 44 of 50 land on real decliners instead of 31.

   Wrong ifThe portfolio changes shape, or the period sits inside a strong seasonal swing.

2. 2Use “old and high-impression” as a filter, never as a ranker.

   EvidenceThat rule scores 12 of 18,010 pages above zero (11 of them declined) and is silent on the rest — every remaining page ties at zero, so its order is arbitrary. ROC-AUC 0.500.

   Wrong ifOn the full warehouse, a properly aligned freshness column could regain its power.

3. 3Treat “impressions spiking without clicks” as the early-warning sign.

   EvidencePrior-window clicks is the top permutation-importance feature (0.077 PR-AUC) with a negative coefficient (−0.583), while prior-window impressions carries the largest positive one (+0.910). Grouped directly: among pages with ≥500 impressions, the lowest click-through quartile declined at 73.6% against 48.9% in the highest (n ≈ 2,780 per quartile).

   Wrong ifThe page deliberately serves an on-SERP informational need, where low CTR is normal.

4. 4Split the work by reason code instead of calling everything a refresh.

   EvidenceOf the top 200: 97 refresh reviews, 62 metadata/intent reviews (clicks falling while impressions hold — a title job, not a rewrite), 27 protect-and-refresh, 5 protect-and-watch on pages that are gaining, and 9 monitor-only.

   Wrong ifThe thresholds behind the reason codes move — they are policy choices, not constants.

5. 5Keep a human gate in front of every action, and automate nothing.

   EvidenceThe model never reads page content, cannot see a sibling page absorbing demand, and cannot separate seasonality from decline. 9 of the top 200 carry only model_pattern_only and are downgraded to monitor by design.

   Wrong ifNothing — this is a design boundary, not a tunable parameter.

The four states, and which of them the engine can act on

The lane asks for pages scored as growing, declining, recovering or worth review. Those states sit on opposite sides of the decision point, so the queue carries them in two separate columns and never mixes them:

• decision_state — actionable. Built from the prior window only: slipping, steady, spiking. In the top 200: 124 slipping, 41 steady, 35 spiking.
• outcome_state — reporting only. What actually happened: declining, recovering, growing, stable. This is label-side information, so it is never a feature and never something the engine claims to predict.

What actually happened to the pages the engine ranked highest. Read the top row against the bottom one: the queue concentrates decline and pushes growth out.

The queue itself

The first eight rows of an 18,010-row queue. IDs are pseudonyms — no client, domain, URL or query appears anywhere in this work.

Confidence: medium at the top of the queue (precision@50 holds at 0.816 ± 0.048 across client holdouts), low through the middle deciles, and out of scope entirely below the 100-impression floor, where the system stays silent rather than reassuring.

07Run it yourself

Reproducibility

Every number on this page is produced by one script and stored as JSON in the repository, and this page is generated from those files — so the paper cannot drift away from the pipeline.

git clone https://github.com/tu-h-nguyn/FlyRank-Machine-Learning-Internship
cd FlyRank-Machine-Learning-Internship
pip install -r requirements.txt

python work/scripts/capstone_pipeline.py   # metrics + figures  (~25s, seed 42)
python work/scripts/build_paper.py         # rebuilds this page

Seed 42 throughout; GroupKFold is deterministic. Python 3.11, pandas 2.x, scikit-learn 1.7.x. Random-forest numbers can move a point or two between scikit-learn versions; the shipped logistic regression is stable. The 18,010-row queue CSV is deliberately not committed — repository policy blocks dataset files — and is regenerated by the command above. No sealed-holdout claim is made anywhere: these are out-of-fold cross-validation numbers on one snapshot.

08Acknowledgments & data credit

Credit

Built on the FlyRank ML Internship dataset — real, anonymized search and content performance data made available for this internship: flyrank.ai. The March 2026 FlyRank research paper is the reference audited in the validation notebook; the questions raised there are methodological, offered in the spirit of making the claims stronger rather than correcting them.