06 — Roadmap: future features, rationale, and suggestions
The roadmap at a glance
MVP (001) Competition analysis · baseline · bounded experiment loop
· research log · MLflow · memory ✅ shipped
│
▼
V2 (002) Strategy Engine — a better experiment chooser
│
▼
V3 (003) Automatic Ensembling — blend / rank-average / stack
│
▼
V4 (004) Cross-Competition Memory — transferable ML research knowledge
│
▼
V5 (005) External research — forum mining, notebook mining, papers
│
▼
V6 (006) Multi-agent research teams — deliberately last
Each item is its own spec-kit feature (own branch, spec, plan, tasks — see 05-development-workflow.md).
Why this order
After a few hundred experiments, the bottleneck changes:
Initially: Problem: no experiments exist.
Later: Problem: too many possible experiments.
The real challenge becomes: which experiment should I run next? That is where a human Kaggle Grandmaster spends much of their value. So:
- Strategy Engine before forum mining. External knowledge = more ideas; the strategy engine = better selection of ideas. A mediocre set of ideas selected intelligently often outperforms a huge set selected randomly.
- Single well-designed agent before multi-agent. A single agent with strong memory and a strong strategy engine usually outperforms a swarm of agents that cannot prioritize experiments.
- One correction to the naive greedy loop: the first post-MVP work should also make the loop an experiment portfolio manager — branch from several strong parent solutions instead of keeping only one incumbent. A greedy keep/reject loop can get stuck on a local optimum; a small frontier of candidates preserves diversity and prepares the ground for ensembling (V3).
V2 — Strategy Engine (feature 002): meta-learning / research strategy
Not a better model generator — a better experiment chooser.
Input: research log, experiment history, leaderboard scores, insights.
Output:
recommended_direction: feature_engineering confidence: 0.82 reason: Last 25 experiments indicate feature changes have produced 85% of total score gains.Mechanism sketch: aggregate score gains by experiment family (feature engineering / hyperparameters / model family / ensembling). Example: 50 feature experiments → +0.025, 20 hyperparameter → +0.003, 10 model switches → +0.002 ⇒ next effort focuses on features. This is the agent learning how to research.
Explore vs exploit: the strategy engine also decides when to try new feature families vs tune what works (formal decision rules over the research log, not vibes).
V3 — Automatic Ensembling (feature 003)
A large share of Kaggle ranking comes from blending, rank averaging, stacking, and weighted ensembles. The agent should automatically discover:
Model A: 0.812
Model B: 0.811
Model C: 0.809
Ensemble: 0.818
Often one of the highest-ROI capabilities to add. Depends on the V2 portfolio (frontier of candidates) and on the diversity term in the acceptance function (04-automlab-lessons.md).
V4 — Cross-Competition Memory (feature 004)
Transfer learned patterns across competitions:
Competition 1: Credit Risk ┐
Competition 2: Fraud Detection ├─► "target encoding often helps",
Competition 3: Customer Churn ┘ "CatBoost handles high-cardinality
categoricals well", "adversarial validation
detects leaderboard shakeup risk"
Every new competition then starts with accumulated experience. At that point the agent looks less like a competition-specific tool and more like an actual machine-learning researcher. Storage: the research memory plus a queryable insight store (vector DB such as Qdrant/LanceDB is a candidate here, not before).
V5 — External research (feature 005)
Forum mining, public-notebook mining, paper reading — a Kaggle-specific “deep research” component. Studies the discussion forum, previous competitions, public notebooks, and papers; stores findings in memory (e.g. “CatBoost performs well here, frequency encoding helps, rank averaging is common, adversarial validation is useful”).
V6 — Multi-agent research teams (feature 006)
Only after 001–005 prove out. Possible decomposition: Competition Analyzer, Research Agent, Feature Engineering Agent, Modeling Agent, Experiment Manager, Ensemble Agent, Submission Agent, Strategy Agent — orchestrated behind the same harness (which stays the single authority on validation, budgets, and memory).
Implementation suggestions (from the architecture research)
Agent runtime: reuse Goose (or Pi) as the runtime — agent loop, model access, tool calling, sessions, recipes. A competition recipe packages instructions, parameters (
competition_slug,time_budget_minutes,maximum_experiments), extensions, and a structured experiment-report schema.Thin Kaggle MCP server exposing high-level, safe tools:
inspect_competition(),profile_dataset(),create_baseline(),propose_experiment(),run_experiment(),evaluate_predictions(),compare_experiments(),record_insight(),create_submission(),request_submission_approval(). Enforcement (fixed folds, timeouts, seeds, artifact collection, failure classification) lives in the tool code, not the prompt.Human cockpit: connect the runtime to Zed via ACP for reviewing hypotheses, diffs, experiment summaries, approving submissions, and interrupting runs. Optional — the CLI remains the stable interface.
Leaderboard strategy (later): understand public/private split, estimate shake-up risk (adversarial validation), and plan submission allocation (e.g. 5/day → 2 exploration, 2 exploitation, 1 ensemble). Model selection itself never uses the public leaderboard.
Experiment record (target shape for the loop, extending feature 001’s schema):
{ "experiment_id": "exp-0007", "parent_id": "exp-0004", "hypothesis": "Frequency encoding will improve high-cardinality categoricals.", "expected_effect": "Increase mean CV AUC by at least 0.001.", "files_changed": ["solution/features.py"], "budget_seconds": 1200, "metric": { "name": "roc_auc", "mean": 0.8421, "std": 0.0019 }, "baseline_metric": 0.8404, "runtime_seconds": 684, "status": "accepted", "conclusion": "Validated with consistent improvements across four of five folds.", "git_commit": "abc123", "next_candidates": ["Combine frequency encoding with CatBoost"] }Self-evaluation after each experiment: was the hypothesis reasonable? was the experiment informative? was the result statistically meaningful? was compute wasted? should similar experiments be avoided? Emit a self-evaluation report alongside every result.
Defer until the protocol is stable: MLflow-only tracking is already in; add a vector DB and a strategy service only once the experiment schema and acceptance function are proven.
Out of scope for the foreseeable roadmap
Vision/NLP competitions, deep-learning training, GPU clusters, distributed systems, Kubernetes, multi-user collaboration, web UI, cloud execution, autonomous Kaggle uploads in the MVP. Success is objectively measurable (leaderboard score), experiments are cheap, and the feedback loop is tight — that is what makes this domain realistic for autonomous AI.
Resources
- karpathy/autoresearch — the autoresearch pattern (mirrored locally in
automlab/) automlab/— working reference: see 04-automlab-lessons.mdopenai-research.md— full architecture conversation (vision, layered stack, MVP spec, post-MVP priorities)TODO.md— original roadmap notesWORKFLOW.md— SDD workflow and feature anatomy
Citation
@online{untitled,
author = {},
url = {https://guillaumegilles.github.io/autokagglab/06-roadmap.html},
langid = {en}
}