We do not replace Huwise. We propose a safe measurement and optimisation loop, on top of its existing tools.
One map, four readings
Click a block: role, inputs/outputs, real tools, example, status and sources. Lines distinguish metadata, schema, data queries, usage feedback and governance.
“Combien y a-t-il de collèges à Dijon ?” (How many middle schools in Dijon?)
Every step actually executed: input, tool, output, time, tokens, source, status. Configuration A uses Huwise MCP tool names, but it is not the Huwise server (an API key is required): it is our agent, same model, with generic tools.
A reproducible error, and what catches it
Real queries on data.education.gouv.fr, replayable by anyone. These are not Huwy errors: they are format traps any agent meets.
nom_commune='DIJON' → 0 rows; the data says 'Dijon' → 23.
where = nom_commune='DIJON' and type_etablissement='Collège'
code_departement='21' → 0 rows; the code is written '021' → 74 middle schools.
where = code_departement='21' and type_etablissement='Collège'
Context Optimizer
What the Optimizer does: on an empty result it asks the portal how the value is written and tells the agent; on the FAST route the value is resolved before the query.
{"total_count": 0, "hint": ["nom_commune: 'DIJON' is stored as 'Dijon'"]}
{"total_count": 0, "hint": ["code_departement: codes are written with leading zeros, try '021'"]}One change at a time, measured, kept or reverted
The lab may change only cards (description, synonyms, query template, note). It never sees the holdout and cannot touch the evaluator, the expected answers, the validator or the log.
Budget per cycle
- max_proposals = 3
- max_llm_calls = 150
- max_seconds = 3600
- control_cases = 10
Quality gate (fixed in advance) gate-2026-11-11.1
- min_fixed = 1
- max_broken_control = 0
- max_new_wrong_confident = 0
- max_new_query_errors = 0
- max_tokens_increase_pct = 25
Log (append-only, chained)
No cycle recorded yet.
Same questions, same model, three configurations
A: generic tools named like Huwise MCP's (not the Huwise server). B: our compiled cards. C: cards + Context Optimizer. Scored without an LLM judge: the value is compared with a reference API query.
The real Huwise MCP server sends facets our A lacks: A probably underestimates Huwise. A real A test needs a portal API key.
No complete A/B/C run yet.
Deterministic Optimizer checks (no model) RECORDED 2026-11-11T15:41:06+00:00
| Recipes: questions taken FAST / correct | education : 4/30 · 4/4 education_v2 : 57/95 · 57/57 |
| Metamorphic relations (public ≤ all, public + private ≤ all) | 18/18 |
| Per-question compression: needed fields kept · tokens saved (median) — too risky, not enabled | 76/117 · −57.6 % |
| FAST route latency (10 runs): cold p50 / p95 · warm p50 / p95 | 54 / 73 ms · 9 / 13 ms |
Earlier internal experiments (A vs B only, 30-question set seen during development) — history, not evidence against Huwy.
| model | A | B | runs | date |
|---|---|---|---|---|
| openai/gpt-5.4-mini | 40.0 % | 90.0 % | #10/#11 | 10.11.2026 |
| nvidia/nemotron-3-ultra-550b-a55b:free | 30.0 % | 80.0 % | #3/#4 | 10.11.2026 |
Prod only changes by a human decision
ctxc database · cards index · Neo4j graph (write) · read-only public MCP · no admin endpoint
ctxc_preprod database · cards_preprod index · shared graph read-only · the lab runs only here · capped model budget
Promotion: a human reads the diff and the measurements, applies the change in prod, watches (canary), and can return to the previous card version in one command.
What we borrow, and what these works do not prove for Huwise
None of these works measures a Huwise portal: only this page's evaluation says anything here.
AutoResearch — Andrej Karpathy
DSPy / MIPRO — Opsahl-Ong et al., EMNLP 2024
LLMCompiler — Kim et al., 2023
RouteLLM — Ong et al., 2024
Metamorphic & property-based testing
OpenTelemetry · SRE (SLI/SLO)
The code, the data, the deployment
| module | rôle / role |
|---|---|
| sources/explore.py | Explore API v2.1 scan: 314 datasets, 6 976 fields → MariaDB |
| ctx/compile.py · ctx/export.py | cards (short ≤ 400, full ≤ 1 500 tokens), versioned · llms.txt + 314 pages |
| projections/es.py · graph.py | Elasticsearch BM25 + usage boost (routing per portal) · Neo4j shared keys / lineage |
| tools/ · mcpserver.py | MCP find_dataset · explain_dataset · query (ODSQL whitelist) · /api/demo/run |
| optimizer/ | recipes · values · router · cache · evidence · pipeline · scenarios · record_traces |
| lab/ | cycle (guard, diagnose, static check, gate, budget) · log (hash chain) · policy.json |
| evals/ | golden sets (education 30, education_v2 95 with dev/val/holdout) · runner (A/B/C, no LLM judge) · improve |
Honest limits
- A is not the Huwise MCP server: measuring the real one needs a portal API key (an account owner's decision).
- Free model (OpenRouter): results vary between runs; sample sizes shown, confidence intervals are wide.
- The 30 legacy questions were seen during development; the v2 holdout is the clean measure, but it was written by the same author.
- Recipes cover one question shape (count establishments by type, status, place); everything else goes through the agent.
- Per-question compression and tool parallelism: described, not enabled; preprod shares the MariaDB/ES/Neo4j servers (separate databases and indexes).
- Huwise internals (storage, ranking, orchestration, Huwy's model) are not public: the map does not claim them.
/data/architecture.json · /data/scenarios.json · /data/comparison.json · /data/lab.json · /llms.txt · MCP: https://context.smiro.dev/mcp