Blog Post
Measuring Whether Your Agent
Configuration Actually Works
A new measurement dimension for rp-why that closes the feedback loop on AGENTS.md configuration. Does your declared configuration actually serve your workflow? Now there is data to answer that.
Context
The Blind Spot
I have been configuring my AI agent for 140+ days. My AGENTS.md file has implicit commands, boundaries, formatting rules, session patterns, workspace routing. It feels comprehensive. It feels like it is working.
But until today, I had no data to prove it.
Every practitioner who configures an AI agent faces the same blind spot: you write instructions, the agent reads them, and then you hope. There is no feedback loop. No telemetry. No way to know which instructions the agent follows reliably, which it ignores, and which address situations that never arise.
Configuration is not free. Every instruction consumes context window tokens on every session. Dead instructions are not neutral - they are a tax on every interaction. And unreliable instructions are worse than no instructions at all, because they create false confidence.
I built a measurement for this. It is called Configuration Effectiveness (CE), and it is coming to the next version of rp-why.
Measurement
What CE Measures
CE answers one question: Is your declared configuration actually serving your workflow?
It does this by parsing your AGENTS.md files, extracting measurable instructions, scanning your session history for evidence of adherence, and computing an effectiveness score.
Phase 1 (shipping now) focuses on implicit commands - the shortcut triggers you define in your AGENTS.md that should fire without confirmation. These are the highest-signal, lowest-ambiguity instructions to measure.
For example, my AGENTS.md declares:
| "proceed" | Continue with the current plan. No confirmation needed. |
| "#N complete" | Mark item N as done on the active list. Show remaining items. |CE scans my session history, finds every instance where I said "proceed" or "#3 complete", and checks whether the agent actually did what the table says. Did it continue without asking? Did it mark the item and show remaining?
Action Fingerprinting
Detection uses observable signals in the session data that indicate the expected behavior occurred. For "proceed": tool calls present (action taken), no question mark in the response (did not ask for confirmation). For "#N complete": response contains a list with items marked done.
Results
What I Found When I Measured
0.28
Initial CE score
9/13
Commands never fired
74.9%
Tokens wasted on dead config
0.75
CE after fix
First measurement: CE 0.28 (Misconfigured). That was a surprise. I thought my config was solid. But the data showed:
9 of 13 implicit commands had never fired in 20 sessions
74.9% of my config tokens were spent on instructions that never triggered
Commands like "refresh the emulator" and "start the dev server" were in my global config but only relevant in specific projects
The fix was straightforward: move project-specific commands out of the global config and into project-level AGENTS.md files where they actually fire.
After the fix and fingerprint calibration: CE 0.75 (Well-tuned). The commands that fire ("proceed" at 3/3, "#N complete" at 2/2) score perfectly. The remaining penalty comes from low-frequency commands ("merged", "sync") that are high-value when they fire but only trigger 1-2x per month.
Trust
How CE Relates to Trust
You cannot safely delegate to an agent that does not follow your instructions.
CE is not a standalone metric. It is an input to Agentic Delegation Trust (ADT) - the existing rp-why dimension that measures how much autonomy you can safely grant your agent. Before CE, ADT was inferred from the relationship between cognitive depth (DOK) and tool sophistication (TM). Now ADT has direct evidence: does the agent actually do what you told it to?
Four quadrants emerge from CE x ADT:
| Quadrant | What It Means |
|---|---|
| Earned Autonomy ADT High, CE High | Delegating heavily and the config is holding. Target state. |
| Reckless Trust ADT High, CE Low | Delegating heavily but instructions are not being followed. Silent failures accumulating. |
| Ready to Trust ADT Low, CE High | Config is proven reliable but delegation has not caught up. Growth edge. |
| Justified Caution ADT Low, CE Low | Not delegating much, and the config is not working anyway. Fix config first. |
My Current Quadrant
Ready to Trust. The configuration is proven. The data says I can safely extend autonomy further.
Architecture
Event-Driven Measurement
CE does not run on every session. That would be wasteful. Instead, it is event-driven:
Config changes (AGENTS.md file hash differs from last recorded version) trigger a new measurement cycle
Window boundaries (every 20 sessions) check for drift even if config is stable
Manual invocation (/rp-why ce) runs on demand
Each config version gets its own score. Over time, you see the impact of your edits:
Config v1 (Jun 1 - Jun 20): CE 0.68 across 15 sessions
Config v2 (Jun 20 - Jul 2): CE 0.75 across 12 sessions
Delta: +0.07 from the updateThis creates a feedback loop that is currently missing from the AGENTS.md ecosystem. Your configuration becomes a living document that improves based on data, not intuition.
Nudges
Growth Nudges
CE does not just score you. It tells you what to do next. The nudges are band-specific (Optimized, Well-tuned, Developing, Under-effective, Misconfigured) and quadrant-specific (Earned Autonomy, Reckless Trust, Ready to Trust, Justified Caution). They meet you where you are.
When things are working
- •"All boundaries held across 20 sessions. Your safety configuration is solid."
- •"'proceed' has fired 10/10 times correctly. This implicit command is well-designed."
When things need attention
- •"These instructions have not fired in 20 sessions. Consider moving them to a project-level config."
- •"You corrected the agent on this topic 4 times. Consider adding an instruction."
When you are ready to grow
- •"Configuration is proven reliable. What is the next workflow you could encode as an implicit command?"
- •"The data says the agent follows your instructions. What is holding you back from delegating more?"
Scope
Why This Matters Beyond Engineering
The AGENTS.md pattern is expanding beyond software engineering. Educators are exploring classroom.agents.md for configuring AI dialogue partners. Creative practitioners configure AI collaborators with style constraints. Organizations use world models as agent context layers.
All of these face the same measurement gap: is the configuration working?
CE generalizes. Any practitioner who configures an AI agent - regardless of domain - benefits from knowing which instructions land and which do not. The mechanism is the same: parse the config, scan the sessions, report adherence.
Shipping
What Is Shipping
CE is implemented as a new module in rp-why (rp_why_ce.py) and available via the /rp-why ce command. Phase 1 covers implicit command detection with action fingerprinting.
Future phases will add:
Formatting rule detection (binary signals like em dash presence)
Boundary violation detection
Verification completion tracking
Override frequency analysis
The code is open source at block/agent-skills (PR #52).
Try It
Try It Yourself
If you use Goose and have an AGENTS.md with implicit commands:
Install the rp-why skill
Run /rp-why ce
See which instructions are earning their token cost
# Install the skill
npx skills add https://github.com/block/agent-skills --skill rp-why
# Run Configuration Effectiveness
/rp-why ce
The first measurement is always surprising. Mine certainly was.
Related
Related Links
rp-why Skill on GitHub
The open-source Goose skill where Configuration Effectiveness ships as a new module.
Case Study: 135 Days of Daily Goose Usage
The empirical foundation that CE builds on - 135 days of instrumented daily practice.
Production Agent Workflow
The AGENTS.md workflow pattern that CE measures. A template for organizing your agent context.
rp-why Data Visualization
Interactive charts showing 141 days of trajectory data, zone diagnostics, and token economics.