Tokens Not Jokin'

Tokens Not Jokin’

1. How API Documentation Format Affects AI Code Generation

Acknowledgments

Chapter 1: Who’s Reading Your Docs Now?

1. The Shift Nobody Planned For
2. The Invisible Integration
3. What AI Actually Does With Your Docs
4. Your Docs Have Two Audiences Now
5. The Metrics Blind Spot
6. What This Book Will Show You

Chapter 2: The Token Budget

1. What Tokens Actually Are
2. The Four-Format Comparison
3. The Attention Problem
4. Why Bigger Windows Don’t Fix This
5. The Compounding Problem
6. The Caching Argument
7. The Cost Math
8. The Real Budget

Chapter 3: The Measurement Gap

1. An Autopsy of Traditional Metrics
2. The Metrics That Matter for AI
3. AI Acceptance Testing
4. Building Your Testing Pipeline
5. The Dashboard You Actually Need
6. Closing the Gap

Chapter 4: Building a Fair Test

1. The Contamination Problem
2. The BookClub API: Simple Baseline
3. The EventForge API: Complexity Gradient
4. Why Two APIs
5. The Four Formats
6. Controlling Variables
7. Why Temperature 0
8. Reproducibility

Chapter 5: The Models

1. Cloud Models
2. Local Models
3. Why These Four
4. What We Ran
5. The Cost of Testing
6. What the Models Tell Us (Preview)

Chapter 6: The Results

1. Cloud Models: 100% Across the Board
2. Small Models: Where Format Matters Most
3. Beyond Pass Rates: Format and Code Patterns
4. The Variance Finding
5. The Four Formats
6. The Summary

Chapter 7: The Four Formats

1. YAML
2. OpenAPI 3.0
3. DON
4. Markdown
5. The Format Summary

Chapter 8: DON: A New Kind of Documentation

1. The Hypothesis
2. The Result: 100% on Cloud Models
3. The Aggregate
4. Explicit vs Implicit: The Critical Distinction
5. Documentation as a Code Quality Control Mechanism
6. Small Model Behavior: The Complexity Wall
7. What Else Could Annotations Influence?
8. DON as a Proof of Concept

Chapter 9: Choosing Your Format

1. The Decision Framework
2. The Dual-Format Strategy
3. Adding Behavioral Annotations
4. What to Cut
5. Migration Checklist

Chapter 10: Building AI Acceptance Tests

1. What You’re Building
2. Prerequisites
3. Step 1: Define Your Tasks
4. Step 2: Prepare Your Documentation
5. Step 3: Build the Test Harness
6. Step 4: Run Your First Test
7. Step 5: Interpret the Results
8. Step 6: Compare Formats
9. Step 7: Add a Local Model
10. Sample Size and Statistical Power
11. Integrating with CI/CD
12. Using Our APIs for Benchmarking
13. Common Pitfalls

Chapter 11: Token Optimization Techniques

1. The Optimization Stack
2. Removing Redundancy
3. Consolidating Descriptions
4. Strategic Ordering
5. Per-Endpoint Delivery
6. The Annotation Budget
7. Before/After: A Real Optimization
8. Measuring Improvement
9. When to Stop

Chapter 12: Serving Docs to AI

1. The Serving Problem
2. Approach 1: Separate Paths
3. Approach 2: Content Negotiation
4. Approach 3: Discovery Mechanisms
5. Per-Endpoint Delivery
6. MCP Server Integration
7. CDN and Caching
8. Versioning Strategy
9. Monitoring
10. Implementation Checklist

Chapter 13: What the Data Shows

1. Format Explains More Than Model Choice
2. The Wrong Question
3. The Documentation Author’s Influence
4. The Cost Argument
5. What API Providers Can Control
6. Implications
7. The One-Sentence Version

Chapter 14: What We Don’t Know Yet

1. Single-Endpoint Tasks
2. Python Only
3. REST APIs Only
4. Four Models at One Point in Time
5. No Models Between 7B and Cloud Scale
6. DON Annotations Beyond Error Handling
7. Determinism and Temperature
8. Long-Term Format Drift
9. What Should Be Tested Next

Chapter 15: What Comes Next

1. Structured Specs as Source of Truth
2. Documentation as Code Quality Input
3. Automated Testing Loops
4. MCP and Documentation Convergence
5. What Standards Could Help
6. What Might Change
7. What to Do This Week

Appendix A: Complete Test Methodology

1. Study Design
2. Test Infrastructure
3. Dataset Size
4. Statistical Methods
5. Power Analysis
6. Task Design
7. Reproducibility
8. Data Availability
9. Threats to Validity

Appendix B: Format Specifications

1. Format Size Comparison
2. YAML Format
3. OpenAPI 3.0 Format
4. DON Format
5. Markdown Format
6. Conversion Examples

Appendix C: The DON Specification

1. Format Structure
2. Annotation Syntax
3. Complete Example: BookClub API in DON
4. Writing Your Own DON Spec
5. Incorporating DON Principles Into Existing Formats

Appendix D: Results Tables

1. Master Pass Rate Table
2. OpenAPI Failure Detail
3. Code Pattern Tables
4. Code Length Tables
5. Variance Explained (Eta-Squared)
6. Determinism Analysis
7. Complex Task Handling (EventForge Cloud)
8. Token Efficiency Summary
9. Statistical Test Reference

Appendix E: Verified Claims Reference

1. Claims You Can Use Freely
2. Claims That Require Qualification
3. Common Misinterpretations to Avoid
4. Every Statistical Claim with Source
5. For Presentations

Appendix F: Tools & Resources

1. Companion Repository
2. Token Counting
3. AI Model APIs
4. Format Conversion
5. Test Frameworks
6. Documentation Platforms
7. Discovery Mechanisms
8. AI Acceptance Testing Starter Kit
9. Repository Contents
10. Further Reading