Discovery Infrastructure for AI Agents: A Practical Course Lesson on llms.txt, agents.json, OpenAPI, and Semantic HTML

Learning Objectives

By the end of this lesson, you will be able to:

Define discovery infrastructure and explain why it matters for AI agents operating autonomously
Distinguish between the four core discovery standards: llms.txt, agents.json, OpenAPI, and semantic HTML
Describe how these standards complement each other in a layered discovery stack
Apply practical implementation patterns when building or consuming agent-ready services
Identify common pitfalls and avoid them in real deployments

1. What is Discovery Infrastructure?

When a human wants to use a web service, they read a landing page, scan documentation, and figure out what the service does. An AI agent cannot do this reliably. It needs structured, machine-parseable signals that describe what a service offers, how to call it, what it costs, and what constraints apply.

Discovery infrastructure is the collection of standards, file formats, and markup conventions that make services legible to AI agents without human mediation.

Think of it as the difference between a restaurant with no menu (a human can ask a waiter) and one with a printed menu, a QR code, and an online ordering API (a system can place an order autonomously). Discovery infrastructure is the menu layer for the agent economy.

Why This Matters Now

The agent economy is shifting from humans-as-orchestrators to agents-as-orchestrators. When an agent needs to:

Find a service that can perform a specific task
Verify the service's capabilities and pricing before committing
Call the service correctly on the first attempt
Handle errors and rate limits without human intervention

...it needs reliable, structured discovery signals. Without them, agents either fail silently, hallucinate API behaviour, or require expensive human-in-the-loop intervention at every integration point.

Discovery infrastructure solves the cold-start problem for agent-to-service interaction.

2. The Four Pillars: llms.txt, agents.json, OpenAPI, and Semantic HTML

These four standards operate at different layers of specificity and serve different consumers. Together they form a complete discovery stack.

Standard	Primary Consumer	Scope	Format
llms.txt	LLMs and crawlers	Site-level intent and permissions	Plain text
agents.json	Autonomous agents	Service capabilities and pricing	JSON
OpenAPI	Developer tools and agents	API endpoint contracts	JSON/YAML
Semantic HTML	Agents and humans	Page-level structured content	HTML with schema.org

2.1 llms.txt: Machine-Readable Service Declarations

What it is: A plain-text file placed at the root of a domain (e.g., https://example.com/llms.txt) that declares how the site owner wants AI systems to interact with their content.

What it contains:

A brief description of the site's purpose and primary audience
Explicit permissions or restrictions for AI training, crawling, and summarisation
Links to structured resources (documentation, APIs, agent-specific endpoints)
Contact or attribution preferences for AI-generated content

Analogy: robots.txt tells crawlers what not to index. llms.txt tells language models what the site is and how it wants to be used.

Example structure:

# Example Service
> A research API providing structured notes on the agent economy.

## Docs
- [API Reference](https://example.com/api/docs): Full endpoint documentation
- [Pricing](https://example.com/pricing): Token-based and subscription tiers

## Notes
- Content may be used for agent reasoning tasks
- Attribution required for republication
- Training use: not permitted without written agreement

Key implementation points:

Keep it short — under 500 words is a strong norm
Use Markdown formatting; most LLM parsers handle it well
Link to your OpenAPI spec and agents.json from here
Update it when your service changes materially

What agents do with it: An agent discovering a new service will fetch llms.txt first to understand intent and permissions before attempting any API calls. It functions as a trust and orientation signal.

2.2 agents.json: Agent Capability Manifests

What it is: A structured JSON file (typically at /agents.json or /.well-known/agents.json) that declares the specific capabilities a service exposes to autonomous agents, along with pricing, authentication requirements, and usage constraints.

What it contains:

Service identity: Name, version, description, and category
Capabilities list: Discrete actions the agent can invoke (e.g., search, summarise, generate_report)
Pricing model: Per-call costs, token rates, subscription tiers
Authentication: Required credential types (API key, OAuth, bearer token)
Rate limits: Requests per minute/hour, concurrency limits
Output formats: What the service returns (JSON, Markdown, structured objects)

Example structure:

{
  "name": "ResearchAPI",
  "version": "2.1",
  "description": "Structured research notes on the agent economy",
  "capabilities": [
    {
      "id": "search_notes",
      "description": "Search published research notes by topic or keyword",
      "input": {"query": "string", "limit": "integer"},
      "output": "application/json",
      "cost_per_call": {"usd": 0.002}
    },
    {
      "id": "fetch_note",
      "description": "Retrieve a full research note by ID",
      "input": {"note_id": "string"},
      "output": "text/markdown",
      "cost_per_call": {"usd": 0.005}
    }
  ],
  "auth": {"type": "api_key", "header": "X-API-Key"},
  "rate_limits": {"requests_per_minute": 60},
  "openapi_url": "https://example.com/openapi.json"
}

Why agents need this beyond OpenAPI: OpenAPI describes how to call an API. agents.json describes whether an agent should call it — given cost, capability fit, and constraints. It is a decision-support layer, not just a technical contract.

Key implementation points:

Keep capability IDs stable; agents may cache them
Be precise about costs — agents use this for budget management
Include a link to your OpenAPI spec for technical detail
Version the file; breaking changes should increment the version field

2.3 OpenAPI: Standardized API Contracts

What it is: The OpenAPI Specification (formerly Swagger) is a widely adopted standard for describing REST APIs in a machine-readable format (JSON or YAML). It defines endpoints, request/response schemas, authentication methods, and error codes.

What it contains:

Paths: Every available endpoint and HTTP method
Parameters: Query, path, header, and body parameters with types and validation rules
Schemas: Request and response object structures
Security: Authentication schemes
Examples: Sample requests and responses

Why it matters for agents: An agent with access to a valid OpenAPI spec can:

Construct correct API calls without trial and error
Validate its own outputs before sending requests
Understand what errors mean and how to handle them
Generate code or tool-use calls programmatically

Example snippet:

openapi: 3.1.0
info:
  title: Research Notes API
  version: "2.1"
paths:
  /notes/search:
    get:
      summary: Search research notes
      parameters:
        - name: query
          in: query
          required: true
          schema:
            type: string
        - name: limit
          in: query
          schema:
            type: integer
            default: 10
      responses:
        "200":
          description: List of matching notes
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: "#/components/schemas/NoteSummary"

Key implementation points:

Use OpenAPI 3.1 — it aligns with JSON Schema and is better supported by modern agent frameworks
Write clear summary and description fields; agents use these for tool selection
Include realistic examples in your schema definitions
Expose your spec at a stable, well-known URL (e.g., /openapi.json)
Validate your spec with a linter before publishing — malformed specs cause agent failures

The agent tooling ecosystem: Most major agent frameworks (LangChain, LlamaIndex, AutoGen, and others) can ingest an OpenAPI spec and automatically generate callable tools. A well-formed spec is therefore a direct path to agent compatibility.

2.4 Semantic HTML: Human-Readable Service Documentation

What it is: HTML pages that use structured markup — particularly schema.org vocabulary embedded via JSON-LD or microdata — to make page content machine-interpretable alongside being human-readable.

Why it matters for agents: Not every agent interaction starts with a known API. Agents browsing the web to discover services will encounter HTML pages. Semantic markup allows them to extract structured facts (service name, pricing, capabilities, contact information) without relying on fragile text parsing.

Key patterns:

JSON-LD blocks: Embed structured data directly in <script type="application/ld+json"> tags
Schema.org types: Use Service, SoftwareApplication, APIReference, Offer types to describe your service
Descriptive headings: Use <h1>–<h3> hierarchy that reflects information structure, not just visual design
ARIA labels: Help agents understand interactive elements
Canonical URLs: Prevent agent confusion from duplicate content

Example JSON-LD for a service page:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "name": "Research Notes API",
  "description": "Structured research notes on the agent economy, accessible via API",
  "applicationCategory": "ResearchTool",
  "offers": {
    "@type": "Offer",
    "price": "0.002",
    "priceCurrency": "USD",
    "description": "Per API call, search endpoint"
  },
  "url": "https://example.com",
  "documentation": "https://example.com/api/docs"
}
</script>

The human-agent bridge: Semantic HTML is the only layer in the discovery stack that serves both humans and agents equally. It is the fallback when dedicated agent files are absent, and it reinforces the signals in llms.txt and agents.json for agents that cross-reference multiple sources.

3. How These Standards Work Together

The four standards form a layered stack, each building on the others:

┌─────────────────────────────────────────────┐
│  llms.txt          — Site intent & permissions  │  Layer 1: Orientation
├─────────────────────────────────────────────┤
│  agents.json       — Capabilities & economics   │  Layer 2: Decision
├─────────────────────────────────────────────┤
│  OpenAPI           — Technical call contracts   │  Layer 3: Execution
├─────────────────────────────────────────────┤
│  Semantic HTML     — Content & fallback layer   │  Layer 4: Fallback/Enrichment
└─────────────────────────────────────────────┘

A typical agent discovery sequence:

Agent receives a task requiring an external research service
Agent fetches llms.txt → confirms the site is agent-friendly and finds links to structured resources
Agent fetches agents.json → evaluates whether the service's capabilities match the task and whether the cost fits its budget
Agent fetches the OpenAPI spec linked from agents.json → constructs the correct API call
If any of the above are missing, agent falls back to parsing semantic HTML to extract what it can

Redundancy is a feature: A service that implements all four layers is resilient to agent framework differences. Some frameworks prioritise OpenAPI; others start with llms.txt. Full implementation ensures compatibility across the ecosystem.

4. Practical Implementation Patterns

4.1 Building a Discovery-Ready Service

Minimum viable discovery stack:

Start with OpenAPI — if you have an API, document it. This is the highest-value single investment.
Add llms.txt — takes under an hour, signals intent immediately to crawlers and LLMs.
Add agents.json — structure your capability and pricing information; link to your OpenAPI spec.
Audit your HTML — add JSON-LD to your main service and documentation pages.

Implementation checklist:

[ ] OpenAPI spec at /openapi.json, validated and versioned
[ ] llms.txt at root with links to OpenAPI and agents.json
[ ] agents.json at /.well-known/agents.json with stable capability IDs
[ ] JSON-LD on homepage and key documentation pages
[ ] Semantic heading hierarchy on all documentation pages
[ ] Canonical URLs set on all pages
[ ] CORS headers configured to allow agent-origin requests

Versioning discipline:

Treat your agents.json and OpenAPI spec as public contracts
Use semantic versioning; increment major version on breaking changes
Keep old versions accessible at versioned URLs for a deprecation window
Announce deprecations in llms.txt and via API response headers

4.2 Consuming Discovery Infrastructure as an Agent

Agent-side discovery workflow:

# Pseudocode: agent service discovery pattern

def discover_service(domain):
    # Step 1: Fetch orientation layer
    llms_txt = fetch(f"{domain}/llms.txt")
    if not permits_agent_use(llms_txt):
        return None  # Respect permissions

    # Step 2: Fetch capability manifest
    agents_json = fetch(f"{domain}/.well-known/agents.json")
    if not fits_task_requirements(agents_json, current_task):
        return None  # Wrong capability or over budget

    # Step 3: Fetch technical contract
    openapi_url = agents_json.get("openapi_url")
    openapi_spec = fetch(openapi_url)

    # Step 4: Build callable tools
    tools = build_tools_from_openapi(openapi_spec)
    return tools

def fallback_discovery(domain):
    # If structured files absent, parse semantic HTML
    html = fetch(domain)
    return extract_schema_org(html)

Caching strategy for agents:

Cache agents.json aggressively — it changes infrequently; use Cache-Control headers
Cache OpenAPI specs per version — invalidate on version change
Do not cache llms.txt for more than 24 hours — permissions can change
Store capability IDs, not full specs, in agent memory for repeated use

Budget-aware consumption:

Parse cost fields from agents.json before calling
Maintain a per-task budget and reject services that would exceed it
Log actual costs against declared costs to detect pricing drift

5. Real-World Use Cases in the Agent Economy

Research and Information Retrieval

An orchestrator agent tasked with producing a market analysis report discovers three research services via their llms.txt files. It compares their agents.json capability manifests, selects the one with the best cost-per-query ratio for its budget, and uses the OpenAPI spec to call the search and fetch endpoints. The entire discovery-to-call sequence requires no human input.

Multi-Agent Capability Delegation

A planning agent needs to delegate a data visualisation subtask. It queries a capability registry (itself an agent-readable service with agents.json) to find visualisation agents, evaluates their declared capabilities and pricing, and delegates the task to the best match. The delegating agent never needed to know about the visualisation agent in advance — discovery infrastructure made the match at runtime.

Automated Service Auditing

A compliance agent periodically crawls a portfolio of vendor services, fetching their agents.json and OpenAPI specs to verify that declared capabilities, pricing, and authentication requirements match observed behaviour. Discrepancies trigger alerts. This is only possible because the services publish structured, machine-readable declarations.

Content Licensing and Attribution

A content aggregation agent uses llms.txt files to determine which sources permit summarisation, which require attribution, and which prohibit AI use entirely. It routes requests accordingly, automatically attaching attribution metadata where required. Without llms.txt, this requires either blanket permission assumptions (legally risky) or human review of every source's terms of service.

6. Common Pitfalls and Best Practices

Pitfall 1: Stale Discovery Files

Problem: agents.json declares capabilities or pricing that no longer reflect the actual service. Agents make decisions based on outdated information, leading to failed calls or unexpected costs.

Fix: Treat discovery files as part of your deployment pipeline. Update them atomically with API changes. Add a last_updated timestamp field to agents.json.

Pitfall 2: Vague Capability Descriptions

Problem: Capability descriptions like "process data" or "handle requests" give agents no basis for task-capability matching. Agents either skip the service or call it inappropriately.

Fix: Write capability descriptions as action statements with explicit input/output types. "Search a corpus of 50,000 research notes by keyword, returning ranked summaries" is actionable. "Search functionality" is not.

Pitfall 3: Missing Error Semantics in OpenAPI

Problem: OpenAPI spec documents the happy path but omits error responses. Agents encountering undocumented errors cannot recover gracefully and escalate to human intervention.

Fix: Document all error responses (400, 401, 429, 500) with schema definitions and human-readable descriptions. Include retry guidance in 429 responses.

Pitfall 4: Ignoring CORS for Agent Clients

Problem: Browser-based agent frameworks are blocked by CORS policies that weren't designed with agent origins in mind.

Fix: Configure CORS to permit requests from agent runtime origins, or provide a server-side proxy endpoint documented in your OpenAPI spec.

Pitfall 5: Conflating llms.txt with robots.txt

Problem: Teams copy their robots.txt rules into llms.txt, producing a permissions file that blocks legitimate agent use cases they actually want to support.

Fix: Treat llms.txt as a positive declaration of intent, not just a restriction list. Lead with what agents can do, then specify restrictions.

Pitfall 6: No Versioning Strategy

Problem: Breaking changes to agents.json or OpenAPI specs break agents that cached the previous version, with no migration path.

Fix: Version all discovery artifacts. Maintain backward compatibility for at least one major version cycle. Signal deprecation in the file itself.

7. Future Evolution and Emerging Standards

Capability Registries

Rather than agents discovering services one domain at a time, centralised and federated capability registries are emerging — directories of agents.json manifests that agents can query by capability type, cost range, or reliability score. These function as marketplaces for agent capabilities.

Signed Discovery Artifacts

As the agent economy matures, trust becomes a critical concern. Research in this area suggests that cryptographically signed agents.json and OpenAPI specs — where the service provider's identity is verifiable — will become a baseline requirement for high-value agent interactions. An agent should be able to verify that the discovery file it fetched was produced by the entity it claims to represent.

Dynamic Capability Negotiation

Current standards are static declarations. Emerging patterns involve agents and services negotiating capabilities at runtime — an agent declaring its task requirements, the service responding with a tailored capability manifest and pricing for that specific task. This moves discovery from a lookup operation to a protocol exchange.

Semantic Capability Matching

Rather than agents matching task descriptions to capability IDs by keyword, semantic embedding-based matching allows agents to find services whose capabilities are semantically similar to the task, even when terminology differs. This requires capability descriptions to be written in natural language (which agents.json already supports) and processed by an embedding model at discovery time.

Standardisation Pressure

As agent-to-service interaction volume grows, pressure for formal standardisation of these formats through bodies like the IETF or W3C is increasing. The current landscape is de facto standards driven by adoption rather than formal specification. Services that implement the emerging conventions now will be better positioned when formal standards crystallise.

Key Takeaways

Discovery infrastructure is the prerequisite for autonomous agent operation. Without it, agents cannot reliably find, evaluate, or call services without human mediation.
The four pillars serve distinct functions: llms.txt orients, agents.json enables decisions, OpenAPI enables execution, and semantic HTML provides fallback and enrichment.
Layered implementation is resilient. Different agent frameworks prioritise different discovery signals. Full implementation ensures compatibility across the ecosystem.
Discovery files are contracts. Stale, vague, or incomplete discovery artifacts cause agent failures just as surely as broken API endpoints.
The minimum viable investment is OpenAPI + llms.txt. These two files unlock the majority of agent compatibility benefits with modest implementation effort.
The agent economy rewards machine-legibility. Services that invest in discovery infrastructure become accessible to the growing population of autonomous agents; those that don't remain invisible to it.

Discovery Infrastructure for AI Agents: A Practical Course Lesson on llms.txt, agents.json, OpenAPI, and Semantic HTML

Discovery Infrastructure for AI Agents: A Practical Course Lesson on llms.txt, agents.json, OpenAPI, and Semantic HTML

Learning Objectives

1. What is Discovery Infrastructure?

Why This Matters Now

2. The Four Pillars: llms.txt, agents.json, OpenAPI, and Semantic HTML

2.1 llms.txt: Machine-Readable Service Declarations

2.2 agents.json: Agent Capability Manifests

2.3 OpenAPI: Standardized API Contracts

2.4 Semantic HTML: Human-Readable Service Documentation

3. How These Standards Work Together

4. Practical Implementation Patterns

4.1 Building a Discovery-Ready Service

4.2 Consuming Discovery Infrastructure as an Agent

5. Real-World Use Cases in the Agent Economy

Research and Information Retrieval

Multi-Agent Capability Delegation

Automated Service Auditing

Content Licensing and Attribution

6. Common Pitfalls and Best Practices

Pitfall 1: Stale Discovery Files

Pitfall 2: Vague Capability Descriptions

Pitfall 3: Missing Error Semantics in OpenAPI

Pitfall 4: Ignoring CORS for Agent Clients

Pitfall 5: Conflating llms.txt with robots.txt

Pitfall 6: No Versioning Strategy

7. Future Evolution and Emerging Standards

Capability Registries

Signed Discovery Artifacts

Dynamic Capability Negotiation

Semantic Capability Matching

Standardisation Pressure

Key Takeaways

Further Reading and Resources