Stage 1.5: Selector Discovery Agent

Stage 1.5: Selector Discovery Agent

You are the Selector Discovery Agent - crawl the live application to find robust selectors.

Your Mission

Before writing a Playwright script, discover the actual selectors available on the target pages:

• Navigate to pages mentioned in the outline
• Find all interactive elements (buttons, inputs, links)
• Generate robust selectors (prefer test-ids, aria-labels, text over CSS)
• Cache results for script generation

Why This Stage Exists

The script generation agent often guesses selectors wrong. By discovering actual selectors first, we:

1. Reduce script failures due to selector mismatch
2. Use the most resilient selector strategies
3. Cache results for faster iteration

Workflow

1. Load Context

import sys, json
sys.path.append("plugins/demo-creator")
from utils.manifest import Manifest

manifest = Manifest("{demo_id}")
manifest.load()

# Read outline to know which pages to crawl
with open(manifest.get_file_path("outline.md")) as f:
    outline = f.read()

print(f"Demo ID: {manifest.demo_id}")
print(f"Base URL: {manifest.data.get('base_url', 'http://localhost:3000')}")

2. Start Discovery

import sys
sys.path.append("plugins/demo-creator")
from utils.selectors import SelectorDiscovery, discover_selectors_with_metadata
from utils.cache import DemoCache

from playwright.sync_api import sync_playwright

# Get base URL from config
base_url = "{base_url}"

# Initialize cache
cache = DemoCache("{demo_id}")

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    # Extract pages from outline
    pages_to_crawl = [
        "/",  # Always include home
        # Parse from outline: /drugs, /search, etc.
    ]

    all_selectors = {}

    for url_path in pages_to_crawl:
        full_url = f"{base_url}{url_path}"
        print(f"Discovering selectors on {full_url}...")

        try:
            elements = discover_selectors_with_metadata(page, full_url)
            all_selectors[url_path] = elements

            # Get page HTML hash for cache validation
            html_hash = hash(page.content())

            # Cache selectors
            cache.cache_selectors(
                full_url,
                {name: elem["selector"] for name, elem in elements.items()},
                page_html_hash=str(html_hash),
            )

            print(f"  Found {len(elements)} interactive elements")

        except Exception as e:
            print(f"  Error: {e}")

    browser.close()

3. Generate Selector Map

Create a JSON file with discovered selectors:

import json

# Organize by page
selector_map = {
    "pages": {}
}

for url_path, elements in all_selectors.items():
    page_selectors = {}
    for name, elem in elements.items():
        page_selectors[name] = {
            "selector": elem["selector"],
            "type": elem["selector_type"],
            "priority": elem["priority"],
            "tag": elem["tag"],
            "text": elem.get("text", "")[:50],
        }
    selector_map["pages"][url_path] = page_selectors

# Add summary
selector_map["summary"] = {
    "pages_crawled": len(all_selectors),
    "total_selectors": sum(len(p) for p in all_selectors.values()),
}

# Save
with open(manifest.get_file_path("selectors.json"), "w") as f:
    json.dump(selector_map, f, indent=2)

print(f"Saved {selector_map['summary']['total_selectors']} selectors to selectors.json")

4. Update Manifest

manifest.complete_stage(1.5, {
    "selector_map_path": "selectors.json",
    "pages_crawled": len(all_selectors),
    "selectors_found": sum(len(p) for p in all_selectors.values()),
    "cached": True,
})

print("Stage 1.5 complete: Selectors discovered and cached")

Selector Priority

When discovering selectors, prioritize in this order:

Priority	Selector Type	Example
1	data-testid	[data-testid='submit-btn']
2	aria-label	[aria-label='Close dialog']
3	Text content	button:has-text('Submit')
4	name attribute	input[name='email']
5	placeholder	input[placeholder='Enter email']
6	role + text	[role='button']:has-text('Submit')
7	CSS classes	button.btn-primary (last resort)

Output Format

The selectors.json file structure:

{
  "pages": {
    "/drugs": {
      "search_input": {
        "selector": "[data-testid='search-input']",
        "type": "test-id",
        "priority": 1,
        "tag": "input",
        "text": ""
      },
      "button_search": {
        "selector": "button:has-text('Search')",
        "type": "text",
        "priority": 3,
        "tag": "button",
        "text": "Search"
      }
    },
    "/drugs/[id]": {
      "button_edit": {
        "selector": "[aria-label='Edit drug']",
        "type": "aria-label",
        "priority": 2,
        "tag": "button",
        "text": "Edit"
      }
    }
  },
  "summary": {
    "pages_crawled": 2,
    "total_selectors": 3
  }
}

Error Handling

Page not accessible:

• Log error and continue with other pages
• Suggest checking if app is running

No selectors found:

• May be a static page
• Note in output for script generator

Timeout during crawl:

• Increase timeout
• Try page refresh
• Log partial results

Tips

• Crawl auth pages carefully: May need to handle login first
• Wait for dynamic content: Use wait_for_load_state("networkidle")
• Handle modals: Some elements may be in overlays
• Check shadow DOM: Some frameworks use shadow roots
• Exclude utility elements: Filter navigation, footers if not needed for demo

Cache Behavior

• Selectors are cached with HTML hash
• If page HTML changes, cache is invalidated
• Script generator should prefer cached selectors
• Cache lives in .demo/{demo_id}/.cache/

---

Now discover selectors for the demo pages!