Screenenv Integration Guide

Screenenv Integration Guide

Screenenv is a Playwright-based tool that runs in Kubernetes Jobs to record browser-based demos. This skill explains how to use it within the demo-creator pipeline.

What is Screenenv?

Screenenv is a headless browser recording tool from HuggingFace that:

• Runs Playwright Python scripts in a sandboxed environment
• Records the screen to high-quality video
• Executes in Kubernetes Jobs for isolation and resource management

Important: Screenenv is NOT an MCP server. You don't call it directly via tools. Instead, you write Playwright scripts that screenenv will execute.

Architecture

┌─────────────────────────────────────────────────────────────┐
│  Phase 1: Script Development (detailed-script agent)        │
│  - Agent writes Playwright Python script                    │
│  - Uses domain knowledge of the app                          │
│  - Tools: Read, Write, Grep, Bash                           │
└─────────────────┬───────────────────────────────────────────┘
                  │
                  ▼
         script.py saved to .demo/{demo_id}/
                  │
                  ▼
┌─────────────────────────────────────────────────────────────┐
│  Phase 2: Recording (record-demo agent)                     │
│  - Uses ScreenenvJobManager Python utility                   │
│  - Creates Helm release with screenenv-job chart            │
│  - Passes script.py to K8s Job                              │
└─────────────────┬───────────────────────────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────────────────────────┐
│  Phase 3: Execution (Kubernetes Job)                        │
│  - Screenenv runs in k8s Pod                                │
│  - Executes Playwright script in headless Chrome            │
│  - Records screen at 1920x1080                              │
│  - Saves video to PVC                                       │
└─────────────────────────────────────────────────────────────┘

Writing Playwright Scripts for Screenenv

When developing demo scripts (in the detailed-script agent), write standard Playwright Python code:

Template Structure

"""
Demo Script: {Feature Name}
Generated by: demo-creator detailed-script agent
"""

from playwright.sync_api import sync_playwright
import time


def run_demo(page):
    """Execute the demo script."""

    # Scene 1: Navigate to Feature
    print("Scene 1: Navigate to Feature")
    page.goto("http://localhost:3000/drugs")
    page.wait_for_load_state("networkidle")
    time.sleep(2)

    # Verify page loaded
    assert page.locator('input[placeholder*="Search"]').is_visible()
    page.screenshot(path="scene_1.png")

    # Scene 2: Interact with UI
    print("Scene 2: Apply Filters")
    page.click('button:has-text("Filter")')
    time.sleep(0.5)

    page.fill('input[name="search"]', "EGFR")
    time.sleep(0.4)

    page.click('button[type="submit"]')
    time.sleep(1.5)

    page.screenshot(path="scene_2.png")


def setup():
    """Run setup commands before demo (optional)."""
    import subprocess

    # Example: Seed test data
    subprocess.run([
        "kubectl", "exec", "-n", "your-namespace",
        "deployment/backend", "--",
        "python", "scripts/seed_demo_data.py"
    ], check=True)


def teardown():
    """Run cleanup commands after demo (optional)."""
    import subprocess

    subprocess.run([
        "kubectl", "exec", "-n", "your-namespace",
        "deployment/backend", "--",
        "python", "scripts/cleanup_demo_data.py"
    ], check=True)


def main():
    """Main entry point - screenenv executes this."""
    setup()

    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        context = browser.new_context(
            viewport={"width": 1920, "height": 1080},
            record_video_dir="./recordings",
            record_video_size={"width": 1920, "height": 1080}
        )
        page = context.new_page()

        try:
            run_demo(page)
        finally:
            context.close()
            browser.close()
            teardown()


if __name__ == "__main__":
    main()

Key Playwright APIs

Navigation:

page.goto("http://localhost:3000/path")
page.wait_for_load_state("networkidle")
page.wait_for_url("**/drugs")

Clicking:

page.click('button:has-text("Submit")')
page.locator('.submit-btn').click()
page.click('button[aria-label="Close"]')

Typing:

page.fill('input[name="search"]', "text")
page.type('input', "text", delay=100)  # With typing delay

Assertions:

assert page.locator('.result').is_visible()
page.wait_for_selector('.result', timeout=5000)

Screenshots:

page.screenshot(path="scene_1.png")
page.locator('.specific-element').screenshot(path="element.png")

Waits:

time.sleep(2)  # Explicit wait (use for timing)
page.wait_for_timeout(2000)  # Playwright wait
page.wait_for_selector('.element')  # Wait for element

Selector Strategy (Priority Order)

1. Text-based (most resilient): page.click('button:has-text("Submit")')
2. Placeholder: page.fill('input[placeholder="Search"]', text)
3. Aria-label: button[aria-label="Close"]
4. Test ID: [data-testid="submit-btn"]
5. CSS classes (last resort, fragile)

Using ScreenenvJobManager

The record-demo agent uses the Python utility to manage K8s Jobs:

import sys
sys.path.append("plugins/demo-creator")
from utils.screenenv_job import ScreenenvJobManager, create_and_run_recording
from utils.manifest import Manifest

# Load manifest
manifest = Manifest("{demo_id}")
manifest.load()

# Get script path
script_path = manifest.get_file_path("script.py")

# Option 1: Convenience function (recommended)
result = create_and_run_recording(
    demo_id=manifest.demo_id,
    script_url=f"http://demo-script-server/scripts/{manifest.demo_id}.py",
    output_path=manifest.get_file_path("raw_recording.mp4"),
    target_url="http://localhost:3000",
    cleanup=True
)

# Option 2: Manual control
manager = ScreenenvJobManager(
    namespace="infra",
    helm_chart_path="k8s/infra/charts/screenenv-job",
    context="k3d-local"
)

# Create job
job_result = manager.create_job(
    demo_id=manifest.demo_id,
    script_url=f"http://demo-script-server/scripts/{manifest.demo_id}.py",
    target_url="http://localhost:3000",
    resolution="1920x1080",
    frame_rate="30"
)

# Wait for completion
wait_result = manager.wait_for_completion(
    demo_id=manifest.demo_id,
    poll_interval=5,
    max_wait=600
)

# Retrieve recording
if wait_result["status"] == "completed":
    success = manager.retrieve_recording(
        demo_id=manifest.demo_id,
        output_path=manifest.get_file_path("raw_recording.mp4")
    )

    if success:
        print("✅ Recording retrieved")

    # Cleanup
    manager.cleanup_job(manifest.demo_id)

Kubernetes Deployment

The screenenv Helm chart is at k8s/infra/charts/screenenv-job/.

Key Configuration

Parameter	Default	Description
demoId	-	Unique demo identifier
scriptUrl	-	URL to Playwright script
targetUrl	http://localhost:3000	App base URL
resolution	1920x1080	Video resolution
frameRate	30	Video frame rate
image.repository	ghcr.io/huggingface/screenenv	Docker image

Job Lifecycle

1. Creation: Helm install creates Job
2. Execution: Pod starts, runs Playwright script
3. Recording: Video saved to PVC at /recordings/{demo_id}/raw_recording.mp4
4. Completion: Job status becomes "succeeded"
5. Retrieval: kubectl cp to extract video
6. Cleanup: Helm uninstall (automatic after 10 minutes)

Debugging

Check Job Status

kubectl get job screenenv-{demo_id} -n infra --context k3d-local

View Logs

kubectl logs job/screenenv-{demo_id} -n infra --context k3d-local

Check Recording

# Get pod name
POD=$(kubectl get pods -n infra -l job-name=screenenv-{demo_id} \
  --context k3d-local -o jsonpath='{.items[0].metadata.name}')

# Check file exists
kubectl exec -n infra $POD --context k3d-local -- \
  ls -lh /recordings/{demo_id}/

Common Issues

"Script not found" Error

The scriptUrl must be accessible from inside the K8s cluster. For local development:

• Use a ConfigMap to embed the script
• Or run a simple HTTP server in the cluster
• Or mount the script via PVC

Recording is Empty/Corrupt

• Check script actually runs: Add print() statements
• Verify browser launches: Check logs for Chromium errors
• Ensure app is accessible from Pod: Test with kubectl exec ... -- curl http://localhost:3000

Job Never Completes

• Check timeout: Default is 10 minutes via --wait --timeout 10m
• Review logs for hangs: Look for page.wait_for_selector() that never resolves
• Verify selectors work: Test script locally with headed browser first

Best Practices

1. Test locally first: Run Playwright script on your machine with headless=False to verify selectors
2. Use resilient selectors: Prefer text-based and aria-label over CSS classes
3. Add generous waits: Network conditions in k8s may be slower
4. Keep scripts simple: Each scene should be 5-10 seconds max
5. Verify assertions: Use assert to catch UI changes that break the script
6. Clean up resources: Always call manager.cleanup_job() after retrieval

Example Workflow (detailed-script agent)

# 1. Load context
import sys
sys.path.append("plugins/demo-creator")
from utils.manifest import Manifest

manifest = Manifest("{demo_id}")
manifest.load()

# Read outline
with open(manifest.get_file_path("outline.md")) as f:
    outline = f.read()

# 2. Write Playwright script based on outline
# (Use domain knowledge of the app, not screenenv MCP calls)
script_content = """
from playwright.sync_api import sync_playwright
import time

def run_demo(page):
    # Based on outline, write the actual interactions
    page.goto("http://localhost:3000/drugs")
    # ... etc
"""

# 3. Save script
with open(manifest.get_file_path("script.py"), "w") as f:
    f.write(script_content)

# 4. Update manifest
manifest.complete_stage(2, {
    "script_path": "script.py",
    "estimated_duration_seconds": 30
})

print("✅ Stage 2 complete: Playwright script created")

Adapting Existing E2E Tests for Demos

If you have existing Playwright pytest files, you can use them as a starting point for demo scripts. Read the test, understand the flow, and adapt it for demo purposes.

What to Keep from Tests

• Working selectors - The test has already figured out how to find elements
• The user flow - The sequence of actions represents a real user journey
• URL paths - Navigation targets are correct

What to Change for Demos

Test Pattern	Demo Adaptation
page.fill('[name="q"]', "text")	page.type('[name="q"]', "text", delay=100) - visible typing
No delays between actions	Add time.sleep(1-2) for pacing
expect(...).to_be_visible()	Remove or minimize assertions
Complex error handling	Simple happy-path only
Fast execution	Deliberate, watchable pacing

Example Adaptation

Original test:

def test_search(self, page):
    page.goto("/drugs")
    page.fill('[name="q"]', "aspirin")
    page.click('button[type="submit"]')
    expect(page.locator(".results")).to_be_visible()

Demo script:

def run_demo(page):
    print("Scene 1: Navigate to search")
    page.goto("http://localhost:3000/drugs")
    time.sleep(2)

    print("Scene 2: Search for aspirin")
    page.type('[name="q"]', "aspirin", delay=100)
    time.sleep(0.5)
    page.click('button[type="submit"]')
    page.wait_for_selector(".results")
    time.sleep(2)

Summary

• ✅ Screenenv runs Playwright scripts in K8s Jobs
• ✅ You write standard Playwright Python code
• ✅ ScreenenvJobManager handles K8s orchestration
• ✅ Recording happens asynchronously in isolated Pods
• ✅ Existing E2E tests can be adapted for demos
• ❌ Screenenv is NOT an MCP server you call directly
• ❌ Don't create .mcp.json for screenenv in plugins

When to use this skill:

• When writing demo scripts (detailed-script agent)
• When setting up recording jobs (record-demo agent)
• When debugging recording failures
• When understanding the demo pipeline architecture