From agent-orchestra
VS Code native browser tool behavior for canvas-based games (Phaser 3, etc.). Use when interacting with HTML canvas elements, clicking game objects, verifying canvas state via browser tools, or when `clickElement` fails on canvas. DO NOT USE FOR: React component tests (use ui-testing) or Playwright E2E tests (use webapp-testing).
How this skill is triggered — by the user, by Claude, or both
Slash command
/agent-orchestra:browser-canvas-testingThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
Domain knowledge for using VS Code native browser tools against HTML canvas applications verified via VS Code source code review.
Domain knowledge for using VS Code native browser tools against HTML canvas applications verified via VS Code source code review.
<canvas>screenshotPagereadPage returns empty on a canvas-only pageclickElement Cannot Target Canvas ObjectsclickElement is selector-based only. The VS Code source (clickBrowserTool.ts) defines these parameters:
pageId, selector, ref, dblClick, button
There is no coordinate or position parameter. The implementation does exactly:
page.locator(selector).click();
A <canvas> element renders its content via GPU/CPU drawing calls — there are no child DOM elements for selectors to target. Even canvas as a CSS selector only clicks the canvas border box center, not a specific game object inside it.
Conclusion: clickElement cannot be used for canvas game interaction. Use runPlaywrightCode instead.
Source: clickBrowserTool.ts
Canvas games use an internal game-space coordinate system that does not map directly to CSS page coordinates. Use getBoundingClientRect to convert.
// 1. Query canvas bounding rect (accounts for scaling and letterboxing)
const rect = await page.evaluate(() => {
const c = document.querySelector("canvas");
if (!c) throw new Error("Canvas element not found");
const r = c.getBoundingClientRect();
return { left: r.left, top: r.top, width: r.width, height: r.height };
});
// 2. Convert game-space coords → CSS page coords
const cssX = rect.left + (gameX / GAME_WIDTH) * rect.width;
const cssY = rect.top + (gameY / GAME_HEIGHT) * rect.height;
// 3. Click via Playwright mouse API
await page.mouse.click(cssX, cssY);
If the page has multiple <canvas> elements (e.g., a main scene canvas plus a HUD overlay), document.querySelector('canvas') returns the first one, which may not be the target. Use a specific selector instead: document.querySelector('#game canvas') or document.getElementById('game-canvas'). Inspect with page.evaluate(() => document.querySelectorAll('canvas').length) to detect multiple canvases.
Prerequisite: The canvas must be rendered before calling getBoundingClientRect(). An unrendered canvas returns {0,0,0,0}, causing the formula to click (0,0) silently. See Section 3 for the render-ready wait pattern.
Variables:
gameX, gameY — coordinates in the game's internal resolution (e.g., from game design docs or layout)GAME_WIDTH, GAME_HEIGHT — the game's declared resolution (e.g., 1920, 1080 for a 1920×1080 game)cssX, cssY — CSS page coordinates passed to PlaywrightThis formula is verified working for Phaser 3 in FIT scale mode at 1920×1080.
Note — scale mode matters: In FIT mode the canvas CSS box scales to the viewport while GAME_WIDTH/GAME_HEIGHT stay constant, so this formula is exact. In RESIZE mode the game's internal resolution changes to match the viewport; use canvas.width and canvas.height read via page.evaluate() for GAME_WIDTH/GAME_HEIGHT instead of hardcoded constants. Other modes (NONE, ENVELOP) may require similar adjustments.
runPlaywrightCode Usage PatternrunPlaywrightCode executes arbitrary Playwright code against the current browser page — the correct tool for all canvas interaction.
Full worked example — click a game object at game-space position (960, 540) in a 1920×1080 game:
// Wait for canvas to be rendered and sized
await page.waitForFunction(() => {
const c = document.querySelector("canvas");
return c !== null && c.getBoundingClientRect().width > 0;
});
const rect = await page.evaluate(() => {
const c = document.querySelector("canvas");
const r = c.getBoundingClientRect();
return { left: r.left, top: r.top, width: r.width, height: r.height };
});
const GAME_WIDTH = 1920;
const GAME_HEIGHT = 1080;
const gameX = 960;
const gameY = 540;
const cssX = rect.left + (gameX / GAME_WIDTH) * rect.width;
const cssY = rect.top + (gameY / GAME_HEIGHT) * rect.height;
await page.mouse.click(cssX, cssY);
Also useful via runPlaywrightCode:
await page.mouse.move(cssX, cssY) — hover without clickingawait page.keyboard.press('Space') — send key events to the gameawait page.evaluate(() => window.gameState) — read exposed JS globals for state assertionsscreenshotPage — Works for Canvas VerificationscreenshotPage does work on canvas content. It uses VS Code's internal browserViewModel.captureScreenshot() — not Playwright's .screenshot() method. This internal capture reads the rendered frame buffer directly, so canvas pixels are captured correctly.
Use screenshotPage to:
Screenshot output: write captured images to .tmp/issue-{N}-{name}.png per skills/terminal-hygiene/SKILL.md ## Scratch & Temp-File Hygiene — never pass a C:\... absolute path to a Bash redirect.
Source: screenshotBrowserTool.ts
readPage — Returns Empty for Canvas-Only PagesreadPage calls playwrightService.getSummary() which returns a DOM accessibility/semantic snapshot. Canvas elements render no accessible child nodes — the entire visual scene is opaque to the accessibility tree.
Result: readPage returns an empty or near-empty response for canvas-only applications.
Do not use readPage to:
Use runPlaywrightCode with page.evaluate() to read game state from exposed JS APIs, or use screenshotPage for visual verification.
Source: readBrowserTool.ts
All behaviors above are verified from VS Code 1.110 source code:
| Tool | Source File |
|---|---|
click_element / clickElement | clickBrowserTool.ts |
screenshot_page / screenshotPage | screenshotBrowserTool.ts |
read_page / readPage | readBrowserTool.ts |
run_playwright_code / runPlaywrightCode | VS Code native Playwright code execution (no single tool source file — functionality available when workbench.browser.enableChatTools: true) |
| Trigger | Gotcha | Fix |
|---|---|---|
Using clickElement to interact with canvas game objects | clickElement is selector-based only — no coordinate parameter exists; canvas internals are not addressable via selector | Use runPlaywrightCode with page.mouse.click(cssX, cssY) and the canvas coordinate formula |
Querying getBoundingClientRect() before the canvas has rendered | Returns {0, 0, 0, 0} silently; click lands at top-left corner of page | Add await page.waitForFunction(() => document.querySelector('canvas')?.width > 0) before querying the rect |
Hardcoding GAME_WIDTH/GAME_HEIGHT in RESIZE scale mode | FIT formula doesn't apply; clicks land at wrong position | Read canvas.width/canvas.height via page.evaluate() rather than using known design dimensions |
Using document.querySelector('canvas') when multiple canvases exist | Always returns the first canvas — may be HUD overlay, not game scene | Use a specific selector (#game canvas) or verify canvas count first |
Using readPage on canvas-only pages | Returns empty — canvas pixels are not DOM text | Use screenshotPage for visual state verification instead |
Guides creation, editing, and verification of skills for AI coding agents using test-driven development with subagent scenarios. Use when authoring or debugging skills.
npx claudepluginhub grimblaz/agent-orchestra --plugin agent-orchestra