fetch-edgenuity-help-articles

Fetching help.imagineedgenuity.com articles

The Imagine Edgenuity help center is the customer-facing documentation surface for both EdgeEX and heritage Edgenuity. It's where product describes feature behavior in customer-readable terms — useful when you want to cite "what customers are told the feature does" alongside engineering-side citations of "what the code does."

Why this skill exists

The site is fronted by Cloudflare bot protection. Both WebFetch and curl --user-agent ... get HTTP 403 with a "Just a moment..." challenge page. The fetch fails before any article content is returned. To get past it, you need a real browser — the mcp__plugin_il_playwright__browser_* tools are sufficient because they drive a real headless Chromium that completes the challenge automatically.

This skill is the "yes, this is annoying; here's the working workflow" reference for that.

Site structure

The help center is structured as: Categories → Sections → Articles.

Stable URL patterns:

Category index:  https://help.imagineedgenuity.com/hc/en-us/categories/{category-id}-{slug}
Section index:   https://help.imagineedgenuity.com/hc/en-us/sections/{section-id}-{slug}
Article:         https://help.imagineedgenuity.com/hc/en-us/articles/{article-id}-{slug}

The numeric IDs are stable. The slugs can drift (article titles get rewritten) but redirects from old slugs to current ones generally work. Prefer the numeric-ID-only form when citing an article you don't want to re-find later:

https://help.imagineedgenuity.com/hc/en-us/articles/{article-id}

The site will redirect that to the slugged URL.

Two starting points worth knowing (verify against the live site if navigation looks wrong — Zendesk reorganizations can rotate these IDs):

• EdgeEX category: /hc/en-us/categories/10238163705879-EdgeEX — top-level EdgeEX docs.
• Course options section: /hc/en-us/sections/12637964872855-Course-options
• Customizing courses section: /hc/en-us/sections/15673548805143-Customizing-courses

Workflow

Step 1 — Index the section first if you're crawling a topic

If you're after a single article you already have a URL for, skip to Step 2.

If you're crawling a topic area (e.g., "everything about course options"), navigate to the section index page first. The section page lists all articles in that section, which lets you build a complete inventory before you start fetching individual articles.

mcp__plugin_il_playwright__browser_navigate
  url: https://help.imagineedgenuity.com/hc/en-us/sections/12637964872855-Course-options

mcp__plugin_il_playwright__browser_snapshot
  filename: <abs-path>/section-index.md

Then extract the article URLs:

grep -oE '/hc/en-us/articles/[0-9]+-[^"]*' <abs-path>/section-index.md | sort -u

That gives you the complete article list to crawl.

Step 2 — Navigate and snapshot each article

Per article:

mcp__plugin_il_playwright__browser_navigate
  url: https://help.imagineedgenuity.com/hc/en-us/articles/{article-id}

mcp__plugin_il_playwright__browser_snapshot
  filename: <abs-path>/article-{article-id}.md

Always pass filename to browser_snapshot. Without it, the full accessibility-tree snapshot comes back inline and pollutes the conversation. With filename, only a confirmation comes back inline and the snapshot is on disk.

The snapshot path you pass must be inside one of the agent's allowed roots. Cleanest is a working directory under your current task — e.g., <repo>/.playwright-mcp/article-{id}.md.

Step 3 — Strip the navigation chrome

Snapshots come back in YAML-like accessibility-tree format with a lot of nav chrome (header, breadcrumbs, footer, sidebar with "Articles in this section"). The actual article content sits between the article heading and the "Articles in this section" sidebar.

Use the extract-article-body.sh helper in this skill's scripts/ directory:

./scripts/extract-article-body.sh <abs-path>/article-12345.md > /tmp/article-body.md

Or inline:

awk 'BEGIN{p=0} /heading "EdgeEX|heading "Edgenuity/{p=1} /Articles in this section/{p=0} p{print}' \
  <abs-path>/article-12345.md > /tmp/article-body.md

The output is still in accessibility-tree format but stripped of chrome. From there grep/Read is sufficient to find the substantive content.

Step 4 — Crawl related articles via the sidebar

Each article snapshot includes an "Articles in this section" sidebar. If you started from a single article and want to expand to the rest of its topic, the sidebar shows the article's siblings. Extract URLs from the snapshot the same way as Step 1.

Step 5 — Close the browser when done

mcp__plugin_il_playwright__browser_close

Cleans up the headless browser session.

Tips & gotchas

• The first navigation may take a few seconds longer than subsequent ones — Cloudflare's challenge-completion happens on the first request and the result is cached for subsequent requests in the same browser session.
• Don't try to bypass Cloudflare with curl/WebFetch. It won't work; you'll get a 403 or a 200 with the "Just a moment..." challenge HTML body. Don't waste time on User-Agent or header tricks.
• Numeric IDs are the durable citation form. When linking to an article from an architecture doc, use the ID-only URL (/hc/en-us/articles/{id}) so the link survives slug renames.
• Snapshot files are big. A typical article snapshot is 200–400 lines of accessibility-tree YAML. Always use filename and read selectively (offset / grep) rather than reading the whole file.
• The site serves both EdgeEX and heritage Edgenuity content. Articles are tagged in their titles ("EdgeEX — …" or just the feature name for heritage). Watch which product an article is describing.
• Some articles have a "Sections in this article" or FAQ that's collapsed. The collapsed content shows up in the snapshot but with [ref=eXX] placeholders for the body — it may not include the actual answer text. If you need the FAQ body, you may need to interact with the page (browser_click) to expand it.
• Permissions and SLA notes inside articles are useful citations. Help-center docs often surface the exact permission name (e.g., "EdgeEX Edit Course Information and Options for All Courses and Sections") — these match the AuthZ checks in the API and are good cross-references.
• If extract-article-body.sh returns nothing, the heading prefix probably doesn't match. The script keys off heading "EdgeEX / heading "Edgenuity — inspect the snapshot for the actual heading shape and either adjust the regex inline or update the script.

When not to use this skill

• Internal Confluence / Jira docs — those have authenticated MCPs (mcp__atlassian__*, mcp__plugin_atlassian_atlassian__*). Use those, not this.
• Any other public site that's not Cloudflare-fronted — try WebFetch first; if it works, you don't need a browser.
• One-off URL retrieval where the user can paste the content directly — if it's faster for the user to paste, that's often less friction than the full Playwright dance.

Example: pulling a topic area end-to-end

A real flow used to produce a research note in a separate edgeex-architecture repo (course customization / supplemental cascade):

1. Navigate to two section indexes (Course options, Customizing courses).
2. Extract ~24 article URLs from the snapshots via grep -oE '/hc/en-us/articles/[0-9]+-[^"]*'.
3. Navigate + snapshot each article in turn, naming the snapshots by article ID.
4. Run extract-article-body.sh over each.
5. Read the extracted bodies and synthesize the relevant findings into the research note.
6. Close the browser.

Total time: about 5 minutes of automation per dozen articles, mostly waiting on the browser. Result: a complete cross-referenced view of how product describes a feature, suitable for citing in architecture docs.

Related Skills

• Atlassian MCP tools (mcp__atlassian__*, mcp__plugin_atlassian_atlassian__*) — for internal Confluence pages and Jira tickets. Use those for IL-internal docs; this skill is only for the public, Cloudflare-fronted help center.
• il:mermaid-render — once you've extracted product flows from help-center articles, render them as diagrams for architecture docs.