web-research

Web Research

Fetch web pages, index them locally, and read selectively using Julie's tools. This replaces dumping entire web pages into context (which wastes thousands of tokens) with a workflow that indexes the content and lets you pull out just the sections you need.

Prerequisites

browser39 must be installed. Check with:

which browser39

If missing, tell the user to download the latest binary release from browser39 releases and add it to their PATH.

Workflow

Step 0 (Optional): Discover URLs from a Topic

If the user gave a research topic rather than a specific URL, find candidate pages first:

browser39 search "your research topic" --output json > /tmp/b39-search.json

Parse the links[] array and pick the URLs worth fetching. Skip this step when the user already provided URLs.

Step 1: Fetch and Save

Determine the target file path from the URL. The directory structure mirrors the URL:

docs/web/
  docs.rs/axum/latest.md
  developer.mozilla.org/Web/API/Fetch_API.md
  github.com/tokio-rs/tokio.md

Fetch the page and save directly to the target file. Never print full page content to stdout.

# Fetch (rm ensures no stale results — --output appends, not overwrites)
echo '{"id":"1","action":"fetch","v":1,"seq":1,"url":"THE_URL","options":{"selector":"article","strip_nav":true,"include_links":true}}' > /tmp/b39-cmd.jsonl
rm -f /tmp/b39-out.jsonl
browser39 batch /tmp/b39-cmd.jsonl --output /tmp/b39-out.jsonl

# Extract markdown directly to file (no stdout leak)
mkdir -p docs/web/TARGET_DOMAIN/TARGET_PATH_DIR
python3 -c "
import sys,json
with open('/tmp/b39-out.jsonl') as f:
    for line in f:
        d=json.loads(line)
        md = d.get('markdown','')
        if md:
            with open('docs/web/TARGET_DOMAIN/TARGET_PATH.md','w') as out:
                out.write(md)
            print(f'Saved {len(md)} chars to docs/web/TARGET_DOMAIN/TARGET_PATH.md')
"

If the page content is empty or too short, retry without the selector option, or try "main", ".content", or "body".

The filewatcher automatically indexes the file within 1-2 seconds.

Step 2: Explore the Content

See the page structure (table of contents):

get_symbols(file_path="docs/web/{domain}/{path}.md", mode="structure")

This returns section headings as a hierarchy, letting you see what's on the page without reading it.

Step 3: Read Selectively

Read a specific section by name:

get_symbols(file_path="docs/web/{domain}/{path}.md", mode="minimal", target="Section Name")

Search across all fetched pages:

fast_search(query="your search terms", file_pattern="docs/web/**")

Get token-budgeted context for a concept:

get_context(query="concept or question", file_pattern="docs/web/**")

Step 4: Follow Links (Optional)

The fetched markdown contains links. If you need more information, fetch additional pages by repeating Steps 1-2 with the linked URLs. The agent decides which links are worth following based on the research goal.

Step 5: Clean Up

When research is complete, suggest removing fetched content:

# Remove a specific page
rm docs/web/{domain}/{path}.md

# Remove all fetched content
rm -rf docs/web/

The filewatcher automatically removes deleted files from the index.

Tips

• GitHub README pages: Use selector: "article" to get just the README content
• Documentation sites: Try selector: "main" or selector: ".content" to skip navigation
• Large pages: Use browser39's max_tokens option to limit the fetch, then paginate with offset
• Multiple pages at once: Write multiple commands to the JSONL file (one per line) for batch fetching
• Searching across fetched content: file_pattern="docs/web/**" scopes any Julie tool to just the fetched web content
• Empty results from get_symbols: The filewatcher indexes within 1-2 seconds. If results are empty, the file may not have been written correctly; check the file exists and has content.