From firecrawl-pack
Choose and implement FireCrawl validated architecture blueprints for different scales. Use when designing new FireCrawl integrations, choosing between monolith/service/microservice architectures, or planning migration paths for FireCrawl applications. Trigger with phrases like "firecrawl architecture", "firecrawl blueprint", "how to structure firecrawl", "firecrawl project layout", "firecrawl microservice".
How this skill is triggered — by the user, by Claude, or both
Slash command
/firecrawl-pack:firecrawl-architecture-variantsThis skill is limited to the following tools:
The summary Claude sees in its skill listing — used to decide when to auto-load this skill
Deployment architectures for Firecrawl web scraping at different scales. Firecrawl's async crawl model, credit billing, and JavaScript rendering support different architectures from simple page scraping to enterprise content ingestion pipelines.
Deployment architectures for Firecrawl web scraping at different scales. Firecrawl's async crawl model, credit billing, and JavaScript rendering support different architectures from simple page scraping to enterprise content ingestion pipelines.
Best for: Single-page scraping, < 500 pages/day, content extraction.
User Request -> Backend -> Firecrawl scrapeUrl -> Parse Content -> Response
app.post('/extract', async (req, res) => {
const result = await firecrawl.scrapeUrl(req.body.url, {
formats: ['markdown'], onlyMainContent: true
});
res.json({ content: result.markdown, title: result.metadata.title });
});
Best for: Content monitoring, 500-10K pages/day, documentation indexing.
Scheduler (cron) -> Crawl Queue -> Firecrawl crawlUrl -> Result Store
|
v
Content Processor -> Search Index
// Scheduled crawler
cron.schedule('0 2 * * *', async () => { // Daily at 2 AM
const sites = await db.getCrawlTargets();
for (const site of sites) {
const crawl = await firecrawl.asyncCrawlUrl(site.url, {
limit: site.maxPages, includePaths: site.paths
});
await db.saveCrawlJob({ siteId: site.id, jobId: crawl.id });
}
});
// Separate worker polls for results
async function processCrawlResults() {
const pending = await db.getPendingCrawlJobs();
for (const job of pending) {
const status = await firecrawl.checkCrawlStatus(job.jobId);
if (status.status === 'completed') {
await indexPages(status.data);
await db.markComplete(job.id);
}
}
}
Best for: Enterprise, 10K+ pages/day, AI training data, knowledge base.
URL Sources -> Priority Queue -> Firecrawl Workers -> Content Validation
|
v
Vector DB + Search Index
|
v
RAG / AI Pipeline
class ContentPipeline {
async ingest(urls: string[], priority: 'high' | 'normal' | 'low') {
const budget = this.creditTracker.canAfford(urls.length);
if (!budget) throw new Error('Daily credit budget exceeded');
const results = await firecrawl.batchScrapeUrls(urls, {
formats: ['markdown'], onlyMainContent: true
});
const validated = results.filter(r => this.validateContent(r));
await this.vectorStore.upsert(validated);
this.creditTracker.record(urls.length);
return { ingested: validated.length, rejected: urls.length - validated.length };
}
}
| Factor | On-Demand | Scheduled | Real-Time Pipeline |
|---|---|---|---|
| Volume | < 500/day | 500-10K/day | 10K+/day |
| Latency | Sync (2-10s) | Async (hours) | Async (minutes) |
| Use Case | Single page | Site monitoring | Knowledge base |
| Cost Control | Per-request | Per-crawl budget | Credit pipeline |
| Issue | Cause | Solution |
|---|---|---|
| Slow scraping in request path | Synchronous scrapeUrl | Move to async pipeline |
| Stale content | Infrequent crawling | Increase crawl frequency |
| Credit overrun | No budget tracking | Implement credit circuit breaker |
| Duplicate content | Re-crawling same pages | Dedup by URL hash before indexing |
< 500 pages/day, user-facing: On-Demand # HTTP 500 Internal Server Error
500-10K pages, batch processing: Scheduled Pipeline # HTTP 500 Internal Server Error
10K+, AI/ML ingestion: Real-Time Pipeline
npx claudepluginhub nickloveinvesting/nick-love-plugins --plugin firecrawl-packCreates, edits, and optimizes skills for Claude Code, including drafting, evaluating with test prompts, iterating on performance, and improving skill descriptions for better triggering accuracy.