es-evidence-ingest

ES Evidence Ingestion Skill

Parse and bulk-ingest forensic evidence into Elasticsearch for threat hunting.

Scripts Location: ${CLAUDE_PLUGIN_ROOT}/skills/threat-hunt/scripts/ and ${CLAUDE_PLUGIN_ROOT}/skills/linux-threat-hunt/scripts/

Prerequisites

• Elasticsearch 9.x running (default: http://localhost:9200)
• evtx_dump (Rust): ~/.cargo/bin/evtx_dump — Install: cargo install evtx
• Python 3.10+ with uv

Supported Formats

Format	Script	Index Suffix
Windows EVTX	ingest_evtx.py	-winevt
Linux syslog	ingest_syslog.py	-syslog
Connection logs	ingest_connections.py	-connections
Bash history	ingest_bash_history.py	-bash-history
Linux full visibility	ingest_linux_full_visibility.py	linux-evidence-*
Windows full visibility	ingest_windows_full_visibility.py	windows-evidence-*

Ingestion Commands

uv run ${CLAUDE_PLUGIN_ROOT}/skills/threat-hunt/scripts/ingest_evtx.py <evtx_file_or_dir> [server_name] [index_name]
uv run ${CLAUDE_PLUGIN_ROOT}/skills/threat-hunt/scripts/ingest_syslog.py [log_dir]
uv run ${CLAUDE_PLUGIN_ROOT}/skills/threat-hunt/scripts/ingest_connections.py <log_file> [index_name]
uv run ${CLAUDE_PLUGIN_ROOT}/skills/threat-hunt/scripts/ingest_bash_history.py [paths...]
uv run ${CLAUDE_PLUGIN_ROOT}/skills/threat-hunt/scripts/create_indices.py
uv run ${CLAUDE_PLUGIN_ROOT}/skills/linux-threat-hunt/scripts/ingest_linux_full_visibility.py --evidence-root <path> --index-prefix linux-evidence
uv run ${CLAUDE_PLUGIN_ROOT}/skills/linux-threat-hunt/scripts/validate_linux_5_layers.py --index-prefix linux-evidence --output <validation-report.json>
uv run ${CLAUDE_PLUGIN_ROOT}/skills/windows-threat-hunt/scripts/ingest_windows_full_visibility.py --evidence-root <path> --index-prefix windows-evidence
uv run ${CLAUDE_PLUGIN_ROOT}/skills/windows-threat-hunt/scripts/validate_windows_5_layers.py --index-prefix windows-evidence --output <validation-report.json>

Ingestion Process

1. Check ES connectivity: bash ${CLAUDE_PLUGIN_ROOT}/skills/threat-hunt/scripts/check-elastic.sh
2. Survey evidence for supported file types
3. Create indices with proper mappings (IP as ip, timestamps as date)
4. Run ingestion scripts with uv run (inline dependencies)
5. Use streaming_bulk with chunk_size=2000, raise_on_error=False
6. Skip near-empty EVTX files (< 70KB = header only)
7. Refresh indices after ingestion

Related Skills

• threat-hunt — Full hunt methodology (index skill)
• linux-threat-hunt — Linux full-visibility + 5-layer verdict gate
• windows-threat-hunt — Windows full-visibility (Zimmerman coverage + more) + 5-layer verdict gate
• elk-stack-queries — ES query languages (EQL, ES|QL, KQL, DSL)

Core Principles

1. verify index mappings first — wrong types prevent queries and aggregations.
2. avoid ingest without timestamps — chronology is the backbone of forensics.
3. use ECS field names — standard naming enables cross-source correlation.
4. Avoid skipping bulk optimization — streaming_bulk with proper chunk sizes.
5. validate record counts — compare ingested docs against source line counts.

Reference

For detailed field mappings, EVTX parsing, and connection log format, see reference.md.