Error Troubleshooting
Common errors and fixes for AIRT attack workflows.
Model Errors
"Model not found" / "Invalid model"
- Cause: Bare model name without provider prefix
- Fix: Always use
provider/model format: openai/gpt-4o, anthropic/claude-sonnet-4-20250514
- Aliases: Use short aliases like
gpt-4o, claude, groq — they auto-resolve
"Rate limit exceeded" / 429
- Cause: Too many concurrent requests to the provider
- Fix: Reduce
n_iterations or switch to a less rate-limited provider
- For Groq: Groq has strict rate limits. Use
groq/llama-3.3-70b-versatile for lower limits
"Authentication failed" / 401
- Cause: Missing or invalid API key for the provider
- Fix: Ensure the correct API key is set as a platform secret (OPENAI_API_KEY, ANTHROPIC_API_KEY, GROQ_API_KEY, etc.)
- Check: API keys are pre-configured in the environment. If missing, ask the user to add them via Settings > Secrets
"Context length exceeded"
- Cause: Conversation history too long for model's context window
- Fix: Reduce
context_depth or n_iterations, or switch to a model with larger context
Transform Errors
"Transform not found: "
- Cause: Wrong transform name. Names must match TRANSFORM_DEFS exactly.
- Fix: Check the transform-reference skill for exact names
- Common mistakes:
base64_encode → base64, rot13_cipher → rot13, caesar_cipher → caesar
"Transform requires LLM but no transform_model"
- Cause: LLM-powered transforms (persuasion, language) need a model
- Fix: Set
transform_model parameter, or it defaults to attacker_model
- LLM transforms:
adapt_language, code_switch, dialectal_variation, transliterate, all persuasion, role_play_wrapper, cognitive_hacking, skeleton_key_framing, many_shot_examples
"Invalid parameter for transform"
- Cause: Wrong parameter syntax
- Fix: Use parentheses:
caesar(5), adapt_language(Zulu), vigenere(SECRET), affine(5,8)
Scorer Errors
"Scorer not found: "
- Cause: Wrong scorer name
- Fix: Check the scorer-reference skill for exact names
- Common mistakes:
memory_injection_detected → memory_injection, goal_hijack_detected → goal_hijacking
Execution Errors
"Workflow execution timed out"
- Cause: Attack took longer than timeout (default 300s, max 600s)
- Fix: Increase timeout with
execute_workflow(filename, timeout=600), or reduce n_iterations
- For long attacks:
rainbow, autodan, beast can take 10+ minutes. Use max timeout.
"Syntax error in generated script"
- Cause: Code generation bug
- Fix: Regenerate with slightly different parameters. If persistent, try a different attack type.
"ImportError: cannot import"
- Cause: SDK version mismatch or missing module
- Fix: This is an environment issue. Try a different attack that uses different SDK modules.
"Connection refused" / "Network error"
- Cause: Sandbox can't reach the API or LLM provider
- Fix: Check SANDBOX_SERVER_URL configuration. For local dev, use
http://host.docker.internal:8000
Agentic Attack Errors
"Agent endpoint returned 4xx/5xx"
- Cause: Wrong agent URL, auth, or request format
- Fix: Verify
agent_url, agent_auth_type, and agent_preset. Test the endpoint manually first.
"Could not extract response text"
- Cause:
agent_response_text_path doesn't match the actual response structure
- Fix: Use
agent_preset if the agent follows OpenAI/Anthropic format. For custom APIs, provide the correct JSONPath.
"No tool calls in response"
- Cause: Agent didn't invoke tools (may not be a tool-using agent)
- Fix: Ensure the agent is tool-enabled. Use more aggressive transforms like
tool_restriction_bypass.
Results Issues
"No analytics files found"
- Cause: Attack didn't complete or output directory is wrong
- Fix: Check
inspect_results(file_type="all") to see what files exist
"ASR is 0.0 but trials ran"
- Cause: No trials scored above threshold (0.5 default)
- Fix: This means the target defended well. Try different attacks or transforms.
"0 trials completed"
- Cause: All trials errored or timed out
- Fix: Check for model/network errors. Reduce complexity (fewer transforms, simpler attack).
Retry Strategy
- First failure: Read the error message, adjust the specific parameter that failed
- Second failure: Try a different model alias (e.g., full path instead of short alias)
- Third failure: Simplify — fewer transforms, different attack type, lower iterations
- Still failing: Try the simplest possible config:
generate_attack(attack_type="prompt", goal=..., target_model="openai/gpt-4o") to verify basic connectivity