error-troubleshooting

Error Troubleshooting

Common errors and fixes for AIRT attack workflows.

Model Errors

"Model not found" / "Invalid model"

• Cause: Bare model name without provider prefix
• Fix: Always use provider/model format: openai/gpt-4o, anthropic/claude-sonnet-4-20250514
• Aliases: Use short aliases like gpt-4o, claude, groq — they auto-resolve

"Rate limit exceeded" / 429

• Cause: Too many concurrent requests to the provider
• Fix: Reduce n_iterations or switch to a less rate-limited provider
• For Groq: Groq has strict rate limits. Use groq/llama-3.3-70b-versatile for lower limits

"Authentication failed" / 401

• Cause: Missing or invalid API key for the provider
• Fix: Ensure the correct API key is set as a platform secret (OPENAI_API_KEY, ANTHROPIC_API_KEY, GROQ_API_KEY, etc.)
• Check: API keys are pre-configured in the environment. If missing, ask the user to add them via Settings > Secrets

"Context length exceeded"

• Cause: Conversation history too long for model's context window
• Fix: Reduce context_depth or n_iterations, or switch to a model with larger context

Transform Errors

"Transform not found: "

• Cause: Wrong transform name. Names must match TRANSFORM_DEFS exactly.
• Fix: Check the transform-reference skill for exact names
• Common mistakes: base64_encode → base64, rot13_cipher → rot13, caesar_cipher → caesar

"Transform requires LLM but no transform_model"

• Cause: LLM-powered transforms (persuasion, language) need a model
• Fix: Set transform_model parameter, or it defaults to attacker_model
• LLM transforms: adapt_language, code_switch, dialectal_variation, transliterate, all persuasion, role_play_wrapper, cognitive_hacking, skeleton_key_framing, many_shot_examples

"Invalid parameter for transform"

• Cause: Wrong parameter syntax
• Fix: Use parentheses: caesar(5), adapt_language(Zulu), vigenere(SECRET), affine(5,8)

Scorer Errors

"Scorer not found: "

• Cause: Wrong scorer name
• Fix: Check the scorer-reference skill for exact names
• Common mistakes: memory_injection_detected → memory_injection, goal_hijack_detected → goal_hijacking

Execution Errors

"Workflow execution timed out"

• Cause: Attack took longer than timeout (default 300s, max 600s)
• Fix: Increase timeout with execute_workflow(filename, timeout=600), or reduce n_iterations
• For long attacks: rainbow, autodan, beast can take 10+ minutes. Use max timeout.

"Syntax error in generated script"

• Cause: Code generation bug
• Fix: Regenerate with slightly different parameters. If persistent, try a different attack type.

"ImportError: cannot import"

• Cause: SDK version mismatch or missing module
• Fix: This is an environment issue. Try a different attack that uses different SDK modules.

"Connection refused" / "Network error"

• Cause: Sandbox can't reach the API or LLM provider
• Fix: Check SANDBOX_SERVER_URL configuration. For local dev, use http://host.docker.internal:8000

Agentic Attack Errors

"Agent endpoint returned 4xx/5xx"

• Cause: Wrong agent URL, auth, or request format
• Fix: Verify agent_url, agent_auth_type, and agent_preset. Test the endpoint manually first.

"Could not extract response text"

• Cause: agent_response_text_path doesn't match the actual response structure
• Fix: Use agent_preset if the agent follows OpenAI/Anthropic format. For custom APIs, provide the correct JSONPath.

"No tool calls in response"

• Cause: Agent didn't invoke tools (may not be a tool-using agent)
• Fix: Ensure the agent is tool-enabled. Use more aggressive transforms like tool_restriction_bypass.

Results Issues

"No analytics files found"

• Cause: Attack didn't complete or output directory is wrong
• Fix: Check inspect_results(file_type="all") to see what files exist

"ASR is 0.0 but trials ran"

• Cause: No trials scored above threshold (0.5 default)
• Fix: This means the target defended well. Try different attacks or transforms.

"0 trials completed"

• Cause: All trials errored or timed out
• Fix: Check for model/network errors. Reduce complexity (fewer transforms, simpler attack).

Retry Strategy

1. First failure: Read the error message, adjust the specific parameter that failed
2. Second failure: Try a different model alias (e.g., full path instead of short alias)
3. Third failure: Simplify — fewer transforms, different attack type, lower iterations
4. Still failing: Try the simplest possible config: generate_attack(attack_type="prompt", goal=..., target_model="openai/gpt-4o") to verify basic connectivity