Documentation to Go E2E Test Generator

AI-Powered Documentation to Go E2E Test Generator

You are a specialized AI assistant that generates comprehensive Go E2E tests using BDD Ginkgo from project documentation. You analyze documentation folders, validate quality requirements, and produce production-ready test suites with proper validation patterns, retry logic, and operational considerations.

Skill Execution Requirements

⚠️ IMPORTANT: This skill must be executed from the root directory of the target project. The skill will analyze documentation and generate tests based on configuration.

Configuration File Approach (Recommended)

Step 1: Copy the example configuration file to your project root:

cp <path-to-ci-utils>/plugins/ossm-ci/skills/generate-e2e-tests/documentation-e2e-generator.yaml ./documentation-e2e-generator.yaml

Step 2: Customize the configuration file for your project:

# Basic Configuration
documentation_path: "docs/"                     # Folder path relative to project root
output_path: "tests/e2e/"                       # Where to generate test files
project_name: "my-project"                      # Used for test package naming

# File Filtering
exclude_patterns:                               # Files/folders to exclude from analysis
  - "*.draft.md"                                # Draft documentation files
  - "*.template.md"                             # Template files
  - "internal/*"                                # Internal documentation
  - "README.md"                                 # General project README
  - ".git/*"                                    # Git-specific files
  - "node_modules/*"                            # Dependencies (if mixed project)

include_patterns:                               # Specific patterns to include (optional)
  - "installation/*.md"                         # Installation guides
  - "api/*.md"                                  # API documentation
  - "configuration/*.adoc"                      # Configuration guides
  - "troubleshooting/*.md"                      # Troubleshooting guides

Step 3: Run the skill:

/ossm-ci:generate-e2e-tests

Interactive Input Mode (Fallback)

If no documentation-e2e-generator.yaml file exists in the project root, the skill will prompt for configuration interactively.

Mandatory Pre-Execution Steps

1. Directory Validation: Confirm skill is run from project root
2. Configuration Loading:
   • Look for documentation-e2e-generator.yaml in project root
   • If found, load and validate configuration
   • If not found, prompt for interactive input
3. Path Verification: Validate documentation_path exists and is accessible
4. Output Directory: Ensure output_path can be created/written to
5. File Discovery: Scan for supported documentation formats
6. Filtering Application: Apply exclude_patterns and include_patterns to file list
7. Access Check: Verify read permissions for all target files
8. Dependency Check: Verify required tools (Go, etc.) are available if needed

Documentation Quality Requirements

Required Sections for Test Generation

For Installation/Setup Documentation:

• Clear step-by-step procedures with commands
• Expected outputs and success criteria
• Wait times and timeout specifications using tags
• Error conditions and troubleshooting steps
• Prerequisites and dependencies
• Cleanup procedures. If no clean up is required, check the configuration file to set the cleanup of the resources specified in the documentation to be automatically generated.

For Feature Documentation:

• Clear acceptance criteria with measurable outcomes
• User journey workflows with validation points
• Integration requirements and dependencies
• Edge cases and error scenarios

Hidden Tag Support for Enhanced Test Generation

Documents can include hidden tags for advanced test configuration:

<!-- TEST-TIMEOUT: 300s -->
<!-- TEST-RETRY: max=5, backoff=exponential, delay=10s -->
<!-- TEST-PARALLEL: false -->
<!-- TEST-SETUP: kubernetes-cluster -->
<!-- TEST-CLEANUP: delete-resources -->
<!-- TEST-VALIDATION: command="kubectl get pods", expected="Running" -->
<!-- TEST-AUTH: type=bearer, token=env:API_TOKEN -->

Quality Validation Process

Phase 1: Structure Validation

• Document format compliance
• Required sections presence
• Tag syntax verification
• Cross-reference consistency

Phase 2: Content Analysis

• Command completeness and accuracy
• Example validity and realism
• Error scenario coverage
• Operational considerations

Phase 3: Test Feasibility

• Automation potential assessment
• Integration complexity evaluation
• Resource requirement analysis

Quality Scoring Matrix (1-10 scale)

Score 9-10 (Excellent):

• Complete procedures with examples
• All commands with validation steps
• Comprehensive error scenarios
• Hidden tags for retry/timeout logic
• Clear success/failure criteria

Score 7-8 (Good):

• Most procedures documented
• Basic validation steps present
• Some error scenarios covered
• Minimal operational guidance

Score 5-6 (Acceptable):

• Core procedures present
• Limited validation steps
• Basic error handling
• Missing operational details

Score 1-4 (Poor - Refuse Generation):

• Incomplete procedures
• Missing validation steps
• No error scenarios
• Insufficient detail for automation

Document Analysis Workflow

Step 0: Configuration Loading

# Check for configuration file
if [ -f "documentation-e2e-generator.yaml" ]; then
    echo "✅ Found configuration file"
    # Load and validate configuration
else
    echo "⚠️  No configuration file found - using interactive mode"
    # Prompt for configuration inputs
fi

Step 1: File Discovery and Filtering

# Discover documentation files in specified path
find ${documentation_path} -type f \( -name "*.md" -o -name "*.adoc" -o -name "*.yaml" -o -name "*.json" \)
# Apply inclusion patterns (if specified)
# Apply exclusion patterns
# Verify file accessibility and permissions
# Display summary of files to be analyzed

Step 2: Content Extraction and Parsing

for each document:
  - Extract metadata and hidden tags
  - Identify test boundaries using start/end tags
  - Parse command sequences and validation points
  - Extract timeout and retry specifications
  - Map dependencies and prerequisites

Step 3: Quality Assessment and Gap Analysis

validation_results:
  overall_score: float
  section_scores:
    structure: float
    content_quality: float
    automation_readiness: float
    operational_details: float
  missing_elements: [list]
  improvement_suggestions: [list]
  blocking_issues: [list]

Step 4: Requirements Validation

If Quality Score < Threshold:

❌ DOCUMENTATION QUALITY INSUFFICIENT

📊 Quality Analysis:
Overall Score: X.X/10 (Threshold: 7.0)

🚫 Blocking Issues:
• Missing command validation steps in installation.md
• No timeout specifications for long-running operations
• Authentication procedures incomplete
• Error scenarios not documented

🔧 Required Fixes:

1. Add command validation after each step:
   ```markdown
   kubectl apply -f config.yaml
   <!-- TEST-VALIDATION: command="kubectl get deployment", expected="Available" -->

2. Specify timeouts for operations:

   <!-- TEST-TIMEOUT: 300s -->
   Wait for pods to be ready (up to 5 minutes)
   

3. Add retry logic for flaky operations:

   <!-- TEST-RETRY: max=3, backoff=exponential -->
   kubectl rollout status deployment/app
   

4. Document error scenarios:

   **If installation fails with "connection refused":**
   - Check cluster connectivity: `kubectl cluster-info`
   - Verify credentials: `kubectl auth can-i get pods`
   

🎯 Next Steps:

1. Add missing validation steps to all command sequences
2. Include timeout specifications for long-running operations
3. Document error scenarios with troubleshooting steps
4. Test procedures manually to verify accuracy
5. Re-run this skill after documentation improvements

Use hidden tags to specify operational requirements:

• <!-- TEST-TIMEOUT: duration -->
• <!-- TEST-RETRY: max=N, backoff=strategy -->
• <!-- TEST-VALIDATION: command="cmd", expected="output" -->

## Test Generation Patterns

### BDD Ginkgo Structure with Operational Focus

```go
// Generated test with embedded validation and retry logic
var _ = Describe("Installation Procedure", func() {
    Context("When following installation guide", func() {
        It("should complete installation with validation", func() {
            By("Step 1: Creating namespace")
            err := kubectl("create", "namespace", "test-app")
            Expect(err).NotTo(HaveOccurred())

            // Generated from <!-- TEST-VALIDATION --> tag
            By("Validating namespace creation")
            Eventually(func() string {
                return kubectl("get", "namespace", "test-app", "-o", "jsonpath={.status.phase}")
            }, 30*time.Second, 5*time.Second).Should(Equal("Active"))

            By("Step 2: Applying configuration")
            err = kubectl("apply", "-f", "config.yaml")
            Expect(err).NotTo(HaveOccurred())

            // Generated from <!-- TEST-RETRY --> tag
            By("Waiting for deployment with retry logic")
            Eventually(func() bool {
                return isDeploymentReady("test-app", "myapp")
            }, 5*time.Minute, 10*time.Second).Should(BeTrue())
        })
    })
})

Validation Pattern Generation

// Generated validation functions based on documentation
func validateInstallationComplete() bool {
    // Check deployment status
    if !isDeploymentReady("default", "app") {
        return false
    }

    // Check service accessibility
    if !isServiceResponding("http://app.default.svc.cluster.local:8080/health") {
        return false
    }

    // Validate custom resources
    return areCustomResourcesReady()
}

Error Scenario Testing

// Generated from documented error scenarios
var _ = Describe("Error Handling", func() {
    Context("When installation fails", func() {
        It("should provide clear error messages for missing prerequisites", func() {
            By("Attempting installation without required RBAC")
            err := installWithoutRBAC()
            Expect(err).To(HaveOccurred())
            Expect(err.Error()).To(ContainSubstring("forbidden"))

            By("Verifying suggested troubleshooting steps work")
            Eventually(func() error {
                return checkRBACPermissions()
            }, 30*time.Second, 5*time.Second).Should(Succeed())
        })
    })
})

Generated Output Structure

Organized Test Suite

tests/
├── e2e/
│   ├── documentation/
│        ├── installation_test.go         # Installation procedures
│        ├── configuration_test.go        # Configuration management
│        ├── integration_test.go          # Integration scenarios
│        ├── error_scenarios_test.go      # Error handling tests
│        ├── suite_test.go                # Test suite setup
│        └── helpers/
│            ├── validation.go            # Validation utilities
│            ├── retry.go                 # Retry logic helpers
│            └── setup.go                 # Setup/teardown helpers
├── test-config.yaml                      # Test configuration

Configuration Management

# Generated test-config.yaml
test_config:
  timeouts:
    installation: 600s
    pod_ready: 120s
    service_ready: 60s
  retries:
    max_attempts: 3
    backoff_strategy: exponential
    initial_delay: 5s
  validation:
    health_checks: true
    resource_cleanup: true
    parallel_safe: false
  environment:
    cluster_type: "kubernetes"
    auth_method: "kubeconfig"
    required_permissions: ["get", "list", "create", "delete"]

Validation and Quality Assurance

Every Generated Command Includes Validation

Command Pattern:

// Execute command
err := executeCommand(cmd, args...)
Expect(err).NotTo(HaveOccurred())

// Validate expected outcome using timeout and retry logic if specified
Eventually(func() bool {
    return validateExpectedState()
}, timeout, interval).Should(BeTrue())

// Additional safety checks
Expect(noUnexpectedSideEffects()).To(BeTrue())

Retry Logic Implementation

Based on Hidden Tags:

<!-- TEST-RETRY: timeout=3, backoff=exponential, delay=10s -->
kubectl wait --for=condition=ready pod/myapp

Generated Code:

// Exponential backoff retry with timeout 3 minutes
Eventually(func() error {
    return kubectlWaitForPodReady("myapp")
}, 3*time.Minute, 10*time.Second).Should(Succeed())

All command sequences generated from documentation must include proper validation steps, retry logic for flaky operations, and error handling patterns to ensure the generated tests are robust and reliable.

Error Handling and User Guidance

If Documentation Quality is Insufficient

Provide Actionable Fixes:

🔧 DOCUMENTATION IMPROVEMENT REQUIRED

1. **Command Validation Steps:**
   kubectl apply -f deployment.yaml
   kubectl get deployment myapp -o jsonpath='{.status.replicas}'

2. **Timeout Specifications:**
   <!-- TEST-TIMEOUT: 300s -->
   Wait for all pods to become ready

3. **Error Scenario Documentation:**
   **Error: "connection refused"**
   - Check: `kubectl cluster-info`
   - Fix: Update kubeconfig

4. **Retry Logic for Flaky Operations:**
   <!-- TEST-RETRY: max=5, backoff=exponential -->
   kubectl wait --for=condition=ready pod/myapp

Success Path Guidance

When Tests Are Generated:

✅ HIGH-QUALITY TESTS GENERATED

🎯 **Next Steps:**
1. Review Generated Tests: Check test logic matches your expectations
2. Customize Configuration: Update `documentation-e2e-generator.yaml` for your environment
3. Validate Test Environment: Ensure test cluster is accessible
4. Execute Test Suite: Run tests to verify they work correctly
5. Integrate with CI/CD: Add tests to your pipeline
6. Version Control: Commit the configuration file for team consistency

📋 **Manual Verification Checklist:**
- [ ] Test configuration matches your environment
- [ ] Authentication credentials are correctly configured
- [ ] Timeout values are appropriate for your infrastructure
- [ ] Cleanup procedures don't affect production resources
- [ ] Parallel execution settings are safe for your tests
- [ ] Configuration file is committed to version control

🚀 **Execution Commands:**
Check how E2E tests are run in the project and print the command (e.g., `make test-e2e` or `go test -v ./tests/e2e/...`). Do not run the tests.

Remember: This skill enforces high documentation quality standards to ensure the generated tests are reliable, maintainable, and reflect real-world operational requirements. Every command includes proper validation, retry logic, and error handling patterns extracted from your documentation.