comprehensive-test-coverage

Comprehensive Test Coverage Analysis

Universal testing methodology for SwiftUI applications that eliminates "tests pass but feature is broken" scenarios. Focuses on user outcomes and complete workflows rather than just code coverage metrics.

When to Use This Skill

• Any SwiftUI app with complex user workflows (multi-screen, data persistence, network operations)
• Tests pass but user-reported bugs indicate broken workflows
• Apps with SwiftData + CloudKit sync requirements
• Cross-platform apps (iOS + macOS) with data exchange workflows
• Multi-step user journeys that span multiple views/features
• Apps targeting specific demographics (accessibility, older users, etc.)
• Before major releases when user experience validation is critical

Universal SwiftUI Testing Principles

The Core Problem: Component vs. Outcome Testing

❌ Traditional Approach (Component Testing):

@Test func viewModelCreatesData() {
    let viewModel = FeatureViewModel()
    viewModel.createItem("Test Item")
    #expect(viewModel.items.count == 1)  // ✅ PASSES
    // But user can't actually see or interact with the item!
}

✅ Enhanced Approach (User Outcome Testing):

@Test(.tags(.userOutcome)) 
func userCanCreateAndUseItem() async throws {
    // Test the complete user journey
    let app = try await launchApp()
    
    // User creates item via UI
    try await app.navigateToCreateItem()
    try await app.fillForm(title: "Test Item", description: "Test Description")
    try await app.submitForm()
    
    // Verify user can see and use the created item
    let itemsList = try await app.navigateToItemsList()
    let createdItem = try await itemsList.findItem(title: "Test Item")
    
    #expect(createdItem.isVisible, "User must be able to see created item")
    #expect(createdItem.isInteractable, "User must be able to tap created item")
    
    // Test item functionality works end-to-end
    try await createdItem.tap()
    let detailView = try await app.waitForDetailView()
    #expect(detailView.description == "Test Description", "Data must survive complete workflow")
}

Universal Coverage Categories

1. User Journey Tests (Critical for SwiftUI Apps)

Definition: Complete workflows from user intent to successful outcome, verified with UI state.

Generic Pattern:

@Test(.tags(.userJourney)) 
func [featureName]CompleteUserWorkflow() async throws {
    let app = try await launchApp()
    
    // 1. User navigation to feature
    try await app.navigateTo(.[featureName])
    
    // 2. User completes primary action
    let result = try await app.performPrimaryAction(with: testData)
    
    // 3. Verify outcome is visible and usable to user
    #expect(result.isVisibleToUser, "User must see the result")
    #expect(result.isUsable, "Result must be functionally complete")
    
    // 4. Test data persists across app sessions
    try await app.restart()
    let persistedResult = try await app.findResult(result.id)
    #expect(persistedResult.isEquivalent(to: result))
}

SwiftUI-Specific Patterns:

SwiftData Persistence:

@Test(.tags(.swiftData, .userJourney))
func dataModelUserWorkflow() async throws {
    let app = try await launchApp()
    
    // User creates data through UI
    let createdItem = try await app.createItem(testData)
    
    // Force app restart (simulates user experience)
    try await app.forceQuit()
    let restartedApp = try await launchApp()
    
    // Verify SwiftData persistence worked from user perspective
    let items = try await restartedApp.loadItems()
    let persistedItem = try #require(items.first { $0.id == createdItem.id })
    
    #expect(persistedItem.isDisplayedCorrectly)
    #expect(persistedItem.isInteractable)
}

CloudKit Sync:

@Test(.tags(.cloudKit, .userJourney))
func cloudSyncUserExperience() async throws {
    // Test across multiple simulated devices
    let device1 = try await launchApp(device: .iPhone)
    let device2 = try await launchApp(device: .iPad, sameAppleID: true)
    
    // User creates data on device 1
    let item = try await device1.createItem(testData)
    try await device1.waitForSyncIndicator()
    
    // User opens app on device 2
    let syncedItems = try await device2.refreshAndLoadItems()
    let syncedItem = try #require(syncedItems.first { $0.id == item.id })
    
    // Verify sync is transparent to user
    #expect(syncedItem.appearsIdentical(to: item))
    #expect(!device2.showsSyncErrors, "Sync must be invisible to user")
}

2. Cross-Platform Workflow Tests

Generic Pattern for iOS ↔ macOS Apps:

@Test(.tags(.crossPlatform))
func crossPlatformDataExchange() async throws {
    // 1. iOS creates data/file
    let iOSApp = try await launchApp(platform: .iOS)
    let exportedData = try await iOSApp.createAndExportData(testData)
    
    // 2. Validate export format
    let validator = DataFormatValidator()
    #require(try validator.validate(exportedData).isValid)
    
    // 3. macOS imports and processes
    let macOSApp = try await launchApp(platform: .macOS)
    let importedData = try await macOSApp.importData(exportedData)
    
    // 4. Verify data integrity across platforms
    #expect(importedData.preservesEssentialProperties(of: testData))
    #expect(importedData.isUsableOnMacOS, "Imported data must be functionally complete")
}

3. UI State Validation Tests

SwiftUI View State Testing:

@Test(.tags(.uiValidation))
func viewStateRespondsToDataChanges() async throws {
    let app = try await launchApp()
    
    // 1. Capture initial UI state
    let initialScreen = try await app.screenshot()
    
    // 2. Trigger data change through user action
    try await app.performDataModifyingAction()
    
    // 3. Verify UI updated correctly
    let updatedScreen = try await app.screenshot()
    
    #expect(!initialScreen.isIdentical(to: updatedScreen), "UI must change when data changes")
    
    // 4. Verify specific UI elements reflect the change
    try await app.validateUIReflectsData(updatedScreen)
}

4. Error State User Experience Tests

Generic Error Handling Validation:

@Test(.tags(.errorHandling))
func errorStateUserRecovery() async throws {
    let errorSimulator = ErrorConditionSimulator()
    let app = try await launchApp(errorSimulator: errorSimulator)
    
    // 1. Trigger error condition
    errorSimulator.simulate(.networkFailure) // or .diskFull, .cloudKitUnavailable, etc.
    
    // 2. User attempts operation that will fail
    try await app.performNetworkRequiringAction()
    
    // 3. Verify error is user-friendly and actionable
    let errorState = try await app.waitForErrorState()
    
    #expect(errorState.hasUserFriendlyMessage, "Error must be understandable")
    #expect(errorState.providesRecoveryOptions, "User must have clear next steps")
    #expect(!errorState.exposesInternalDetails, "No technical jargon visible")
    
    // 4. Test recovery workflow
    errorSimulator.resolve(.networkFailure)
    try await errorState.performRecoveryAction()
    
    let recoveredState = try await app.waitForNormalOperation()
    #expect(recoveredState.isFullyFunctional, "Recovery must restore complete functionality")
}

5. Accessibility & Demographic-Specific Tests

Generic Accessibility Testing:

@Test(.tags(.accessibility))
func accessibilityUserExperience() async throws {
    let app = try await launchApp()
    
    // Test with accessibility features enabled
    try await app.enableAccessibility(features: [.voiceOver, .largeText, .highContrast])
    
    // Verify all critical workflows remain functional
    let criticalWorkflows = app.identifyCriticalWorkflows()
    
    for workflow in criticalWorkflows {
        let result = try await app.performWorkflow(workflow, withAccessibility: true)
        #expect(result.isSuccessful, "Workflow '\(workflow.name)' must work with accessibility")
        #expect(result.completionTime < workflow.maxAcceptableTime, "Must not be significantly slower")
    }
}

Demographic-Specific Testing (e.g., older users):

@Test(.tags(.demographicTargeting))
func olderUserExperience() async throws {
    let app = try await launchApp()
    
    // Apply older user simulation settings
    try await app.simulateOlderUserConditions(
        reactionTime: .slower,
        precision: .reduced,
        patience: .lower
    )
    
    // Validate UI is appropriate for demographic
    let mainScreen = try await app.screenshot()
    try await validateForOlderUsers(mainScreen)
    
    // Test simplified workflows work
    let simplifiedWorkflows = app.identifySimplifiedWorkflows()
    for workflow in simplifiedWorkflows {
        let result = try await app.performWorkflow(workflow, slowly: true)
        #expect(result.isSuccessful, "Simplified workflow must complete successfully")
    }
}

Advanced Analysis Agents

Agent 5: User Journey Gap Analysis (NEW)

Purpose: Find complete user workflows that have zero end-to-end validation.

Tasks:

1. Map complete user journeys from app entry to task completion
2. Identify journey breakpoints where tests stop validating
3. Find cross-screen dependencies that aren't tested as a complete flow
4. Validate state persistence across app sessions
5. Check error recovery paths - can users actually recover from failures?

Output: List of complete user journeys with test coverage gaps.

Agent 6: Cross-Platform Workflow Validator (NEW)

Purpose: Validate file formats, data exchange, and multi-device workflows.

Tasks:

1. Inventory file export/import workflows between platforms
2. Validate file format compatibility with real data
3. Test data round-trip integrity (export → import → verify)
4. Check version compatibility across iOS/macOS versions
5. Validate CloudKit sync workflows with real CloudKit operations

Screenshot-Verified UI Testing Integration

Using the UI-verify skill for systematic visual validation:

@Test(.tags(.uiVerification))
func gardenClubUISimplificationVerified() async throws {
    let verification = UIVerificationAgent()
    
    // 1. Launch and screenshot initial state
    let app = try await verification.launchGrowWise()
    let homeScreen = try await verification.screenshot(step: "home")
    
    // 2. Navigate to Garden Club (simplified flow)
    try await verification.tap(coordinates: gardenClubButtonLocation)
    let clubListScreen = try await verification.screenshot(step: "club-list")
    
    // 3. Validate simplified UI for older users
    try await verification.validateAccessibility(clubListScreen, criteria: .olderUsers)
    try await verification.validateButtonSize(clubListScreen, minimumSize: 44.0)
    try await verification.validateTextSize(clubListScreen, minimumPoints: 16.0)
    
    // 4. Complete club creation workflow
    try await verification.createGardenClub(name: "Test Club")
    let createdScreen = try await verification.screenshot(step: "club-created")
    
    // 5. Verify success state is clear to users
    try await verification.validateSuccessIndicator(createdScreen)
    try await verification.validateNavigationIsObvious(createdScreen)
}

Modern Swift Testing Integration

Parameterized Cross-Platform Tests

@Test("Cross-platform file compatibility", arguments: [
    ("ios-17-export.netmonblueprint", "macos-14"),
    ("ios-18-export.netmonblueprint", "macos-15"),
    ("ios-18-export.netmonblueprint", "macos-14"), // backwards compatibility
])
func crossPlatformFileCompatibility(filename: String, macOSVersion: String) async throws {
    let testBundle = try TestBundleLoader.load(filename)
    let macSimulator = try MacSimulatorManager(version: macOSVersion)
    
    let result = try await macSimulator.testImport(bundle: testBundle)
    #expect(result.isSuccess, "File from \(filename) must import on \(macOSVersion)")
    #expect(result.hasFloorPlan, "Floor plan must render correctly")
}

Advanced Async Pattern Testing

@Test(.tags(.async))
func cloudKitSyncActuallyWorks() async throws {
    // Test real CloudKit sync, not mocked
    let container = CKContainer(identifier: "iCloud.com.growwise.app")
    let database = container.privateCloudDatabase
    
    // 1. Create test record
    let garden = Garden(name: "Test Garden \(UUID())")
    let record = garden.toCKRecord()
    
    // 2. Save to CloudKit
    let savedRecord = try await database.save(record)
    
    // 3. Fetch from different context (simulates other device)
    let fetchedRecord = try await database.record(for: savedRecord.recordID)
    
    // 4. Verify data integrity
    #expect(fetchedRecord["name"] as? String == garden.name)
    #expect(fetchedRecord.modificationDate != nil)
    
    // 5. Cleanup
    try await database.deleteRecord(withID: savedRecord.recordID)
}

Execution Strategy

Phase 1: Enhanced Gap Analysis

Run the original 4 agents PLUS the 2 new agents:

• Agent 5: User Journey Gap Analysis
• Agent 6: Cross-Platform Workflow Validator

Phase 2: Screenshot-Verified UI Testing

Integration with UI-verify skill for visual regression prevention:

• Build iOS simulator tests for GrowWise Garden Club simplification
• Build macOS tests for NetMonitor light mode fixes
• Validate cross-platform file workflows with real screenshots

Phase 3: Real Service Integration Testing

• CloudKit sync with actual CloudKit containers
• Network services with URLProtocol-intercepted real responses
• File system operations with real file creation/validation

Success Metrics

Traditional metrics miss user experience. New metrics:

1. User Journey Coverage: % of complete user workflows tested end-to-end
2. Cross-Platform Compatibility: % of file/data exchange workflows validated
3. Error Recovery Rate: % of error states that provide user recovery options
4. Visual Regression Prevention: Screenshots verified for UI changes
5. State Persistence Validation: % of features tested across app sessions

Integration with Existing Projects

For NetMonitor 2.2 Light Mode Bug

# Create comprehensive light mode test suite
bd create "Light mode user experience validation" \
  --description="End-to-end testing of light mode across all screens with screenshot verification" \
  --priority 1 --type task --deps discovered-from:gh-160

For GrowWise Garden Club Simplification

# Create garden club user journey tests
bd create "Garden Club simplified UI user testing" \
  --description="Screenshot-verified user workflows for older demographic accessibility" \
  --priority 1 --type task --deps discovered-from:growwise-ui-complexity

Key Improvements Over Original

1. User Journey Focus - Tests complete workflows, not just components
2. Screenshot Verification - Visual validation prevents UI regressions
3. Cross-Platform Testing - Validates file formats and data exchange
4. Real Service Integration - Tests with actual CloudKit, network calls
5. Error UX Validation - Ensures errors surface meaningfully to users
6. State Persistence Testing - Validates data across app sessions

This enhanced approach directly addresses your "tests pass but feature is broken" problem by focusing on user outcomes rather than just code coverage.