GCP WAF Reliability Review
Purpose
Evaluate GCP workload reliability against the Google Cloud Well-Architected Framework (WAF) reliability pillar. This skill supports consistent performance, resilience to failures, and graceful recovery for workloads running on GCP.
When to use
Use this skill for:
- Assessing availability requirements and SLO definitions for a workload
- Designing highly available, resilient architectures on GCP
- Auditing a workload against the nine WAF reliability principles
- Evaluating redundancy topology, observability coverage, and failure testing maturity
- Reviewing postmortem culture and data recovery practices
WAF Reliability Pillar Overview
The GCP Well-Architected Framework reliability pillar provides guidance across nine core principles. Together they ensure workloads deliver consistent performance to users, remain resilient under fault conditions, and recover gracefully from both partial and complete failures. The pillar emphasizes user-experience-driven SLOs, error budget discipline, horizontal scalability, and continuous improvement through postmortems.
Core Principles
1. Define Reliability Based on User-Experience Goals
Anchor reliability targets to user-observable outcomes rather than internal infrastructure metrics. Define SLIs that reflect what users actually care about — latency, availability, correctness, and throughput.
2. Set Realistic Targets for Reliability
Establish SLOs and error budgets that are achievable, meaningful, and aligned with business risk tolerance. Avoid aspirational targets that drain engineering capacity without delivering user value.
3. Build Highly Available Systems Through Resource Redundancy
Eliminate single points of failure by distributing resources across multiple zones and regions. Design for fault isolation so that the failure of any single component does not cascade to a full system outage.
4. Take Advantage of Horizontal Scalability
Design systems to scale out rather than up. Use managed instance groups, GKE node pools, and serverless platforms to handle load increases without manual intervention and without architectural bottlenecks.
5. Detect Potential Failures by Using Observability
Instrument workloads with metrics, logs, and traces that provide actionable signals before failures reach users. Use SLI-based alerting to distinguish user-impacting events from noise.
6. Design for Graceful Degradation
Architect systems to maintain partial functionality under failure conditions. Implement circuit breakers, retries with exponential backoff, bulkheads, rate limiting, and fallback paths to prevent cascading failures.
7. Perform Testing for Recovery from Failures
Validate resilience assumptions through chaos engineering and game days. Test failure modes including zone outages, dependency timeouts, traffic surges, and misconfigurations in pre-production and, where safe, in production.
8. Perform Testing for Recovery from Data Loss
Validate backup and restore procedures regularly. Test recovery time objective (RTO) and recovery point objective (RPO) compliance under realistic failure scenarios, not just in tabletop exercises.
9. Conduct Thorough Postmortems
Build a blameless postmortem culture to systematically capture incident learnings. Track action items to completion and share findings to prevent recurrence across teams.
Relevant GCP Products
- Compute & Containers: Compute Engine Managed Instance Groups (MIGs), Google Kubernetes Engine (GKE), Cloud Run, Cloud Run Functions
- Load Balancing & Networking: Cloud Load Balancing (global and regional), Cloud CDN, Cloud DNS with routing policies, Traffic Director
- Data & Storage: Cloud Storage (multi-region buckets), Cloud SQL with HA and read replicas, Cloud Spanner (multi-region), Bigtable, Firestore
- Observability: Cloud Monitoring, Cloud Logging, Cloud Trace, Google Cloud Managed Service for Prometheus, Error Reporting, SLO monitoring
- Backup & Resilience: Backup and DR Service, Cloud Storage versioning, Cloud SQL automated backups, Spanner PITR
- Autoscaling: MIG autoscaler, GKE Horizontal Pod Autoscaler (HPA), GKE Vertical Pod Autoscaler (VPA), GKE cluster autoscaler, Cloud Run concurrency-based scaling
Assessment Question Bank
SLO/SLI Definition and Error Budgets
- Are SLIs defined based on user-observable outcomes (latency, availability, correctness) rather than infrastructure metrics?
- Is an SLO set for each user-facing service with an associated error budget?
- Is the error budget actively tracked and used to gate feature velocity vs. reliability investment?
- Are SLOs documented in a service-level agreement (SLA) or internal reliability contract?
- Are SLOs reviewed and updated when user expectations or product requirements change?
- Is there a process for alerting when error budget burn rate is too high?
Realistic Reliability Targets
- Were SLO targets derived from user research, business requirements, or dependency constraints rather than set arbitrarily?
- Is the cost of each additional nine (99.9% vs. 99.99%) understood and justified?
- Are dependencies' reliability levels accounted for when setting the workload's own SLO?
- Is there a tiering model distinguishing critical vs. non-critical services with different reliability targets?
- Are error budget policies defined (e.g., freeze new features when budget is exhausted)?
High Availability Topology
- Are workloads deployed across at least two availability zones within a region?
- Are stateful services (databases, caches) configured for multi-zone or multi-region replication?
- Are managed instance groups or GKE deployments configured with anti-affinity rules to spread replicas across zones?
- Is Cloud Load Balancing configured with health checks that remove unhealthy backends automatically?
- Are DNS failover or routing policies used to redirect traffic during regional impairments?
- Is there a documented recovery procedure for a full zone failure?
- Are global load balancers used for latency-sensitive workloads requiring cross-region failover?
Horizontal Scalability
- Are workloads designed to scale out (add instances) rather than scale up (increase instance size)?
- Are autoscaling policies configured and tested for all stateless compute tiers?
- Is there capacity headroom available to absorb unexpected traffic spikes without manual intervention?
- Are stateful components (databases, queues) able to scale horizontally or handle peak load without bottlenecks?
- Are scaling decisions based on meaningful metrics (request latency, queue depth) rather than just CPU utilization?
- Are scale-in operations safe — do they wait for in-flight requests to complete before terminating instances?
Observability
- Are the four golden signals (latency, traffic, errors, saturation) instrumented for each service?
- Are SLI metrics available in Cloud Monitoring with SLO policies configured?
- Is distributed tracing enabled to diagnose cross-service latency issues?
- Are logs structured and indexed for efficient querying during incidents?
- Are alerting policies calibrated to user-impacting events — not just infrastructure anomalies?
- Is there an on-call runbook for every critical alert?
- Is Managed Service for Prometheus used for workload-level metrics in GKE environments?
- Are dashboards available that show SLO burn rate in real time?
Graceful Degradation
- Are circuit breakers implemented for calls to external dependencies and downstream services?
- Are retries bounded with exponential backoff and jitter to prevent retry storms?
- Are rate limits and concurrency controls applied to protect services from overload?
- Are fallback paths defined — what does the system do when a non-critical dependency is unavailable?
- Are bulkheads or resource pools used to isolate failures in one subsystem from others?
- Are timeouts set on all synchronous outbound calls?
- Is load shedding implemented to preserve core functionality under extreme load?
Failure Recovery Testing
- Are chaos engineering experiments (e.g., zone failure simulation, dependency injection) performed in pre-production?
- Is a game day schedule maintained and followed at least annually for critical workloads?
- Are failure scenarios documented with expected behavior and actual results compared?
- Are runbooks validated by actually executing them during game days rather than just reviewing them?
- Is the blast radius of each experiment scoped and approved before execution?
Data Recovery Testing
- Are backup procedures automated and scheduled for all stateful services?
- Are backup restores tested on a scheduled cadence (e.g., monthly) — not just at setup time?
- Are RTO and RPO targets defined and validated through restore tests?
- Are backups stored in a separate project or region from the primary data to protect against account-level failures?
- Is point-in-time recovery (PITR) enabled for databases where fine-grained recovery is required?
- Are backup success/failure alerts configured and monitored?
Postmortems
- Is a blameless postmortem required for every P0 and P1 incident?
- Are postmortems completed within a defined SLA (e.g., 5 business days after incident resolution)?
- Are action items assigned owners and tracked to completion in a system of record?
- Are postmortem findings shared across teams to prevent similar incidents?
- Is there a process for identifying recurring themes across postmortems?
- Are follow-up reliability improvements validated before the next incident review cycle closes?
Validation Checklist
SLI/SLO Monitoring
Cross-Zone Redundancy
Autoscaling
Health Checks
Backup Testing
Graceful Degradation Patterns
Game Days and Chaos Engineering
Postmortem Process
Response Shape
- Scope — workload name, GCP resource hierarchy scope, evidence level (live / sanitized / documentation-based / inference)
- SLO/SLI Assessment — review of SLI definitions, SLO targets, error budget tracking, and alerting calibration
- HA Topology Review — evaluation of redundancy patterns, load balancing configuration, and zone/region coverage
- Observability Gaps — missing metrics, alerts, dashboards, or tracing coverage that reduce incident detection capability
- Failure Testing Status — game day schedule, chaos experiment coverage, runbook validation status
- Recommendations — ordered by reliability risk (Critical / High / Medium / Low), each with minimum required change, validation step, and rollback procedure
- Open Risks — items that could not be assessed due to missing evidence, with recommended evidence to gather