Three production applications down simultaneously on AWS requires immediate and systematic triage to identify the shared point of failure. Here’s the definitive action plan:

Immediate Triage Actions:

1. ·AWS Service Health Dashboard — Check for regional or service-specific outages affecting your applications. This is the first step to rule out AWS-side issues.
2. ·Shared Infrastructure Audit — Inspect common dependencies across all three apps:
   • ·Networking: VPC, NACLs, security groups, route tables, NAT gateways
   • ·Databases/Caches: RDS/Aurora clusters, ElastiCache, DynamoDB
   • ·Load Balancers: ALB/NLB health, listener rules, certificates
   • ·IAM Roles/Policies: Verify no recent changes broke permissions
3. ·CloudWatch & Logs — Search for concurrent spikes in errors, latency, or throttling. Check:
   • ·EC2 instance logs (/var/log/)
   • ·CloudWatch Logs for app-specific errors
   • ·RDS/Aurora metrics (CPU, connections)
4. ·Recent Changes — Review:
   • ·CI/CD deployments (code or IaC updates)
   • ·CloudTrail for unauthorized API calls

Likely Root Causes:

• ·AWS Region/Service Outage (if apps are co-located)
• ·Shared Database/Cache Failure (e.g., single RDS instance)
• ·IAM or Networking Misconfiguration (e.g., route table update, NACL change)
• ·DDoS/WAF Rate Limiting (if apps share a Shield/WAF)

Recovery Steps:

• ·Failover: Activate multi-AZ/multi-region fallbacks if available.
• ·Rollback: Revert recent deployments or IaC changes if correlated.
• ·AWS Support: Escalate with Urgent severity if AWS-side.
• ·Communicate: Update stakeholders with clear timelines/actions.

Debugging Commands (Key Examples): bash

Check ALB target health

aws elbv2 describe-target-health --target-group-arn <arn>

Verify RDS connectivity

telnet <rds-endpoint> 3306

Simulate IAM permissions

aws iam simulate-principal-policy --policy-source-arn <role-arn> --action-names ec2:DescribeInstances

Long-term Prevention:

• ·Fault Isolation: Avoid shared single points of failure.
• ·Multi-Region Design: Critical apps should span regions.
• ·Automated Playbooks: Use AWS Systems Manager for rapid response.

The key is methodical elimination—trace dependencies to isolate the common failure. Simultaneous outages are rarely coincidental.