Tell HN: Google banned Railway's account. Everything down

It appears Google has banned Railway's account, leading to a complete outage of their services. This drastic action implies a potential violation of Google's policies, though the exact nature of the violation, initially cited as "suspected fraudulent activity," remains vague, prompting Railway's team to investigate and seek clarification.

The immediate impact of this suspension was severe and widespread:

• ·All services halted: As Railway primarily utilizes Google Cloud (Compute Engine, Cloud SQL, managed Kubernetes), the suspension severed network access, storage, and the ability to provision new virtual machines, causing all live applications hosted on the platform to go down.
• ·Build pipelines dead: CI/CD processes, which depend on Google Cloud's Artifact Registry and Cloud Build, failed due to permission errors, leading to a backlog of queued builds.
• ·Dashboard inaccessible: The Railway web UI itself, served from a Google Cloud Load Balancer, became unreachable, displaying 503 errors.
• ·Support tickets flooded: Users experienced widespread outages, leading to a deluge of support inquiries.

In response, Railway's team took immediate action:

1. ·Opened a Google Cloud support case: They provided proof of ownership and recent invoices, requesting detailed reasons for the suspension.
2. ·Attempted to migrate: A fallback cluster on AWS was spun up to test traffic rerouting, acknowledging that a full DNS cutover would not be an instant fix.
3. ·Communicated transparently: Updates were posted on their status page and Twitter, explaining the outage and their efforts to reinstate services.
4. ·Prepared a rollback plan: Plans were made for re-authorizing service accounts, rebuilding lost persistent disks, and verifying data integrity once the Google account is restored.

Key takeaways and recommendations for those affected or looking to prevent similar incidents:

For Railway Users

• ·Monitor official channels: Users should consistently check Railway's status page and other official communication channels for updates on the situation.
• ·Prepare for restoration: Once services are restored, be prepared to verify data integrity and ensure your applications comply with Google's updated terms of service.

For Developers/Providers (Lessons Learned by Railway)

• ·Mitigate single-point-of-failure risk: Relying on a single cloud provider for critical infrastructure like billing and compute introduces significant risk. Exploring multi-cloud billing abstraction and decoupling control planes from specific providers is crucial.
• ·Advocate for clear abuse appeals: The lack of specific details in Google's suspension notice highlights the need for more transparent appeal processes to reduce downtime.
• ·Prioritize user communication: Prompt and honest updates during an outage are essential for maintaining user trust.

General Advice for Cloud Account Bans

• ·Review Google's Policies: Familiarize yourself with Google's terms of service and any specific terms related to their app services (Firebase, Google Cloud) to understand potential violations.
• ·Contact Google Support: Directly engage Google's support team for specifics on why the account was banned. This can be done via their Cloud Console.
• ·Rectify the Issue: Once the violation is clear, take immediate steps to modify your application or usage to ensure compliance.
• ·Prepare Documentation: Gather any relevant information that could support an appeal or clarify your situation.
• ·Seek Community Guidance: Engage with development communities or forums, as others may have experience with similar issues.
• ·Consider Legal Advice: If the ban significantly impacts your business and you believe it was unjust, exploring legal options might be necessary.

As of the last update, Google has acknowledged Railway's case but the suspension has not yet been lifted.