On October 24th, 2025 from approximately 01:53AM UTC until 04:24AM UTC our package download service experienced an outage affecting customer access. This was confirmed to have impacted Debian and RubyGems.
At 01:13AM UTC we updated the S3 Bucket Policy for an S3 bucket containing customer packages via Terraform as part of a larger re-architecting project.
We received customer reports about Debian package download issues from 01:53AM UTC and started doing isolated, per-customer investigations. At 03:40AM UTC we concluded that the issues were not isolated and declared the incident. More members from our team were called to help investigate the problem, where we found that service for pulling RubyGems had also been impacted. Around 04:00 UTC, the new bucket policy deployed at 01:13AM UTC was identified as a main factor to the access failure and was removed.
At 04:19AM UTC our team identified that updating the IAM policy via Terraform did not have the expected result of merging with the pre-existing policy, but rather inadvertently overwrote the pre-existing access policies, causing the loss of access permissions necessary for package downloads. Afterwards it became apparent to our team making these changes that this particular behavior is a known issue with the Terraform AWS provider, which contributed to the difficulty of detection and prevention, thereby increasing the likelihood of the incident occurring for those unaware.
After redeploying the pre-existing S3 bucket IAM policy, we confirmed service functionality for Debian package downloads was restored at 04:24 AM UTC. The team subsequently verified that pulling RubyGems and other supported package types were also functioning as expected, and the incident was closed at 04:47 AM UTC.
We are revising our monitoring approaches to identify and address service disruptions proactively, while also improving our communication processes to ensure timely and accurate customer updates.