Customer Impact:

On Wednesday, 10 May 2017, approximately 5% of our users experienced failures when interacting with the service. The issue started at 19:32 UTC and lasted for 40 minutes. During this time, VSTS users experienced errors signing into VSTS or accessing new pages . The issue was mitigated at 20:12 UTC.

What went wrong:

VSTS relies on Azure Active Directory for both for user authentication as well as internal service-to-service token authorization. At 19:32 UTC, a configuration change was incorrectly deployed to the Azure Active Directory production environment which impacted authentication calls for all dependent services worldwide.

Our monitoring detected these failures and alerted our Site Reliability team 5 minutes into the incident. While trying to engage our partners in Azure Active Directory team, we discovered that they were already aware of the issue and working on a fix. The issue was finally resolved when the configuration change was rolled back to resolve the authentication errors.

Next Steps:

Within the VSTS service, we identified opportunities to improve our service-to-service interaction mechanisms.  Additionally, our partners in Azure Active Directory have identified several safe deployment practices related improvements which are outlined here: https://azure.microsoft.com/en-us/status/history/ (see ‘RCA – Intermittent Authentication Failures due to Underlying Azure Active Directory issues’ on 5/10).