Azure Front Door - Connectivity issues

Incident Report for Centrilogic

Resolved

Update from Azure:

The AFD service is now operating with 100% availability. All the customers and services are mitigated, completed the tail-end recovery for all the impacted customers and services.

MS status page shows There are currently no active events

Posted Oct 29, 2025 - 21:01 EDT

Update

Starting at approximately 16:00 UTC on 29 October 2025, customers and Microsoft services leveraging Azure Front Door (AFD) may have experienced latencies, timeouts, and errors. We have confirmed that an inadvertent configuration change was the trigger event for this issue.

Affected Azure services may have included, but were not limited to:

App Service, Azure Active Directory B2C, Azure Communication Services, Azure Databricks, Azure Healthcare APIs, Azure Maps, Azure Portal, Azure SQL Database, Azure Virtual Desktop, Container Registry, Media Services, Microsoft Defender External Attack Surface Management, Microsoft Entra ID (Mobility Management Policy Service, Identity & Access Management, and User Management UX), Microsoft Purview, Microsoft Sentinel (Threat Intelligence), and Video Indexer.

Current status:

We have completed deployment of our ‘last known good’ configuration, and recovery is progressing steadily. We are currently recovering nodes and re-routing traffic through healthy nodes across the global fleet. As recovery continues, some requests may still land on unhealthy nodes, resulting in intermittent failures or reduced availability for a subset of customers.

This recovery effort involves reloading configurations and rebalancing traffic across a large number of nodes to restore full operational scale. The process is gradual by design, ensuring stability and preventing overload as dependent services recover.

The AFD service is now operating above 98% availability. While the majority of customers and services are mitigated or seeing strong improvement across affected regions, we are continuing to work on tail-end recovery for remaining impacted customers and services. We have revised our mitigation time and are currently tracking toward full mitigation by 00:40 UTC on 30 October 2025, though we will communicate if mitigation is achieved sooner.

Customer configuration changes remain temporarily blocked to prevent new deployments that could interfere with recovery. We will notify customers once this block has been lifted.

We will provide another update on our progress within the next 60 minutes, or sooner if warranted.

Posted Oct 29, 2025 - 19:49 EDT

Update

Azure Front Door - Connectivity issues - Observing recovery

Starting at approximately 16:00 UTC on 29 October 2025, customers and Microsoft services leveraging Azure Front Door (AFD) may experience latencies, timeouts, and errors. We have confirmed that an inadvertent configuration change was the trigger event for this issue.

Affected Azure services include, but are not limited to: App Service, Azure Active Directory B2C, Azure Communication Services, Azure Databricks, Azure Healthcare APIs, Azure Maps, Azure Portal, Azure SQL Database, Container Registry, Media Services, Microsoft Defender External Attack Surface Management, Microsoft Entra ID, Microsoft Purview, Microsoft Sentinel, Video Indexer, and Virtual Desktop.

Current status: We initiated the deployment of our ‘last known good’ configuration, which has now successfully completed. We are currently recovering nodes and re-routing traffic through healthy nodes.

As recovery progresses, some requests may still land on unhealthy nodes, resulting in intermittent failures or reduced availability until more nodes are fully restored. This recovery effort involves reloading configurations and rebalancing traffic across a large volume of nodes to restore full operational scale. The process is gradual by design, ensuring stability and preventing overload as dependent services recover. We expect continued improvement across affected regions. This means we expect recovery to happen by 23:20 UTC [19:20 EDT] on 29 October 2025

Customer configuration changes remain temporarily blocked to prevent new deployments that could interfere with recovery. We will notify customers once this block has been lifted. Customers can failover to origins if they decide to.

Some customers may also have experienced issues accessing the Azure management portal. We have failed the portal away from AFD to mitigate these access issues. Customers should now be able to access the Azure portal directly, and while most portal extensions are functioning as expected, a small number of endpoints (e.g., Marketplace) may still experience intermittent loading problems.

We will provide another update on our progress within two hours, or sooner if warranted. Although we are seeing signs of recovery, customers may also consider implementing existing failover strategies using Azure Traffic Manager to redirect traffic from Azure Front Door to their origin servers as an interim measure. https://learn.microsoft.com/azure/frontdoor/routing-methods

Posted Oct 29, 2025 - 17:41 EDT

Update

Update from Azure:

"Starting at approximately 16:00 UTC, customers and Microsoft services leveraging Azure Front Door (AFD) may have experienced latencies, timeouts, and errors. We have confirmed that an inadvertent configuration change was the trigger event for this issue.

Current status:

We initiated the deployment of our ‘last known good’ configuration, which has now successfully been completed. Customers may have begun to see initial signs of recovery. We are currently recovering nodes and routing traffic through healthy nodes, and as we make progress in this workstream, customers will continue to see improvement.

Customer configuration changes will remain temporarily blocked while we continue mitigation efforts. We will notify customers once this block has been lifted.

Some customers may also have experienced issues accessing the Azure management portal. We have failed the portal away from AFD to mitigate these access issues. Customers should now be able to access the Azure portal directly, and while most portal extensions are functioning as expected, a small number of endpoints (e.g., Marketplace) may still experience intermittent loading problems.

At this stage, we anticipate full mitigation within the next four hours as we continue to recover nodes. We will provide another update on our progress within two hours, or sooner if warranted.

Although we are seeing signs of recovery and have an estimated timeline, customers may also consider implementing failover strategies using Azure Traffic Manager to redirect traffic from Azure Front Door to their origin servers as an interim measure.

Learn more about Azure Front Door failover strategies for AFD: https://learn.microsoft.com/en-us/azure/architecture/guide/networking/global-web-applications/overview"

Posted Oct 29, 2025 - 15:24 EDT

Update

As per Azure status page, they have initiated the deployment of last known good configuration, which is expected to complete within 30 minutes. As this deployment progresses, customers should begin to see initial signs of recovery. Once completed, They will begin recovering nodes and routing traffic through these healthy nodes.

Customer configuration changes will remain temporarily blocked while they continue mitigation efforts. Customers will be notified once this block has been lifted.

Some customers may also have experienced issues accessing the Azure management portal. Azure have failed the portal away from AFD to mitigate these access issues. Customers should now be able to access the Azure portal directly, and while most portal extensions are functioning as expected, a small number of endpoints (e.g., Marketplace) may still experience intermittent loading problems.

We do not yet have an ETA for full mitigation.

Customers may also consider implementing failover strategies using Azure Traffic Manager to redirect traffic from Azure Front Door to their origin servers as an interim measure.

Posted Oct 29, 2025 - 14:38 EDT

Update

Azure has confirmed that an inadvertent configuration change triggered this issue.

Azure is taking several concurrent actions:

Blocking all changes to AFD services, including customer configuration changes.

Rolling back the AFD configuration to the last known good state, ensuring the problematic configuration does not reapply upon recovery.

Customers may have experienced difficulties accessing the Azure management portal. The portal has been failed away from AFD to mitigate access issues. While the portal is now accessible and most extensions are functioning, a small number of endpoints (e.g., Marketplace) may still experience issues.

There is currently no ETA for completion of the rollback.

Customers can consider implementing failover strategies with Azure Traffic Manager, to fail over from Azure Front Door to your origins: https://learn.microsoft.com/azure/architecture/guide/networking/global-web-applications/overview

Posted Oct 29, 2025 - 13:59 EDT

Update

Starting at approximately 12:00 PM ET, Azure began experiencing Azure Front Door (AFD) issues, resulting in a loss of availability of some services. This was likely triggered by an inadvertent configuration change.

Azure is taking two concurrent actions to resolve the issue:

1. Blocking all changes to AFD services and disabling the problematic route identified.

2. Rolling back to the last known good configuration.

The portal has been failed away from AFD, and customers should be able to access the Azure management portal directly.

There is currently no ETA for completion of the rollback.

Posted Oct 29, 2025 - 13:21 EDT

Identified

As per Azure status update starting at approximately 12:00 PM ET, customers may experience issues accessing the Azure Portal due to DNS-related disruptions affecting service availability. Customers can continue to access and manage resources using programmatic methods such as PowerShell, CLI, or SDKs if the portal is unavailable.

Azure has rerouted the portal away from Azure Front Door (AFD) to mitigate access issues and is actively assessing failover options. Investigation into contributing factors and additional recovery workstreams is ongoing.

Posted Oct 29, 2025 - 13:07 EDT

This incident affected: Services (Dedicated and Shared hosting, Colocation, CPE) (CA - Services, USA-Services, UK-Services, Calgary DC).