Internal BGP misconfiguration leading to public outage

Incident Report for APNIC Services Status

Resolved

Start Time: 13 December 2023 10:54 (UTC +10)
End Time: 13 December 2023 11:25 (UTC +10)

Duration: 31 minutes

Services Affected:
Partial Outage:
* arena-pac.net
* rrdp.apnic.net

Complete Outage:
* apidt.org
* apnic.foundation
* rrdp.sub.apnic.net
* rpki.sub.apnic.net
* wq.apnic.net

Due to a mistake in BGP configuration, some peers in our core environment were shutdown. All affected services were restored after fixing the configuration. We are working on improvements to our monitoring so the cause of outages affecting multiple services is more visible to our on-call operations team to help diagnose this type of problem faster in future.

Posted Dec 13, 2023 - 00:54 UTC