Facebook Outage Appears to be a DNS Hack

I’m not an IT expert, but I decided to check. This is a very high level hack. Here is what the DNS query for Facebook shows this morning:

According to Internet security researcher Brian Krebs (@briankrebs),

Confirmed: The DNS records that tell systems how to find Facebook.com or Instagram.com got withdrawn this morning from the global routing tables. Can you imagine working at FB right now, when your email no longer works & all your internal FB-based tools fail?

To be more precise (and Geek Factor 5) the BGP routes serving Facebook’s authoritative DNS were withdrawn, rendering all Facebook domains inaccessible. That’s per @DougMadory , who knows a few things about BGP/DNS.

Domain Name Servers (DNS) are the most important thing on the Internet. It is like a phone book, where you type a name, and the DNS record says, oh, Facebook is here at this IP numerical address.

Without DNS records, Facebook and Instagram are DOA.

[UPDATE]

Disclose.tv
@disclosetv
JUST IN – Facebook employees reportedly can’t enter buildings to evaluate the Internet outage because their door access badges weren’t working (NYT)

Get notified when a new post is published.
Subscribe today!
5 42 votes
Article Rating
180 Comments
Inline Feedbacks
View all comments
October 5, 2021 7:11 am

Revoked certificates 😀 could be hack, unhappy pappy engineer, or those rare but inevitable dopey screw-ups that even the best are capable of.

Reply to  Mark - Helsinki
October 5, 2021 7:42 am

See (because no one reads the previous posts): https://engineering.fb.com/2021/05/20/networking-traffic/peering-automation/

“So we’ve developed a new automated method [using BGP], which allows for faster self-service peering configuration.”

Dated MAY 20, 2021.

BEST GUESS ESTIMATE would be their algorithm ‘went south’ in an “own goal” type failure …

October 5, 2021 9:33 am

Couldn’t happen to a nicer group of people. And a gift so appropriate for Mark Zuckerberg.

Vuk
Reply to  Pat Frank
October 5, 2021 2:15 pm

That is fortuitous that engineers are to blame, what kind of engineers he does employ?
Now, if it was hack the billion of users data would be compromised and some of the authorities throughout the world would impose heavy penalties (and they would well deserve it!) and that would not do, would it?
Their data security has been abysmal, I registered about a decade ago, for reasons of log-in into various sites, but within weeks someone stolen my email for login into their FB account. On dozens occasions I tried to recover my email FB log in, but no avail.

October 5, 2021 3:27 pm

DNS just advertises what the IP address is assigned to a URL. It could be multiple addresses. BGP and other routing protocols advertise what IPs they own and what their neighbors are advertising. Down the routes, this form tables of paths (and it can be more than one path) and the ‘penalty’ time it takes to get to an IP address. They then forward the traffic along those path of routers. Each doing the same. If BGP breaks, there is no path to the site. And since their access system broke, and those are usually in secure segments internally, it sounds like their core routers lost their minds. Their DNS records may also have a time to live as they may change their IP addresses frequently (for load considerations). No DNS advertisement from them because the router is broken, then the DNS records disappear.

BillJ
October 5, 2021 11:36 pm

Well this post didn’t age well. Reminds me of Hanlon’s Razor:

Never attribute to malice that which can be adequately explained by stupidity.”