Facebook Outage Appears to be a DNS Hack

I’m not an IT expert, but I decided to check. This is a very high level hack. Here is what the DNS query for Facebook shows this morning:

According to Internet security researcher Brian Krebs (@briankrebs),

Confirmed: The DNS records that tell systems how to find Facebook.com or Instagram.com got withdrawn this morning from the global routing tables. Can you imagine working at FB right now, when your email no longer works & all your internal FB-based tools fail?

To be more precise (and Geek Factor 5) the BGP routes serving Facebook’s authoritative DNS were withdrawn, rendering all Facebook domains inaccessible. That’s per @DougMadory , who knows a few things about BGP/DNS.

Domain Name Servers (DNS) are the most important thing on the Internet. It is like a phone book, where you type a name, and the DNS record says, oh, Facebook is here at this IP numerical address.

Without DNS records, Facebook and Instagram are DOA.

[UPDATE]

Disclose.tv
@disclosetv
JUST IN – Facebook employees reportedly can’t enter buildings to evaluate the Internet outage because their door access badges weren’t working (NYT)

Get notified when a new post is published.
Subscribe today!
5 42 votes
Article Rating
180 Comments
Inline Feedbacks
View all comments
Steve Rice
October 4, 2021 10:53 am

Sure looks like it.

Vuk
Reply to  Steve Rice
October 4, 2021 12:45 pm

Do you believe in coincidence?
I tend to be cause & consequence minded.
Two days ago (2 October, 2021):
Britain to carry out ‘offensive’ cyber attacks from new £5bn digital warfare centre.
In an interview with The Telegraph, Ben Wallace (UK defence secretary,) says Britain will act in response to assaults by ‘hostile states’ such as Russia”
https://www.telegraph.co.uk/politics/2021/10/02/britain-capable-launching-offensive-cyber-attacks-against-russia/

Todsy (4 October 2021) Some other cyber warrior:
Let me show you Mr.Wallace what I can do.

Reply to  Vuk
October 4, 2021 2:28 pm

Surprisingly, it’s the Guardian that is reporting why the world’s biggest news provider has been shutdown today:
Pandora papers reveal hidden riches of Putin’s inner circle | Vladimir Putin | The Guardian
Who benefits from suppressing news distribution until the news cycle moves on?

To bed B
Reply to  M Courtney
October 4, 2021 2:45 pm

A lot of people benefit from it. Some large companies as well as people wealthy enough to hire someone savvy enough to do it. Putin merely has access to geeks ready to do it quickly. Not sure that it’s worth it, though.

Derg
Reply to  To bed B
October 4, 2021 3:24 pm

Are we sure it’s not a 3 letter agency pulling this off?

Russia sure seems to be an easy excuse…why not China?

I am no defender of Putin but there are lots of bad actors and many of them work for us.

Sara
Reply to  Derg
October 4, 2021 6:34 pm

Nah. It’s the North Koreans. They have an army of hackers who are quite capable of this kind of thing.

Civilization grinding to a halt because of this would make them very happy.

niceguy
Reply to  Sara
October 4, 2021 7:10 pm

Just like when they hacked SONY, for no reason?

Reply to  Sara
October 4, 2021 10:58 pm

The civilised don’t do Farcebook, only the uncivil

Reply to  Derg
October 4, 2021 11:15 pm

Space Force has more than three letters.

maddog
Reply to  Steve Rice
October 4, 2021 1:31 pm

God I hope they were cancelled!

Carbon Bigfoot
Reply to  Steve Rice
October 4, 2021 2:53 pm

No it is not. The Storm is Upon Us:

https://beforeitsnews.com/prophecy/2021/10/october-red-update-2524367.html

Green New Deal Is Dead

LetsGoViking
Reply to  Carbon Bigfoot
October 4, 2021 5:23 pm

Well, that link was a study in incoherence.

Carbon Bigfoot
Reply to  LetsGoViking
October 5, 2021 12:57 am

This sysop has been planned for over twenty years. I don’t expect you to understand it as you are one of the terminally stupid—and you are in the majority.

Curious George
Reply to  Carbon Bigfoot
October 10, 2021 3:20 pm

Sysop definition is – the administrator of a computer message board.

Dave
Reply to  Carbon Bigfoot
October 4, 2021 6:58 pm

Presumably the Demand for the new Facebook currency fell off the cliff as well.

PaulH
October 4, 2021 10:53 am

According to Internet security researcher Brian Krebs (@briankrebs),

Confirmed: The DNS records that tell systems how to find Facebook.com or Instagram.com got withdrawn this morning from the global routing tables. Can you imagine working at FB right now, when your email no longer works & all your internal FB-based tools fail?

To be more precise (and Geek Factor 5) the BGP routes serving Facebook’s authoritative DNS were withdrawn, rendering all Facebook domains inaccessible. That’s per @DougMadory , who knows a few things about BGP/DNS.

Reply to  PaulH
October 4, 2021 10:59 am

I read, Instagram and Whatsapp are down too.

SMC
Reply to  Krishna Gans
October 4, 2021 11:19 am

I know Whatsapp is owned by Facebook. I think Instagram is also.

Reply to  Krishna Gans
October 4, 2021 12:20 pm

Instagram appears somewhet up to me. It has a DNS entry at least

Reply to  Krishna Gans
October 4, 2021 1:54 pm

Yes both Instagram and WattsApp down

Craig from Oz
Reply to  Krishna Gans
October 4, 2021 4:14 pm

Pity – I actually like and use Instagram. See something that makes me smile, photo, caption, back to real world.

Not as intrusive as FB. FB wants to remind me of friends I actually can’t remember and put together little memory packages so I can recall some insignificant date in the past.

Only reason I still have FB is for messenger and Instagram.

SMC
Reply to  PaulH
October 4, 2021 11:31 am

Seems to me either someone screwed up or somebody with serious horse power is unhappy.

Vuk
Reply to  SMC
October 4, 2021 12:14 pm

List is not very long but there are some serious game players in the field: Iran, China, Russia, North Korea and one or two lunatics who would like to earn easy way shedload of bitcoins.

Reply to  SMC
October 4, 2021 12:21 pm

Not serious horsepower, serious smarts

SMC
Reply to  Leo Smith
October 4, 2021 2:43 pm

Them too.

Reply to  SMC
October 4, 2021 3:27 pm

I guess Facebook got Fact Checked – there were so much bias they just cancelled the whole platform.

PaulH
Reply to  PaulH
October 4, 2021 12:17 pm

A short write-up from @briankrebs on the ongoing outages at Facebook, Instagram and WhatsApp.

Facebook and its sister properties Instagram and WhatsApp are suffering from ongoing, global outages. We don’t yet know why this happened, but the how is clear: Earlier this morning, something inside Facebook caused the company to revoke key digital records that tell computers and other Internet-enabled devices how to find these destinations online.

Fred Middleton
Reply to  PaulH
October 4, 2021 1:12 pm

Test. Trinity. Megaton. Triggering device. High Tech. = We found the problem folks.

Mr.
October 4, 2021 11:00 am

Let’s hope it’s permanent.

John Bell
October 4, 2021 11:06 am

What is DNS?

Reply to  John Bell
October 4, 2021 11:12 am

Google is your friend… and foe.

Reply to  Joel O'Bryan
October 4, 2021 1:34 pm

Or duckduckgo

Greytide
Reply to  John Bell
October 4, 2021 11:28 am

Domain Name Server. Maps a website name like http://WWW.Wattsupththat.com to an actual IP address 192.164.xxx.xxx etc

Reply to  Greytide
October 4, 2021 3:24 pm

IP addresses are just abstractions for MAC addresses.

Patrick MJD
Reply to  Shoki Kaneda
October 4, 2021 4:34 pm

Not at all.

mcswell
Reply to  Shoki Kaneda
October 4, 2021 6:58 pm

MAC addresses are visible to your local router, which (in the simple case) assigns each one to a local IP address. From outside your Local Area Network (LAN), your router probably has a single external (non-local) IP address, and your Internet Service Provider (ISP) knows that IP address. Your ISP probably does not know the MAC addresses on your LAN.

For others reading this, MAC stands for Media Access Control (not Macintosh, in this context). Each network card has its own MAC address (or motherboards that include network controllers).

Things can get more complicated for other than home networks.

That’s probably more than you want to know, and probably more than I know, to be truthful.

John Dilks
Reply to  Shoki Kaneda
October 4, 2021 7:26 pm

MAC addresses are not routeable, IP addresses are.

Patricia Langdon
Reply to  John Bell
October 4, 2021 2:43 pm

Domain Name Server

Reply to  John Bell
October 4, 2021 10:00 pm

Domain Name Server

Maryanne
October 4, 2021 11:06 am

Now I can get some real work done.

Robert of Texas
October 4, 2021 11:14 am

Good, may they remain closed forever.

Sparko
October 4, 2021 11:16 am

Oops, I’ll plug it back in.
Sorry folks

SMC
Reply to  Sparko
October 4, 2021 11:23 am
Notanacademic
Reply to  Sparko
October 4, 2021 12:00 pm

Please don’t, I think the thought of the cancellers being cancelled is wonderful. Hope it lasts for a long time.

Jeff Alberts
Reply to  Sparko
October 4, 2021 2:36 pm

When I was working network operations in the 90s, we had a customer who’s network connection (dedicated 9600bps line) would drop out at about the same time every evening when no one was at the company. But some of their automated processes would fail at the same time.

After weeks of troubleshooting, checking the switches out our (Telenet/Sprintnet) network office, having the local telco checking their equipment, performing loopback tests, etc, nothing ever showed a problem.

So one evening, someone at the company stayed and decided to watch the modem to see what was happening. Then the cleaning lady comes in, unplugs the modem, plugs in her vacuum cleaner, and goes merrily about her business.

Richard Page
Reply to  Jeff Alberts
October 4, 2021 2:54 pm

Oh not that story again. First time I heard it was about air traffic control at Heathrow then it was another company – all because of the little 3 pin plug and the cleaning lady!

Alexy Scherbakoff
Reply to  Sparko
October 4, 2021 5:14 pm

Could have been the cleaning lady who was unplugging life support to use her vacuum cleaner.

Reply to  Alexy Scherbakoff
October 7, 2021 11:39 am

Same lady…born in Russia.

October 4, 2021 11:23 am

Escapees almost always have outside help.

48BC4D36-29E7-4EC6-8FC9-EDB446025E46.png
October 4, 2021 11:28 am

Guess they’ll need to use Parler. LOL

Opus
October 4, 2021 11:29 am

Now, FB knows how it feels to be cancelled.

Bob Vislicky
October 4, 2021 11:35 am

I blame climate change, lol.

Interestingly, last night about half way through the 60 minutes report on the Facebook whistleblower our local CBS channel cut away to a test pattern!!

Richard Page
Reply to  Bob Vislicky
October 4, 2021 4:07 pm

If FB was based in the UK then it would be due to Brexit, Covid, supply chain issues and driver shortages, in that order!

Donna K. Becker
Reply to  Bob Vislicky
October 5, 2021 9:29 am

Somehow, this event brings to mind Galt’s speech.

Robert of Texas
October 4, 2021 11:45 am

Good news. I hope they stay down. The mental health of millions will be improved.

Jeff Labute
October 4, 2021 11:58 am

If these IP addresses are accurate, you still cannot reach Facebook by IP nor can you ping the following IP addresses.

  • 69.63.176.13
  • 69.63.181.15
  • 69.63.184.142
  • 69.63.187.17
  • 69.63.187.18
  • 69.63.187.19
  • 69.63.181.11
  • 69.63.181.12

Problem goes beyond DNS alone to include route tables(?).
Maybe they didn’t pay their yearly DNS registration fee? 🙂

Hallelujah

Reply to  Jeff Labute
October 4, 2021 12:25 pm

Lack of ping response unfortunately means nothing
But you are right – they seem not to be responding to valid http/https requests.

Hmm. There may also be a BGP hack. There seems to be no route to those addresses either.

Neo
Reply to  Jeff Labute
October 4, 2021 1:35 pm

Apparently, Facebook was able to cycle down the servers but can’t physically get back in to restart the servers because the IoT badge readers go to one of their own servers.

H.R.
Reply to  Neo
October 4, 2021 6:25 pm

🤣 🤣 🤣

Unbelievable. Just how smart are these people?

I’m not sure why someone hasn’t just taken a sledgehammer to one or two doors, or a front window or two.

You can call maintenance to replace the doors with a keyed lock type for the short term.

Maybe someone has thought of that by now. A-a-a-n-d… maybe not.

It would be the funniest thing EVAH! if the employees are still standing outside the door.

October 4, 2021 12:00 pm

Do we HAVE to fix it?

Reply to  TonyG
October 4, 2021 12:29 pm

I’ve spent 5 minutes looking. Whoever has done this has done something major and pretty catastrophic.

freaking with DNS is possibly something a hacker in a basement can do., If BGP is involved they need to be a serious ISP, which implicates someone at a much deeper level.

Well I’ll hand it over to the pros now.

Whoever has done it is going to be blessed as much as cursed

Richard Page
Reply to  Leo Smith
October 4, 2021 4:10 pm

Apparently, FB are saying it was due to a new update that wasn’t debugged or tested properly before installation. I think whoever has done it is going to get fired tomorrow morning!

Patrick MJD
Reply to  Richard Page
October 4, 2021 4:34 pm

Nah. It affected Whatsapp too.

October 4, 2021 12:11 pm

Here is another guess at what is going on…..BREAKING: Facebook, Instagram Down After ’60 Minutes’ Whistleblower Story…good explanation, read the entire article.
https://www.toddstarnes.com/media/breaking-facebook-instagram-down-60-minutes-whistleblower-story/

Users on Monday reported Facebook, Instagram, and Whatsapp being down one day after a whistleblower accused the company of putting profits over safety.

The blackout happened after Frances Haugen, Facebook’s product manager on the civic misinformation team, revealed her identity Sunday on the CBS television program “60 Minutes,” as the whistleblower who provided the documents to the Wall Street Journal investigation and a Senate hearing on Instagram’s harm to teen girls, Reuters reports.

Haugen gave “tens of thousands” of pages to the WSJ, claiming Facebook’s algorithm purposefully shows users content to make them angry.

“Facebook has realized that if they change the algorithm to be safer, people will spend less time on the site, they’ll click on less ads, they’ll make less money,” Haugen told “60 Minutes.”

Reply to  TEWS_Pilot
October 4, 2021 12:41 pm

What a coincidence. Not….

It is being reported that employees cannot gain entrance into some buildings via the access control system also.

Reply to  Ossqss
October 4, 2021 1:28 pm

While they’re trying to get in the front door, Zuckerberg is sneaking out the back door with a large bag marked “Swag”. He’s headed to where there is no extradition treaty but plenty of rum drinks.

Reply to  TEWS_Pilot
October 4, 2021 3:47 pm

What!!??? A private for-profit enterprise puts profits before “the public good”? Hold the presses!

The so-called “whistleblower” complains that Facebook is supposedly prioritizing “profit” over removing “misinformation” and “hate speech”. More proof that there is no end to the meddling sociopathic Leftists like Haugen want to do in your life.

Here’s a thought, Big Tech: how about not censoring anything unless it violates the law? That seems like a morally and legally sound policy to me.

October 4, 2021 12:15 pm

The official DNS delegation records for “facebook.com” are:

;; AUTHORITY SECTION:
facebook.com. 172800 IN NS a.ns.facebook.com.
facebook.com. 172800 IN NS b.ns.facebook.com.
facebook.com. 172800 IN NS c.ns.facebook.com.
facebook.com. 172800 IN NS d.ns.facebook.com.

;; ADDITIONAL SECTION:
a.ns.facebook.com. 172800 IN A 129.134.30.12
a.ns.facebook.com. 172800 IN AAAA 2a03:2880:f0fc:c:face:b00c:0:35
b.ns.facebook.com. 172800 IN A 129.134.31.12
b.ns.facebook.com. 172800 IN AAAA 2a03:2880:f0fd:c:face:b00c:0:35
c.ns.facebook.com. 172800 IN A 185.89.218.12
c.ns.facebook.com. 172800 IN AAAA 2a03:2880:f1fc:c:face:b00c:0:35
d.ns.facebook.com. 172800 IN A 185.89.219.12
d.ns.facebook.com. 172800 IN AAAA 2a03:2880:f1fd:c:face:b00c:0:35

So there should be four reachable DNS servers providing authoritative answers for DNS queries for “facebook.com” records. Two are in US/ARIN-allocated address blocks and two are in RIPE/Europe blocks.

The 129.134.0.0/16 network is a direct ARIN allocation to Facebook.

185.89.216.0/22 network is a direct RIPE allocation to Facebook.

I can’t reach any of those addresses. I don’t use IPv6 so I’m just checking the IPv4 routes. My internet provider is AT&T and the routes appear to be missing from their network.

This is not a “DNS Hack” per-se; it is a routing failure that happens to hit all the networks where the authoritative DNS servers for Facebook reside.

No doubt a lot of other things on the same networks are also unreachable; but since DNS resolution needs to come first, people tend to label this as a “DNS problem”.

Why the routes are gone is an interesting question. Could be a hack or it could be an error. Some months back one of the big content delivery providers (I think it was CloudFlare, but I could be mis-remembering) pretty much went down because they introduced a routing misconfiguration that cause virtually all traffic to be routed through their Atlanta hub, overloading all the circuits.

In this case the routes are simply missing. Either they are not being advertised or the the route advertisements are not being accepted.

rbabcock
Reply to  Alan Watt, Climate Denialist Level 7
October 4, 2021 12:19 pm

Maybe it’s the hidden code on Chinese router chips. Doing a test run on bringing the financial system to a standstill.

Reply to  rbabcock
October 4, 2021 1:31 pm

The chips in the vaxxes are activated 😀

Richard Page
Reply to  Krishna Gans
October 4, 2021 2:56 pm

You’re a very strange person!

Reply to  Richard Page
October 4, 2021 4:04 pm

Why ? 😀 Irony detector broken ? 😀 😀
No Idea what in other countries people says about the “Anti vaxxers” but the nano-chips in the vaxxes are often taken as cause they don’t want to be “vaxxed”. if you read critics about here in Germany

Richard Page
Reply to  Krishna Gans
October 4, 2021 4:16 pm

Krishna Gans- I upvoted your comment and couldn’t help but put up a tongue-in-cheek comment, forgetting (of course) that some humour simply doesn’t travel well. I do apologise if 2 people and/or yourself failed to appreciate the humour in that post but it’s a bit too late to do anything about it now. I keep forgetting that British humour so often has to be explained to others.

Reply to  Alan Watt, Climate Denialist Level 7
October 4, 2021 1:51 pm

strange that two entirely different networks are affected

Reply to  Leo Smith
October 4, 2021 2:31 pm

Enemy action.
Although the enemy of Facebook is my…

Chuck no longer in Houston
Reply to  Alan Watt, Climate Denialist Level 7
October 6, 2021 12:59 pm

I used to administer the three authoritative DNS servers for a well known oil field services company. In addition to the sysadmin, I was also the DNS admin for the company. These were authoritative for about 12-15 separate domains. These three servers (Sun/Oracle Sparc running Solaris 11 at the time) were located in different parts of the country and worked to load balance and back each other up. To me this says someone screwed up multiple DNS servers at once. You have to really go out of your way to do this. Automated admin tools are very good at this sort of thing.

Alex
October 4, 2021 12:17 pm

facebook.com is on sale!

October 4, 2021 12:20 pm

DNS is not BGP and routing.
DNS tells you where they are.
BGP is how to get there

a quick check indicates that facebook.com no longer exists on global DNS
Nor does whatsapp.com

linkedin.com does.

A neat hack to the root nameservers of the .com domain.
Well done hackers. Microsoft is hated by computer professionals. Sad I cant talk to my family round the world, but worth it to see Microsoft sweat.

Bill Rocks
Reply to  Leo Smith
October 4, 2021 12:57 pm

Why do you refer to Microsoft? Facebook et al is the broken item. Do you mean that Microsoft could be victimized by a similar attack?

I agree that Facebook is a menace and MS has sold useful but often defective products for many years.

Reply to  Bill Rocks
October 4, 2021 1:35 pm

sorry was thinking of skype, but ms owns a large chunk of facebook as well

Reply to  Leo Smith
October 4, 2021 2:57 pm

No. The root DNS servers were not compromised; they continued to hold the proper delegation records. The routes to Facebook’s DNS servers disappeared.

Now that the routes are back it appears that the circuits are provided by AT&T and Facebooks is operating their own DNS servers on their own networks.

sycomputing
Reply to  Alan Watt, Climate Denialist Level 7
October 4, 2021 6:28 pm

“When considering the cause of IT failures, never ascribe to conspiracy that which can be reasonably attributed equally to: ‘Oops . . . I didn’t think that would happen.'”

A paraphrase from an astute twaddling I saw today on “the Twitter.”

October 4, 2021 12:31 pm

My Yahoo links to WUWT don’t work either. –AGF

Russell Cook
October 4, 2021 12:34 pm

Would be funny if all the content with a Facebook address was replaced with ClimateGate emails, version 3, or everything from Hunter Biden’s laptop.

Mike
October 4, 2021 12:38 pm

BGP is Border Gateway Protocol, think of an internet cop directing traffic through the best routes that can be used to reach a specific address.

In 2008 through a mistake by Pakistan telecom. just about all the global you tube traffic ended up in a black hole in Pakistan, all those wasted electrons!

The vision of that odious parasite Zuckerberg spending the rest of his life trying to escape from a black hole is hilarious to me.

Jeff Labute
October 4, 2021 12:40 pm

Probably partly caused Facebook stock to take a dip too. -5.4%.
(In addition to claims of harming mental health, human trafficking, arms deals, etc)
Outages make people with addictions, nervous.

Dave
Reply to  Jeff Labute
October 4, 2021 7:13 pm

Want to buy some Facebook crypto coins? What could go wrong?

October 4, 2021 12:42 pm

I don’t think this Internet attack is a coincidence given that Chinese PLAAF and PLAN aircraft incursions into Taiwan airspace have ramped up dramatically over the past 3 days. A lot of overseas country’s populations like the Taiwanese depend on Facebook and and its various message and photo products to stay informed.

The long expected Chinese invasion of at least taking Taiwan’s Dongsha Island could happen within days or even hours now.

Reply to  Joel O'Bryan
October 4, 2021 1:33 pm

And meanwhile, General Milley is distracted because he can’t post to his Facebook fan page “Milley is Marvelous”. Insidious, those Chinese. 🙂

Jerome P Koch
October 4, 2021 12:53 pm

One analyst says the hack occurred via BGP routing protocols. BGP routes via DNS, and for whatever reason, there was a bad BGP update, which emptied all references to FB and its apps. This was confirmed by Cloudflare. A reddit user also confirmed that a BGP update occurred seconds before FB went down.

https://arstechnica.com/information-technology/2021/10/facebook-instagram-whatsapp-and-oculus-are-down-heres-what-we-know/

Reply to  Jerome P Koch
October 4, 2021 1:37 pm

BGP does not route via DNS.
DNS may route via BGP though

1 2 3 4