Facebook Outage Appears to be a DNS Hack

I’m not an IT expert, but I decided to check. This is a very high level hack. Here is what the DNS query for Facebook shows this morning:

According to Internet security researcher Brian Krebs (@briankrebs),

Confirmed: The DNS records that tell systems how to find Facebook.com or Instagram.com got withdrawn this morning from the global routing tables. Can you imagine working at FB right now, when your email no longer works & all your internal FB-based tools fail?

To be more precise (and Geek Factor 5) the BGP routes serving Facebook’s authoritative DNS were withdrawn, rendering all Facebook domains inaccessible. That’s per @DougMadory , who knows a few things about BGP/DNS.

Domain Name Servers (DNS) are the most important thing on the Internet. It is like a phone book, where you type a name, and the DNS record says, oh, Facebook is here at this IP numerical address.

Without DNS records, Facebook and Instagram are DOA.

[UPDATE]

Disclose.tv
@disclosetv
JUST IN – Facebook employees reportedly can’t enter buildings to evaluate the Internet outage because their door access badges weren’t working (NYT)

5 42 votes
Article Rating
180 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Steve Rice
October 4, 2021 10:53 am

Sure looks like it.

Vuk
Reply to  Steve Rice
October 4, 2021 12:45 pm

Do you believe in coincidence?
I tend to be cause & consequence minded.
Two days ago (2 October, 2021):
Britain to carry out ‘offensive’ cyber attacks from new £5bn digital warfare centre.
In an interview with The Telegraph, Ben Wallace (UK defence secretary,) says Britain will act in response to assaults by ‘hostile states’ such as Russia”
https://www.telegraph.co.uk/politics/2021/10/02/britain-capable-launching-offensive-cyber-attacks-against-russia/

Todsy (4 October 2021) Some other cyber warrior:
Let me show you Mr.Wallace what I can do.

M Courtney
Reply to  Vuk
October 4, 2021 2:28 pm

Surprisingly, it’s the Guardian that is reporting why the world’s biggest news provider has been shutdown today:
Pandora papers reveal hidden riches of Putin’s inner circle | Vladimir Putin | The Guardian
Who benefits from suppressing news distribution until the news cycle moves on?

To bed B
Reply to  M Courtney
October 4, 2021 2:45 pm

A lot of people benefit from it. Some large companies as well as people wealthy enough to hire someone savvy enough to do it. Putin merely has access to geeks ready to do it quickly. Not sure that it’s worth it, though.

Derg
Reply to  To bed B
October 4, 2021 3:24 pm

Are we sure it’s not a 3 letter agency pulling this off?

Russia sure seems to be an easy excuse…why not China?

I am no defender of Putin but there are lots of bad actors and many of them work for us.

Sara
Reply to  Derg
October 4, 2021 6:34 pm

Nah. It’s the North Koreans. They have an army of hackers who are quite capable of this kind of thing.

Civilization grinding to a halt because of this would make them very happy.

niceguy
Reply to  Sara
October 4, 2021 7:10 pm

Just like when they hacked SONY, for no reason?

Redge
Reply to  Sara
October 4, 2021 10:58 pm

The civilised don’t do Farcebook, only the uncivil

Big Al
Reply to  Derg
October 4, 2021 11:15 pm

Space Force has more than three letters.

maddog
Reply to  Steve Rice
October 4, 2021 1:31 pm

God I hope they were cancelled!

Carbon Bigfoot
Reply to  Steve Rice
October 4, 2021 2:53 pm

No it is not. The Storm is Upon Us:

https://beforeitsnews.com/prophecy/2021/10/october-red-update-2524367.html

Green New Deal Is Dead

LetsGoViking
Reply to  Carbon Bigfoot
October 4, 2021 5:23 pm

Well, that link was a study in incoherence.

Carbon Bigfoot
Reply to  LetsGoViking
October 5, 2021 12:57 am

This sysop has been planned for over twenty years. I don’t expect you to understand it as you are one of the terminally stupid—and you are in the majority.

Reply to  Carbon Bigfoot
October 10, 2021 3:20 pm

Sysop definition is – the administrator of a computer message board.

Dave
Reply to  Carbon Bigfoot
October 4, 2021 6:58 pm

Presumably the Demand for the new Facebook currency fell off the cliff as well.

PaulH
October 4, 2021 10:53 am

According to Internet security researcher Brian Krebs (@briankrebs),

Confirmed: The DNS records that tell systems how to find Facebook.com or Instagram.com got withdrawn this morning from the global routing tables. Can you imagine working at FB right now, when your email no longer works & all your internal FB-based tools fail?

To be more precise (and Geek Factor 5) the BGP routes serving Facebook’s authoritative DNS were withdrawn, rendering all Facebook domains inaccessible. That’s per @DougMadory , who knows a few things about BGP/DNS.

Reply to  PaulH
October 4, 2021 10:59 am

I read, Instagram and Whatsapp are down too.

SMC
Reply to  Krishna Gans
October 4, 2021 11:19 am

I know Whatsapp is owned by Facebook. I think Instagram is also.

Reply to  Krishna Gans
October 4, 2021 12:20 pm

Instagram appears somewhet up to me. It has a DNS entry at least

Reply to  Krishna Gans
October 4, 2021 1:54 pm

Yes both Instagram and WattsApp down

Craig from Oz
Reply to  Krishna Gans
October 4, 2021 4:14 pm

Pity – I actually like and use Instagram. See something that makes me smile, photo, caption, back to real world.

Not as intrusive as FB. FB wants to remind me of friends I actually can’t remember and put together little memory packages so I can recall some insignificant date in the past.

Only reason I still have FB is for messenger and Instagram.

SMC
Reply to  PaulH
October 4, 2021 11:31 am

Seems to me either someone screwed up or somebody with serious horse power is unhappy.

Vuk
Reply to  SMC
October 4, 2021 12:14 pm

List is not very long but there are some serious game players in the field: Iran, China, Russia, North Korea and one or two lunatics who would like to earn easy way shedload of bitcoins.

Reply to  SMC
October 4, 2021 12:21 pm

Not serious horsepower, serious smarts

SMC
Reply to  Leo Smith
October 4, 2021 2:43 pm

Them too.

Ozonebust
Reply to  SMC
October 4, 2021 3:27 pm

I guess Facebook got Fact Checked – there were so much bias they just cancelled the whole platform.

PaulH
Reply to  PaulH
October 4, 2021 12:17 pm

A short write-up from @briankrebs on the ongoing outages at Facebook, Instagram and WhatsApp.

Facebook and its sister properties Instagram and WhatsApp are suffering from ongoing, global outages. We don’t yet know why this happened, but the how is clear: Earlier this morning, something inside Facebook caused the company to revoke key digital records that tell computers and other Internet-enabled devices how to find these destinations online.

Fred Middleton
Reply to  PaulH
October 4, 2021 1:12 pm

Test. Trinity. Megaton. Triggering device. High Tech. = We found the problem folks.

Mr.
October 4, 2021 11:00 am

Let’s hope it’s permanent.

John Bell
October 4, 2021 11:06 am

What is DNS?

Reply to  John Bell
October 4, 2021 11:12 am

Google is your friend… and foe.

Barnes Moore
Reply to  Joel O'Bryan
October 4, 2021 1:34 pm

Or duckduckgo

Greytide
Reply to  John Bell
October 4, 2021 11:28 am

Domain Name Server. Maps a website name like http://WWW.Wattsupththat.com to an actual IP address 192.164.xxx.xxx etc

Shoki Kaneda
Reply to  Greytide
October 4, 2021 3:24 pm

IP addresses are just abstractions for MAC addresses.

Patrick MJD
Reply to  Shoki Kaneda
October 4, 2021 4:34 pm

Not at all.

mcswell
Reply to  Shoki Kaneda
October 4, 2021 6:58 pm

MAC addresses are visible to your local router, which (in the simple case) assigns each one to a local IP address. From outside your Local Area Network (LAN), your router probably has a single external (non-local) IP address, and your Internet Service Provider (ISP) knows that IP address. Your ISP probably does not know the MAC addresses on your LAN.

For others reading this, MAC stands for Media Access Control (not Macintosh, in this context). Each network card has its own MAC address (or motherboards that include network controllers).

Things can get more complicated for other than home networks.

That’s probably more than you want to know, and probably more than I know, to be truthful.

John Dilks
Reply to  Shoki Kaneda
October 4, 2021 7:26 pm

MAC addresses are not routeable, IP addresses are.

Patricia Langdon
Reply to  John Bell
October 4, 2021 2:43 pm

Domain Name Server

mario lento
Reply to  John Bell
October 4, 2021 10:00 pm

Domain Name Server

Last edited 1 month ago by mario lento
Maryanne
October 4, 2021 11:06 am

Now I can get some real work done.

Robert of Texas
October 4, 2021 11:14 am

Good, may they remain closed forever.

Sparko
October 4, 2021 11:16 am

Oops, I’ll plug it back in.
Sorry folks

SMC
Reply to  Sparko
October 4, 2021 11:23 am
Notanacademic
Reply to  Sparko
October 4, 2021 12:00 pm

Please don’t, I think the thought of the cancellers being cancelled is wonderful. Hope it lasts for a long time.

Jeff Alberts
Reply to  Sparko
October 4, 2021 2:36 pm

When I was working network operations in the 90s, we had a customer who’s network connection (dedicated 9600bps line) would drop out at about the same time every evening when no one was at the company. But some of their automated processes would fail at the same time.

After weeks of troubleshooting, checking the switches out our (Telenet/Sprintnet) network office, having the local telco checking their equipment, performing loopback tests, etc, nothing ever showed a problem.

So one evening, someone at the company stayed and decided to watch the modem to see what was happening. Then the cleaning lady comes in, unplugs the modem, plugs in her vacuum cleaner, and goes merrily about her business.

Richard Page
Reply to  Jeff Alberts
October 4, 2021 2:54 pm

Oh not that story again. First time I heard it was about air traffic control at Heathrow then it was another company – all because of the little 3 pin plug and the cleaning lady!

Alexy Scherbakoff
Reply to  Sparko
October 4, 2021 5:14 pm

Could have been the cleaning lady who was unplugging life support to use her vacuum cleaner.

Mortimer Zilch
Reply to  Alexy Scherbakoff
October 7, 2021 11:39 am

Same lady…born in Russia.

gringojay
October 4, 2021 11:23 am

Escapees almost always have outside help.

48BC4D36-29E7-4EC6-8FC9-EDB446025E46.png
Buckeyebob
October 4, 2021 11:28 am

Guess they’ll need to use Parler. LOL

Opus
October 4, 2021 11:29 am

Now, FB knows how it feels to be cancelled.

Bob Vislicky
October 4, 2021 11:35 am

I blame climate change, lol.

Interestingly, last night about half way through the 60 minutes report on the Facebook whistleblower our local CBS channel cut away to a test pattern!!

Richard Page
Reply to  Bob Vislicky
October 4, 2021 4:07 pm

If FB was based in the UK then it would be due to Brexit, Covid, supply chain issues and driver shortages, in that order!

Donna K. Becker
Reply to  Bob Vislicky
October 5, 2021 9:29 am

Somehow, this event brings to mind Galt’s speech.

Robert of Texas
October 4, 2021 11:45 am

Good news. I hope they stay down. The mental health of millions will be improved.

Jeff Labute
October 4, 2021 11:58 am

If these IP addresses are accurate, you still cannot reach Facebook by IP nor can you ping the following IP addresses.

  • 69.63.176.13
  • 69.63.181.15
  • 69.63.184.142
  • 69.63.187.17
  • 69.63.187.18
  • 69.63.187.19
  • 69.63.181.11
  • 69.63.181.12

Problem goes beyond DNS alone to include route tables(?).
Maybe they didn’t pay their yearly DNS registration fee? 🙂

Hallelujah

Last edited 1 month ago by Jeff Labute
Reply to  Jeff Labute
October 4, 2021 12:25 pm

Lack of ping response unfortunately means nothing
But you are right – they seem not to be responding to valid http/https requests.

Hmm. There may also be a BGP hack. There seems to be no route to those addresses either.

Neo
Reply to  Jeff Labute
October 4, 2021 1:35 pm

Apparently, Facebook was able to cycle down the servers but can’t physically get back in to restart the servers because the IoT badge readers go to one of their own servers.

Last edited 1 month ago by Neo
H.R.
Reply to  Neo
October 4, 2021 6:25 pm

🤣 🤣 🤣

Unbelievable. Just how smart are these people?

I’m not sure why someone hasn’t just taken a sledgehammer to one or two doors, or a front window or two.

You can call maintenance to replace the doors with a keyed lock type for the short term.

Maybe someone has thought of that by now. A-a-a-n-d… maybe not.

It would be the funniest thing EVAH! if the employees are still standing outside the door.

TonyG
October 4, 2021 12:00 pm

Do we HAVE to fix it?

Reply to  TonyG
October 4, 2021 12:29 pm

I’ve spent 5 minutes looking. Whoever has done this has done something major and pretty catastrophic.

freaking with DNS is possibly something a hacker in a basement can do., If BGP is involved they need to be a serious ISP, which implicates someone at a much deeper level.

Well I’ll hand it over to the pros now.

Whoever has done it is going to be blessed as much as cursed

Richard Page
Reply to  Leo Smith
October 4, 2021 4:10 pm

Apparently, FB are saying it was due to a new update that wasn’t debugged or tested properly before installation. I think whoever has done it is going to get fired tomorrow morning!

Patrick MJD
Reply to  Richard Page
October 4, 2021 4:34 pm

Nah. It affected Whatsapp too.

TEWS_Pilot
October 4, 2021 12:11 pm

Here is another guess at what is going on…..BREAKING: Facebook, Instagram Down After ’60 Minutes’ Whistleblower Story…good explanation, read the entire article.
https://www.toddstarnes.com/media/breaking-facebook-instagram-down-60-minutes-whistleblower-story/

Users on Monday reported Facebook, Instagram, and Whatsapp being down one day after a whistleblower accused the company of putting profits over safety.

The blackout happened after Frances Haugen, Facebook’s product manager on the civic misinformation team, revealed her identity Sunday on the CBS television program “60 Minutes,” as the whistleblower who provided the documents to the Wall Street Journal investigation and a Senate hearing on Instagram’s harm to teen girls, Reuters reports.

Haugen gave “tens of thousands” of pages to the WSJ, claiming Facebook’s algorithm purposefully shows users content to make them angry.

“Facebook has realized that if they change the algorithm to be safer, people will spend less time on the site, they’ll click on less ads, they’ll make less money,” Haugen told “60 Minutes.”

Last edited 1 month ago by TEWS_Pilot
Ossqss
Reply to  TEWS_Pilot
October 4, 2021 12:41 pm

What a coincidence. Not….

It is being reported that employees cannot gain entrance into some buildings via the access control system also.

Mumbles McGuirck
Reply to  Ossqss
October 4, 2021 1:28 pm

While they’re trying to get in the front door, Zuckerberg is sneaking out the back door with a large bag marked “Swag”. He’s headed to where there is no extradition treaty but plenty of rum drinks.

stinkerp
Reply to  TEWS_Pilot
October 4, 2021 3:47 pm

What!!??? A private for-profit enterprise puts profits before “the public good”? Hold the presses!

The so-called “whistleblower” complains that Facebook is supposedly prioritizing “profit” over removing “misinformation” and “hate speech”. More proof that there is no end to the meddling sociopathic Leftists like Haugen want to do in your life.

Here’s a thought, Big Tech: how about not censoring anything unless it violates the law? That seems like a morally and legally sound policy to me.

Last edited 1 month ago by stinkerp
Alan Watt, Climate Denialist Level 7
October 4, 2021 12:15 pm

The official DNS delegation records for “facebook.com” are:

;; AUTHORITY SECTION:
facebook.com. 172800 IN NS a.ns.facebook.com.
facebook.com. 172800 IN NS b.ns.facebook.com.
facebook.com. 172800 IN NS c.ns.facebook.com.
facebook.com. 172800 IN NS d.ns.facebook.com.

;; ADDITIONAL SECTION:
a.ns.facebook.com. 172800 IN A 129.134.30.12
a.ns.facebook.com. 172800 IN AAAA 2a03:2880:f0fc:c:face:b00c:0:35
b.ns.facebook.com. 172800 IN A 129.134.31.12
b.ns.facebook.com. 172800 IN AAAA 2a03:2880:f0fd:c:face:b00c:0:35
c.ns.facebook.com. 172800 IN A 185.89.218.12
c.ns.facebook.com. 172800 IN AAAA 2a03:2880:f1fc:c:face:b00c:0:35
d.ns.facebook.com. 172800 IN A 185.89.219.12
d.ns.facebook.com. 172800 IN AAAA 2a03:2880:f1fd:c:face:b00c:0:35

So there should be four reachable DNS servers providing authoritative answers for DNS queries for “facebook.com” records. Two are in US/ARIN-allocated address blocks and two are in RIPE/Europe blocks.

The 129.134.0.0/16 network is a direct ARIN allocation to Facebook.

185.89.216.0/22 network is a direct RIPE allocation to Facebook.

I can’t reach any of those addresses. I don’t use IPv6 so I’m just checking the IPv4 routes. My internet provider is AT&T and the routes appear to be missing from their network.

This is not a “DNS Hack” per-se; it is a routing failure that happens to hit all the networks where the authoritative DNS servers for Facebook reside.

No doubt a lot of other things on the same networks are also unreachable; but since DNS resolution needs to come first, people tend to label this as a “DNS problem”.

Why the routes are gone is an interesting question. Could be a hack or it could be an error. Some months back one of the big content delivery providers (I think it was CloudFlare, but I could be mis-remembering) pretty much went down because they introduced a routing misconfiguration that cause virtually all traffic to be routed through their Atlanta hub, overloading all the circuits.

In this case the routes are simply missing. Either they are not being advertised or the the route advertisements are not being accepted.

rbabcock
Reply to  Alan Watt, Climate Denialist Level 7
October 4, 2021 12:19 pm

Maybe it’s the hidden code on Chinese router chips. Doing a test run on bringing the financial system to a standstill.

Reply to  rbabcock
October 4, 2021 1:31 pm

The chips in the vaxxes are activated 😀

Richard Page
Reply to  Krishna Gans
October 4, 2021 2:56 pm

You’re a very strange person!

Reply to  Richard Page
October 4, 2021 4:04 pm

Why ? 😀 Irony detector broken ? 😀 😀
No Idea what in other countries people says about the “Anti vaxxers” but the nano-chips in the vaxxes are often taken as cause they don’t want to be “vaxxed”. if you read critics about here in Germany

Richard Page
Reply to  Krishna Gans
October 4, 2021 4:16 pm

Krishna Gans- I upvoted your comment and couldn’t help but put up a tongue-in-cheek comment, forgetting (of course) that some humour simply doesn’t travel well. I do apologise if 2 people and/or yourself failed to appreciate the humour in that post but it’s a bit too late to do anything about it now. I keep forgetting that British humour so often has to be explained to others.

Last edited 1 month ago by Richard Page
Reply to  Alan Watt, Climate Denialist Level 7
October 4, 2021 1:51 pm

strange that two entirely different networks are affected

M Courtney
Reply to  Leo Smith
October 4, 2021 2:31 pm

Enemy action.
Although the enemy of Facebook is my…

Chuck no longer in Houston
Reply to  Alan Watt, Climate Denialist Level 7
October 6, 2021 12:59 pm

I used to administer the three authoritative DNS servers for a well known oil field services company. In addition to the sysadmin, I was also the DNS admin for the company. These were authoritative for about 12-15 separate domains. These three servers (Sun/Oracle Sparc running Solaris 11 at the time) were located in different parts of the country and worked to load balance and back each other up. To me this says someone screwed up multiple DNS servers at once. You have to really go out of your way to do this. Automated admin tools are very good at this sort of thing.

Alex
October 4, 2021 12:17 pm

facebook.com is on sale!

October 4, 2021 12:20 pm

DNS is not BGP and routing.
DNS tells you where they are.
BGP is how to get there

a quick check indicates that facebook.com no longer exists on global DNS
Nor does whatsapp.com

linkedin.com does.

A neat hack to the root nameservers of the .com domain.
Well done hackers. Microsoft is hated by computer professionals. Sad I cant talk to my family round the world, but worth it to see Microsoft sweat.

Bill Rocks
Reply to  Leo Smith
October 4, 2021 12:57 pm

Why do you refer to Microsoft? Facebook et al is the broken item. Do you mean that Microsoft could be victimized by a similar attack?

I agree that Facebook is a menace and MS has sold useful but often defective products for many years.

Reply to  Bill Rocks
October 4, 2021 1:35 pm

sorry was thinking of skype, but ms owns a large chunk of facebook as well

Alan Watt, Climate Denialist Level 7
Reply to  Leo Smith
October 4, 2021 2:57 pm

No. The root DNS servers were not compromised; they continued to hold the proper delegation records. The routes to Facebook’s DNS servers disappeared.

Now that the routes are back it appears that the circuits are provided by AT&T and Facebooks is operating their own DNS servers on their own networks.

sycomputing
Reply to  Alan Watt, Climate Denialist Level 7
October 4, 2021 6:28 pm

“When considering the cause of IT failures, never ascribe to conspiracy that which can be reasonably attributed equally to: ‘Oops . . . I didn’t think that would happen.'”

A paraphrase from an astute twaddling I saw today on “the Twitter.”

A G Foster
October 4, 2021 12:31 pm

My Yahoo links to WUWT don’t work either. –AGF

Russell Cook
October 4, 2021 12:34 pm

Would be funny if all the content with a Facebook address was replaced with ClimateGate emails, version 3, or everything from Hunter Biden’s laptop.

Last edited 1 month ago by Russell Cook
Mike
October 4, 2021 12:38 pm

BGP is Border Gateway Protocol, think of an internet cop directing traffic through the best routes that can be used to reach a specific address.

In 2008 through a mistake by Pakistan telecom. just about all the global you tube traffic ended up in a black hole in Pakistan, all those wasted electrons!

The vision of that odious parasite Zuckerberg spending the rest of his life trying to escape from a black hole is hilarious to me.

Jeff Labute
October 4, 2021 12:40 pm

Probably partly caused Facebook stock to take a dip too. -5.4%.
(In addition to claims of harming mental health, human trafficking, arms deals, etc)
Outages make people with addictions, nervous.

Last edited 1 month ago by Jeff Labute
Dave
Reply to  Jeff Labute
October 4, 2021 7:13 pm

Want to buy some Facebook crypto coins? What could go wrong?

October 4, 2021 12:42 pm

I don’t think this Internet attack is a coincidence given that Chinese PLAAF and PLAN aircraft incursions into Taiwan airspace have ramped up dramatically over the past 3 days. A lot of overseas country’s populations like the Taiwanese depend on Facebook and and its various message and photo products to stay informed.

The long expected Chinese invasion of at least taking Taiwan’s Dongsha Island could happen within days or even hours now.

Mumbles McGuirck
Reply to  Joel O'Bryan
October 4, 2021 1:33 pm

And meanwhile, General Milley is distracted because he can’t post to his Facebook fan page “Milley is Marvelous”. Insidious, those Chinese. 🙂

Jerome P Koch
October 4, 2021 12:53 pm

One analyst says the hack occurred via BGP routing protocols. BGP routes via DNS, and for whatever reason, there was a bad BGP update, which emptied all references to FB and its apps. This was confirmed by Cloudflare. A reddit user also confirmed that a BGP update occurred seconds before FB went down.

https://arstechnica.com/information-technology/2021/10/facebook-instagram-whatsapp-and-oculus-are-down-heres-what-we-know/

Reply to  Jerome P Koch
October 4, 2021 1:37 pm

BGP does not route via DNS.
DNS may route via BGP though

2hotel9
October 4, 2021 12:54 pm

Blame cbs, they pissed Markeemark off and he took his ball and went home. 😉

Editor
October 4, 2021 1:00 pm

Wow! I don’t spend much time at Facebook anyway, but I just tried and can’t get there.

Vuk
Reply to  Andy May
October 4, 2021 1:15 pm

I use the whatsapp messages all the time to keep in touch with people around the world, and find thr outage inconvenient. Registered with FB some 10 or so years ago and someone hijacked it before I menaged to put anything there. Instagram is an unknown to me.

Pamela Matlack-Klein
Reply to  Vuk
October 4, 2021 1:43 pm

I also use Whatsapp almost daily, a very useful thing.

Reply to  Vuk
October 4, 2021 2:18 pm

Since month I use Signal instead, left WA.

Nick Haag
October 4, 2021 1:04 pm

Tee-hee! It just shows how vulnerable the world is to this sort of thing.

Neo
October 4, 2021 1:10 pm

Not only has FB apparently fubar’d its own DNS settings, I Choose To Believe the Sources that are Saying that the employees can’t even get into the rooms where the routers et al are stored because, guess what! All the badge swipe security is IoT and with the FB servers being down, there’s no way to swipe the badges.

TEWS_Pilot
Reply to  Neo
October 4, 2021 1:44 pm

Like the NG providers who couldn’t manipulate valves manually, these folks outsmarted themselves….oh, well, doors can be opened with fire axes.

Neo
October 4, 2021 1:17 pm

From AppleNews

PCMag: How To Completely Disappear From The Internet

Vuk
October 4, 2021 1:18 pm

Reuters reports (unrelated?):
KYIV, Oct 4 (Reuters) – Ukrainian police said on Monday they had arrested a 25-year-old man who hacked more than 100 foreign companies and caused damage worth more than $150 million.

https://www.reuters.com/technology/ukrainian-police-arrest-hacker-who-caused-150-million-damage-global-firms-2021-10-04/

Richard Page
Reply to  Vuk
October 4, 2021 3:02 pm

Not surprised. Only surprised that there aren’t more – presumably he was the only one that Ukrainian government didn’t want to hire.

Mumbles McGuirck
October 4, 2021 1:20 pm

Damn that Donald Trump! Will he stop at nothing??

😉

Reply to  Mumbles McGuirck
October 4, 2021 10:11 pm

I’m so old I that I can remember it used to be: “Rove, you magnificent b*st*rd!”

Retired_Engineer_Jim
October 4, 2021 1:22 pm

No, no, no – Facebook was working so hard to censor “disinformation” that they managed to come up with a “disinformation” algorithm that censored them. Love the fact that they can’;t even get in the front door.

October 4, 2021 1:23 pm

In other humor news…Elon Musk sez the USA must double its electricity output to accommodate EVs……better wind those windmills up and clean those solar cells. In the meantime some USA utilities are looking at burning oil instead of the high priced NG ….you can’t make this stuff up.

Last edited 1 month ago by Anti_griff
Gunga Din
Reply to  Anti_griff
October 4, 2021 3:38 pm

If I remember correctly, it only took a 3hp electric motor to open and close the original lock gates on the Panama Canal. Well built.
Perhaps the could hook up 30hp electric (or gas) motor to the windmills to give the appearance the windmills are still working?
That’s all they’re after anyway, the appearance the things work to get more subsidies and “damage to the environment” exemptions. (How many endangered birds and bats have these things swatted out of the air?)

Glenn
Reply to  Gunga Din
October 4, 2021 5:26 pm

They already have motors built in.

October 4, 2021 1:24 pm
Reply to  SteveT
October 4, 2021 1:48 pm

best information yet.
P****ing myself at staff being locked out of the building

October 4, 2021 1:28 pm
richard
October 4, 2021 1:28 pm
Neo
October 4, 2021 1:33 pm

Apparently, the Facebook algorithm to censor misinformation is finally working.

Stephen W
October 4, 2021 1:36 pm

Coincidentally, covid solved overnight

Trying to Play Nice
October 4, 2021 1:38 pm

This is exactly the reason I wish I could buy products with self-contained software or that would run on my network rather than in the cloud. When the your part of the cloud is down you are SOL. Why do door locks need to go through the internet so hackers can open your door? Why do baby monitors need to go through the internet so hackers can watch your child?

CarGuy Pete
Reply to  Trying to Play Nice
October 4, 2021 2:42 pm

@ Playing Nice, I used to enjoy playing games on the computer. Now you cannot play a game on the computer unless you are connected to the internet. I don’t play any games on the computer anymore.

ScarletMacaw
Reply to  CarGuy Pete
October 4, 2021 4:18 pm

I play games on the computer. Of course they’re all 20 years old. Check out GOG.com, they sell old computer games.

jdgalt1
October 4, 2021 1:48 pm

This has been discussed quite heavily on gab.com this morning. There are several other news items related to Facebook that may or may not be related to the outage.

  1. Project Veritas said over the weekend that they will release, today (Monday), an interview with a whistleblower who works at Facebook.
  2. Someone posted on Gab that Facebook employees can’t get into the office this morning. Their key cards don’t work.
  3. Within the past couple of days, a hacker offered for sale the private information of 1.5 billion Facebook users. This was reported on BleepingComputer and KrebsOnSecurity.

When I heard about the outage I thought that Facebook must have taken its own system down until they could block all discussion of the Veritas interview. But now I suspect that the hacker may be responsible for both the outage and the employee lock-out.

Reply to  jdgalt1
October 4, 2021 1:57 pm

The theory that the staff cant get in because they need internet access to a system that is down, sounds plausible.
Since the root issue seems to be not that soeme has diverted BGP, but that BGP adverts have simply stopped, its probably an issue inside the company.

It should be possible to hack it with static routes at some level

Reply to  Leo Smith
October 4, 2021 3:46 pm

Update failour 😀

Reply to  Krishna Gans
October 4, 2021 10:10 pm

AS I liked to put errors of this type: “Configuration Error”.

J N
October 4, 2021 1:48 pm

Ahhhh. I was worried why, suddenly, people started to talk to each other instead of looking to the mobile phone. I was trying to explain it with climate change but I decided to look at real data and, as a matter of fact, that crap is down.

October 4, 2021 1:51 pm

Yup – FB is down (Belgium)

Editor
October 4, 2021 2:10 pm

You knows how many careers will be saved by people prevented from self-destruction by Facebook and Instagram indiscretions.

Richard Page
Reply to  Kip Hansen
October 4, 2021 3:06 pm

Oh not to worry; once it’s back up they’ll be going all out to make up for lost time. Some people just seem to be drawn to fb like flies to shit.

John Kelly
October 4, 2021 2:15 pm

Lets see which does more phycological harm. Climate change or no Facebook.

Stephen W
Reply to  John Kelly
October 4, 2021 2:29 pm

Possibly a typo, but I’m not sure how Facebook affects plant life

John Kelly
Reply to  Stephen W
October 4, 2021 2:35 pm

FB doesn’t affect plant life but it affects the life and mental health of hundreds of millions of kids around world. Going cold turkey on FB will have an impact on these kids and the longer FB is out of action the bigger the impact. Did you forget Gretel and her mentally ill minions?

Richard Page
Reply to  John Kelly
October 4, 2021 3:08 pm

Really? How interesting. I guess you should have spelled it ‘psychological’ then if that was your meaning?

Gunga Din
Reply to  John Kelly
October 4, 2021 3:52 pm

FB and the rest social media along with our current education system in the US seem to be devoted to producing vegetables that can’t think for themselves but are (or will be) eligible to vote.
Maybe it wasn’t a typo?

Reply to  Gunga Din
October 4, 2021 7:12 pm

Yes, NOT a typo …

Walter Sobchak
October 4, 2021 2:22 pm

Couldn’t have happened to a nicer bunch of guys.

Upfrontaussie
October 4, 2021 2:27 pm

Never signed up for any of them, I hope it is permanent.

October 4, 2021 2:28 pm

The Anonymous hacking group has been busy over the last 4 days posting 3 rounds of embarrassing and annoying pictures, videos, and cartoons to various Chinese government web pages via various password hacks on vulnerable Chinese government servers.

https://www.taiwannews.com.tw/en/news/4305341

CCP has ordered what is probably counter-hacking attacks back at the West.

Peter
October 4, 2021 2:36 pm

Check the insider trading… Fb and Zucks are cashing in on the “fluctuation” no doubt.

TEWS_Pilot
October 4, 2021 2:38 pm

On another blog that has been tracking this fiasco all day they say Zuckerberg has lost $7 BILLION in net worth.

Reply to  TEWS_Pilot
October 4, 2021 2:51 pm

Not enough 😀

Richard Page
Reply to  TEWS_Pilot
October 4, 2021 3:10 pm

Possibly more in the long term. The longer this goes on, the more people will turn to alternatives.

Patricia Langdon
October 4, 2021 2:41 pm

I hope FB feels vulnerable right now… Lost control? Ha! How’s it feel?

Chip Henry
October 4, 2021 2:55 pm

Does anyone thing Team Trump has anything to do with this, as in, the big take-down Americans have been waiting for?

SMC
Reply to  Chip Henry
October 4, 2021 7:08 pm

Doubt it but, it makes for a nice conspiracy theory.

Kyle
October 4, 2021 3:02 pm

Yeah you’re obviously not an IT expert because the quote you posted from Krebs explained the actual problem, their BGP routes were withdrawn from the internet. Has absolutely nothing to do w/ DNS, or a “DNS hack” (although a side effect of the loss of BGP was also DNS failures). BGP routes being withdrawn from the internet is an issue caused by a router misconfiguration.

Reply to  Kyle
October 4, 2021 3:44 pm

Seems to have been an update failour, missing access to the building and / or serverrooms

October 4, 2021 3:04 pm

About five minutes before Facebook’s DNS stopped working we saw a large number of BGP changes (mostly route withdrawals) for Facebook’s ASN.

Source

https://twitter.com/jgrahamc/status/1445068309288951820https://twitter.com/jgrahamc/status/1445068309288951820

Reply to  Krishna Gans
October 4, 2021 6:35 pm

WHAT did you do to screw up that link?

https://twitter.com/jgrahamc/status/1445068309288951820

October 4, 2021 3:07 pm

seems to run now

Gunga Din
October 4, 2021 3:14 pm

I don’t a cellphone so don’t do any of the cell phone related social media. I briefly was on facebook in order to view some wedding pictures. Aside from accepting an avalanche of friend request, all I ever did was post a message saying I was going to cancel the account in two weeks. I only did that because Facebook didn’t send a message that my account was cancelled. Instead, they sent a message saying all my friends were “unfriended”.

Jeff
October 4, 2021 3:24 pm

Incidentally, I couldn’t access anything on my phone about 5 hours ago indicating that my network in the UK went down. Oddly, I could access YouTube just fine, but I couldn’t go online and nothing else online worked. Why YouTube was unaffected I don’t know.

However, when I changed the DNS to Google’s servers everything was OK.

October 4, 2021 3:25 pm

It was NOT a DNS hack. Their load blanacers went south….
.
 wget 31.13.71.36

–2021-10-04 16:26:17– http://31.13.71.36/

Connecting to 31.13.71.36:80… connected
.
HTTP request sent, awaiting response… 503 No server is available for the request

2021-10-04 16:26:35 ERROR 503: No server is available for the request.

Reply to  Ronald Stein
October 4, 2021 3:42 pm

Connected to facebooks register request – seems to be online again now.

October 4, 2021 4:04 pm

Does no one anymore maintain their own favorites Domain Names file? Back when we worried about decentralizing – distributing the ‘Internet’ many maintained their own DNs files.

Reply to  Doug Huffman
October 5, 2021 7:56 am

Facebook post dated May 20, 2021: “So we’ve developed a new automated method [using BGP], which allows for faster self-service peering configuration.”

https://engineering.fb.com/2021/05/20/networking-traffic/peering-automation/

Last edited 1 month ago by _Jim
October 4, 2021 4:30 pm

re Krebs: “We don’t know how or why the outages persist at Facebook and its other properties, but the changes had to have come from inside the company, as Facebook manages those records internally. Whether the changes were made maliciously or by accident is anyone’s guess at this point.”

What is the French word for self-imolation? Oh yeah – sabotage
.
.
Etymology — The English word derives from the French word saboter, meaning to “bungle, botch, wreck or sabotage“.

Last edited 1 month ago by _Jim
October 4, 2021 4:56 pm

Hmm. I’d be looking at short sale transactions. ALL of the :”tech giants” were way down at the end of the day, as people flipped out. (Interestingly, Twitter was down the most, not FB.)

John
October 4, 2021 5:18 pm

Fantastic
can we make a routine to make facebook a non entity permanently
It would save billions of tonnes of CO2 and improve worktime productivity by 10,000 percent and we wouldn’t have all those sill storys

October 4, 2021 5:27 pm

Couldn’t happen to a nicer bunch !!!!

dk_
October 4, 2021 5:30 pm

A piece on Bleeping Computer that I found just now on problem and the fix. https://www.bleepingcomputer.com/news/technology/facebook-instagram-and-whatsapp-back-online-after-bgp-fix/#

Streetcred
October 4, 2021 5:36 pm

Doug Madory, director of internet analysis at the network monitoring company Kentik, said someone at Facebook caused an update to be made to the company’s BGP records, which resulted in the company’s system taking away the map telling the world’s computers how to find its various online properties.

dk_
Reply to  Streetcred
October 5, 2021 10:05 am

Makes sense that it was self-inflicted, in light of what we have been told.

4E Douglas
October 4, 2021 6:00 pm

Was the offending update from Microsoft?

H.R.
October 4, 2021 6:02 pm

Sorry. Sucks to be them.

Here’s a link to all the sympathy I can muster.
comment image

Sara
October 4, 2021 6:32 pm

Seriously, I started giggling over this, as I do not indulge in FB or the other “social media” sites. Got enough other stuff to do.

niceguy
October 4, 2021 7:09 pm

Domain Name Servers (DNS) are the most important thing on the Internet.”

Nope. As the text you just quoted shows, BGP is.

October 4, 2021 7:20 pm

MOST likely related (and the ‘root’ of the problem as indicted by more than just a few people) –

“Peering automation at Facebook”
POSTED ON MAY 20, 2021 TO Networking & Traffic
https://engineering.fb.com/2021/05/20/networking-traffic/peering-automation/

Backup copy: https://archive.is/NvJR4

Opening excerpt: Traffic on the internet travels across many different kinds of links. A fast and reliable way to exchange traffic between different networks and service providers is through peering.

Initially, we managed peering via a time-intensive manual process. Reliable peering is essential for Facebook and for everyone’s internet use. But there is no industry standard for how to set up a scalable, automatic peering management system. So we’ve developed a new automated method

Last edited 1 month ago by _Jim
Redge
October 4, 2021 10:56 pm

Truly put the farce in Farcebook

Steve
October 5, 2021 2:54 am

I hope this is a 30-day ban at least and not just a 3-day ban.

ozspeaksup
October 5, 2021 3:38 am

love the couldnt enter bit;-) as aussies found out re their homes lights etc when our net fell over
Aus media reported it as a wiring in servers issue and turn it off then on again..
hilarious
antisocial media could stay off forever and do us all a favour

James Bull
October 5, 2021 5:36 am

I have never and will never join this so I’m sitting on the sidelines watching people melt down about it.
Also changed from whatsapp to Telegram last year when they did their ‘upgrade’ that wanted all my private data available to them. No thank You.I don’t have any shady stuff as far as I’m aware but still doesn’t mean I want unknown people looking.

James Bull

leowaj
October 5, 2021 6:51 am

I know I’m late to the party. FB is crap and I don’t use it. However, I work in IT and often the DNS records are managed by a mostly-automated process for a website. I say “mostly” because there still is a human or humans who trigger the automated process to update DNS records. It’s rare but sometimes the human element screws up. I imagine that’s what happened at FB, though I don’t exclude the narrow possibility of outside nefarious actors.

Reply to  leowaj
October 5, 2021 7:40 am

See (because no one reads the previous posts): https://engineering.fb.com/2021/05/20/networking-traffic/peering-automation/

“So we’ve developed a new automated method [using BGP], which allows for faster self-service peering configuration.”

Dated MAY 20, 2021.

leowaj
Reply to  _Jim
October 5, 2021 1:00 pm

Jim, “peering” is a common solution that high speed networks use to deliver content faster. It is, in effect, prioritized traffic. It cuts out the extra hops between your Internet Service Provider and the source of the data (a video, movie, game, etc.). It is a solution that has been around for some time. It does not directly relate to DNS records.

Reply to  leowaj
October 5, 2021 7:22 pm

re: “Jim, “peering” is a common solution that blah blah blah.”

You completely miss the point in my post – completely missed it. Care to read it again instead of just knee-jerking a response?

To SPELL it out for you and you alone – FB instituted an automatic BGP process. Got that? Extrapolate that now knowing the ‘knot’ it created in the system on the fourth …

Last edited 1 month ago by _Jim
leowaj
Reply to  _Jim
October 6, 2021 8:31 am

Jim, it was not a knee-jerk response. Remember that you first responded to me.

If you will not respect the fact that I do know what peering is and I do know what Facebook’s problem was according to what has been reported, then the problem is with your pride. I cannot help you with that.

Peering has nothing to do with Facebook’s problem. Period.

Mark - Helsinki
October 5, 2021 7:11 am

Revoked certificates 😀 could be hack, unhappy pappy engineer, or those rare but inevitable dopey screw-ups that even the best are capable of.

Reply to  Mark - Helsinki
October 5, 2021 7:42 am

See (because no one reads the previous posts): https://engineering.fb.com/2021/05/20/networking-traffic/peering-automation/

“So we’ve developed a new automated method [using BGP], which allows for faster self-service peering configuration.”

Dated MAY 20, 2021.

BEST GUESS ESTIMATE would be their algorithm ‘went south’ in an “own goal” type failure …

Pat Frank
October 5, 2021 9:33 am

Couldn’t happen to a nicer group of people. And a gift so appropriate for Mark Zuckerberg.

Vuk
Reply to  Pat Frank
October 5, 2021 2:15 pm

That is fortuitous that engineers are to blame, what kind of engineers he does employ?
Now, if it was hack the billion of users data would be compromised and some of the authorities throughout the world would impose heavy penalties (and they would well deserve it!) and that would not do, would it?
Their data security has been abysmal, I registered about a decade ago, for reasons of log-in into various sites, but within weeks someone stolen my email for login into their FB account. On dozens occasions I tried to recover my email FB log in, but no avail.

George Ellis
October 5, 2021 3:27 pm

DNS just advertises what the IP address is assigned to a URL. It could be multiple addresses. BGP and other routing protocols advertise what IPs they own and what their neighbors are advertising. Down the routes, this form tables of paths (and it can be more than one path) and the ‘penalty’ time it takes to get to an IP address. They then forward the traffic along those path of routers. Each doing the same. If BGP breaks, there is no path to the site. And since their access system broke, and those are usually in secure segments internally, it sounds like their core routers lost their minds. Their DNS records may also have a time to live as they may change their IP addresses frequently (for load considerations). No DNS advertisement from them because the router is broken, then the DNS records disappear.

BillJ
October 5, 2021 11:36 pm

Well this post didn’t age well. Reminds me of Hanlon’s Razor:

Never attribute to malice that which can be adequately explained by stupidity.”

%d bloggers like this: