I’m not an IT expert, but I decided to check. This is a very high level hack. Here is what the DNS query for Facebook shows this morning:
According to Internet security researcher Brian Krebs (@briankrebs),
Confirmed: The DNS records that tell systems how to find Facebook.com or Instagram.com got withdrawn this morning from the global routing tables. Can you imagine working at FB right now, when your email no longer works & all your internal FB-based tools fail?
To be more precise (and Geek Factor 5) the BGP routes serving Facebook’s authoritative DNS were withdrawn, rendering all Facebook domains inaccessible. That’s per @DougMadory , who knows a few things about BGP/DNS.
Domain Name Servers (DNS) are the most important thing on the Internet. It is like a phone book, where you type a name, and the DNS record says, oh, Facebook is here at this IP numerical address.
Without DNS records, Facebook and Instagram are DOA.
[UPDATE]
Disclose.tv
@disclosetv
JUST IN – Facebook employees reportedly can’t enter buildings to evaluate the Internet outage because their door access badges weren’t working (NYT)
I hope FB feels vulnerable right now… Lost control? Ha! How’s it feel?
Does anyone thing Team Trump has anything to do with this, as in, the big take-down Americans have been waiting for?
Doubt it but, it makes for a nice conspiracy theory.
Yeah you’re obviously not an IT expert because the quote you posted from Krebs explained the actual problem, their BGP routes were withdrawn from the internet. Has absolutely nothing to do w/ DNS, or a “DNS hack” (although a side effect of the loss of BGP was also DNS failures). BGP routes being withdrawn from the internet is an issue caused by a router misconfiguration.
Seems to have been an update failour, missing access to the building and / or serverrooms
About five minutes before Facebook’s DNS stopped working we saw a large number of BGP changes (mostly route withdrawals) for Facebook’s ASN.
Source
https://twitter.com/jgrahamc/status/1445068309288951820https://twitter.com/jgrahamc/status/1445068309288951820
WHAT did you do to screw up that link?
https://twitter.com/jgrahamc/status/1445068309288951820
seems to run now
I don’t a cellphone so don’t do any of the cell phone related social media. I briefly was on facebook in order to view some wedding pictures. Aside from accepting an avalanche of friend request, all I ever did was post a message saying I was going to cancel the account in two weeks. I only did that because Facebook didn’t send a message that my account was cancelled. Instead, they sent a message saying all my friends were “unfriended”.
Incidentally, I couldn’t access anything on my phone about 5 hours ago indicating that my network in the UK went down. Oddly, I could access YouTube just fine, but I couldn’t go online and nothing else online worked. Why YouTube was unaffected I don’t know.
However, when I changed the DNS to Google’s servers everything was OK.
It was NOT a DNS hack. Their load blanacers went south….
.
wget 31.13.71.36
–2021-10-04 16:26:17– http://31.13.71.36/
Connecting to 31.13.71.36:80… connected
.
HTTP request sent, awaiting response… 503 No server is available for the request
2021-10-04 16:26:35 ERROR 503: No server is available for the request.
Connected to facebooks register request – seems to be online again now.
Does no one anymore maintain their own favorites Domain Names file? Back when we worried about decentralizing – distributing the ‘Internet’ many maintained their own DNs files.
Facebook post dated May 20, 2021: “So we’ve developed a new automated method [using BGP], which allows for faster self-service peering configuration.”
https://engineering.fb.com/2021/05/20/networking-traffic/peering-automation/
re Krebs: “We don’t know how or why the outages persist at Facebook and its other properties, but the changes had to have come from inside the company, as Facebook manages those records internally. Whether the changes were made maliciously or by accident is anyone’s guess at this point.”
What is the French word for self-imolation? Oh yeah – sabotage …
.
.
Etymology — The English word derives from the French word saboter, meaning to “bungle, botch, wreck or sabotage“.
Hmm. I’d be looking at short sale transactions. ALL of the :”tech giants” were way down at the end of the day, as people flipped out. (Interestingly, Twitter was down the most, not FB.)
Fantastic
can we make a routine to make facebook a non entity permanently
It would save billions of tonnes of CO2 and improve worktime productivity by 10,000 percent and we wouldn’t have all those sill storys
Couldn’t happen to a nicer bunch !!!!
A piece on Bleeping Computer that I found just now on problem and the fix. https://www.bleepingcomputer.com/news/technology/facebook-instagram-and-whatsapp-back-online-after-bgp-fix/#
Doug Madory, director of internet analysis at the network monitoring company Kentik, said someone at Facebook caused an update to be made to the company’s BGP records, which resulted in the company’s system taking away the map telling the world’s computers how to find its various online properties.
Makes sense that it was self-inflicted, in light of what we have been told.
Was the offending update from Microsoft?
Sorry. Sucks to be them.
Here’s a link to all the sympathy I can muster.
Seriously, I started giggling over this, as I do not indulge in FB or the other “social media” sites. Got enough other stuff to do.
“Domain Name Servers (DNS) are the most important thing on the Internet.”
Nope. As the text you just quoted shows, BGP is.
MOST likely related (and the ‘root’ of the problem as indicted by more than just a few people) –
“Peering automation at Facebook”
POSTED ON MAY 20, 2021 TO Networking & Traffic
https://engineering.fb.com/2021/05/20/networking-traffic/peering-automation/
Backup copy: https://archive.is/NvJR4
Opening excerpt: Traffic on the internet travels across many different kinds of links. A fast and reliable way to exchange traffic between different networks and service providers is through peering.
Initially, we managed peering via a time-intensive manual process. Reliable peering is essential for Facebook and for everyone’s internet use. But there is no industry standard for how to set up a scalable, automatic peering management system. So we’ve developed a new automated method …
Truly put the farce in Farcebook
I hope this is a 30-day ban at least and not just a 3-day ban.
love the couldnt enter bit;-) as aussies found out re their homes lights etc when our net fell over
Aus media reported it as a wiring in servers issue and turn it off then on again..
hilarious
antisocial media could stay off forever and do us all a favour
I have never and will never join this so I’m sitting on the sidelines watching people melt down about it.
Also changed from whatsapp to Telegram last year when they did their ‘upgrade’ that wanted all my private data available to them. No thank You.I don’t have any shady stuff as far as I’m aware but still doesn’t mean I want unknown people looking.
James Bull
I know I’m late to the party. FB is crap and I don’t use it. However, I work in IT and often the DNS records are managed by a mostly-automated process for a website. I say “mostly” because there still is a human or humans who trigger the automated process to update DNS records. It’s rare but sometimes the human element screws up. I imagine that’s what happened at FB, though I don’t exclude the narrow possibility of outside nefarious actors.
See (because no one reads the previous posts): https://engineering.fb.com/2021/05/20/networking-traffic/peering-automation/
“So we’ve developed a new automated method [using BGP], which allows for faster self-service peering configuration.”
Dated MAY 20, 2021.
Jim, “peering” is a common solution that high speed networks use to deliver content faster. It is, in effect, prioritized traffic. It cuts out the extra hops between your Internet Service Provider and the source of the data (a video, movie, game, etc.). It is a solution that has been around for some time. It does not directly relate to DNS records.
re: “Jim, “peering” is a common solution that blah blah blah.”
You completely miss the point in my post – completely missed it. Care to read it again instead of just knee-jerking a response?
To SPELL it out for you and you alone – FB instituted an automatic BGP process. Got that? Extrapolate that now knowing the ‘knot’ it created in the system on the fourth …
Jim, it was not a knee-jerk response. Remember that you first responded to me.
If you will not respect the fact that I do know what peering is and I do know what Facebook’s problem was according to what has been reported, then the problem is with your pride. I cannot help you with that.
Peering has nothing to do with Facebook’s problem. Period.