I’m not an IT expert, but I decided to check. This is a very high level hack. Here is what the DNS query for Facebook shows this morning:
According to Internet security researcher Brian Krebs (@briankrebs),
Confirmed: The DNS records that tell systems how to find Facebook.com or Instagram.com got withdrawn this morning from the global routing tables. Can you imagine working at FB right now, when your email no longer works & all your internal FB-based tools fail?
To be more precise (and Geek Factor 5) the BGP routes serving Facebook’s authoritative DNS were withdrawn, rendering all Facebook domains inaccessible. That’s per @DougMadory , who knows a few things about BGP/DNS.
Domain Name Servers (DNS) are the most important thing on the Internet. It is like a phone book, where you type a name, and the DNS record says, oh, Facebook is here at this IP numerical address.
Without DNS records, Facebook and Instagram are DOA.
[UPDATE]
Disclose.tv
@disclosetv
JUST IN – Facebook employees reportedly can’t enter buildings to evaluate the Internet outage because their door access badges weren’t working (NYT)
Sure looks like it.
Do you believe in coincidence?
I tend to be cause & consequence minded.
Two days ago (2 October, 2021):
Britain to carry out ‘offensive’ cyber attacks from new £5bn digital warfare centre.
In an interview with The Telegraph, Ben Wallace (UK defence secretary,) says Britain will act in response to assaults by ‘hostile states’ such as Russia”
https://www.telegraph.co.uk/politics/2021/10/02/britain-capable-launching-offensive-cyber-attacks-against-russia/
Todsy (4 October 2021) Some other cyber warrior:
Let me show you Mr.Wallace what I can do.
Surprisingly, it’s the Guardian that is reporting why the world’s biggest news provider has been shutdown today:
Pandora papers reveal hidden riches of Putin’s inner circle | Vladimir Putin | The Guardian
Who benefits from suppressing news distribution until the news cycle moves on?
A lot of people benefit from it. Some large companies as well as people wealthy enough to hire someone savvy enough to do it. Putin merely has access to geeks ready to do it quickly. Not sure that it’s worth it, though.
Are we sure it’s not a 3 letter agency pulling this off?
Russia sure seems to be an easy excuse…why not China?
I am no defender of Putin but there are lots of bad actors and many of them work for us.
Nah. It’s the North Koreans. They have an army of hackers who are quite capable of this kind of thing.
Civilization grinding to a halt because of this would make them very happy.
Just like when they hacked SONY, for no reason?
The civilised don’t do Farcebook, only the uncivil
Space Force has more than three letters.
God I hope they were cancelled!
No it is not. The Storm is Upon Us:
https://beforeitsnews.com/prophecy/2021/10/october-red-update-2524367.html
Green New Deal Is Dead
Well, that link was a study in incoherence.
This sysop has been planned for over twenty years. I don’t expect you to understand it as you are one of the terminally stupid—and you are in the majority.
Sysop definition is – the administrator of a computer message board.
Presumably the Demand for the new Facebook currency fell off the cliff as well.
According to Internet security researcher Brian Krebs (@briankrebs),
I read, Instagram and Whatsapp are down too.
I know Whatsapp is owned by Facebook. I think Instagram is also.
Instagram appears somewhet up to me. It has a DNS entry at least
Yes both Instagram and WattsApp down
Pity – I actually like and use Instagram. See something that makes me smile, photo, caption, back to real world.
Not as intrusive as FB. FB wants to remind me of friends I actually can’t remember and put together little memory packages so I can recall some insignificant date in the past.
Only reason I still have FB is for messenger and Instagram.
Seems to me either someone screwed up or somebody with serious horse power is unhappy.
List is not very long but there are some serious game players in the field: Iran, China, Russia, North Korea and one or two lunatics who would like to earn easy way shedload of bitcoins.
Not serious horsepower, serious smarts
Them too.
I guess Facebook got Fact Checked – there were so much bias they just cancelled the whole platform.
A short write-up from @briankrebs on the ongoing outages at Facebook, Instagram and WhatsApp.
Test. Trinity. Megaton. Triggering device. High Tech. = We found the problem folks.
Let’s hope it’s permanent.
What is DNS?
Google is your friend… and foe.
Or duckduckgo
Domain Name Server. Maps a website name like http://WWW.Wattsupththat.com to an actual IP address 192.164.xxx.xxx etc
IP addresses are just abstractions for MAC addresses.
Not at all.
MAC addresses are visible to your local router, which (in the simple case) assigns each one to a local IP address. From outside your Local Area Network (LAN), your router probably has a single external (non-local) IP address, and your Internet Service Provider (ISP) knows that IP address. Your ISP probably does not know the MAC addresses on your LAN.
For others reading this, MAC stands for Media Access Control (not Macintosh, in this context). Each network card has its own MAC address (or motherboards that include network controllers).
Things can get more complicated for other than home networks.
That’s probably more than you want to know, and probably more than I know, to be truthful.
MAC addresses are not routeable, IP addresses are.
Domain Name Server
Domain Name Server
Now I can get some real work done.
Good, may they remain closed forever.
Oops, I’ll plug it back in.
Sorry folks
https://youtu.be/LJBZmuv7FtQ
Please don’t, I think the thought of the cancellers being cancelled is wonderful. Hope it lasts for a long time.
When I was working network operations in the 90s, we had a customer who’s network connection (dedicated 9600bps line) would drop out at about the same time every evening when no one was at the company. But some of their automated processes would fail at the same time.
After weeks of troubleshooting, checking the switches out our (Telenet/Sprintnet) network office, having the local telco checking their equipment, performing loopback tests, etc, nothing ever showed a problem.
So one evening, someone at the company stayed and decided to watch the modem to see what was happening. Then the cleaning lady comes in, unplugs the modem, plugs in her vacuum cleaner, and goes merrily about her business.
Oh not that story again. First time I heard it was about air traffic control at Heathrow then it was another company – all because of the little 3 pin plug and the cleaning lady!
Could have been the cleaning lady who was unplugging life support to use her vacuum cleaner.
Same lady…born in Russia.
Escapees almost always have outside help.
Guess they’ll need to use Parler. LOL
Now, FB knows how it feels to be cancelled.
Hmmmm….
I blame climate change, lol.
Interestingly, last night about half way through the 60 minutes report on the Facebook whistleblower our local CBS channel cut away to a test pattern!!
If FB was based in the UK then it would be due to Brexit, Covid, supply chain issues and driver shortages, in that order!
Somehow, this event brings to mind Galt’s speech.
Good news. I hope they stay down. The mental health of millions will be improved.
If these IP addresses are accurate, you still cannot reach Facebook by IP nor can you ping the following IP addresses.
Problem goes beyond DNS alone to include route tables(?).
Maybe they didn’t pay their yearly DNS registration fee? 🙂
Hallelujah
Lack of ping response unfortunately means nothing
But you are right – they seem not to be responding to valid http/https requests.
Hmm. There may also be a BGP hack. There seems to be no route to those addresses either.
Apparently, Facebook was able to cycle down the servers but can’t physically get back in to restart the servers because the IoT badge readers go to one of their own servers.
🤣 🤣 🤣
Unbelievable. Just how smart are these people?
–
I’m not sure why someone hasn’t just taken a sledgehammer to one or two doors, or a front window or two.
–
You can call maintenance to replace the doors with a keyed lock type for the short term.
–
Maybe someone has thought of that by now. A-a-a-n-d… maybe not.
It would be the funniest thing EVAH! if the employees are still standing outside the door.
Do we HAVE to fix it?
I’ve spent 5 minutes looking. Whoever has done this has done something major and pretty catastrophic.
freaking with DNS is possibly something a hacker in a basement can do., If BGP is involved they need to be a serious ISP, which implicates someone at a much deeper level.
Well I’ll hand it over to the pros now.
Whoever has done it is going to be blessed as much as cursed
Apparently, FB are saying it was due to a new update that wasn’t debugged or tested properly before installation. I think whoever has done it is going to get fired tomorrow morning!
Nah. It affected Whatsapp too.
Here is another guess at what is going on…..BREAKING: Facebook, Instagram Down After ’60 Minutes’ Whistleblower Story…good explanation, read the entire article.
https://www.toddstarnes.com/media/breaking-facebook-instagram-down-60-minutes-whistleblower-story/
Users on Monday reported Facebook, Instagram, and Whatsapp being down one day after a whistleblower accused the company of putting profits over safety.
The blackout happened after Frances Haugen, Facebook’s product manager on the civic misinformation team, revealed her identity Sunday on the CBS television program “60 Minutes,” as the whistleblower who provided the documents to the Wall Street Journal investigation and a Senate hearing on Instagram’s harm to teen girls, Reuters reports.
Haugen gave “tens of thousands” of pages to the WSJ, claiming Facebook’s algorithm purposefully shows users content to make them angry.
“Facebook has realized that if they change the algorithm to be safer, people will spend less time on the site, they’ll click on less ads, they’ll make less money,” Haugen told “60 Minutes.”
What a coincidence. Not….
It is being reported that employees cannot gain entrance into some buildings via the access control system also.
While they’re trying to get in the front door, Zuckerberg is sneaking out the back door with a large bag marked “Swag”. He’s headed to where there is no extradition treaty but plenty of rum drinks.
What!!??? A private for-profit enterprise puts profits before “the public good”? Hold the presses!
The so-called “whistleblower” complains that Facebook is supposedly prioritizing “profit” over removing “misinformation” and “hate speech”. More proof that there is no end to the meddling sociopathic Leftists like Haugen want to do in your life.
Here’s a thought, Big Tech: how about not censoring anything unless it violates the law? That seems like a morally and legally sound policy to me.
The official DNS delegation records for “facebook.com” are:
;; AUTHORITY SECTION:
facebook.com. 172800 IN NS a.ns.facebook.com.
facebook.com. 172800 IN NS b.ns.facebook.com.
facebook.com. 172800 IN NS c.ns.facebook.com.
facebook.com. 172800 IN NS d.ns.facebook.com.
;; ADDITIONAL SECTION:
a.ns.facebook.com. 172800 IN A 129.134.30.12
a.ns.facebook.com. 172800 IN AAAA 2a03:2880:f0fc:c:face:b00c:0:35
b.ns.facebook.com. 172800 IN A 129.134.31.12
b.ns.facebook.com. 172800 IN AAAA 2a03:2880:f0fd:c:face:b00c:0:35
c.ns.facebook.com. 172800 IN A 185.89.218.12
c.ns.facebook.com. 172800 IN AAAA 2a03:2880:f1fc:c:face:b00c:0:35
d.ns.facebook.com. 172800 IN A 185.89.219.12
d.ns.facebook.com. 172800 IN AAAA 2a03:2880:f1fd:c:face:b00c:0:35
So there should be four reachable DNS servers providing authoritative answers for DNS queries for “facebook.com” records. Two are in US/ARIN-allocated address blocks and two are in RIPE/Europe blocks.
The 129.134.0.0/16 network is a direct ARIN allocation to Facebook.
185.89.216.0/22 network is a direct RIPE allocation to Facebook.
I can’t reach any of those addresses. I don’t use IPv6 so I’m just checking the IPv4 routes. My internet provider is AT&T and the routes appear to be missing from their network.
This is not a “DNS Hack” per-se; it is a routing failure that happens to hit all the networks where the authoritative DNS servers for Facebook reside.
No doubt a lot of other things on the same networks are also unreachable; but since DNS resolution needs to come first, people tend to label this as a “DNS problem”.
Why the routes are gone is an interesting question. Could be a hack or it could be an error. Some months back one of the big content delivery providers (I think it was CloudFlare, but I could be mis-remembering) pretty much went down because they introduced a routing misconfiguration that cause virtually all traffic to be routed through their Atlanta hub, overloading all the circuits.
In this case the routes are simply missing. Either they are not being advertised or the the route advertisements are not being accepted.
Maybe it’s the hidden code on Chinese router chips. Doing a test run on bringing the financial system to a standstill.
The chips in the vaxxes are activated 😀
You’re a very strange person!
Why ? 😀 Irony detector broken ? 😀 😀
No Idea what in other countries people says about the “Anti vaxxers” but the nano-chips in the vaxxes are often taken as cause they don’t want to be “vaxxed”. if you read critics about here in Germany
Krishna Gans- I upvoted your comment and couldn’t help but put up a tongue-in-cheek comment, forgetting (of course) that some humour simply doesn’t travel well. I do apologise if 2 people and/or yourself failed to appreciate the humour in that post but it’s a bit too late to do anything about it now. I keep forgetting that British humour so often has to be explained to others.
strange that two entirely different networks are affected
Enemy action.
Although the enemy of Facebook is my…
I used to administer the three authoritative DNS servers for a well known oil field services company. In addition to the sysadmin, I was also the DNS admin for the company. These were authoritative for about 12-15 separate domains. These three servers (Sun/Oracle Sparc running Solaris 11 at the time) were located in different parts of the country and worked to load balance and back each other up. To me this says someone screwed up multiple DNS servers at once. You have to really go out of your way to do this. Automated admin tools are very good at this sort of thing.
facebook.com is on sale!
DNS is not BGP and routing.
DNS tells you where they are.
BGP is how to get there
a quick check indicates that facebook.com no longer exists on global DNS
Nor does whatsapp.com
linkedin.com does.
A neat hack to the root nameservers of the .com domain.
Well done hackers. Microsoft is hated by computer professionals. Sad I cant talk to my family round the world, but worth it to see Microsoft sweat.
Why do you refer to Microsoft? Facebook et al is the broken item. Do you mean that Microsoft could be victimized by a similar attack?
I agree that Facebook is a menace and MS has sold useful but often defective products for many years.
sorry was thinking of skype, but ms owns a large chunk of facebook as well
No. The root DNS servers were not compromised; they continued to hold the proper delegation records. The routes to Facebook’s DNS servers disappeared.
Now that the routes are back it appears that the circuits are provided by AT&T and Facebooks is operating their own DNS servers on their own networks.
“When considering the cause of IT failures, never ascribe to conspiracy that which can be reasonably attributed equally to: ‘Oops . . . I didn’t think that would happen.'”
A paraphrase from an astute twaddling I saw today on “the Twitter.”
My Yahoo links to WUWT don’t work either. –AGF
Would be funny if all the content with a Facebook address was replaced with ClimateGate emails, version 3, or everything from Hunter Biden’s laptop.
BGP is Border Gateway Protocol, think of an internet cop directing traffic through the best routes that can be used to reach a specific address.
In 2008 through a mistake by Pakistan telecom. just about all the global you tube traffic ended up in a black hole in Pakistan, all those wasted electrons!
The vision of that odious parasite Zuckerberg spending the rest of his life trying to escape from a black hole is hilarious to me.
Probably partly caused Facebook stock to take a dip too. -5.4%.
(In addition to claims of harming mental health, human trafficking, arms deals, etc)
Outages make people with addictions, nervous.
Want to buy some Facebook crypto coins? What could go wrong?
I don’t think this Internet attack is a coincidence given that Chinese PLAAF and PLAN aircraft incursions into Taiwan airspace have ramped up dramatically over the past 3 days. A lot of overseas country’s populations like the Taiwanese depend on Facebook and and its various message and photo products to stay informed.
The long expected Chinese invasion of at least taking Taiwan’s Dongsha Island could happen within days or even hours now.
And meanwhile, General Milley is distracted because he can’t post to his Facebook fan page “Milley is Marvelous”. Insidious, those Chinese. 🙂
One analyst says the hack occurred via BGP routing protocols. BGP routes via DNS, and for whatever reason, there was a bad BGP update, which emptied all references to FB and its apps. This was confirmed by Cloudflare. A reddit user also confirmed that a BGP update occurred seconds before FB went down.
https://arstechnica.com/information-technology/2021/10/facebook-instagram-whatsapp-and-oculus-are-down-heres-what-we-know/
BGP does not route via DNS.
DNS may route via BGP though
Blame cbs, they pissed Markeemark off and he took his ball and went home. 😉
Wow! I don’t spend much time at Facebook anyway, but I just tried and can’t get there.
I use the whatsapp messages all the time to keep in touch with people around the world, and find thr outage inconvenient. Registered with FB some 10 or so years ago and someone hijacked it before I menaged to put anything there. Instagram is an unknown to me.
I also use Whatsapp almost daily, a very useful thing.
Since month I use Signal instead, left WA.
Tee-hee! It just shows how vulnerable the world is to this sort of thing.
Not only has FB apparently fubar’d its own DNS settings, I Choose To Believe the Sources that are Saying that the employees can’t even get into the rooms where the routers et al are stored because, guess what! All the badge swipe security is IoT and with the FB servers being down, there’s no way to swipe the badges.
Like the NG providers who couldn’t manipulate valves manually, these folks outsmarted themselves….oh, well, doors can be opened with fire axes.
From AppleNews
PCMag: How To Completely Disappear From The Internet
Reuters reports (unrelated?):
KYIV, Oct 4 (Reuters) – Ukrainian police said on Monday they had arrested a 25-year-old man who hacked more than 100 foreign companies and caused damage worth more than $150 million.
https://www.reuters.com/technology/ukrainian-police-arrest-hacker-who-caused-150-million-damage-global-firms-2021-10-04/
Not surprised. Only surprised that there aren’t more – presumably he was the only one that Ukrainian government didn’t want to hire.
Damn that Donald Trump! Will he stop at nothing??
😉
I’m so old I that I can remember it used to be: “Rove, you magnificent b*st*rd!”
No, no, no – Facebook was working so hard to censor “disinformation” that they managed to come up with a “disinformation” algorithm that censored them. Love the fact that they can’;t even get in the front door.
In other humor news…Elon Musk sez the USA must double its electricity output to accommodate EVs……better wind those windmills up and clean those solar cells. In the meantime some USA utilities are looking at burning oil instead of the high priced NG ….you can’t make this stuff up.
If I remember correctly, it only took a 3hp electric motor to open and close the original lock gates on the Panama Canal. Well built.
Perhaps the could hook up 30hp electric (or gas) motor to the windmills to give the appearance the windmills are still working?
That’s all they’re after anyway, the appearance the things work to get more subsidies and “damage to the environment” exemptions. (How many endangered birds and bats have these things swatted out of the air?)
They already have motors built in.
1743 UTC from The Register
https://www.theregister.com/2021/10/04/facebook_sites_outage/
best information yet.
P****ing myself at staff being locked out of the building
Down Detector.
https://downdetector.com/status/facebook/
Facebook headquarters- https://www.youtube.com/watch?v=jUzGF401vLc
Apparently, the Facebook algorithm to censor misinformation is finally working.
Coincidentally, covid solved overnight
This is exactly the reason I wish I could buy products with self-contained software or that would run on my network rather than in the cloud. When the your part of the cloud is down you are SOL. Why do door locks need to go through the internet so hackers can open your door? Why do baby monitors need to go through the internet so hackers can watch your child?
@ Playing Nice, I used to enjoy playing games on the computer. Now you cannot play a game on the computer unless you are connected to the internet. I don’t play any games on the computer anymore.
I play games on the computer. Of course they’re all 20 years old. Check out GOG.com, they sell old computer games.
This has been discussed quite heavily on gab.com this morning. There are several other news items related to Facebook that may or may not be related to the outage.
When I heard about the outage I thought that Facebook must have taken its own system down until they could block all discussion of the Veritas interview. But now I suspect that the hacker may be responsible for both the outage and the employee lock-out.
The theory that the staff cant get in because they need internet access to a system that is down, sounds plausible.
Since the root issue seems to be not that soeme has diverted BGP, but that BGP adverts have simply stopped, its probably an issue inside the company.
It should be possible to hack it with static routes at some level
Update failour 😀
AS I liked to put errors of this type: “Configuration Error”.
Ahhhh. I was worried why, suddenly, people started to talk to each other instead of looking to the mobile phone. I was trying to explain it with climate change but I decided to look at real data and, as a matter of fact, that crap is down.
Yup – FB is down (Belgium)
You knows how many careers will be saved by people prevented from self-destruction by Facebook and Instagram indiscretions.
Oh not to worry; once it’s back up they’ll be going all out to make up for lost time. Some people just seem to be drawn to fb like flies to shit.
Lets see which does more phycological harm. Climate change or no Facebook.
Possibly a typo, but I’m not sure how Facebook affects plant life
FB doesn’t affect plant life but it affects the life and mental health of hundreds of millions of kids around world. Going cold turkey on FB will have an impact on these kids and the longer FB is out of action the bigger the impact. Did you forget Gretel and her mentally ill minions?
Really? How interesting. I guess you should have spelled it ‘psychological’ then if that was your meaning?
FB and the rest social media along with our current education system in the US seem to be devoted to producing vegetables that can’t think for themselves but are (or will be) eligible to vote.
Maybe it wasn’t a typo?
Yes, NOT a typo …
Couldn’t have happened to a nicer bunch of guys.
Never signed up for any of them, I hope it is permanent.
The Anonymous hacking group has been busy over the last 4 days posting 3 rounds of embarrassing and annoying pictures, videos, and cartoons to various Chinese government web pages via various password hacks on vulnerable Chinese government servers.
https://www.taiwannews.com.tw/en/news/4305341
CCP has ordered what is probably counter-hacking attacks back at the West.
Check the insider trading… Fb and Zucks are cashing in on the “fluctuation” no doubt.
On another blog that has been tracking this fiasco all day they say Zuckerberg has lost $7 BILLION in net worth.
Not enough 😀
Possibly more in the long term. The longer this goes on, the more people will turn to alternatives.
I hope FB feels vulnerable right now… Lost control? Ha! How’s it feel?
Does anyone thing Team Trump has anything to do with this, as in, the big take-down Americans have been waiting for?
Doubt it but, it makes for a nice conspiracy theory.
Yeah you’re obviously not an IT expert because the quote you posted from Krebs explained the actual problem, their BGP routes were withdrawn from the internet. Has absolutely nothing to do w/ DNS, or a “DNS hack” (although a side effect of the loss of BGP was also DNS failures). BGP routes being withdrawn from the internet is an issue caused by a router misconfiguration.
Seems to have been an update failour, missing access to the building and / or serverrooms
About five minutes before Facebook’s DNS stopped working we saw a large number of BGP changes (mostly route withdrawals) for Facebook’s ASN.
Source
https://twitter.com/jgrahamc/status/1445068309288951820https://twitter.com/jgrahamc/status/1445068309288951820
WHAT did you do to screw up that link?
https://twitter.com/jgrahamc/status/1445068309288951820
seems to run now
I don’t a cellphone so don’t do any of the cell phone related social media. I briefly was on facebook in order to view some wedding pictures. Aside from accepting an avalanche of friend request, all I ever did was post a message saying I was going to cancel the account in two weeks. I only did that because Facebook didn’t send a message that my account was cancelled. Instead, they sent a message saying all my friends were “unfriended”.
Incidentally, I couldn’t access anything on my phone about 5 hours ago indicating that my network in the UK went down. Oddly, I could access YouTube just fine, but I couldn’t go online and nothing else online worked. Why YouTube was unaffected I don’t know.
However, when I changed the DNS to Google’s servers everything was OK.
It was NOT a DNS hack. Their load blanacers went south….
.
wget 31.13.71.36
–2021-10-04 16:26:17– http://31.13.71.36/
Connecting to 31.13.71.36:80… connected
.
HTTP request sent, awaiting response… 503 No server is available for the request
2021-10-04 16:26:35 ERROR 503: No server is available for the request.
Connected to facebooks register request – seems to be online again now.
Does no one anymore maintain their own favorites Domain Names file? Back when we worried about decentralizing – distributing the ‘Internet’ many maintained their own DNs files.
Facebook post dated May 20, 2021: “So we’ve developed a new automated method [using BGP], which allows for faster self-service peering configuration.”
https://engineering.fb.com/2021/05/20/networking-traffic/peering-automation/
re Krebs: “We don’t know how or why the outages persist at Facebook and its other properties, but the changes had to have come from inside the company, as Facebook manages those records internally. Whether the changes were made maliciously or by accident is anyone’s guess at this point.”
What is the French word for self-imolation? Oh yeah – sabotage …
.
.
Etymology — The English word derives from the French word saboter, meaning to “bungle, botch, wreck or sabotage“.
Hmm. I’d be looking at short sale transactions. ALL of the :”tech giants” were way down at the end of the day, as people flipped out. (Interestingly, Twitter was down the most, not FB.)
Fantastic
can we make a routine to make facebook a non entity permanently
It would save billions of tonnes of CO2 and improve worktime productivity by 10,000 percent and we wouldn’t have all those sill storys
Couldn’t happen to a nicer bunch !!!!
A piece on Bleeping Computer that I found just now on problem and the fix. https://www.bleepingcomputer.com/news/technology/facebook-instagram-and-whatsapp-back-online-after-bgp-fix/#
Doug Madory, director of internet analysis at the network monitoring company Kentik, said someone at Facebook caused an update to be made to the company’s BGP records, which resulted in the company’s system taking away the map telling the world’s computers how to find its various online properties.
Makes sense that it was self-inflicted, in light of what we have been told.
Was the offending update from Microsoft?
Sorry. Sucks to be them.
Here’s a link to all the sympathy I can muster.
Seriously, I started giggling over this, as I do not indulge in FB or the other “social media” sites. Got enough other stuff to do.
“Domain Name Servers (DNS) are the most important thing on the Internet.”
Nope. As the text you just quoted shows, BGP is.
MOST likely related (and the ‘root’ of the problem as indicted by more than just a few people) –
“Peering automation at Facebook”
POSTED ON MAY 20, 2021 TO Networking & Traffic
https://engineering.fb.com/2021/05/20/networking-traffic/peering-automation/
Backup copy: https://archive.is/NvJR4
Opening excerpt: Traffic on the internet travels across many different kinds of links. A fast and reliable way to exchange traffic between different networks and service providers is through peering.
Initially, we managed peering via a time-intensive manual process. Reliable peering is essential for Facebook and for everyone’s internet use. But there is no industry standard for how to set up a scalable, automatic peering management system. So we’ve developed a new automated method …
Truly put the farce in Farcebook
I hope this is a 30-day ban at least and not just a 3-day ban.
love the couldnt enter bit;-) as aussies found out re their homes lights etc when our net fell over
Aus media reported it as a wiring in servers issue and turn it off then on again..
hilarious
antisocial media could stay off forever and do us all a favour
I have never and will never join this so I’m sitting on the sidelines watching people melt down about it.
Also changed from whatsapp to Telegram last year when they did their ‘upgrade’ that wanted all my private data available to them. No thank You.I don’t have any shady stuff as far as I’m aware but still doesn’t mean I want unknown people looking.
James Bull
I know I’m late to the party. FB is crap and I don’t use it. However, I work in IT and often the DNS records are managed by a mostly-automated process for a website. I say “mostly” because there still is a human or humans who trigger the automated process to update DNS records. It’s rare but sometimes the human element screws up. I imagine that’s what happened at FB, though I don’t exclude the narrow possibility of outside nefarious actors.
See (because no one reads the previous posts): https://engineering.fb.com/2021/05/20/networking-traffic/peering-automation/
“So we’ve developed a new automated method [using BGP], which allows for faster self-service peering configuration.”
Dated MAY 20, 2021.
Jim, “peering” is a common solution that high speed networks use to deliver content faster. It is, in effect, prioritized traffic. It cuts out the extra hops between your Internet Service Provider and the source of the data (a video, movie, game, etc.). It is a solution that has been around for some time. It does not directly relate to DNS records.
re: “Jim, “peering” is a common solution that blah blah blah.”
You completely miss the point in my post – completely missed it. Care to read it again instead of just knee-jerking a response?
To SPELL it out for you and you alone – FB instituted an automatic BGP process. Got that? Extrapolate that now knowing the ‘knot’ it created in the system on the fourth …
Jim, it was not a knee-jerk response. Remember that you first responded to me.
If you will not respect the fact that I do know what peering is and I do know what Facebook’s problem was according to what has been reported, then the problem is with your pride. I cannot help you with that.
Peering has nothing to do with Facebook’s problem. Period.
Revoked certificates 😀 could be hack, unhappy pappy engineer, or those rare but inevitable dopey screw-ups that even the best are capable of.
See (because no one reads the previous posts): https://engineering.fb.com/2021/05/20/networking-traffic/peering-automation/
“So we’ve developed a new automated method [using BGP], which allows for faster self-service peering configuration.”
Dated MAY 20, 2021.
BEST GUESS ESTIMATE would be their algorithm ‘went south’ in an “own goal” type failure …
Couldn’t happen to a nicer group of people. And a gift so appropriate for Mark Zuckerberg.
That is fortuitous that engineers are to blame, what kind of engineers he does employ?
Now, if it was hack the billion of users data would be compromised and some of the authorities throughout the world would impose heavy penalties (and they would well deserve it!) and that would not do, would it?
Their data security has been abysmal, I registered about a decade ago, for reasons of log-in into various sites, but within weeks someone stolen my email for login into their FB account. On dozens occasions I tried to recover my email FB log in, but no avail.
DNS just advertises what the IP address is assigned to a URL. It could be multiple addresses. BGP and other routing protocols advertise what IPs they own and what their neighbors are advertising. Down the routes, this form tables of paths (and it can be more than one path) and the ‘penalty’ time it takes to get to an IP address. They then forward the traffic along those path of routers. Each doing the same. If BGP breaks, there is no path to the site. And since their access system broke, and those are usually in secure segments internally, it sounds like their core routers lost their minds. Their DNS records may also have a time to live as they may change their IP addresses frequently (for load considerations). No DNS advertisement from them because the router is broken, then the DNS records disappear.
Well this post didn’t age well. Reminds me of Hanlon’s Razor:
“Never attribute to malice that which can be adequately explained by stupidity.”