Guest essay by Leo Goldstein
A Method of Google Search Bias Quantification and Its Application in Climate Debate and General Political Discourse
Abstract
The percentage of domain traffic, referred by Google Search, net of brand searches (PGSTN), tends to be in or around the range 25%-30% for a broad class of web domains. This hypothesis is tested by calculating the correlation between the popularity of news/opinions websites and their PGSTN, and finding it to be near zero. Thus, PGSTN can be used rigorously to detect and even quantify Google Search intentional bias. Intentional bias is the bias that has been introduced by internal Google decisions, and unrelated to external factors, such as the dominance of particular viewpoints on the web. Here, the PGSTN method is applied for intentional bias detection about climate debate and in general political discourse.
Google Search is found to be extremely biased in favor of climate alarmism and against climate realism. The PGSTN ranges for climate realism and climate alarmism do not even overlap! Some of the most important climate realist domains, including low-controversial judithcurry.com, have such a low PGSTN that they can be considered blacklisted by Google.
Google Search is found to be biased in favor of left/liberal domains and against conservative domains with a confidence of 95%. Further, certain hard-Left domains have such a high PGSTN that their standing raises suspicions that they have been hand-picked for prominent placement. Certain respected conservative domains are blacklisted.
Introduction
Left-liberal political bias in Google Search has been noticed for years. See Robert Epstein et al, A Method for Detecting Bias in Search Rankings, with Evidence of Systematic Bias Related to the 2016 Presidential Election; Todd Dunning’s 2015-09-15 comment; Leo Goldstein, Why are Search Engines so Hostile to Climate Realism?
These observations have not completely resolved the question whether the bias was intentional, or reflected the biased web content. Recently, Google’s official Search Evaluation Guidelines have shown intentional bias against climate realism. At least one former Google employee claimed intentional demotion of “anything non-PC” by the Google Search team.
This paper uses published SEO data from multiple sources, including BrightEdge Research, 2017: Organic Search Is Still the Largest Channel, updating its 2014 report. From here on, the term ‘bias’ means intentional bias. This paper formulates, substantiates, and applies a quantitative method of bias detection in Google Search.
Methods
It is known that Google Search provides 25%-30% of the user’s traffic to an average website. As Google executives and PR repeated many times, Google Search service exists to provide the most relevant and useful results to the user’s queries. Google Chairman Eric Schmidt even joked that there should be only one result for each query – the result that the user seeks. Google servers crawl the whole web, extracting text, links, and other data from trillions of pages. Google constantly and successfully fights attempts to artificially promote websites through collusive linking, and other search engine optimization techniques. In its undertaking, Google also uses an enormous amount of off-web information, which it collects through Chrome browser, other Google applications and services, analytics beacons, domains registrar status, and so on. This information includes domains popularity and ownership. Google also processes immediate feedback from the users in the form of frequency of clicks on the results, bounce rate, the frequency of repeated searches with modified terms, etc.
Google is very good at its job. Sites and domains that are less popular with the visitors tend to be less likely to receive traffic from Google, and vice versa. The effect is that percentage of net traffic that domains receive from Google Search tends to be similar across web domains! This fact is illustrated by nearly zero correlation between domain popularity and percentage of net traffic it receives from Google within each of the sets of the left/liberal media and conservative media, despite the domain popularity (according to Alexa.com, lower values mean higher popularity), varying from 24 to 1,469 for the left/liberal media set, and from 56 to 12,795 for the conservative media set. Traffic from Google ads is about 5% of the total Google traffic, so it is not a factor. “Net traffic,” used throughout this research, excludes traffic received from the users, intentionally searching for the website by its name (i.e., searches for ‘foxnews’ and ‘fox news’ are excluded from the net traffic for foxnews.com). Net traffic better reflects Google intent, because Google Search does not have much choice when the user searches for a website by its name or brand. Alexa.com provides information which allows PGSTN calculation for hundreds of thousands of web domains.
Given the robustness of PGSTN, I conclude that statistically significant difference in PGSTN between a priori defined sets of comparable domains is due to intentional bias by Google, unless there is another good explanation.
Methodological Details
All the data in this research is based on Alexa (free version) snapshots from September 4, 2017. For each domain, Google Search Total was taken from “Upstream Sites | Which sites did people visit immediately before this site?” table. Branded traffic was taken from the “Top Keywords from Search Engines | Which search keywords send traffic to this site?” table. It should be noted that only five top values, appearing in the free Alexa snapshots, were used. All Google search domains shown in the table were included (google.com, google.ca, google.co.uk, google.co.in etc.) If the total of the branded traffic were less than 5%, the value 1% was entered. PGSTN was calculated by deducting branded search traffic from the total Google Search traffic.
PGSTN is not expected to provide sufficient certainty for individual domains, because multiple factors influence it, including possible error in Alexa data. Nevertheless, the Google attitude toward a domain has been provisionally noted and color coded in the attached spreadsheet PGSTN-Domains.xlsx as follows:
Whitelist / Green Light: >36%
Normal: 20%-36%
Grey Area: 12%-20%
Blacklist: <=12%
Most domains were expected (based on the cited SEO research) to have PGSTN in the 20%-36% range. This expectation has been met. PGSTN <= 12% provisionally indicates that the domain is blacklisted by Google. Everything between the blacklist and the normal range is considered a grey area. Finally, PGSTN > 36% provisionally indicates unusual favoritism by Google.
Google Bias in Climate Debate
The domains were selected mostly according to Alexa classification. Detection of extreme bias Google Search has against climate realism did not require statistical methods.
There is a huge gap between PGSTN of realism domains (6.3% – 17.4%), and PGSTN of climate alarmism domains (23.5%-52.4%). The gap is 6.1%. Except for drroyspencer.com, all climate realism domains are blacklisted by Google (PGSTN is 6.3% – 11.0%).
On the other hand, self-appointed “fact checkers,” including snopes.com and politifact.com have PGSTN about 50%. That gives ground to the suspicion that they had been hand-picked by Google for prioritization. Another two sites with suspiciously high PGSTN are sourcewatch.org (PGSTN = 50.1%) and prwatch.org (PGSTN = 40.9%). These two sites grossly exchange links (they refer to each other as the source), have overlapping content, and are known to Google to belong to the same organization, the Center for Media and Democracy. These are well-known signs of spam – yet Google has not only failed to downrank them as spam, but likely manually prioritized them.
This section includes netrootsnation.org, a site of a radical left conference, not specifically geared toward climate alarmism. Its PGSTN = 44.5%. This domain could have been hand-picked or its owners had been advised by Google insiders on gaming the rankings. Google has funded the conference, and Google representatives attended it and made presentations on relevant subjects, like this one. A quote:
“We’ll share some ways to leverage the power of online video and how to integrate Google and YouTube’s tools with other advocacy efforts.”
All other alarmist domains have PGSTN in the whitelisted or normal range.
Google Bias in General Political Discourse
To quantify Google general political bias, I selected top U.S. news and opinions sites by their ranking in Alexa, then added some lower ranking conservative sites based on my personal knowledge and/or Alexa suggestions. There was an element of subjectivity in selection and classification, and I omitted some domains that I could not classify. Nevertheless, the most popular domains in both left/liberal (including Left, Mainstream Liberal, and Mainstream Center) and conservative (including Conservative and Mainstream Conservative) categories have been selected and classified rigorously, and use of weighted statistics minimized the element of subjectivity in the results.
The results show that Google Search is heavily biased against conservative domains, and some respectable conservative domains seem to be blacklisted:
thegatewaypundit.com
pjmedia.com
americanthinker.com
redstate.com
powerlineblog.com
drudgereport.com
There might be an alternative or additional explanation for low PGSTN of the Drudge Report – the site mostly consists of links to articles on other sites, a practice Google looks down on.
On average, the conservative domains have almost two times lower PGSTN than the left/liberal ones: conservative 15.5% (standard deviation 5.1%) vs. left/liberal 27.4% (standard deviation 4.9%). Hypothesis of Google Search left/liberal bias is confirmed with a confidence of 95%.
Discussion
Although PGSTN of individual domains is not sufficient for conclusions, I cannot avoid noticing that extremist websites, such as dailystormer.com (PGSTN = 13.6%; ceased to exist by the time of the research) and dailykos.com (PGSTN = 20.2%) are preferred by Google over many conservative and climate realist domains.
Conclusions
Google Search is biased in favor of left/liberal websites against conservative websites, and is extremely biased in favor of climate alarmism against climate realism.
Disclosure
I hold short positions in Google stock.
References
The references are in the body of the article.
Supplementary Materials
Alexa snapshots are available from https://defyccc.com/data/PGSTN-Snapshots.7z (compressed with 7-Zip).
Contact Author: Leo Goldstein, DefyCCC.com, ah@defyccc.com
Another good research, showing intentional left bias in Google search results:
http://www.canirank.com/blog/analysis-of-political-bias-in-internet-search-engine-results/
Another good article on the intentional leftist bias in Google search results:
http://www.canirank.com/blog/analysis-of-political-bias-in-internet-search-engine-results/
I began the long & arduous task of decoupling myself from Google a few weeks back. While the smart (!) people debate about whether Silicon Valley should be regulated I’ll be doing the 1 thing I know how to, i.e. stopping my money from flowing to Google. After that I’ll have a hard look at Amazon, Paypal, etc. – ugh, this freedom crap ain’t easy.
My guess is that dailystormer.com was favored because it was a false-flag site run by the Left.
There is a lot of that going on. For instance, this well-known singer saw people getting off the same buses wearing “BLM” and “KKK” T-shirts, and acting chummy with each other until the news cameras started to arrive.
One question for the people who did the study: Did you compare other search engines? I suspect that Bing and Yahoo are just as bad. (And I’ve heard from another source that DuckDuckGo is just a front-end for Bing, and that there are doubts about its claim not to track its users.)
There is a lot of SEO research on Google ranking. Google “officially” uses about 200 parameters to determine page ranking. I guess that unofficially it also uses input from the Democrat Party, EU bureaucrats, the the UN politicians.
Is there a way to avoid google bias by using search terms including + /- or terms used when searching other data bases such as those used by government departments or libraries.
I am astonished to learn that people are apparently using Google to “search” for generic and general terms like “climate change”, or maybe “food” or “music”. Even without intentional bias, this is always bound to be a failure. That’s not what Google Search (or any search engine) is good for, because the answers to such broad-brush questions are always TOO MANY for the user to peruse them all, even with the help of the snippets provided by Google Search. The Internet generally is not very suitable for people who have no idea about a subject to begin with. At least it does not provide instant answers. Could it be that the overhang of Google-induced traffic at alarmist sites is not intentional, but caused by them peddling their goods to the Great Unwashed in tabloid fashion, so that those who google generic layman’s terms will end up with the alarmists, while those (few) that know the subject properly, who will search for technical terms or for an intelligent combination of details, are directed to the scientific-minded sites like this one. I mean, once you have found and liked WUWT, you do not google aimlessly around but either use a bookmark or (like I do because I dislike long bookmark lists) use Google Search to find WUWT and click the link there. As the original poster said already: Asking for specific information does not leave Google much leeway to bias the result. The fault is IMHO with those whose minds are too blurry to even know what the specific information is to find in Google what they need. Google Search is not a teacher, neither a well-meaning nor an ill-meaning one, just a database. What you get from it depends on YOUR skill – it’s not rocket science!
Hmmmm.Let’s see now, what can we do to help you the most? We already know that a well-reasoned and scientific argument would only fall on deaf ears because conservatives don’t want to hear anything that doesn’t match their worldview. Besides, I believe it’s been statistically proven more than once that conservatives like Fox News, the Republican Party, and Donald Trump all have a high percentage of telling lies. let’s face it, a quick tour through YouTube will yield no telling how many examples of Donald Trump saying, “I never said so and so,” with clear, unambiguous, video of a human being who looks exactly like the human being that said he never said that, namely Donald John Trump, saying exactly that.
I have been following the antics of the GOP organized crime gang since the time of Nixon. I know you think you fool a lot of people, but there are a lot of us who remember every minute ever since Nixon was kicked out of office of repeated GOP and conservative subterfuge to the point that most everyone would consider “The Boy Who Cried Wolf” quite believable compared to any of you. To be fair, there is no doubt that there was a wolf, because the wolf finally ate him!
So, all of this disingenuous pissing and moaning from conservatives has finally just become background noise. you have to be the biggest bunch of cry babies that have ever lived on the face of the planet Earth. You don’t mind having your bullies at Fox News or Mr. LimpBlab, et al. speak clearly identified and well-documented hate speech and lies through every outlet they can find 24/7.
So could we maybe tag this with a scientific name? Possibly “projection” but I’m not sure that’s really it. So, let’s keep it simple and in common language.
I believe what you’re seeing on Google and many other places, is what is often referred to as a “backlash”, often enjoyed by yours truly when watching Wile E Coyote cartoons as a child. It appears conservatives are just like Mr. Coyote in that they keep using the same failed products from the Acme company and never seem to learn anything from it. Newton’s third law is frequently a guest.
You think, after you finally pulled out the worst idiot that you can possibly find, amazingly worse than George W. Bush, and did not find a way to stop him, and worse, seem to refuse to be making any effort to stop him, a lot of people are catching on to the fact that the set known as “conservative” pretty much contains a majority of nuts. Now, we are aware that nuts are quite easily manipulated with propaganda because they have a tendency to be willfully ignorant, but there’s still quite a few of us old liberal scientists who studied other things, including literature and philosophy, who aren’t fooled by your antics. Unfortunately for you some of the #Trumpanzees are beginning to listen to us.
Thus, your inevitable downfall is in full swing. And just like a baseball coming in contact with Babe Ruth’s bat, you are headed in the opposite direction now. So I am not surprised that Google has wisely chosen to squelch some of your lies and propaganda and other subterfuge to the point that it’s harder to find because when people use Google they want to find facts, not fiction, unless their intent is to find fiction.
I believe your use of language especially the English language is sorely lacking. There is not a bias against conservatives, I believe what is going on is people are more interested in reading the truth! But I’ll be honest, over history once Liars have been fitted out into the light of day, the masses tend to eventually turn against them. But I believe the proper terminology should be Google is tending towards finding truth and attempting to eliminate some of the absolute garbage, idiocy, propaganda, corporatist manipulations, and whatever else you have you, the conservatives have been spewing for over 40 years.
Being from The South, where Nixon’s “Southern Strategy” started a coordinated campaign from that point forward of using dirty tactics in politics as much as possible, I have made good use of my front row seat watching White Supremacists and neo-nazis drive around with their Confederate flags and swastikas on their pickup trucks swearing that they will eventually win the Civil War. It’s major contributors being Lee Atwater, Karl Rove, and Satan’s own right hand man, Roger Ailes. Thank goodness that two of the three are enjoying the unusually very warm climate at their current location in the afterlife.
After all, we know the Lord works in mysterious ways, and like we assume he must have used some way to signal the animals to come to Noah’s Ark, it could be argued that Donald Trump is actually being used by the Almighty God to draw The Deplorables out into the light of day so that they can be easily caught on the many surveillance cameras all over the nation now to be identified, rounded up, and since it’s their wish anyway for Armageddon to come, send them on their merry way to really enjoy The Father of Lies, commonly referred to as “The Devil”, or “Satan”, where it’s never, ever, ever, going to be cold and I’m sure Fox News is on 24/7×infinity. Most of the poor rubes will think they are in Heaven until their favorite sports season comes around. Not being able to change the channel from Fox will give them a hint that they ain’t where they thought they was. Poor things will have to suffer watching beer commercials and pretty young ladies scantily clad while realizing that there will be no beer or girls available to them forevermore.
Seriously? You are trying to use scientific methods to somehow prove that people don’t like you, right? Why? All you have to do is talk to anyone who has an IQ of two or more digits and you will find out that people who aren’t stupid do in fact hate conservatives. As much as all of you seem to find hate so useful, us liberals thought we would return the favor. So how do you like it so far? Being hated for just being who you truly are? You don’t need a paper or statistics to know that the majority of the world population has been wise to your con game for a long time and are sick and tired of you.
This time though, you really messed up. The proof will be coming out soon of what liars and cheaters you are, and hopefully many of you end up prison where you belong. Aren’t you glad us liberals aren’t in favor of the death penalty? Because in the good old days that you seem to love so much, people planning sedition and committing treason were typically hung from the nearest tree, or removed from their prison cells before dawn, taken to the nearest solid wall, offered a blindfold, and then the command to “fire!” would be given to the firing squad. I kind of miss those old days. Just like you have to spray to the kill all the roaches to finally be rid of them, having been raised with a conservative background, I still believe that evil should be destroyed whenever it’s found.
As far as climate change is concerned, whether you believe what scientists say or not, I have a feeling that the Holy and Just Almighty God is sending us a message buy once again hitting Texas with a terrible hurricane, followed by Florida. Maybe it’s his way of talking to us now. Considering their roles in some of the worst things to happen to America, he’s sending a reminder that he can change the climate anytime he wants to, and I’m afraid conservatives have been so evil that he’s giving second thoughts to flooding the whole surface of the Earth again. my friendly word of advice would be: pay attention this time
Well done zealot’s screed, puddin. You do have the view of someone who sees their teeth from the backside (as you have your head so far up. . . ).
All “truth” is owned by your side? The mainstream media is totally reliable? No scientist has ever gone down a metaphoric rabbit hole on theory? Check your meds or reconsider your “education”.
It would be interesting if you were to compare Bing which should at least have some different algorithms or adjustments.