Guest Post by Willis Eschenbach
I was reading an interesting article, paywalled sadly, called “Detecting Novel Associations in Large Data Sets“, by David N. Reshef et al. It describes a subtle method for detecting relationships in datasets. Their method is called “MIC” for maximal information coefficient. The MIC coefficient measures the strength of the association of the two datasets. A value of 1 indicates a very close association, while a value of 0 means random noise. Their MIC coefficient outperforms traditional indicators for a range of complex non-linear types of relationships, including sinusoidal, circular, and multiple additive relationships. This is because it makes no assumptions about the shape or form of the association. It’s a fascinating method, one I want to learn more about.
One of their test cases involved looking for a relationship between a range of global indicators. Here is a list of their results, sorted by MIC.
Figure 1. Significant associations as indicated by the maximal information coefficient. Traditionally, the association would be measured by the Pearson coefficient.
The odd one out in this list is MIC rank 3, the association between oil consumption per person and income per person. The Pearson rank of this one was 207, while the MIC rank was 3. So I was motivated to take another look at the question of energy and development.
To do so, I used “Gapminder World”, an amazing online tool for visualizing data. Figure 2 shows an example. This is a comparison of average energy use and income, both on a per capita basis. Each country is represented by a “bubble” in the diagram.
Figure 2. Bubble plot, by country, of per capita energy use (vertical scale) versus income per person (horizontal scale). Note that both scales are logarithmic. Size of the individual bubbles shows total energy production by that country. Color of bubbles shows total oil production by that country. Units of energy use are tonnes of oil equivalent (TOE) per person per year. SOURCE
As you can see, there is a clear and quite tight linear relationship between energy use and income. This leads to an inexorable conclusion. You can’t get out of poverty without having access to affordable energy. Figure 3 below shows the same data, with larger energy producing nations identified.
Figure 3. Bubble plot, by country, of per capita energy use (vertical scale) versus income per person (horizontal scale). Note that both scales are logarithmic. Size of the individual bubbles shows total energy production by that country. Color of bubbles shows total oil production by that country. Units of energy use are tonnes of oil equivalent (TOE) per person per year. SOURCE
The bubble size shows that the US and China are about tied for top country regarding total energy production. Russia is third, the Saudis fourth, and surprising to me, India fifth. The colors show that for Russia and the Saudis most of the energy is produced from oil (red) while for China and the US coal is also a major source. India’s energy is mostly coal.
Figure 4 below shows the same basic energy vs. income chart, but in a different way. In Figure 4 the bubble size is energy production per capita, rather than total energy production. All of the bubbles are in the same location, but are changed in size.
Figure 4. Bubble plot by country of per capita energy use (vertical scale) versus income per person (horizontal scale). Note that both scales are logarithmic. Size of the individual bubbles shows total energy production per capita. Color of bubbles shows total oil production. Units of energy use are tonnes of oil equivalent (TOE) per person per year. SOURCE
We can draw some fresh conclusions from Figures 3 and 4. One is that you don’t have to produce a lot of energy, either per capita or in total, to have a modern industrial developed economy (lots of small bubbles at upper right). The Netherlands and Japan are examples. The second is that if you have high energy production per capita, it is easier to have high per capita income (preponderance of large bubbles at upper right).
The Gapminder website also allows us to look at the history of the various countries. Here is how some countries have evolved over time. Label lines show the start of each record.

Figures 5 and 6. Same as Figure 3, but showing the evolution of some countries 1971-2007. Size of the individual bubbles shows per capita energy production by that country. Color of bubbles shows total oil production by that country. “Trails” show the year by year values. Note that both scales are logarithmic. Fig. 4 SOURCE1 Fig. 5 SOURCE2:
Some comments on the historical figures. First, the direction you’d love for your country to be moving over time would be down and to the right. This would mean using less energy while making more money. Generally, almost nobody is moving in that direction overall.
The bad direction would be up and to the left. That would be using more energy to make less money. Ugly. The Saudis have moved that way in recent years.
Some countries took the worst quadrant, down and to the left. This is where you are using less energy, and you’re also making less money. Zimbabwe and the “Democratic” Republic of Congo did that. Bad sign. It’s de-development, and it generally involves suffering for both humans and the environment.
That leaves the fourth quadrant, moving up and to the right. Using more energy and making more money. Commonly called “development”, AKA getting out of poverty. Making enough money to be able to afford to protect the environment.
The game is to move to the right as much as you can (increased money) and upwards as little as you can (increased energy). So Bangladesh is not doing as well as India in that regard, since it is moving upwards more steeply. China was doing as well as India in the 70s and 80s, but has sloped upwards in the last decade of the record. Note that India is producing the majority of its energy from coal.
Russia went down and to the left in the early nineties, but has since recovered and nearly doubled its income without much increase in energy. Curiously, the income is now back to the 1990 level, but the energy use is less. The same is true of Uzbekistan and many other former members of the Soviet Empire. To their credit, they have fought back from the breakup of the Empire and returned in a more efficient form. Indeed, the Uzbeks have gone down and to the right in the last decade, and that’s the holy grail of development, doing more with less energy.
The poor Saudis, on the other hand, ended up going almost straight up (more energy used to provide the same income), and even lost a little ground. And Senegal has gone nowhere at all.
Japan, China, Mexico, and Australia have increased their per capita energy production over the period (bubble size), while the US and Russia have stayed about the same. The total oil production for the US has fallen (bubble color) while for China it has increased. Russian oil production fell and then has come back up.
The US and the UK have done a curious thing. Per capita energy use for each country in 2007 was about the same as in 1979. But income went up. Both countries nearly doubled their per capita income, with basically no increase in per capita energy use. Not sure what they are doing right, but we should figure it out and clone it …
Conclusions and final notes:
1. Development is energy, and energy is development. Although efficiency and conservation can help you, in general you must increase energy use in order to increase personal income enough to get out of poverty. If you make energy expensive, it is hugely regressive, as the poor countries and the poor people will simply not be able to afford it. Carbon taxes, “cap-and-trade”, or other energy taxes are a crime against the less favored inhabitants of our planet.
2. Large countries with higher transportation costs will use more energy per dollar of income than do small countries.
3. Within the limits of the “cloud” of countries shown in Figure 3, it is possible to increase energy efficiency, and to make more money using the same amount of energy.
4. Countries that produce lots of energy tend to waste it more than countries that produce little.
5. The preferable place for any given income level to be is on the lower edge of the “cloud” of countries with that income. This is where you get the most bucks for your bang. Many European countries are in this position. The US and Canada are about in the middle of the cloud. However, as noted they are much larger than the European countries.
6. China, India, and Bangladesh had about the same per capita income in 1971, ~ $700 per year. The differences in their current positions are large, with Bangladesh at $1,400, India at $2,600, and China at $6,000 per year.
7. Sadly, the datasets only go up to 2007 … it would be interesting to see the reduction in both energy use and income due to the global financial meltdown.
8. Finally, when someone says the word “technology”, many environmentalists think “bulldozers”. Instead, they should think “energy efficiency”. At the end of the day, technology is about doing more with less. Technology is what allows us to use less gasoline to go a mile. Through some combination of conservation and technological advances, the US and the UK were able to double their income on the same expenditure of energy. This technological advance is to the benefit of everyone including the environment.
Regards to all,
w.
PS—The source links below each chart goes to the corresponding live chart on the Gapminder website, where you can play with the variables or investigate the histories of countries other than the ones at which I looked.
TOE data comes from The World Bank.
http://www.worldbank.org/
The World Bank motto: Working for a World Free of Poverty
I wouldn’t trust their data as far as I can throw it when they have a mission statement like that.
But that’s just me.
@Gail Combs
Arm yourself with data not conjecture.
For instance:
http://minerals.usgs.gov/ds/2005/140/ironsteel.xls
http://minerals.usgs.gov/ds/2005/140/cement.xls
Steel and cement production are two of the biggest industrial energy hogs there are. US cement production, for instance, was at new record highs in this decade and only dropped beginning in 2007 with the recession. US Iron and steel production reached a record high in 1973 and then fell off a cliff as the oil crisis created a huge demand for small cheap high mileage jap cars and trucks instead of the venerable 8-cylinder big block Detroit Iron epitomized and idolized in the muscle car era (sigh, good times). The US auto industry never recovered and neither did US steel production.
Dave Springer says:
December 19, 2011 at 5:08 am
Dave, once again you’ve let your unreasoning hatred for me cloud your judgement.
Consider a flat tax of $50,000 per person. By your definition, it would not be regressive, since as you say “the price is constant”, and is the same for everyone.
However, in fact it would hit the poor the hardest.
Your problem is the assumption that a flat tax on fuel, assessed say on the basis of carbon content, somehow translates to the same percentage of their income for different purchasers. It does not. It is a larger percentage of their income for the poor. I read the other day of a lady who is paying most of her social security to heat her home.
If you think a $1 per gallon fuel tax will take the same percentage of her income that it would of Bill Gates income, you’re mad. There is no way that such a tax is “neither progressive nor regressive”. In practice, it hits the poor the hardest, and that is a regressive tax.
w.
Mike Smith says:
December 19, 2011 at 6:15 am
You should learn to read, I did not ignore efficiency and reducing waste. I said
This is because I’m talking about countries and people getting out of poverty (moving from the lower left end of the charts above to the upper right end). You’re not going to do that by insulating their homes.
w.
P. Solar says:
December 19, 2011 at 4:04 am
>>
“How about this interpretation: “richer people can afford more energy”
inexorable !!”
Chicken and the Egg argument. The age-old problem of correlation confused with causation. So which is it? Cheap energy creates wealth or wealth creates cheap energy?
There are only three ways to create wealth: “Make it, mine it, or grow it” and all three need energy. The cheaper the energy, the more wealth you can create (the more things you can mine like Lithium for batteries, the more things you can make like hybrid cars, and the more things you can grow like corn for biofuels).
In old times you’d use slaves as the cheap energy source, or you’d exploit child labor or immigrants, etc. Nasty stuff. Nowadays we can use cheap fossil fuels. Without energy, you simply can’t create wealth. So cheap energy allows you to attain wealth. Your wealth cannot create cheap energy.
Dave Springer tries to insult me by saying:
December 19, 2011 at 7:54 am
I love it when someone tries to insult me by calling me an “amateur”, because I am in fact an amateur scientist. However, I’m an amateur scientist who just published his fourth peer-reviewed scientific paper, including a “Communication Arising” in Nature magazine … so the claim redounds to my credit.
The best, though, is when some charmingly illiterate fellow tries out the insult and he spells it “amatuer” … that makes my whole day.
w.
http://www.eia.gov/pub/international/iealf/tablee1.xls
Above is energy use per dollar of GDP generated listed by nation and continent, annually, from 1980 to present.
Interestingly Eurasia sticks out like a sore thumb consuming almost twice as much energy per dollar GDP as any other region.
Europe is the most efficient. The United States is typical but has been becoming more efficient every year.
Judging by this data what brings a country out of poverty generally isn’t rising energy consumption per capita but rather rising efficiency in converting energy consumption to gross domestic product.
Technology then appears to be the key. A nation will just spend itself further into the poor house by simply consuming more fuel. The consumption has to efficiently generate GDP. Consumption isn’t, in and of itself, enough. The key is HOW it is consumed not HOW MUCH is consumed.
Dave Springer says:
December 19, 2011 at 8:15 am
As usual, and as you would know if you had followed the citations to look at the original, you’re wrong in your assumption, Dave. I didn’t make the label up myself, it’s straight from Gapminder.
So take your pissy little complaints and your incorrect assumptions to them … and next time, do your homework. You’re just making yourself look foolish with your dickish comments.
w.
Dave Springer: In tax and government (read political) terms something is called regressive if it impacts those with less income harder. For instance people argue against the sales tax on food because we all eat about the same amount rich or poor (within roundoff error, the rich do buy more expensive food, but not enough to significantly skew the numbers in the example.) IF it costs $5,000 to feed a person, with a 10% tax rate that comes to $500 dollars of tax per person each year. If a poor person makes $16,000 per year, that is 3% of there income. If a middle class person makes $50,000 that is 1% of there income. If a rich person makes $500,000 that is 0.1% of there income. Because such a tax hits the poor with a higher tax percentage the politicians call that a regressive tax. As the price is influenced by a tax and it will hit those with the least disposable income the hardest it is by political definition “regressive”. That this does not meet your current political desire, “regressive” being a political pejorative and all, does not allow you to ignore the common usage.
Your example on electricity is not a good example. Most poor people live in older, less well repaired abodes that require much more power to keep at a livable temperature. California’s insanity only works because the climate in California where most of the poor live is fairly benign. The folks with the expensive places in the mountains or who like their A/C at 68 are the ones who get hit hard, but they make the choices and can afford it. The problem is that sometimes they use the same amount of electricity keeping their new, high-efficiency house at 68 that the poor in their old, inefficient houses do to keep their places at 78.
You are being intentionally obtuse in using your definition.
Dave Springer says:
December 19, 2011 at 9:02 am
You “don’t trust their data.” But you have not pointed out a single problem with their data. Not one. You have not indicated why you think their data is bad. As far as you’ve told us, you can find nothing wrong with their data.
But you don’t like it because they are working to reduce poverty in the world?
Yeah, that’s a mean, nasty, ugly thing to do, working to reduce poverty, you’re right to groundlessly mistrust someone who does that …
You are correct, however, and thankfully so, when you say
because most everyone else is totally unaffected by whatever it is that has so badly damaged and warped your judgement. We base our judgements regarding data on whether or not the data is good and accurate, and not on who produced it. And particularly, not on the mission statement of whoever produced it.
w.
PS—Truly, I don’t get your unreasoning hatred for me, Dave, but you really should do something about it. It’s leading you to make very foolish postings whose only result is that people point and laugh at you. I doubt that that is your intention … but I can assure you that that is the usual result.
Now plot it all against External debt
http://en.wikipedia.org/wiki/List_of_countries_by_external_debt
Looks like a Ponzi scheme to me.
Dave Springer says: December 19, 2011 at 7:54 am
… Thanks for nothing, pal. This is meat for the opposition…
Dave Springer has a certain amount of rudeness but I think he also has some valid points to answer. However in your defence Willis I think this piece is tremendous, at least as a starter, and I trust we can refine our tools as we go, with help from Dave et al.
I’d like to see Bjorn Lomborg’s stats here. And I’d luuuuurve to see these tremendous Gapminder tools used to demonstrate to Gapminder’s founder that AGW is (a) near-total scam (b) put out as propaganda in which debate, and even the existence of serious challenges, is stifled and denied existence. And I’d like opinions from the statistically savvy as to its wonder-working Maximal Informational Coefficient.
Dave Springer says:
December 19, 2011 at 9:49 am
No, it’s not. Your citation is to a table of Primary Energy Consumption, not energy use per dollar of GDP as you mistakenly claim. The table is headed:
E.1 World Primary Energy Consumption (Btu), 1980-2006
(Quadrillion (10^15) Btu)
You can’t even cite your own facts right, and you call me an “amatuer”??
w.
son of mulder says:
December 19, 2011 at 10:00 am
Thanks, s.o.m. Remember, however, that the external debt is owed to someone … so you need to consider NET external debt as well.
w.
“1. Development is energy, and energy is development.”
And the biggest problem in the US is that the “progressives” are stopping all progress through litigation, for which the various levels of government pay the legal costs. Result, no power plants, no refineries, less mining of coal, less drilling for oil, etc., etc. The liberals will even approve projects knowing that their sycophants will litigate and stop them with the cooperation of liberal judges.
Willis Eschenbach says:
December 19, 2011 at 9:44 am
“…who just published his fourth peer-reviewed scientific paper, including a “Communication Arising” in Nature magazine … ”
Congrats! No small feat!
I wonder…how many publications does one need before they are no longer considered amateur. I went to grad school with plenty of people that got PhD’s and didn’t publish even 2 papers. And I guess they would be considered “professional”.
I also wonder if all the aeronautic professionals and academics at the turn of the century called the Wright brothers “amateurs”.
Heck Willis are you fast
Thanks again for this post, which makes an essential point. The last world wars were fought over energy far more than is realized IMO. Germany was trying to build a land-based pipeline to the Middle East through Sarajevo IIRC. The Brits at that point understood very well the importance of oil.
Fred H. Haynie says:
December 19, 2011 at 6:44 am
It would be better to reverse your x and y to show that the efficient use of energy produces wealth. Thus, restricting it’s use should be expected to produce poverty. Just compare food productivity methods as a function of efficient energy use per person.
______________________________________
Very good idea. The well being of a country can be measure by how much food a farmer produces.
Subsistence farming means feeds me and my family and a bit more for taxes.
In 1790, the time period the Neo-Luddites seem to want us to return to, Farmers made up about 90% of labor force. With equipment inventions this was reduced to 69% of labor force by 1800 finally dropping to 2.6% by 1990. 1840 is when factory made farm equipment became available.
It took about 250-300 labor-hours to produce 100 bushels of wheat in 1830. By 1987 it took 2-3/4 labor-hours.
THAT is what t energy use is really all about. Freeing up humans from the drudgery of producing the necesities of life (food,shelter & clothing) so they can produce more wealth. Now the UN/IPCC wants to return to shackling everyone to a farm like the serfs of old. (UN Agenda 21 considers land too important to be owned by an individual)
SOURCE: http://inventors.about.com/library/inventors/blfarm4.htm
To P. Solar:
Regarding your belief that rich countries use more energy because they are rich.
Let’s use a little logic. Assume first that energy is very expensive, so much so that only a few can afford it at all. It should be obvious that only a few people, those selling the energy, would benefit from such a situation.
Now, assume that energy is cheap and limitless. Obviously everyone would benefit economically by such a situation, except perhaps the energy producers.
Further, assume that some countries have abundant energy, while others have little and it is more expensive. Would you locate a factory of any kind in a country with limited and expensive energy, or energy that was only available when the sun shined or the wind blew?
The people that champion wind and solar energy see no problem with increasing the cost of energy, and in fact see it as a means to an end. When Obama said that “energy costs would necessarily skyrocket”, he was saying it proudly. By advocating such a thing, he is advocating a lower standard of living for everyone, and higher costs for everything.
Wilis:
Einstein was a patent clerk when he published his most important papers. Was an amateur scientist when he published his paper on Special Relativity?
Curiousgeorge says:
December 19, 2011 at 8:12 am
Apparently the EPA wants to decide what is Sustainable and what isn’t. I can hardly wait for that! 🙁
=======================================================
” EXCLUSIVE: EPA Ponders Expanded Regulatory Power In Name of ‘Sustainable Development’
Read more: http://www.foxnews.com/politics/2011/12/19/epa-ponders-expanded-regulatory-power-in-name-sustainable-development/?test=latestnews#ixzz1gzuT6GQY “
==================================
Oh great now the bureaucrats want to bypass Congress and take their orders DIRECT from the United Nations!
Why don’t we just abolish the US federal government and get rid of the parasites in DC since the UN and WTO are now writing all our laws anyway and all Congress does is rubber stamp them.
GRRrrrrr
@Gail
It’s a bit of an urban legend that the US became service oriented. We became high technology oriented. Who invented the transister, the microprocessor, the personal computer, and the internet? The information age was pretty much “Made in the USA” just as the auto industry before it was. What’s next? I’m betting on synthetic biology which will reduce the cost of energy and manufacturing by an order of magnitude over the next several decades. I think the information age is a brief but necessary predecessor to what will be a far more transformative technology.
You can produce an incredible amount of energy for very little cost if you could grow a plant that was an arbitrarily large green mat you could water with municipal waste water and it has a tap at the end of a collection network that produces peanut oil which can power a diesel engine, a boiler for an electric plant, a furnace for a home…
Evolution never produced anything like that because there’s nothing for natural selection to select for but we are getting close to being able to take what evolution has provided and cut & paste to get new things tailored to what WE want instead of what natural selection “wants”. What evolution has provided is pretty wide ranging. There are bacteria which make a living without sunlight eating iron hydroxide and crapping pure magnetic iron for instance.
http://www.ornl.gov/info/ornlreview/rev32_3/amazing.htm
So it’s only a matter of time before synthetic biology can pretty much cut & paste microbial capabilities in new ways and then direct their coordinated actions by the quadrillion. The big green living mat that produces pure peanut oil out of a tap is something that can be biologically engineered one way or another. Craig Venter is already able to assemble a working genome for a minimal free-living bacteria from a computer specification of the genetic sequence and bring it to life. It’s expensive to do but the cost/performance of automated laboratory equipment to take the time & labor out of it is improving at the rate of Moore’s Law for semiconductors. That’s probably because it’s moving towards being done on semiconductor size devices grown in same way that traditional semiconductors are made.
Just because US workforce is less blue collar doesn’t mean it’s producing less of what raises living standards. The global network for instance probably carved 20% off the typical end to end supply chain cost of a widget. When you buy something off the shelf at Walmart the relevent information instantly and seamlessly flows all over the world to everyone who played a part in manufacturing that widget or any of its pieces. All billing and payment, tracking bits in transit, all instant and paperless. That was a huge gain in productivity that the entire world benefitted from greatly. Is that “service” oriented?
Lucy Skywalker says:
December 19, 2011 at 10:02 am
Thanks, Lucy. Which points?
The MIC was actually of much more interest to me than the energy/income stuff. Any comments welcome.
w.
Keith Battye says:
December 19, 2011 at 8:29 am
I have said this several times here and elsewhere . . .
Electrify rural Africa and you will stop deforestation in it’s tracks. At the same time people’s lives will improve immeasurably….
Everything in life that is not religious depends on energy. The more affordable energy the more prosperous the population and the better their environment.
Several people I know grew up on dirt poor farms here in the USA. We are only a generatio or two away from that now in the USA:
1940
+ 58% of all farms had cars
+ 25% had telephones
+ 33% had electricity
1954
+ 70.9% of all farms had cars
+ 49% had telephones
+ 93% had electricity
1975
+ 90% of all farms had phones
+ 98.6% had electricity.
SOURCE: http://inventors.about.com/library/inventors/blfarm3.htm
More on MIC here.
http://www.exploredata.net/
Some code in both Java and R.
http://www.exploredata.net/Downloads
Just a matter of time before its ported to Cuda. Maybe if I’ve nothing else to do over the break.