JAXA timing worst ever – switching Arctic Sea Ice software, right as the minimum is about to happen

Looking for an explanation of the anomalous 2week straightline during mid-August in the DMI link at WUWT, I see that DMI is revising their charting as well:
Old: http://ocean.dmi.dk/arctic/plots/icecover/icecover_current.png
New: http://ocean.dmi.dk/arctic/plots/icecover/icecover_current_new.png
Some questions seem to be raised:
http://ocean.dmi.dk/arctic/plots/icecover/osisaf_nh_iceextent_monthly-09_en.png

The new algorithm creates a downshift bias that is instantly apparent in the curves. In particular, it makes little difference in years with comparatively high sea ice at minimum, and a lot of difference in years with comparatively low sea ice minimum. This fairly obviously makes sense — if the entire arctic were ice coast to coast, the correction wouldn’t do anything at all. The more the coverage is marginal or broken up, the more opportunities there are for the correction to kick in.
The bad news for the new algorithm is that it is self-evident proof that their algorithm corrected or not is either terribly converged and hence inaccurate from the beginning or else has artifacts in it both before and after of unknown sign. There is no way a correction like this should have this large an effect unless coastal coverage is nearly fractal at minimum or unless there is a large systematic bias in the old way it was computed (which seems unlikely — nearly any averaging with sufficiently small pixel resolution for the average itself to be meaningful should not be seriously affected by repixellating the boundary. The two should scale like perimeter to area, that is, quadratically, and with anything like adequate pixellation only a small fraction of the pixels should be coastal in the first place. For example, if the entire water area were circular and the pixel size was , the ratio of coastal pixels to open water (unaffected) pixels should be . In general, the ratio of the coastal area to the interior volume should be proportional to where is the size of the coarse grained area element (pixel) and is the length scale associated with the total area. For non-smooth surfaces the constant of proportionality might well be larger than 2, but it should not in general be as much as an order of magnitude greater unless the area itself is terribly resolved because $\Delta r$ is much too large to get meaningful results in the first place.
Let’s see if this is the scaling of the results above. At first glance this is unreasonable. To put it in simple terms, to lose 250,000 square kilometers from the perimeter of a 16 million square kilometers square with sides 4000 kilometers in length, you have to remove a strip of width 250,000/16,000 = 16 km in width from the whole thing. That is, we can estimate as being order of 10 km. Are the pixels used in the sea ice estimate really that large? That’s terrible resolution. But it is a lot worse than this — in order to lose this much, the area exposed to be lost has to be much larger, because not ever coastal pixel will suddenly convert from being covered with sea ice to not being covered with sea ice. In fact, if the previous resolution were close to adequate, the only places where one could lose pixels in a biased way is at boundaries between open water and sea ice right up to the coast, where the open water can reasonably shift a pixel one way or the other around the perimeter. If the prior granularity of ice on the entire boundary was one pixel (every other pixel sea ice and open water) and every sea ice pixel disappeared, it would only be half of the pixels that were formerly ice, so the pixel size would have to be at least twice the estimate above, some 30 km, and the entire coastal boundary would have to be covered by an alternating checkerboard of ice and open water 30 km to the side. Obviously this is almost certainly not the case — it would be surprising if the granularity were less than ten times this scale, exposing only 1/10th of the pixels to “sudden” conversion from sea ice to open water. Again, even if 100% of those pixels lost the conversion algorithm decision, this means we’re up to pixels hundreds of kilometers across where I would have expected them to be order of one kilometer across at worst. If the granularity of coastal ice-covered regions was order of as little as ten kilometers in size (where hundreds is a lot more reasonable) there is simply no way to lose 250,000 square kilometers of sea ice from any algorithmic conversion, even one biased to lose every contested pixel!
Unless, of course, my Fermi-estimate arithmetic sucks (entirely possible), but it is pretty simple stuff and enough to make me wonder if they’ve done sufficient common sense checking to ensure that their algorithm change produced a reasonable result or if the shift is the result of some artifact in the code. Or, maybe the pixels from which they make the estimate are huge AND there is a bias — that would do it too, but in that case they really need to include an error estimate per point in the original graph that is equally huge because the empirical resolution of the actual data is directly related to the number of pixels that can convert from a shift like this — the shift from a simple unbiased algorithm change SHOULD HAVE been on the order of the uncertainty in the curves themselves, and should not really have been systematic. To me it looks like the algorithm is taking lots of pixels that are square in the middle of coastal regions that are fairly solidly ice covered and just randomly flipping them to free in a way that would not make sense if one smoothed from the ocean side as well.
rgb

The Land Expanded Mask, and Land Filter reference-images (dated 1996) appear to be an application of the well-known Convolution Matrix. These techniques are heavily used in image-processing, and DSP digital signal processing.
Very brief intro: http://stackoverflow.com/questions/2219386/how-does-a-convolution-matrix-work
As such, a close relative of what JAXA is doing here (convolutions are commonly used for “edge-detection”, in images) has already received intense scrutiny. The ups, downs and artifacts of the mathematical methodology have been extensively explored & characterized.
Indeed, the citation visible in the second Land Mask (image convolutions are also referred to as masks; the JAXA case may be a 2-D DSP function) is dated 1996 … so this has been around long enough to be a ‘known-known’, in the literature.
There is something resembling the Laws of Thermodynamics at work, with such digital operations. You can trade or convert one form of noise to/from another form … but entirely “disappearing” stuff you don’t like, usually breaks a fundamental principle.
The devil is in the specific side-effects of the specific manipulation employed.

JAXA has been underreporting for a long time, I thought. I prefer the 30% DMI graph, which is honestly produced by DMI’s own staff, unlike the 15% OSISAF replacement. Maybe DMI can be lobbied to keep the 30% graph going.
The satellite data is much beholden to the orientation of its polarizing lens, which is turned to suit conditions at the pole of interest — causing the Arctic & Antarctic ice growth to appear to be anti-correlated.

One other remark, after reading a few of the latter comments. Yes, the artifact could have been in the prior version, but even so the pixel resolution and bias have to be pretty staggeringly poor to have a 250,000 square kilometer (or more — Anthony is eyeballing and to my eyeballs it looks more like 300,000) effect.
The second comment is even simpler. Why doesn’t anybody ever bother to validate this at the local level? Surely one can afford to fly a plane along the coastline and take high resolution photos that allow the correct classification of every coastal pixel to be unambiguously resolved. In fact, the misclassification of water-covered ice can similarly be addressed by means of Monte Carlo sampling actual pixels from the interior volume. If I were writing ANY sort of classifier algorithm, I wouldn’t be just saying “oh, this algorithm sucks we need a new one” I’d be saying “lets get the actual precisely correct data and compare the algorithms available to see which one best approximates it”.
It doesn’t SOUND like this has been done, although I’m sure that there is a lot of literature out there beyond the mere announcement. If the algorithm has been verified by comparing its accuracy at actually getting the right answer as determined by looking at the crib sheet over a year or so, then while the disappearing ice is still puzzling and difficult to make sense of via scaling arguments, accurate as in agreement with direct observation is accurate as in agreement with direct observation. Accurate as in a supposedly “better algorithm” doesn’t mean anything at all without direct validation.
rgb