Guest post by Dennis Ray Wingo
Introduction
The foundation of all observational science is data. This is true whether the data is temperature measurements from ground networks, satellites, or any other thing in nature that can be observed, quantified, and recorded. After data is recorded it must be archived so that future researchers who seek to extend or question conclusions drawn from that data can go back to the original source to replicate results. This is a fundamental premise of the scientific method, without it we can make no reliable statements about nature and call it science. This is true whether or not the subject is climate change, planetary motion, or any other scientific discipline. This missive is about the supremely important subject of data archival and how you the reader can support our lunar data archival project. First a historical digression.
The Importance of the Recording and Archival of Scientific Data
In the era before computers and the Internet, data archival was the responsibility of the scientist who obtained and recorded scientific observations. Johannes Kepler used Tycho Brahe’s archived records of meticulous observations of planetary motion to calculate the elliptical orbit of Mars and thus developed his laws of planetary motion. After the laws were published, anyone could check Kepler by going to the observatory and do their own calculations based on the archived data. The archived work of Brahe and Kepler underpinned Sir Isaac Newton’s formulation of his theory of gravity. Without archived data, Newton would have had no basis for his calculations. A scientist’s archives, stored at institutes of learning, has been the standard method of preserving data and results until the era of the computer.
Data Archiving in the Modern Age
In recent times a structural deficiency has emerged in the sciences related to the storage, archiving, and the availability of original data. Beginning in the world war two years and exploding afterward, scientific data in many fields of the physical sciences began to be obtained though electronic means. Strip charts, oscilloscopes, and waveforms from analog and digital sensors began to be fed into calculating programs, and results obtained. These results were and are used to develop and or confirm hypotheses. This exploded in the 1960’s and has continued to where today it is ubiquitous. However, there has been a decoupling in the scientific process regarding the recording and archiving of data and the ability to replicate results. The following example is just one of a legion of problems that exist in this realm.
In the 1960’s when data was obtained and fed into the computer, the data was often truncated due to memory limitations and computational speed of computers of the era. For example a paper was published by NASA as NASA TM X-55954 entitled:
The Radiation balance of the Earth-Atmosphere System Over Both Polar Regions Obtained From Radiation Measurements of the Nimbus II Meteorological Satellite;
This is probably the first definitive study of the radiation balance of the Earth-Atmosphere system published in the space era. Figure 1 is a figure from that paper:
Figure 1: Radiation Balance of the Earth-Atmosphere System From Nimbus 1966
This is an important paper in climate studies as it was the first paper to quantify the radiation balance based on data from satellites. However, the question is, where is the original data was fed into the computers to come up with these results?
Recovering the
HRIR Data
In the paper the primary data used to produce the temperature gradients was obtained from the Medium Resolution Infrared Radiometer (MRIR) that flew on the Nimbus I-III meteorological satellite, the first satellite to carry this high quality of sensor. Where is that data today? I actually don’t know much about the MRIR data but I do know quite a lot about the High Resolution Infrared Radiometer (HRIR) that was a companion experiment on the early Nimbus birds.
During the missions the data from the spacecraft was transmitted in analog form to ground stations where it was recorded and from there it was sent for processing at NASA Goddard Spaceflight Center in Greenbelt Maryland. Figure 2 shows the design of the HRIR instrument and the computerized method of processing of the data:
Figure 2a, 2b: HRIR Calibration and HRIR Data Processing
Looking at Figure 2a on the left you see that a laboratory calibration was done against a known blackbody target. An in flight calibration standard was measured at the same time and a reference calibration for the instrument obtained. The same in flight calibration reference blackbody (shown in the upper left) is scanned on each swath (a swath is a line of recording representing an 8.25 x 1100 km section of the Earth), providing a continuous means to maintain calibration of the instrument in flight. Figure 3 shows a trace of a swath of HRIR analog data:
Figure 3: Nimbus HRIR Swath Trace With and Without Calibration Stair Step
In 2009 my company, as a result of our work on the 1966 Lunar Orbiter data, was contracted by the National Snow and Ice Data Center (NSIDC) to take raw Nimbus HRIR data, correct errors, and translate it into a modern NetCDF-4 format so that it could be used in studies of pre 1979 Arctic and Antarctic ice extent. The HRIR data had been digitized by the diligent effort of NASA Goddard scientists who had retrieved the surviving tapes from the federal records center. Since no tape drives exist anymore that can read the tapes, a company was contracted to use an MRI type machine to read these low data density tapes. This worked remarkably well and the data from over 1700 of these tapes were provided to us. However, it turns out that the data tapes do not have the original analog data. It turns out that the original analog tapes no longer exist.
The digitized data that we used are, as best as we can tell, is an intermediate product derived from the IBM 1704 computer processing. The swaths no longer have the calibration stair step or sync pulses but each one does have a metadata file with geo-positioning data. We reprocessed the data and re-gridded it to comply with modern Net-CDF4 conventions. The HRIR images produced are then used by the NSIDC to find the edges of the polar ice. We took the files and translated them into .kml files for display on Google Earth with dramatic effect. Our work is described in an AGU Poster (IN41A-1108, 2009). Figure 4 is a .kml file mapped in Google Earth.
Figure 4: Google Earth .kml File of the Nimbus II HRIR Data, August 23, 1966
This image is centered near Indonesia. Bluer temperatures are colder and clearly show the Monsoon clouds. The contrast between the ocean and Australia is clearly evident. Colder temps in the Himalayas are seen as is the heat of the Persian gulf and the deep cool temperatures of the clouds in the upper right from typhoon Helen and Ida. The HRIR data can be used for many purposes but due to the loss of calibration, only a relative comparison with modern IR data can be obtained. This also renders replication of the findings of the radiation balance paper nearly impossible. So, what the heck does all of this have to do with Lunar images?
The Lunar Orbiter Image Recovery Project (LOIRP)
In 1966-67 NASA sent five spacecraft to orbit the Moon as a photoreconnaissance mission to scout landing sites for the Apollo landings. Today’s reader must remember that prior to these missions mankind had never seen the Moon up close. The first three Lunar Orbiters were in a near equatorial orbit and the last two in polar orbits for general mapping. Each carried two visible light cameras, a 24” focal length instrument obtaining images at about 1 meter resolution, and an 8” focal length instrument at about 5-7 meters resolution on the on the lunar near side. The images were recorded on 70mm SO-243 photographic film which was processed on board. This film was then scanned with a 5 micron spot beam that modulated an analog signal that was transmitted to the Earth. This is shown in figure 4:
Figure 4: Lunar Orbiter Image Capture, Scan, Transmit, Storage and Print Process
The images were captured on the Earth via two dissimilar processes. At the lower left, of the most interest to our project, was the recording of the pre-demodulated combined raw analog and digital data on a 2” Ampex FR-900 Instrumentation tape drive. The second process demodulated the signal to produce a video signal that was sent to a long persistence phosphor called a kinescope. The resulting image was photographed by a 35mm film camera. The 35mm film strip positives were then assembled into a larger sub-image that was filmed again to create a 35mm large negative that was processed to create a 35mm print that was used by the photo analysts to look for landing sites. However, as one might suspect, there was degradation of the quality of the images in going through this many steps.
I was aware of this quality reduction as I had worked with the film records in the late 1980’s at the University of Alabama Huntsville. At that time I had researched the tapes but was informed that the tapes were unavailable, though rumors were that someone was digitizing them. However, this never happened and all the archived images, such as the excellent repositories at the USGS in Flagstaff Arizona and at the Lunar and Planetary Laboratory (LPI) in Houston were derived from the films and were the only high resolution images of the Moon available.
In 2007 quite by accident I read a newsgroup posting that Nancy Evans, a retired JPL researcher, was retiring from her second career as a veterinarian and had a four FR-900 tape drives that she wanted to give away. I later found that she was the responsible official at NASA JPL in the 1980’s that had saved the original Lunar Orbiter analog tapes and that they were still in storage at JPL. I contacted Nancy and JPL and she was willing to donate the tape drives and JPL was willing to loan the tapes to NASA Ames were we had donated facilities to attempt to restore the tape drives and read the tapes. I raised a bit of funding from NASA Watch editor Keith Cowing. We loaded two trucks with the 1478 tapes weighing over 28,000 lbs and the four tape drives weighing a thousand pounds each and drove to NASA Ames.
The reason that previous efforts by Nancy Evans and engineer Mark Nelson from Cal Tech had been unsuccessful was that NASA was not convinced of the value of the original data. I had known of the tapes before but we had to quantify the benefits to NASA before we could obtain funding. We found the money quote as we called it in an obscure NASA memo from 1966. This memo said in brief (figure 5):
Figure 5: NASA Memo Regarding Superiority of Mag Tape Lunar Images
This had originally been suggested by NASA contractor Bellcomm employee Charles Byrne as a means to improve the methods that would be used to analyze landing sites for the dangers from large boulders and to analyze the slope of the landing sites. If rocks were too big or the slope more than eleven degrees, it would be a bad day for the crews seeking to land. With this memo in hand NASA headquarters provided us with initial funding to get one tape drive out of the four operational and to see if we could produce one image. We had three questions to answer.
1. Could we get a 40+ year old tape drive operational again?
2. Even if the tape drive is operational, is there any data still on the tapes?
3. Even if there is surviving data, is it of higher quality than the USGS and LPI archives of the film images?
Suffice to say we answered all three questions in the affirmative and in November of 2008 we unveiled to the world our first image, which just happened to be the famous “Earthrise” image of the Earth as seen from lunar orbit from August 23, 1966. The original image and our restored image is shown in figure 6:
Figure 6: Earthrise 1966 and Earthrise 2008!
The improvement in dynamic range we found from the documentation was a factor of four due to the reduced (250 to 1 on film vs 1000 to 1 on the tapes) dynamic range of the ground 35mm film. The raw data also preserves the sync pulses used to rectify each line of the data and when we used oversampling techniques (10x in frequency and bit depth) we can produce much larger images (the Earthrise image at full resolution is 60’ x 25’ at 300 dpi). With modern digitizing cards and inexpensive terabyte class drives this became a very manageable affair. For more information, this link is from a lunch presentation that I gave at Apple’s worldwide developer conference (WWDC) in 2009. Here is a link to an LPI paper.
Where We are in 2013
After our success NASA headquarters Exploration Systems Mission Directorate provided further funding. However, since ours was basically an unsolicited proposal that funding was limited. Each of the Lunar Orbiters (LO) acquired approximately 215 medium and high resolution images. The most important images are from Lunar Orbiter II, III, followed by LO-V, then I, then IV. The reason is that LO-II and III have the best high resolution images on the near side equatorial region. The digitized raw images best preserves the data in a form that can then be integrated into a multilayer dataset that best compares with today’s data which we have done on an experimental basis. In contrast to the Nimbus HRIR data the LO data fully preserves the calibration marks, which are on the tapes every 22 seconds. LO-I lost its image compensation sensor early in the mission resulting in blurred high resolution images. The medium resolution images are fine though they are less relevant for comparison purposes due to their lower resolution. LO-V has almost all of its high resolution images at 2 meters, thus being a good comparison to LRO. The lowest priority are the LO-IV images, which were obtained from a much higher altitude than the other missions and are thus of mostly historical value.
Our project has successfully digitized 98% of the LO-III images, with only six images lost to tape related causes (erased tapes), while we have found several images that are not in the existing USGS and LPI archives. We have so far digitized about 40% of the LO-II images, and about 10% of the LO-V, LO-IV, and LO-1 images.
We Need Your Help
We are today raising funds through the crowd funding site;
http://www.rockethub.com/projects/14882-lunar-orbiter-image-recovery-project
We are doing this as we do not expect further NASA funding and there is only a limited amount of time still available to digitize these tapes. The FR-900 tape drives use a head with four iron tips that rotate at 15,000 rpm. These heads are in direct contact with the tapes that are moving by at 12.5 inches per second, creating a sandpaper effect that quickly wears the heads down. Here is a video from a couple of years ago with a tour of the lab, which by is in an old MacDonald’s at the old Navy Base at Moffett field CA. Only a few dozen tapes can be played before the heads wear out, necessitating a refurbishment that costs well over $7000 each time.
We also have to pay our engineer to maintain the drive, our students to manage, assemble, and quality check the images as well as myself to manage the project, operate the tape drives (I worked in video production for years and thus do the operations and real time quality control during image capture). We are also preparing this data for subsequent archiving at the National Space Science Data Center though we also have the images archived at the NASA Lunar Science Institute and at our www.moonviews.com site where anyone is welcome to download them. We also have a Lunar Orbiter Facebook page that you are welcome to join.
Scientific Value
The images that we are producing and the raw data will be available to anyone for their own purposes. We have students who have been doing real science of comparing the LOIRP digitized images with the latest images from the NASA LRO mission. Why is this important? Since the Moon has no atmosphere, even the smallest meteors impact the surface and make a crater. With a resolution on both LO and LRO ~one meter we can examine the lunar surface in detail over thousands of square kilometers over a period of almost half a century. We can then see what the frequency of small impactors are on the Moon. Not only does this provide information for crew safety while out on the surface of the Moon, it provides a statistical representation of the asteroid risk in near Earth space. The bolide that exploded over Russia is thought to represent a risk of a one in one hundred year event. What if that risk is higher? Our images, coupled with the LRO LROC camera images can help to better bound this risk.
Our project has been honored by congress and our images were used in a presentation by NASA to the president in 2009 and were part of a package of NASA photos provided in the inaugural package this year. We have had extensive coverage of our efforts in what we have termed “techno-archeology” or literally the archeology of technology. Many of these links are at the end of this article. However, with all of that it is a very difficult funding environment and that is why we need your help.
What is on the Crowdfunding Site
We are offering a lot of stuff for your donation on the site. We have collectable and historical images that were printed back during the Apollo era for varying price ranges. We have models of the Lunar Orbiter with a stand, suitable for your desk. We have microfilm from the original photographs and if you cannot afford any of that, you can just make a donation!
This is what we call citizen science, the chance to have a part in an ongoing effort to archive data that can never been archived again. Our tapes are gradually degrading and the tape drives cannot function without heads. Our engineering team is comprised of retired engineers who won’t be around forever. NASA JPL in 2008 estimated that to recreate what we have would cost over $6 million dollars. We have done what we have done with a tenth of that amount of money and with your generous donation we will complete our task by the end of this September.
The Big Picture
Stories like ours regarding the actual and potential loss of valuable original data is not a rarity. Due to funding cuts to NASA on October 1, 1977 they turned off the Apollo lunar surface experiments that we spent billions putting there. The majority of the data that was obtained up until the experiments were turned off was in great danger of being lost. Retired scientists and interested parties at NASA recently put together a team that retrieved these records from as far away as Perth Australia and the NASA Lunar Science Institute has a focus group dedicated to this effort. Sadly some of this data is still in limbo and may indeed be lost forever due to poor record keeping and preservation of the original data.
For the reader of WUWT most of you are well aware of the issues associated with the adjustments of original data in the field of climate science. The integrity of science is preconditioned on the ability to replicate results and the archival of data and the preservation of that original data is one of the highest priorities in science. We are doing our small part here with the Lunar Orbiter images. One of our team members is Charles Byrne, who just happened to be the one who wrote the original memo that resulted in the purchase of the tape drives. In talking with Charlie he never in a million years thought that a generation later he would be able to work with the original data. He has developed several algorithms that we are currently using to remove instrument related artifacts from our images. Charlie is still doing original science with Lunar Orbiter images and is the author of the near side mega-basin theory.
One of the reasons that I started thinking about original data was that at the same time I was working with the forth generation lunar orbiter film in the late 1980’s Dr. John Christy was working just down the hall from me at UAH recovering satellite data from the 1970’s that for all practical purposes was the genesis of the era of the climate skeptic. Did he think that his work would have had such a long lasting effect? Just think, did Brahe in his wildest dreams think that his meticulous work would lead to the theory of gravitation? We don’t know what may come in the future from the raw data that we are preserving but we do know that having an original record from 1966-67 could not be replicated at any price and with your support we will preserve this record for posterity.
A selection of published Articles About Our Project
http://news.cnet.com/2300-11386_3-10004237.html
http://www.theregister.co.uk/2009/07/22/destination_moon/print.html
http://www.sciencebuzz.org/buzz-tags/dennis-wingo
http://news.nationalgeographic.com/news/2009/05/090505-moon-photos-video-ap.html
http://articles.latimes.com/2009/mar/22/nation/na-lunar22
http://www.nasa.gov/topics/moonmars/features/LOIRP/index.html
http://boingboing.net/2012/07/12/inside-the-lunar-orbiter-image.html
http://news.cnet.com/8301-13772_3-10097025-52.html
Apple Worldwide Developer Conference Slide Show
Wikipedia Page
http://en.wikipedia.org/wiki/Lunar_Orbiter_Image_Recovery_Project
LOIRP Gigapans
http://gigapan.com/profiles/loirp
This is a marvelous effort and I strongly endorse funding it [I joined myself]. Archival of original raw data is mandatory for science. In a sense data must be kept ‘forever’ and become more valuable the longer they are kept and the older they are. A sad example from my own work: The Zurich observatory archived all observations of sunspots from hundreds of observers over time. When Director Waldmeier retired in 1979 all those archives were tossed out and not a shred of the original data as archived exists any more. That makes reconstruction and recalibration and just simple checking of the sunspot number very difficult. Perhaps that was the purpose behind the destruction of the archives. Strangely enough, some people applaud this as one should ‘mess with the precious historical record’.
So, do your part and contribute!
Remember DRW, that LO and all other photo’s from the moon, Mars, etc are processed first through Malin Space Science Systems, a privately owned company set up by the gov’t, before ANYONE gets to see them. There is more than ample evidence that all the photo’s have been diddled and by the time they are released to the public we only get highly compressed jpeg’s, even if the release is in some other format.
lsvalgaard says:
February 26, 2013 at 8:31 am
Strangely enough, some people applaud this as one NOT should ‘mess with the precious historical record’.
Anthony – this article, and others like it, is one of the reasons that people will be voting for this website in the Bloggies. In fact, I already have done. Super stuff, thank you.
Fantastic effort. Thanks.
Great article and I will definitely forward some funds. This is a very important and relatively inexpensive project. I’ve bookmarked the contribution page.
I do have one question. In the paragraph that starts “For the reader of WUWT most of you are well aware of the issues associated with the adjustments of original data in the field of climate science.” and near the end states that Charles Byrne “has developed several algorithms that we are currently using to remove instrument related artifacts from our images.”, are these adjustments he is making not very similar to the adjustments climate scientists make for TOBs, instrument changes, station moves, UHI, etc.?
You’re asking for money for the recovering of these wonderful detailed images, and you’re not selling posters?
Thanks for the post. I agree, lost data is probably the greatest sin that science commits today – right along side of pimping for politics, that is. I did what I could and put in my donation. Hope everyone that takes the time to read the post does the same.
Dennis,
I recall a couple of years ago when you, I believe, asked on WUWT about the old tape drives. Many had used these in their time (mid-60s ?) but few actually knew where they disappeared to. Every time I’ve seen your name here I have wondered whether or not you were successful with the project. However, I remember also someone investigating photos of Antarctica (?) and then Climategate happened . . , … , and . . . It’s a long list.
Thanks for the work and the post. Quoting Leif: “This is a marvelous effort . . . ”
Contribution made. Now alerting friends and family that this is a cause they may wish to support.
Jason Miller says:
February 26, 2013 at 9:36 am
are these adjustments he is making not very similar to the adjustments climate scientists make for TOBs, instrument changes, station moves, UHI, etc.?
Removal of known artifacts and problems are necessary and thus vital for the science. This is OK as long as the original data still exists.
Jason Miller says:
February 26, 2013 at 9:36 am
are these adjustments he is making not very similar to the adjustments climate scientists make for TOBs, instrument changes, station moves, UHI, etc.?
Removal of known artifacts and problems are necessary and thus vital for the science. This is OK as long as the original data still exists.
If possible, please provide the name(s) of the responsible party. It may be too late to throw them a blanket party, but at least we can heap scorn and derision on their names in front of our children as an object lesson of the antithesis of what it means to be a scientist!
Great effort maybe a place you can find money is at the now shuttered National Drug Intelligence Centre it closed June 2012 but for some reason still has a 20 Yes 20 million dollar Budget! OBM reported.
Great to see efforts at allowing future scientists (and public) to be able to benefit as they should from past commitments to the importance of understanding reality through close and objective observation and recording.
Disturbing that with all the money sloshed around and over dubious undertakings on merely socially fashionable grounds that there is no meaningful interest in this or, no doubt, similar essential reference points and information.
Unsurprising that the functionaries dictating money flows care nothing for such things, and extremely unsurprising that this manifested itself in the late 1970’s as shown in this case in 1977, and in the example quoted by lsvalgaard above in 1979, as the poisonous seeping of the 1960’s “values”, “standards” and priorities into general society began to be reflected structurally.
Remarkably – or not, perhaps, as the ’60’s derived “mind” might consider this possible evidence of a sort of cosmic karma demonstrated in a minor way (and thus readily get funding to “study” it) – the above ties in with your comment made on Willis’s recent story/post “In Goal Again” or some such, the posting of which, almost contemporaneously with yours now, gave that opportunity.
That Willis evaded responding to your point regarding values (and ignored entirely my much longer comment on the same line) which is obviously abundantly demonstrated by the priorities shown in above two cases illustrated, is a wonderfully opportune illustration of the contemporary disregard for the foundational basis for anything at all, and the means by which the ’60’s mind can continue to follow its inclinations blissfully untroubled by such extraneous considerations, which can only hamper the required gratification and validation.
Credit cards only. No PayPal, etc. So much for my pittance.
ls: neither your original not corrected sentence makes sense.
typo:
notnorD.J. Hawkins says:
February 26, 2013 at 10:23 am
If possible, please provide the name(s) of the responsible party. It may be too late to throw them a blanket party, but at least we can heap scorn and derision on their names in front of our children as an object lesson of the antithesis of what it means to be a scientist!
As far as I can determine it seems to be Max Waldmeier [died 2000]. The motive? I don’t know, but my hunch is that he wanted to make sure that nobody could challenge his sunspot values [1946-1979] by going to the original sources. This a severe accusation, so take it many grains of salt.
lsvalgaard says:
February 26, 2013 at 10:09 am
Removal of known artifacts and problems are necessary and thus vital for the science. This is OK as long as the original data still exists.
I agree completely with your statement. The original data must be preserved.
Jason Miller says:
February 26, 2013 at 11:19 am
I agree completely with your statement. The original data must be preserved.
And just as important: known errors and instrumental problems must be corrected before the data is used for anything.
I’d like to add the importance of submitting code itself to the archive.
Even articles drowning in details don’t tend to provide enough details for -exact- replication unless the code is entirely ‘canned’ and documented. That is, available off-the-shelf, or OSS, or for download by the author.
And all three aren’t necessarily “archived for posterity”.
There is more than ample evidence that all the photo’s have been diddled and by the time they are released to the public we only get highly compressed jpeg’s, even if the release is in some other format.
This is absolutely NOT CORRECT for our images. I have the ORIGINAL tapes from the ground stations. We are digitizing directly from these tapes. Not only are our processed data (processed to correctly frame them and then to remove the W artifacts in the framelets ) at the original resolution available at the links above (www.moonviews and at the NASA Lunar Science Institute website), we are providing the unprocessed raw data files to the National Space Science Data center so that when people come up with improved software they can go back and improve on our work.
Alan S. Blue says:
February 26, 2013 at 11:47 am
I’d like to add the importance of submitting code itself to the archive.
I don’t think that is so important. What the original data allows is to make independent review with other code [so not reproducing any errors] possible. Wading through tens of thousands lines of code is usually not practical. Have you tried? I have.
are these adjustments he is making not very similar to the adjustments climate scientists make for TOBs, instrument changes, station moves, UHI, etc.?
No, they are not. Here is the deal.
The images as they were taken on the spacecraft are actually a 70mm negative image from both the medium and high resolution cameras. That film was scanned by a 5 micron light beam that modulated a Vestigial Sideband Modulator, which was combined with the telemetry coming from the spacecraft and then further FM modulated. The 5 micron beam scanned across a 0.11″ wide section of the film and 2″ down across the film. That resulted in several thousand line scans per framelet. A medium resolution image has 28 framelets and a high resolution image has 96 framelets.
The 5 micron scanning beam is an intense spot of white light that scanned across a rotating glass drum. Due to thermal effects of the mounts of the drum, a W pattern is seen in each line scanned. This results in a variation in the grey scale across an individual frame let. Visually it looks like striping that you can see in the original earth rise image. There are also marks that were embedded on the glass drum that produces an artifact that looks like stitching a the 0.05″ and 0.05″ from the edges of each framelet. This is an overlap between each framelet that was used by NASA to use a machine to reassemble the framelets before they were then photographed again to create the negative for the positive prints.
Charlies software does the registration between each framelet to merge the overlaps and then remove the stitches. The W pattern, which is a repetitive pattern, is also balanced out by inserting an inverse of the W pattern into the processing for each framelet.
This produces a very high resolution very nice looking print that you can see on our moonviews and NASA NLSI site.
HOWEVER
We are also providing the original raw data (around 40 terabytes), to the National Space Science Data Center (NSSDC) where it will also be available to the community. The raw data is simply the digitizer running during the operation of the instrumentation tape machine as the tapes are being read.
I’d like to add the importance of submitting code itself to the archive.
<strongI don’t think that is so important. What the original data allows is to make independent review with other code [so not reproducing any errors] possible. Wading through tens of thousands lines of code is usually not practical. Have you tried? I have.
This is an extremely difficult issue. One of the problems is that sometimes data is adjusted before it is sent to the archives with no discussion of the changes that were made or why. I had a scientist that worked on the Nimbus HRIR data tell me that directly that factors were added that were never documented. Since we no longer have the original data, we will never know what it originally looked like.
As far as computer generated data goes, one thing that is crucially important is that the metadata and the format of the data be also included in any archive. We are fortunate in our Lunar Orbiter project that NASA has a fantastic archive for documents at http://ntrs.nasa.gov where we were able to find most of the original documents for lunar orbiter.
Something that was much more difficult was the documentation for the tape drives themselves. Very little of that survives, even in the Ampex archive at Stanford. We have been very fortunate to have Kenneth Zin, Al Sturm, and some of the other old Ampex engineers to help us. If we did not have that it would have been far more difficult to make this happen. With the Nimbus data they had to use a completely new technology (a modified MRI machine) to read those tapes. This is a very real problem that science PI’s need to address in their archival efforts. It should be FORCED by the funding agencies to properly archive this data.
Yes, I have.
The particular value of the original code isn’t necessarily in the “wading through” the code line-by-line so much as being able to see certain crucial lines and thus in elimination of sources of error.
When you have two people nominally applying exactly the same method and coming up with different answers, it is mighty handy to be able to “debug” and go “Aha! They used Runge-Kutta (or whatever), but they only used floats!” Therefore one can say “I’m more confident in my results and here’s why.”
Otherwise you end up with “Person A says X”, “Person B says Y”. If they’re black boxes, you have have quite a few people finding “Y” and hearing “You are doing it wrong. Somewhere.”
I don’t have a credit card.
You have a little over a month to make some other payment form that I can use.
DaveE.
Alan S. Blue says:
February 26, 2013 at 12:19 pm
being able to see certain crucial lines
And how to find those? and miss some obscure detail somewhere that makes a big difference? Or scale it up to millions of lines of code which modern systems approach. The only way to be sure is to replicate the analysis with new code. This may be impossible because meta data is missing. Granted that meta data can sometimes be extracted from the old code, but that can be extremely difficult. The code can be subtle. A real-life example [from a COBOL program I once saw]: MOVE A TO B, making a local copy of A. Problems was that A was signed and B was not. The code used -1 for missing data, so you see the problem: all the sudden B = 1 became good data even if A was missing.
There are companies that can coat the recorder heads with CVD diamond. The life of the heads is extended considerably.
“We loaded two trucks with the 1478 tapes weighing over 28,000 lbs and the four tape drives weighing a thousand pounds each and drove to NASA Ames.”
The bandwidth of this kind of data transfer surely beats the fastest link available on the internet today.
There are companies that can coat the recorder heads with CVD diamond. The life of the heads is extended considerably.
These are not that type of head. What you are talking about works on heads like for an 8 track player or other low frequency heads. These heads have to have a flat response out to 20 MHz.
If you have any links that say otherwise we are always willing to listen.
Folks
Here is an example of an image that we recently captured.
individual subframes
http://lunarscience.arc.nasa.gov/files/LOIRP/5041_H1.tif
http://lunarscience.arc.nasa.gov/files/LOIRP/5041_H2.tif
http://lunarscience.arc.nasa.gov/files/LOIRP/5041_H3.tif
assembled subframes
http://lunarscience.arc.nasa.gov/files/LOIRP/5041_FULL.tif
This is the same as this image at the Lunar and Planetary Institute website derived from the film data.
http://www.lpi.usra.edu/resources/lunarorbiter/frame/?5041
The difference is readily apparent.
On the very relevant question about “transmitting” data at Fedex speeds ….
The real answer for various storage devices is under the http://www.xkcd.com website, answering that very question about Fedex bandwidth: Read the complete explanation at http://what-if.xkcd.com/31/ The short-term answer follows:
FedEx Bandwidth
If you want to transfer a few hundred gigabytes of data, it’s generally faster to FedEx a hard drive than to send the files over the internet. This isn’t a new idea—it’s often dubbed SneakerNet—and it’s how Google transfers large amounts of data internally.
But will it always be faster?
Cisco estimates that total internet traffic currently averages 167 terabits per second. FedEx has a fleet of 654 aircraft with a lift capacity of 26.5 million pounds daily. A solid-state laptop drive weighs about 78 grams and can hold up to a terabyte.
That means FedEx is capable of transferring 150 exabytes of data per day, or 14 petabits per second—almost a hundred times the current throughput of the internet.
Sneakernet
Yep. Here are the specs for the FR-900
http://www.google.com/url?sa=t&rct=j&q=ampex%20fr-900%20brochure&source=web&cd=1&ved=0CDIQFjAA&url=http%3A%2F%2Fimages.spaceref.com%2Fnews%2F2011%2FFR-900.Brochure.pdf&ei=cD8tUdudNefZ2QXinIHAAg&usg=AFQjCNHNKzZSThx8X4wkzOEj2aCSn5FDlg&sig2=wUtNxzqLYJ4hKrnf15kX1g&bvm=bv.42965579,d.b2I
The recorder was the highest density data storage device of the 1960’s. When used in a digital mode it recorded 20 megabits/sec. 20 x 60 x 3600 = 73.913 gigabytes per hour. Each tape is a one hour tape so multiply by 1478 = 109.243 terabytes.
The drive from Moorpark (where the tapes were in storage) to Ames is about 420 miles and we did it in about 8 hours. Thus the Sneakernet rate was about 30 gigabits/sec.
A marvellous project, I’ve signed up and will be forwarding this link to friends and family.
Did you really say one meter resolution? Reading between the lines …
This had originally been suggested by NASA contractor Bellcomm employee Charles Byrne as a means to improve the methods that would be used to analyze landing sites for the dangers from large boulders and to analyze the slope of the landing sites. If rocks were too big or the slope more than eleven degrees, it would be a bad day for the crews seeking to land.
The bold fragment suggests to me the existence of elevation data. Is this project only about images, or is there any elevation data also, and if so are you planning to recover the elevation data too? A 1m DEM of the lunar surface, even just a part of it, mapped to similar resolution images mapped to that terrain … for someone with a hobby interest in digital elevation mapping, that would be a Very Good Thing.
denniswingo says:
February 26, 2013 at 2:12 pm
assembled subframes
http://lunarscience.arc.nasa.gov/files/LOIRP/5041_FULL.tif
at 2GB this is too big for my image viewer…
Brian H. and Dale E., both sans credit cards:
Are you also sans friends who actually do have credit cards? You slip’em the green, they make the equivalent donation. They might even throw in some extra for such a cause.
The bold fragment suggests to me the existence of elevation data. Is this project only about images, or is there any elevation data also, and if so are you planning to recover the elevation data too? A 1m DEM of the lunar surface, even just a part of it, mapped to similar resolution images mapped to that terrain … for someone with a hobby interest in digital elevation mapping, that would be a Very Good Thing.
Derek
The slope information was derived indirectly from only the images as we did not have laser altimeters at that time. Here is a great document related to the effort that Bellcomm was doing for NASA at the time. Google “project slope” for more information.
http://ntrs.nasa.gov/search.jsp?R=19660029202
We actually do have some marvelous 1 meter resolution LIDAR data that has come from the Lunar Reconniassance Orbiter in its mission. We have been doing some work in the polar regions where the data has the highest density and have been having a great time with it. Also check out the LROC imaging camera that has made some awesome pictures of the Moon, the only ones comparable with Lunar Orbiter.
at 2GB this is too big for my image viewer…
You can download the subframes and view them separately.
They are quite amazing…
As old as that tape is, it’s possible the lubrication has dried up resulting in additional wear on the heads. You might try contacting tape manufactures to see if they can suggest a solution to restore the lubrication on the tape before you attempt to read them.
Why bother archiving data? In another few decades the full effects of the economic and social damages done by King Obama and the EU technocrats on western nations will result in food & energy riots, anarchy, chaos and an end to liberal democracies. The complete break down of western society will render the question of science moot. Looking forward to the great asteroid strike, or what ever other natural calamity strikes, when we will find out how woefully unprepared human kind is to adapt to natural environmental change. Our brains are already significantly smaller than the much smarter early modern man, and our socialist education system is out to ensure that evolution in reverse continues by making competition an unheard of concept, We are being turned into HG Wells’ Eloi. The progressives seem to not be aware that being forced to adapt to rapid environmental changes was the driving force behind the evolution of our large brain. I would be very surprised if, when the time comes, more than a handful of people in our civilization will have the cognitive and physical capacity to adapt, and adapt they will have to, as the one constant in the earth’s history is change and the extinction of those who can not adapt.
lsvalgaard says:
February 26, 2013 at 12:54 pm
Alan S. Blue says:
February 26, 2013 at 12:19 pm
being able to see certain crucial lines
And how to find those?
==========
Without the code there is no possibility that you can find the error in the original work. The original result is your reference data. Your new result is what you compare to the reference. If there is a mismatch then the hunt is on. You are looking for either the error in your new code or the error in the old code.
A secondary issue is the problem with code revisions. Code is rarely static. So by archiving the code one of the first questions to be answered is whether the archived code can generate the archived result from the archived raw data. Or as more likely, was the result generated with an earlier version of the code than the version archived?
I’m with Alan on this. The code has huge value as does the data from an archive point of view. However, like old data there are problems in simply running old code. You may find that your CP/M version of VisiCalc with all the original data and formulas won’t run on any machine you have available.
Just wow! Thanks Dennis. I’m a plant phys guy by training but got roped into the AGW thing via my father’s collab with Fred Singer (Unstoppable Global Warming book). While far afield of my training, the discussions here always remind me of fundamental science basics. Data is key. Without it we’re lost. Truly at the mercy of ideologues. And keeping data for future scientists is just as critical.
A few years back, I had to nearly sue to get North Carolina to cough up its water quality data. Why did the NC DENR not want me to have the taxpayer-funded data? Because it showed that the hog farmers in the eastern part of the state had not ruined the water quality (it was better 10 years after 500% increase in hog population in the watersheds than before significant hog farming ops). When I tried to repeat the analysis in other states (to defend their farmers from scurilous eco-wacktivist attacks), I found that was impossible because the states had never collected the data in the first place (b/c it was another unfunded mandate of the Feds, so the states ignored!). But even NC was hard because they only had 8 years of data from 1970s. Without data, the politicos and wactivists can make up any lie they want and were getting away with it until we showed them there was no there there.
The funny thing is, farmers were afraid of the data and most told me if it’d been up to them, they never would have wanted the WQ data collected — and i just exploded with ernestness back to them, “The data is your only defense! Without it, you’re screwed!”
And that is SOOOOO true with the GW issue, as we here at Watt’s site know so well.
Again, THANKS Dennis! 🙂
From school days, I gained a love of science. I continue to read about science and its principles.
While this data endeavor is great, the entire post, and comments on the post, show weaknesses in the general conceptions of science as held by fellow science lovers.
First of all: “DATA” IS PLURAL.
Yes, I am shouting. Please stop showing your ignorance, or weakness in grammar. “Datum” is singular, and “data” is plural. To say, “Data is necessary…” is to be wrong. Wrong. Data are necessary. A datum is necessary, etc.
This error is repeated a lot here. I will post a couple other issues regarding science once I review this post to capture the obvious ones.
“I do have one question. In the paragraph that starts “For the reader of WUWT most of you are well aware of the issues associated with the adjustments of original data in the field of climate science.” and near the end states that Charles Byrne “has developed several algorithms that we are currently using to remove instrument related artifacts from our images.”, are these adjustments he is making not very similar to the adjustments climate scientists make for TOBs, instrument changes, station moves, UHI, etc.?”
Yes. If you have seen the raw data from temperature stations you’ll quickly understand that raw data is filled with errors. -15000C, the same figure repeated for days. Wrong units, about 10-20 different classifications of mistakes. Then there are the changes in operation.
changing the time of observation ( something almost unique to the US ) and changing the instrumentation.
What most of us have argued for is this.
1. a copy of the raw data where it exists. We term this level 0 data or the first report.
2. a copy of the tool,code,proceedure, used to make the adjustment
3. Proper treatment of the uncertainties.
With TOBS I can tell you that skeptics have looked at this adjustment three ways from sunday and the adjustment is required. JerryB, a commenter at ClimateAudit had his independent analysis posted at John Daly’s. That file is still there. I re ran his entire analysis back in 2007-08
when we discussed TOBS at Climate audit. The adjustment for TOBS is needed. When you change the time of observation it does change the min/max recorded. AT first I could not see how, but after going over the data prepared by a skeptic, I was convinced. Later I would look at CRN data and find the same thing. Change the time of observation from one time to another and you change Mins and Maxs. Almost 6 years later people still discuss TOBS as if it were a conspiracy. Yet the data sits there at John Daly’s for anyone to look at. To prove it to yourself.
The CRN data sits there, so you can prove it to yourself. But 99.9% of people refuse to lift a finger to prove it to themselves. They want somebody else to prove it to them
Let me tell you why. Most folks know that you can fight any argument anyone raises. Any argument. You cant convince me of something I refuse to be convinced of. But if I do the work myself. If I look at it myself, then I have no choice but to accept my work..
So.
Start here http://www.john-daly.com/tob/TOBSUM.HTM
Download the files. 190 stations with hourly data. The files will show you what happens if you change the Time of observation.
If you dont want to look at the data for yourself. Then ask yourself why not.
Back in 2007 I thought TOBS was a crock. It made no sense. I took the data.
I looked for myself. I didnt ask someone to prove it to me, I proved it to myself.
First step. recognize that an adjustment is needed. If you cant prove that to yourself by looking at the data, then there really is no point in discussing it
“The foundation of all observational science is data.”
I have no idea what “observational science” is, apart from science itself. Science necessarily requires observations. Those are compared to theory/hypothesis-based predictions to judge the degree that we ought to continue entertaining the hypothesis, or discard it altogether, or work with it (a la Hegelian dialectic).
Possibly, what is meant is observational analyses, where an investigator does not manipulate an independent variable, as contrasted with experimental analysis, where the scientist does manipulate the IV.
We cannot manipulate stars. We can.however, set up hypotheses about their nature, and figure out how any hypothesis might be tested by an observation, then conduct the oservation, then evaluate the fit of the observed data with what the hypothesis would have predicted, and so have a scientific test of phenomena we cannot manipulate.
That is the difference between dealing with observational and experimental data. I am guessing this is what meant by “observational science.” Much of climatology is observational analyses, while things such as the recreation of global warming with CO2 in soda bottles and various light exposures would be experimental.
Steven Mosher says:
“If you dont want to look at the data for yourself. Then ask yourself why not.”
“After data is [sic] recorded it must be archived so that future researchers who seek to extend or question conclusions drawn from that data can go back to the original source to replicate results.]
Sigh. Replicability does not refer to an ability to re-run some mathematical analysis on the numeric data; it refers to the scientific assumption that the nature of everything in the universe has some underlying essential nature that is physical, and that causes lead to effects in orderly, lawly, predictable ways, and so that if an observation was carried out, of natural phenomena, in once place, and was offered up as “knowledge,” then it ought to be replicable eslewhere – ultimately, at any locale in our universe.
Mathematically, if you take the same data and run the same operations, tautologically, and by definition, you wil get the same result. Proofs are proofs. Laws and rules of math are laws and rules. Unless you make a mistake with your pencil, or the computer program has been changed, the same analysis run with the same data by the same operations will yield the same results time after time.
All that to say that the “replicability” ethos of science is not a matter of re-running the same data, but of being able to gather yet another set of data that is predicted by a theory, and yet again testing the hypothesis.
This is fundamental, essential science.
IF you can get the ‘media’ (5 1/4″ or 8″ disks etc) read, “Virtual machines” can do it …
Google: CP/M “virtual machine” z80
.
When discussing science, this ethos or value is rarely made explicit, but is is essential: honesty.
The reason to make data available is so that others can figure out whether you are being honest, or deceptive.
In my mind, this is a big deal. It is flaunted a lot.
This is often described as “transparency.”
If you won’t “show your work,” in my humble opinion, you are not conducting “science.”
I believe that whole-heartedly. Even though that has huge ramifications. If you develop a secret formula – an adhesive, or a drug, perhaps – you cannot keep it private, and also consider it “scientific.”
A patentable drug: possibly, you could contract with an outside group to replicate your study while agreeing to not reveal your secret. That could move “authority” (“trust me”) toward “science,” but it still would not be science.
CERN, etc. Sure, you should be able to benefit from your original work. Why give away the data the day after you publish?
I naively believe that science requires this. Otherwise, “knowledge” and “evidence” are a matter of “authority,” not “science.”
“Trust us – we ran calculations and found the Higgs Boson – but you cannot examine our raw data or algorithms.”
That is not transparency. It is just a tough aspect od science that it does not have, built-in, a way to ensure career or commercial success for some observation. But science is not a means for building a career or commerical success. It is about gaining knowledge, and that alone.
Enough of my opinions for now.
Steven Mosher and lsvalgaard
I worked with large, often error filled, data sets for years. I was paid to “manipulate” the data so it was usable. I always kept the original data, but used the “manipulated” for analysis. To have known errors in the data and not correct for those errors would give an incorrect interpretation of that data. My point is that the original data needs to be kept and then needs to be edited thoroughly to be useful.
Steven Mosher says:
February 26, 2013 at 7:04 pm
…
If you dont want to look at the data for yourself. Then ask yourself why not.
Back in 2007 I thought TOBS was a crock. It made no sense. I took the data.
I looked for myself. I didnt ask someone to prove it to me, I proved it to myself.
…
———————————–
This is one of the reasons I know Steve Mosher is one of the good guys. I wish people would remember before dishing out the horsehockey that Mosher obviously looks at the data and accepts what it tells him, like it or not. Not saying the guy is God or that he doesn’t make mistakes. I am saying it’s clear he does his homework. I’m willing to bet Mosher doesn’t particularly enjoy having to eat the abuse from people who not only don’t want to hear it but who also aren’t willing or able to go figure out the data themselves and show the same level of integrity he does standing by the results, popular or not.
Thanks Steve.
thelastdemocrat says:
February 26, 2013 at 6:56 pm
First of all: “DATA” IS PLURAL.
What is correct is what people actually use. Language is about communication and as long as everybody agree what the word means it doesn’t matter that the word is ‘wrong’. Similar arguments can be made about visa, agenda, criteria, and many others.
Jason Miller says:
February 26, 2013 at 7:46 pm
My point is that the original data needs to be kept and then needs to be edited thoroughly to be useful.
Absolutely, and then the edited data has also to be kept, because we don’t want for everyone to do the same editing over and over. Now, one could make the argument [and I would lean in that direction] that data that is KNOWN to be wrong [e.g. a value for November 31st – which I recently came across] be deleted up front and not be carried forward forever.
As old as that tape is, it’s possible the lubrication has dried up resulting in additional wear on the heads. You might try contacting tape manufactures to see if they can suggest a solution to restore the lubrication on the tape before you attempt to read them.
Dena
Thanks. We actually have talked to the guys that designed the heads from Ampex, since most of them are still local to Silicon Valley. These particular heads were purchased by the government and when they recorded the data they did it with a lot of what is called “tip penetration”. When you read them you have to use a similar level of tip penetration, which leads to a lot of wear. I have on staff an engineer with almost 50 years worth of experience with these machines and there are some things that you just have to deal with. Go to the website and see some of the videos to see how the heads operate. Thanks for your idea though.
I’m with Leif.
Yes, we all learned and know that “datum” is the singular and “data” the plural form of the Latin word, but we don’t speak Latin, English is a remarkably (and historically) maleable language, and in today’s modern communication, “data” has (like it or not, “correct” of not) become the term used to describe “all the measurements” made scientifically. And we refer to “it”, being the block/chunk/pile/stack/?? of measurements.
So quible all you want, but it isn’t productive or even remotely impressive of your scientific knowledge. It’s just petty.
Cheers and graciousness to all. 🙂
Alex Avery
I am the former System Administrator for the data archive for the Hubble Space Telescope. The policy for the Hubble was that all data became public domain 1 year after it was captured. Anybody can get an account and request data and start looking for hidden secrets of the galaxy. No need to be part of any team or show your academic qualifications. Also, if you publish a paper using Hubble data, you can be sure that soon after it is published somebody will be checking your work with the exact same data. That is real science!
Absolutely, and then the edited data has also to be kept, because we don’t want for everyone to do the same editing over and over. Now, one could make the argument [and I would lean in that direction] that data that is KNOWN to be wrong [e.g. a value for November 31st – which I recently came across] be deleted up front and not be carried forward forever.
Leif
This is exactly what happened to the Nimbus data. For example the Nimbus HRIR instrument data was only calibrated down to about 210 kelvin. In our searching we did find that one of the things that the data acquisition software on the CDC computer did is if a temperature reading of 210 kelvin was read the software would automatically change the number to 190 kelvin to show that it was out of the calibration range. This way that number could be looked for and the data tossed as it was known to not be within the range of calibration of the instrument.
There is no reason to go back and to that effort a second time.
No concise reason given why a TOBs adjustment is necessary … of course, it is easy to understand the case where an “average” temperature calculation could be affected by a “time of observation” change where the two read values do not occur on or ‘in’ the same ‘day’ period before and after a “time of obs” change or shift … but a “strict” max or min temp reading? Only if the **reading device** were affected in some material way.
To that end, I can see cases where a (1) min/max Mercury thermometer might have an issue (given mechanics and fluid ‘friction’ etc) involving the little indicator trapped inside …. but not an electronic min/max ‘capture’ device IF said electronic device readout was (2) digital in nature (i.e., an A/D converter driving a numeric display, which THEN captures the min/or max reading) as opposed to: (3) an analog peak-detect circuit (in which the cap holding the detected ‘charge’ would be prone to inevitable leak-down) which drives a D’Arsonval meter movement. Cases (2) and (3) probably use thermistors (from a cost standpoint), which seem to have some small amount of hysteresis I should note.
Cases (1) and (3) above could have notable min/max issues. Case (1) requires mechanical resetting after a read. Case (3) would/might have to hold a ‘charge’ for 20 plus hrs until the next reading point.
Case (2) is both friction-less, ‘leak-less’ and hysteresis-less (exc for the thermistor) in operation.
Otherwise, it boggles the mind … (so the explanation might be, after all, a ‘data’ trick, but don’t quote me yet.)
.
“Alan S. Blue says:
February 26, 2013 at 12:19 pm”
Totally correct! This is my experience too. Shame Phil Jones was so sloppy in the UEA CRU office moves in the 90’s where all their raw data was lost.
@ thelastdemocrat Feb 26 2013 7.28 pm.
Your focus on what the actual nature of science is in your various posts is what is needed, whether your particular comments are considered accurate or not.
It is currently considered to be defined, to the public and politicians, and to some or a significant degree, to a portion of practitioners themselves, by the process of accreditation and ratification itself, and the ability to attract sufficient funds to carry out a of course of action.
Thus for example, on this site, a contribution may be made that meets the core requirements – observably so – of science yet may come from a source (person, institution) that in effect disqualifies it from consideration by those for whom it is convenient to do so.
In this it has a commonality with many fields of contemporary endeavor, which is universally negative in effect. An expression of functionaries not expertise.
Your point about this being at heart about honesty and the ability to confirm that is absolutely true not just in science but across the board. Whilst this might have always been an issue, it has now established itself as normal that an individual or group “owns” what they claim to be knowledge and this has one instinct behind it and only one end: the degradation of the possibility of any knowledge existing at all.
I suspect that in the contemporary culture even those who do understand and practice the pursuit of knowledge in a rigorous manner are slightly discomforted by the prospect of complete openness.
All people and processes come to reflect whatever dominant system in place to some degree – it is unavoidable. So for example, someone doing great work in a private sector company might not be in a hurry to address such issues, because they might not readily see the implications of an alternative.
@ lsvalgaard. Feb 26 2013. 7.59 pm. and @ Alex Avery 8.34pm.
You are correct in the observation that language is always changing.
Also that if is there is a unanimity of understanding exists, there is no confusion as to meaning within the parameters that any given word or phrase can embrace at that time.
However you are assuming, in saying it doesn’t matter, that any change in usage in itself reflects nothing as to the ability of that language to communicate something of use. That the redefined meaning has an equivalent (or superior) if different utility. Or you are saying that the original distinction now has no value.
In this case, you are saying that a distinction between singular and plural is no longer of importance. This constitutes a broadening of definition at the cost of precision.
Rather than having adjusted the meaning of the word “data” to better meet a contemporary interpretation of what “data” by its nature might be, the adjustment extinguishes a core reality within its traditional meaning whilst adding nothing.
This is the antithesis of the process of discrimination that has allowed any language to develop beyond grunts in the first place. Or any development in mathematical or other language.
An indifference to precision in one element of any process is indicative of a lack of precision in others. A loss of the capacity to describe anything in detail is always negative. What language – of brevity – is now used to say: “Look: I mean that particular thing”?
This change in “data” has been the result of ignorance and pretension, it has not occurred to meet a new requirement for precision. It is observable – in fact it is one of the defining characteristics of the age – that words that sound “important” and “technical” have been purloined by those simply seeking to elevate their position.
The practice of precision or otherwise is a question of discipline of the mind. Life is habit.
It may well be convenient to use “data” generically. It is also sloppy. That has implications.
Whilst a certain age or type of person might know and retain awareness of the derivation and distinction between “data” and “datum” it is a certainty that the average 30 yo does not. In fact they are probably unaware of the existence of “datum” at all.
This cannot be divorced from the readiness for all and sundry: marketers, politicians, activists, anyone, from throwing around any collection of numbers and collation of statements and calling this “data” to legitimize their claims. A lack of precision has allowed anything that is supposedly independent of any proponent to be “data”.
In reality, this is where the alteration in meaning has come from, not from scientists.
If the desire is to have people in any field encouraged and ultimately judged on their capacity to focus and discriminate – that is to fully exercise their intelligence- this matters.
js says: “Whilst a certain age or type of person might know and retain awareness of the derivation and distinction between “data” and “datum” it is a certainty that the average 30 yo does not. In fact they are probably unaware of the existence of “datum” at all.”
The only really prominent usage of “datum” that the average 30 yo observes today is the name of a surface-matching spheroid. Its plural is “datums”.
As a foreigner I can tell you than it is not my inability to pronounce words correctly or to use a correct grammatical form that makes me sound foreign. People know I am not “from around here” when they notice I do not violate the standard English grammar in exactly the same way they do. When I use a commonly recognised foreign word in its original, unadapted, foreign meaning, it gives them a similar impression.
Following on from above post @ 3.40 am
Alex Avery @ 8.34 pm.
“And we refer to “it”, being the block/chunk/pile/stack/?? of measurements.”
Are you sure that you are in effect making the point you want to make?
Or are you providing evidence – bizarrely, within a very short statement of claim to the opposite – that supports my above post of 3.40 pm in principle if not in detail?
I wish that archiving of scientific data were mandatory. I’ve seen terabytes of original, irreplaceable data from spacecraft being tossed into dumpsters for “lack of storage space”. Many more terabytes of deep-space probe data (i.e., Pioneer, Voyager) sit on 9-track tapes, slowly disintegrating, never even looked at by the PI. Shameful. Should be a crime.
What an excellent article. I was aware of this project, having found out about it earlier via some link or other, but had not followed up on my initial reading. It is really heartening to read about those who are still interested enough to try to restore the discarded original. Thanks to a involved, happy to add a modest sum to the ongoing effort..
c says:
February 27, 2013 at 3:40 am
In this case, you are saying that a distinction between singular and plural is no longer of importance. This constitutes a broadening of definition at the cost of precision.
I’m saying that it doesn’t matter much. I have never seen a situation where the distinction between datum and data has been a source of confusion. Nor of visum and visa, etc. When people lack precision in their expressions it is most often because of lack of precision of thought rather than of language.
jc, I’m saying essentially the same thing as Leif: that in specific discussions,” I have never seen a situation where the distinction between datum and data has been a source of confusion.”
That said, I agree with you about the devolution of science to claims of legitimacy based on title/institution/other not-so-relevant stuff rather than the facts/data. I am constantly amazed at how politics and emotion trump observable reality and provable facts. I spent 20 years of my life banging my head against a bureaucratic/political/media machine bent on misdirecting the uninformed, ignorant public to further their agenda (agendums, agendum? see, such quibling can actually get in the way of communication!).
Thanks for the respectful discussion, JC.
Cheers,
AAA
This comment is tangentially related to the above post by Mr. Wingo. I noticed that the paper, NASA TMX 55954 linked above, was co-authored by Fritz Moller. That reminded me of a search we did on CA a few years ago for a paper by Moller on CO2 concentration and radiation balance in JGR vol. 68 issue 13, July 1, 1963. Issues 13-24 were inexplicably missing from the online edition, but I eventually tracked down a physical copy. Weart’s book, “The Discovery of Global Warming” lists it as a discredited paper.
Anyway, the last 12 issues (but still missing issue 21) in vol. 68 are now online at Wiley having been added finally on Jan. 29. Several of these old papers from issue 13 deal with items of interest to readers of WUWT as they are from the “International Symposium on Trace Gasses and Radioactivity.”
This comment is tangentially related to the above post by Mr. Wingo. I noticed that the paper, NASA TMX 55954 linked above, was co-authored by Fritz Moller.
A lot of these papers are available online at ntrs.nasa.gov, especially if you have the title or paper number. NTRS is updated frequently so what may not be there today could be there in six months.
lsvalgaard says: February 26, 2013 at 8:31 am
This is a marvelous effort and I strongly endorse funding it [I joined myself]. Archival of original raw data is mandatory for science. In a sense data must be kept ‘forever’ and become more valuable the longer they are kept and the older they are.
_______________________________________
A nice sentiment, Leif, but are there any ‘forever’ storage mediums? Modern data all seems so ‘fragile’.
Is anyone investing in ‘forever’ storage, not just for data, but also for all our historical and cultural data and imagery??
.
Silver Ralph,
There are no forever mediums. When I worked on the Hubble, the long term media we used had a lifetime of 50 years. When I left, we were about to go to our 3rd storage medium. Each time the a new storage medium was introduced, all of the data was moved to the new medium. The problem is not the medium, the problem is having something that can physically read the old medium. When I left, we were using Sony WDD-600 drives with 6.4GB WDM6DL0 WORM drives in large jukeboxes. This drive was was first built in 1989 and was state of the art at the time. The previous disks were larger and only held 2GB.
As for this article, the Hubble archive is now called MAST and includes data from many other missions, I wonder what it would take to get them to take this data also? You could contact them at archive@stsci.edu and ask. I am sure your data would not be much of a load and I bet they might like the positive public relations it would bring.
Sralph, what about this from Hitachi? (storing binary data as dots on thin sheets of quart embedded in glass)
http://www.google.com/url?sa=t&rct=j&q=forever%20data%20storage&source=web&cd=1&cad=rja&ved=0CEgQFjAA&url=http%3A%2F%2Fphys.org%2Fnews%2F2012-09-japan-hitachi.html&ei=jmAuUZDXG5OG9gSW9IGgBw&usg=AFQjCNGF6o5Em_1kYJYNSKPi1GJB34aj7w
I’ll take the category: “Forever storage medium” for $1,000 Alex …
What is an “IBM Tape Farm”?
(A living, breathing entity which lives for the sole purpose of periodically copy and ‘rotating’ the ‘tape’ datasets under scheduled robot control unto perpetuity and also offering ‘interface’ utilities e.g. Librarian)
.
oops, dots in layers on thin sheets of quartz glass — currently up to 4 layers per “sheet” of glass, but more layers per “sheet” is apparently possible.
Silver Ralph says:
February 27, 2013 at 10:48 am
Is anyone investing in ‘forever’ storage, not just for data, but also for all our historical and cultural data and imagery??
Yes this is an ongoing concern and an active research area. Digital data can live forever if moved [in suitable format to a new medium in time.
An interesting article for us amateurs to read on Nimbus 1:
“Polar Exploration with Nimbus Meteorological Satellite” by Popham and Samuelson in “Arctic” Dec. 1965. Includes AVCS photos of Petermann Glacier, the Weddell Sea, McMurdo Sound, dry valleys, calving ice shelves. There was great interest in the Nimbus program as an ice reconnaisance tool at the time. Have you done all that can be done to recover the Nimbus data or are you essentially done due to the missing tapes?
In my opinion, the media is not the problem for data archiving. I have a non-archival quality CD-ROM disk I created in 1998 that still works(media was over $2 per disk in 1998). The problem is the availability of something to read the old media. Long term storage media usually involves some mechanical reader in a non-standard form factor that rapidly becomes out of date and reaches commercial end of life. And none of this media has the form factor longevity of the CD-ROM. So, the problem is that the moving parts of the reader will fail long before the media. The only good way I have seen to make an archive last is continue to migrate to new technologies and move your old data to these new technologies.
Folks, questions and comments are good, but please pitch in $$$ a bit more. The progress is slow.
Leif
Thanks a lot. We are working very hard on this and if you folks would at least toss us $10-20-30 bucks it would really help.
I just had a meeting today about our Nimbus HRIR images that we mapped to Google Earth KML and it may be used as an input into hurricane path studies to see how they changed from 1966 until today. This may bring some money in to help, but we have put a lot of time and effort into acquiring these old data sets and providing them to the world. You can’t bitch about what the other side is doing if you won’t even lift your hands to help efforts at providing original data, wherever the chips may fall.
Have you done all that can be done to recover the Nimbus data or are you essentially done due to the missing tapes?
Our effort was to take the HRIR TAP files (derived from the MRI scans), of the Nimbus II instrument, regrid them to a modern grid format compatible with modern climate data sets, and then convert them to Net CDF-4 format. We did that and since the instrument on Nimbus I and Nimbus III have the same format, then the NSIDC would do the rest of the conversions.
About two years ago someone at the Federal Records Center at Suitland MD found the film output (after all the computer processing was completed). It is my understanding that NASA paid the NSIDC to scan some if not all of those AVCS images. Here is a search term to some of the great work that Dave Gallaher and Walt Meir have done recovering these images for their applications to climate science.
1960s Nimbus Sea Ice Extent – Project Basis – NOAA
It is my understanding that all of this data is at some point going to be archived and made available.
Here is the image that is in my article above, viewed instead from over the central pacific. Overlaid on the HRIR image mapped to Google Earth are hurricane path traces from a Japanese source. These multilayer data sets are extremely valuable to integrate disparate researcher’s work into a coherent gestalt for studying climate, hurricanes, and other earth-atmosphere science.
http://www.panoramio.com/photo_explorer#view=photo&position=0&with_photo_id=86712068&order=date_desc&user=141000
The data from Apollo’s ALSEP is in much worse state:
http://poikiloblastic.wordpress.com/2013/02/22/the-long-road-to-alsep-data-recovery/
“Records show something like 3270 ARCSAV tapes made between April 1973 to February 1976 were sent to WNRC. Records also show a massive withdrawal of analog tapes from WNRC, prompted by the tape shortage in 1980, which included ~2800 ARCSAV tapes as Goddard Space Flight Center staff searched high and low for reusable/recyclable tapes. Fortunately, the 7-track digital tapes were not their target (the Apollo 11 landing footage tapes were probably not as fortunate), but instead of returning them to WNRC the ARCSAV tapes were stored in the basement of GSFC. Many of these were later destroyed in a 1990 building flood, and the trail of surviving tapes goes cold after they were removed from the basement of GSFC during cleanup.”
1980 tape shortage?
o.O
Yikes.
Tony
Yea we know about that. The story is much darker than what this article intimates. It is a very sad tale.