Guest post by Dennis Ray Wingo
Introduction
The foundation of all observational science is data. This is true whether the data is temperature measurements from ground networks, satellites, or any other thing in nature that can be observed, quantified, and recorded. After data is recorded it must be archived so that future researchers who seek to extend or question conclusions drawn from that data can go back to the original source to replicate results. This is a fundamental premise of the scientific method, without it we can make no reliable statements about nature and call it science. This is true whether or not the subject is climate change, planetary motion, or any other scientific discipline. This missive is about the supremely important subject of data archival and how you the reader can support our lunar data archival project. First a historical digression.
The Importance of the Recording and Archival of Scientific Data
In the era before computers and the Internet, data archival was the responsibility of the scientist who obtained and recorded scientific observations. Johannes Kepler used Tycho Brahe’s archived records of meticulous observations of planetary motion to calculate the elliptical orbit of Mars and thus developed his laws of planetary motion. After the laws were published, anyone could check Kepler by going to the observatory and do their own calculations based on the archived data. The archived work of Brahe and Kepler underpinned Sir Isaac Newton’s formulation of his theory of gravity. Without archived data, Newton would have had no basis for his calculations. A scientist’s archives, stored at institutes of learning, has been the standard method of preserving data and results until the era of the computer.
Data Archiving in the Modern Age
In recent times a structural deficiency has emerged in the sciences related to the storage, archiving, and the availability of original data. Beginning in the world war two years and exploding afterward, scientific data in many fields of the physical sciences began to be obtained though electronic means. Strip charts, oscilloscopes, and waveforms from analog and digital sensors began to be fed into calculating programs, and results obtained. These results were and are used to develop and or confirm hypotheses. This exploded in the 1960’s and has continued to where today it is ubiquitous. However, there has been a decoupling in the scientific process regarding the recording and archiving of data and the ability to replicate results. The following example is just one of a legion of problems that exist in this realm.
In the 1960’s when data was obtained and fed into the computer, the data was often truncated due to memory limitations and computational speed of computers of the era. For example a paper was published by NASA as NASA TM X-55954 entitled:
The Radiation balance of the Earth-Atmosphere System Over Both Polar Regions Obtained From Radiation Measurements of the Nimbus II Meteorological Satellite;
This is probably the first definitive study of the radiation balance of the Earth-Atmosphere system published in the space era. Figure 1 is a figure from that paper:
Figure 1: Radiation Balance of the Earth-Atmosphere System From Nimbus 1966
This is an important paper in climate studies as it was the first paper to quantify the radiation balance based on data from satellites. However, the question is, where is the original data was fed into the computers to come up with these results?
Recovering the
HRIR Data
In the paper the primary data used to produce the temperature gradients was obtained from the Medium Resolution Infrared Radiometer (MRIR) that flew on the Nimbus I-III meteorological satellite, the first satellite to carry this high quality of sensor. Where is that data today? I actually don’t know much about the MRIR data but I do know quite a lot about the High Resolution Infrared Radiometer (HRIR) that was a companion experiment on the early Nimbus birds.
During the missions the data from the spacecraft was transmitted in analog form to ground stations where it was recorded and from there it was sent for processing at NASA Goddard Spaceflight Center in Greenbelt Maryland. Figure 2 shows the design of the HRIR instrument and the computerized method of processing of the data:
Figure 2a, 2b: HRIR Calibration and HRIR Data Processing
Looking at Figure 2a on the left you see that a laboratory calibration was done against a known blackbody target. An in flight calibration standard was measured at the same time and a reference calibration for the instrument obtained. The same in flight calibration reference blackbody (shown in the upper left) is scanned on each swath (a swath is a line of recording representing an 8.25 x 1100 km section of the Earth), providing a continuous means to maintain calibration of the instrument in flight. Figure 3 shows a trace of a swath of HRIR analog data:
Figure 3: Nimbus HRIR Swath Trace With and Without Calibration Stair Step
In 2009 my company, as a result of our work on the 1966 Lunar Orbiter data, was contracted by the National Snow and Ice Data Center (NSIDC) to take raw Nimbus HRIR data, correct errors, and translate it into a modern NetCDF-4 format so that it could be used in studies of pre 1979 Arctic and Antarctic ice extent. The HRIR data had been digitized by the diligent effort of NASA Goddard scientists who had retrieved the surviving tapes from the federal records center. Since no tape drives exist anymore that can read the tapes, a company was contracted to use an MRI type machine to read these low data density tapes. This worked remarkably well and the data from over 1700 of these tapes were provided to us. However, it turns out that the data tapes do not have the original analog data. It turns out that the original analog tapes no longer exist.
The digitized data that we used are, as best as we can tell, is an intermediate product derived from the IBM 1704 computer processing. The swaths no longer have the calibration stair step or sync pulses but each one does have a metadata file with geo-positioning data. We reprocessed the data and re-gridded it to comply with modern Net-CDF4 conventions. The HRIR images produced are then used by the NSIDC to find the edges of the polar ice. We took the files and translated them into .kml files for display on Google Earth with dramatic effect. Our work is described in an AGU Poster (IN41A-1108, 2009). Figure 4 is a .kml file mapped in Google Earth.
Figure 4: Google Earth .kml File of the Nimbus II HRIR Data, August 23, 1966
This image is centered near Indonesia. Bluer temperatures are colder and clearly show the Monsoon clouds. The contrast between the ocean and Australia is clearly evident. Colder temps in the Himalayas are seen as is the heat of the Persian gulf and the deep cool temperatures of the clouds in the upper right from typhoon Helen and Ida. The HRIR data can be used for many purposes but due to the loss of calibration, only a relative comparison with modern IR data can be obtained. This also renders replication of the findings of the radiation balance paper nearly impossible. So, what the heck does all of this have to do with Lunar images?
The Lunar Orbiter Image Recovery Project (LOIRP)
In 1966-67 NASA sent five spacecraft to orbit the Moon as a photoreconnaissance mission to scout landing sites for the Apollo landings. Today’s reader must remember that prior to these missions mankind had never seen the Moon up close. The first three Lunar Orbiters were in a near equatorial orbit and the last two in polar orbits for general mapping. Each carried two visible light cameras, a 24” focal length instrument obtaining images at about 1 meter resolution, and an 8” focal length instrument at about 5-7 meters resolution on the on the lunar near side. The images were recorded on 70mm SO-243 photographic film which was processed on board. This film was then scanned with a 5 micron spot beam that modulated an analog signal that was transmitted to the Earth. This is shown in figure 4:
Figure 4: Lunar Orbiter Image Capture, Scan, Transmit, Storage and Print Process
The images were captured on the Earth via two dissimilar processes. At the lower left, of the most interest to our project, was the recording of the pre-demodulated combined raw analog and digital data on a 2” Ampex FR-900 Instrumentation tape drive. The second process demodulated the signal to produce a video signal that was sent to a long persistence phosphor called a kinescope. The resulting image was photographed by a 35mm film camera. The 35mm film strip positives were then assembled into a larger sub-image that was filmed again to create a 35mm large negative that was processed to create a 35mm print that was used by the photo analysts to look for landing sites. However, as one might suspect, there was degradation of the quality of the images in going through this many steps.
I was aware of this quality reduction as I had worked with the film records in the late 1980’s at the University of Alabama Huntsville. At that time I had researched the tapes but was informed that the tapes were unavailable, though rumors were that someone was digitizing them. However, this never happened and all the archived images, such as the excellent repositories at the USGS in Flagstaff Arizona and at the Lunar and Planetary Laboratory (LPI) in Houston were derived from the films and were the only high resolution images of the Moon available.
In 2007 quite by accident I read a newsgroup posting that Nancy Evans, a retired JPL researcher, was retiring from her second career as a veterinarian and had a four FR-900 tape drives that she wanted to give away. I later found that she was the responsible official at NASA JPL in the 1980’s that had saved the original Lunar Orbiter analog tapes and that they were still in storage at JPL. I contacted Nancy and JPL and she was willing to donate the tape drives and JPL was willing to loan the tapes to NASA Ames were we had donated facilities to attempt to restore the tape drives and read the tapes. I raised a bit of funding from NASA Watch editor Keith Cowing. We loaded two trucks with the 1478 tapes weighing over 28,000 lbs and the four tape drives weighing a thousand pounds each and drove to NASA Ames.
The reason that previous efforts by Nancy Evans and engineer Mark Nelson from Cal Tech had been unsuccessful was that NASA was not convinced of the value of the original data. I had known of the tapes before but we had to quantify the benefits to NASA before we could obtain funding. We found the money quote as we called it in an obscure NASA memo from 1966. This memo said in brief (figure 5):
Figure 5: NASA Memo Regarding Superiority of Mag Tape Lunar Images
This had originally been suggested by NASA contractor Bellcomm employee Charles Byrne as a means to improve the methods that would be used to analyze landing sites for the dangers from large boulders and to analyze the slope of the landing sites. If rocks were too big or the slope more than eleven degrees, it would be a bad day for the crews seeking to land. With this memo in hand NASA headquarters provided us with initial funding to get one tape drive out of the four operational and to see if we could produce one image. We had three questions to answer.
1. Could we get a 40+ year old tape drive operational again?
2. Even if the tape drive is operational, is there any data still on the tapes?
3. Even if there is surviving data, is it of higher quality than the USGS and LPI archives of the film images?
Suffice to say we answered all three questions in the affirmative and in November of 2008 we unveiled to the world our first image, which just happened to be the famous “Earthrise” image of the Earth as seen from lunar orbit from August 23, 1966. The original image and our restored image is shown in figure 6:
Figure 6: Earthrise 1966 and Earthrise 2008!
The improvement in dynamic range we found from the documentation was a factor of four due to the reduced (250 to 1 on film vs 1000 to 1 on the tapes) dynamic range of the ground 35mm film. The raw data also preserves the sync pulses used to rectify each line of the data and when we used oversampling techniques (10x in frequency and bit depth) we can produce much larger images (the Earthrise image at full resolution is 60’ x 25’ at 300 dpi). With modern digitizing cards and inexpensive terabyte class drives this became a very manageable affair. For more information, this link is from a lunch presentation that I gave at Apple’s worldwide developer conference (WWDC) in 2009. Here is a link to an LPI paper.
Where We are in 2013
After our success NASA headquarters Exploration Systems Mission Directorate provided further funding. However, since ours was basically an unsolicited proposal that funding was limited. Each of the Lunar Orbiters (LO) acquired approximately 215 medium and high resolution images. The most important images are from Lunar Orbiter II, III, followed by LO-V, then I, then IV. The reason is that LO-II and III have the best high resolution images on the near side equatorial region. The digitized raw images best preserves the data in a form that can then be integrated into a multilayer dataset that best compares with today’s data which we have done on an experimental basis. In contrast to the Nimbus HRIR data the LO data fully preserves the calibration marks, which are on the tapes every 22 seconds. LO-I lost its image compensation sensor early in the mission resulting in blurred high resolution images. The medium resolution images are fine though they are less relevant for comparison purposes due to their lower resolution. LO-V has almost all of its high resolution images at 2 meters, thus being a good comparison to LRO. The lowest priority are the LO-IV images, which were obtained from a much higher altitude than the other missions and are thus of mostly historical value.
Our project has successfully digitized 98% of the LO-III images, with only six images lost to tape related causes (erased tapes), while we have found several images that are not in the existing USGS and LPI archives. We have so far digitized about 40% of the LO-II images, and about 10% of the LO-V, LO-IV, and LO-1 images.
We Need Your Help
We are today raising funds through the crowd funding site;
http://www.rockethub.com/projects/14882-lunar-orbiter-image-recovery-project
We are doing this as we do not expect further NASA funding and there is only a limited amount of time still available to digitize these tapes. The FR-900 tape drives use a head with four iron tips that rotate at 15,000 rpm. These heads are in direct contact with the tapes that are moving by at 12.5 inches per second, creating a sandpaper effect that quickly wears the heads down. Here is a video from a couple of years ago with a tour of the lab, which by is in an old MacDonald’s at the old Navy Base at Moffett field CA. Only a few dozen tapes can be played before the heads wear out, necessitating a refurbishment that costs well over $7000 each time.
We also have to pay our engineer to maintain the drive, our students to manage, assemble, and quality check the images as well as myself to manage the project, operate the tape drives (I worked in video production for years and thus do the operations and real time quality control during image capture). We are also preparing this data for subsequent archiving at the National Space Science Data Center though we also have the images archived at the NASA Lunar Science Institute and at our www.moonviews.com site where anyone is welcome to download them. We also have a Lunar Orbiter Facebook page that you are welcome to join.
Scientific Value
The images that we are producing and the raw data will be available to anyone for their own purposes. We have students who have been doing real science of comparing the LOIRP digitized images with the latest images from the NASA LRO mission. Why is this important? Since the Moon has no atmosphere, even the smallest meteors impact the surface and make a crater. With a resolution on both LO and LRO ~one meter we can examine the lunar surface in detail over thousands of square kilometers over a period of almost half a century. We can then see what the frequency of small impactors are on the Moon. Not only does this provide information for crew safety while out on the surface of the Moon, it provides a statistical representation of the asteroid risk in near Earth space. The bolide that exploded over Russia is thought to represent a risk of a one in one hundred year event. What if that risk is higher? Our images, coupled with the LRO LROC camera images can help to better bound this risk.
Our project has been honored by congress and our images were used in a presentation by NASA to the president in 2009 and were part of a package of NASA photos provided in the inaugural package this year. We have had extensive coverage of our efforts in what we have termed “techno-archeology” or literally the archeology of technology. Many of these links are at the end of this article. However, with all of that it is a very difficult funding environment and that is why we need your help.
What is on the Crowdfunding Site
We are offering a lot of stuff for your donation on the site. We have collectable and historical images that were printed back during the Apollo era for varying price ranges. We have models of the Lunar Orbiter with a stand, suitable for your desk. We have microfilm from the original photographs and if you cannot afford any of that, you can just make a donation!
This is what we call citizen science, the chance to have a part in an ongoing effort to archive data that can never been archived again. Our tapes are gradually degrading and the tape drives cannot function without heads. Our engineering team is comprised of retired engineers who won’t be around forever. NASA JPL in 2008 estimated that to recreate what we have would cost over $6 million dollars. We have done what we have done with a tenth of that amount of money and with your generous donation we will complete our task by the end of this September.
The Big Picture
Stories like ours regarding the actual and potential loss of valuable original data is not a rarity. Due to funding cuts to NASA on October 1, 1977 they turned off the Apollo lunar surface experiments that we spent billions putting there. The majority of the data that was obtained up until the experiments were turned off was in great danger of being lost. Retired scientists and interested parties at NASA recently put together a team that retrieved these records from as far away as Perth Australia and the NASA Lunar Science Institute has a focus group dedicated to this effort. Sadly some of this data is still in limbo and may indeed be lost forever due to poor record keeping and preservation of the original data.
For the reader of WUWT most of you are well aware of the issues associated with the adjustments of original data in the field of climate science. The integrity of science is preconditioned on the ability to replicate results and the archival of data and the preservation of that original data is one of the highest priorities in science. We are doing our small part here with the Lunar Orbiter images. One of our team members is Charles Byrne, who just happened to be the one who wrote the original memo that resulted in the purchase of the tape drives. In talking with Charlie he never in a million years thought that a generation later he would be able to work with the original data. He has developed several algorithms that we are currently using to remove instrument related artifacts from our images. Charlie is still doing original science with Lunar Orbiter images and is the author of the near side mega-basin theory.
One of the reasons that I started thinking about original data was that at the same time I was working with the forth generation lunar orbiter film in the late 1980’s Dr. John Christy was working just down the hall from me at UAH recovering satellite data from the 1970’s that for all practical purposes was the genesis of the era of the climate skeptic. Did he think that his work would have had such a long lasting effect? Just think, did Brahe in his wildest dreams think that his meticulous work would lead to the theory of gravitation? We don’t know what may come in the future from the raw data that we are preserving but we do know that having an original record from 1966-67 could not be replicated at any price and with your support we will preserve this record for posterity.
A selection of published Articles About Our Project
http://news.cnet.com/2300-11386_3-10004237.html
http://www.theregister.co.uk/2009/07/22/destination_moon/print.html
http://www.sciencebuzz.org/buzz-tags/dennis-wingo
http://news.nationalgeographic.com/news/2009/05/090505-moon-photos-video-ap.html
http://articles.latimes.com/2009/mar/22/nation/na-lunar22
http://www.nasa.gov/topics/moonmars/features/LOIRP/index.html
http://boingboing.net/2012/07/12/inside-the-lunar-orbiter-image.html
http://news.cnet.com/8301-13772_3-10097025-52.html
Apple Worldwide Developer Conference Slide Show
Wikipedia Page
http://en.wikipedia.org/wiki/Lunar_Orbiter_Image_Recovery_Project
LOIRP Gigapans
http://gigapan.com/profiles/loirp
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
This is a marvelous effort and I strongly endorse funding it [I joined myself]. Archival of original raw data is mandatory for science. In a sense data must be kept ‘forever’ and become more valuable the longer they are kept and the older they are. A sad example from my own work: The Zurich observatory archived all observations of sunspots from hundreds of observers over time. When Director Waldmeier retired in 1979 all those archives were tossed out and not a shred of the original data as archived exists any more. That makes reconstruction and recalibration and just simple checking of the sunspot number very difficult. Perhaps that was the purpose behind the destruction of the archives. Strangely enough, some people applaud this as one should ‘mess with the precious historical record’.
So, do your part and contribute!
Remember DRW, that LO and all other photo’s from the moon, Mars, etc are processed first through Malin Space Science Systems, a privately owned company set up by the gov’t, before ANYONE gets to see them. There is more than ample evidence that all the photo’s have been diddled and by the time they are released to the public we only get highly compressed jpeg’s, even if the release is in some other format.
lsvalgaard says:
February 26, 2013 at 8:31 am
Strangely enough, some people applaud this as one NOT should ‘mess with the precious historical record’.
Anthony – this article, and others like it, is one of the reasons that people will be voting for this website in the Bloggies. In fact, I already have done. Super stuff, thank you.
Fantastic effort. Thanks.
Great article and I will definitely forward some funds. This is a very important and relatively inexpensive project. I’ve bookmarked the contribution page.
I do have one question. In the paragraph that starts “For the reader of WUWT most of you are well aware of the issues associated with the adjustments of original data in the field of climate science.” and near the end states that Charles Byrne “has developed several algorithms that we are currently using to remove instrument related artifacts from our images.”, are these adjustments he is making not very similar to the adjustments climate scientists make for TOBs, instrument changes, station moves, UHI, etc.?
You’re asking for money for the recovering of these wonderful detailed images, and you’re not selling posters?
Thanks for the post. I agree, lost data is probably the greatest sin that science commits today – right along side of pimping for politics, that is. I did what I could and put in my donation. Hope everyone that takes the time to read the post does the same.
Dennis,
I recall a couple of years ago when you, I believe, asked on WUWT about the old tape drives. Many had used these in their time (mid-60s ?) but few actually knew where they disappeared to. Every time I’ve seen your name here I have wondered whether or not you were successful with the project. However, I remember also someone investigating photos of Antarctica (?) and then Climategate happened . . , … , and . . . It’s a long list.
Thanks for the work and the post. Quoting Leif: “This is a marvelous effort . . . ”
Contribution made. Now alerting friends and family that this is a cause they may wish to support.
Jason Miller says:
February 26, 2013 at 9:36 am
are these adjustments he is making not very similar to the adjustments climate scientists make for TOBs, instrument changes, station moves, UHI, etc.?
Removal of known artifacts and problems are necessary and thus vital for the science. This is OK as long as the original data still exists.
Jason Miller says:
February 26, 2013 at 9:36 am
are these adjustments he is making not very similar to the adjustments climate scientists make for TOBs, instrument changes, station moves, UHI, etc.?
Removal of known artifacts and problems are necessary and thus vital for the science. This is OK as long as the original data still exists.
If possible, please provide the name(s) of the responsible party. It may be too late to throw them a blanket party, but at least we can heap scorn and derision on their names in front of our children as an object lesson of the antithesis of what it means to be a scientist!
Great effort maybe a place you can find money is at the now shuttered National Drug Intelligence Centre it closed June 2012 but for some reason still has a 20 Yes 20 million dollar Budget! OBM reported.
Great to see efforts at allowing future scientists (and public) to be able to benefit as they should from past commitments to the importance of understanding reality through close and objective observation and recording.
Disturbing that with all the money sloshed around and over dubious undertakings on merely socially fashionable grounds that there is no meaningful interest in this or, no doubt, similar essential reference points and information.
Unsurprising that the functionaries dictating money flows care nothing for such things, and extremely unsurprising that this manifested itself in the late 1970’s as shown in this case in 1977, and in the example quoted by lsvalgaard above in 1979, as the poisonous seeping of the 1960’s “values”, “standards” and priorities into general society began to be reflected structurally.
Remarkably – or not, perhaps, as the ’60’s derived “mind” might consider this possible evidence of a sort of cosmic karma demonstrated in a minor way (and thus readily get funding to “study” it) – the above ties in with your comment made on Willis’s recent story/post “In Goal Again” or some such, the posting of which, almost contemporaneously with yours now, gave that opportunity.
That Willis evaded responding to your point regarding values (and ignored entirely my much longer comment on the same line) which is obviously abundantly demonstrated by the priorities shown in above two cases illustrated, is a wonderfully opportune illustration of the contemporary disregard for the foundational basis for anything at all, and the means by which the ’60’s mind can continue to follow its inclinations blissfully untroubled by such extraneous considerations, which can only hamper the required gratification and validation.
Credit cards only. No PayPal, etc. So much for my pittance.
ls: neither your original not corrected sentence makes sense.
typo:
notnorD.J. Hawkins says:
February 26, 2013 at 10:23 am
If possible, please provide the name(s) of the responsible party. It may be too late to throw them a blanket party, but at least we can heap scorn and derision on their names in front of our children as an object lesson of the antithesis of what it means to be a scientist!
As far as I can determine it seems to be Max Waldmeier [died 2000]. The motive? I don’t know, but my hunch is that he wanted to make sure that nobody could challenge his sunspot values [1946-1979] by going to the original sources. This a severe accusation, so take it many grains of salt.
lsvalgaard says:
February 26, 2013 at 10:09 am
Removal of known artifacts and problems are necessary and thus vital for the science. This is OK as long as the original data still exists.
I agree completely with your statement. The original data must be preserved.
Jason Miller says:
February 26, 2013 at 11:19 am
I agree completely with your statement. The original data must be preserved.
And just as important: known errors and instrumental problems must be corrected before the data is used for anything.
I’d like to add the importance of submitting code itself to the archive.
Even articles drowning in details don’t tend to provide enough details for -exact- replication unless the code is entirely ‘canned’ and documented. That is, available off-the-shelf, or OSS, or for download by the author.
And all three aren’t necessarily “archived for posterity”.
There is more than ample evidence that all the photo’s have been diddled and by the time they are released to the public we only get highly compressed jpeg’s, even if the release is in some other format.
This is absolutely NOT CORRECT for our images. I have the ORIGINAL tapes from the ground stations. We are digitizing directly from these tapes. Not only are our processed data (processed to correctly frame them and then to remove the W artifacts in the framelets ) at the original resolution available at the links above (www.moonviews and at the NASA Lunar Science Institute website), we are providing the unprocessed raw data files to the National Space Science Data center so that when people come up with improved software they can go back and improve on our work.
Alan S. Blue says:
February 26, 2013 at 11:47 am
I’d like to add the importance of submitting code itself to the archive.
I don’t think that is so important. What the original data allows is to make independent review with other code [so not reproducing any errors] possible. Wading through tens of thousands lines of code is usually not practical. Have you tried? I have.
are these adjustments he is making not very similar to the adjustments climate scientists make for TOBs, instrument changes, station moves, UHI, etc.?
No, they are not. Here is the deal.
The images as they were taken on the spacecraft are actually a 70mm negative image from both the medium and high resolution cameras. That film was scanned by a 5 micron light beam that modulated a Vestigial Sideband Modulator, which was combined with the telemetry coming from the spacecraft and then further FM modulated. The 5 micron beam scanned across a 0.11″ wide section of the film and 2″ down across the film. That resulted in several thousand line scans per framelet. A medium resolution image has 28 framelets and a high resolution image has 96 framelets.
The 5 micron scanning beam is an intense spot of white light that scanned across a rotating glass drum. Due to thermal effects of the mounts of the drum, a W pattern is seen in each line scanned. This results in a variation in the grey scale across an individual frame let. Visually it looks like striping that you can see in the original earth rise image. There are also marks that were embedded on the glass drum that produces an artifact that looks like stitching a the 0.05″ and 0.05″ from the edges of each framelet. This is an overlap between each framelet that was used by NASA to use a machine to reassemble the framelets before they were then photographed again to create the negative for the positive prints.
Charlies software does the registration between each framelet to merge the overlaps and then remove the stitches. The W pattern, which is a repetitive pattern, is also balanced out by inserting an inverse of the W pattern into the processing for each framelet.
This produces a very high resolution very nice looking print that you can see on our moonviews and NASA NLSI site.
HOWEVER
We are also providing the original raw data (around 40 terabytes), to the National Space Science Data Center (NSSDC) where it will also be available to the community. The raw data is simply the digitizer running during the operation of the instrumentation tape machine as the tapes are being read.
I’d like to add the importance of submitting code itself to the archive.
<strongI don’t think that is so important. What the original data allows is to make independent review with other code [so not reproducing any errors] possible. Wading through tens of thousands lines of code is usually not practical. Have you tried? I have.
This is an extremely difficult issue. One of the problems is that sometimes data is adjusted before it is sent to the archives with no discussion of the changes that were made or why. I had a scientist that worked on the Nimbus HRIR data tell me that directly that factors were added that were never documented. Since we no longer have the original data, we will never know what it originally looked like.
As far as computer generated data goes, one thing that is crucially important is that the metadata and the format of the data be also included in any archive. We are fortunate in our Lunar Orbiter project that NASA has a fantastic archive for documents at http://ntrs.nasa.gov where we were able to find most of the original documents for lunar orbiter.
Something that was much more difficult was the documentation for the tape drives themselves. Very little of that survives, even in the Ampex archive at Stanford. We have been very fortunate to have Kenneth Zin, Al Sturm, and some of the other old Ampex engineers to help us. If we did not have that it would have been far more difficult to make this happen. With the Nimbus data they had to use a completely new technology (a modified MRI machine) to read those tapes. This is a very real problem that science PI’s need to address in their archival efforts. It should be FORCED by the funding agencies to properly archive this data.