Breaking news on the ISEE-3 mission: it may not be lost – it's those "O" rings again

A few days ago, my heart sank when I heard this news: Space Probe Might Lack Nitrogen to Push It Home It seemed a deal-killer if the nitrogen tanks were empty. Now, one of the team leaders has given me an inside track into the issue, and the mission may not be lost, thanks to the collective consciousness of the Internet.

We all know of the importance of a simple thing like an “O” ring, which Dr. Richard Feynman showed was the singular cause of the Challenger disaster, due to cold.

Now, it is “O” rings again, due to high temperature. Workarounds are being engineering as I write this.

Dennis Wingo, team member for the ISEE-3 reboot mission writes on his blog:

We Are Borg: Crowdsourced Engineering and the Collective Mind of the Internet

In the science fiction universe of Star Trek, set several hundred years in the future, when we are a spacefaring civilization, humanity encounters a species called the Borg.  The Borg are a conglomeration of species who are assimilated into a collective mind numbering in the hundreds of billions.  All of the borg are connected to each other through a communications link that allows each of them to share each others thoughts, though in a manner that erases individuality.

This week, with the call that our ISEE-3 reboot team put out to the internet for help in debugging our propulsion system problem, I have come to realize that a significant portion of humanity has reached a Borg like state, one where the internet has become a collective mind for communications and knowledge sharing.  We still have our individuality, we can still decouple at will from the collective mind, but in a way that few philosophers or technologists have envisioned, we are connected in a way never before thought possible.  The implications are staggering, and here is how our little ISEE-3 project is an example of the operation of the collective mind.

ISEE-3 Trajectory Correction Maneuver Attempt

On July 8th and 9th the ISEE-3 reboot team attempted to fire the radial thrusters on the spacecraft.  Figure 1 shows the configuration of the thrusters:

The thruster firing on the 8th was a follow up to the successful spin up maneuver that was performed on July 2nd, 2014.  The engine firing on both the 8th and the 9th was unsuccessful in producing the thruster needed to change the course of the spacecraft to what is necessary for the lunar flyby on August 10th.  After the largely unsuccessful Trajectory Correction Maneuver (TCM), on the 8th, our team decided on a strategy to both use alternate thrusters and to troubleshoot the propulsion system.  On the 9th the TCM was also largely unsuccessful (a few pulses out but no substantial change in course).  After a bit of team depression that the mission was over, a failure investigation was started.  Following are the results of that failure investigation, and our plan for recovery.

First TCM Attempt July 8th

After the successful July 2nd spin up maneuver we had high hopes for the TCM.  We had learned quite a bit about the operation of the spacecraft and the spin up maneuver made us and our NASA partners confident in the next steps.  Indeed it was the successful spin up maneuver that allowed NASA to give us permission for the TCM.  However, as events would unfold, this confidence was to be misplaced.

As those who have read of our exploits know, the ISEE-3 is a spinning spacecraft.  It spins quite rapidly, at 19.75 (19.787 right now) RPM or about one rotation every 3.04 seconds.  The thrusters for the propulsion system that allows for an in plane change in velocity are located on panels 1 and 9 (redundant) of the spacecraft.  This is shown in figure 2:

Figure 3 shows the schematic of the propulsion system from the same AIAA paper from 1979:

The spacecraft has two completely redundant fuel, latch valve, and thruster systems.  The thrusters for dV maneuvers are located on the exterior of the spacecraft around the circumference as shown in figure 2.  These are the “radial dV” thrusters.  For the maneuver on the 8th we used thruster’s F and N, which are connected to the HPS 1 tank system.  Everything proceeded normally as we followed the worksheets provided by our documentation.  Things went much better from a commanding perspective as we were able to leave the transmitter and power amplifier in transmit mode at Arecibo while we received telemetry pipelined to us over the internet from the radio telescope at Bochum Germany, which is operated by the AMSAT DL team there.

When we fired the thrusters for the TCM maneuver, initially things looked great, but soon we saw a fall off in thruster performance as monitored by an accelerometer designed for this purpose.  This is shown in figure 4:

The upper graph is of the accelerometer output during the three firings.  The middle graph is the plot of the Fine Sun Sensor (FSS), which gives orientation.  The bottom graph is the telemetry value indicating whether or not the Latch Valve opened or not.   At this scale it is hard to see the actual acceleration values and they are in counts, not m/sec squared.  It is quite clear the fall off in thrust in the first maneuver.  We now know that the latch valve was not opened (we did not then).  Later, and before we did the second attempt, we see the sun angle decrease.  This so far has not been explained adequately.

The third attempt is the most interesting.  We started wondering about the latch valve and then sent the command for it to open again.  It opened.  The difference is that we did not have the +28 volts on the first time as none of this is in the procedures that we currently have available.  After we commanded the latch valve to open with + 28 volts on we saw telemetry confirmation opening.  Figure five shows what we saw in detail regarding the FSS sensors:

While this is probably hard to read on a screen but figure six is toward the end here when we first got confirmation that the latch valve between the tanks and the propellant lines was opened:

The latch valve opens at 9200 seconds and there is an immediate effect on the accelerometer and the FSS sensor.  The data rate is 8 hz for the accelerometer and 1/8th hz for the FSS sensor so there is a lot of aliasing but the influence is clear to see.  This is normally in the form of vibrations on the spacecraft that gives the appearance of a change in FSS pointing, though not in the longer term.  The end result is that we never got the thrust that we expected on July 8th, whether or not we had indications of latch valve open.

Evaluation of July 8th Attempt and Plan for the 9th

Our evaluation of what happened that day is different than what we know now but it informed how we operated on the ninth and thus is instructive to deciphering the nature of the overall problem.  Our evaluation that day was that we thought that the hydrazine tanks in fuel system 1 (on the left in figure 3) was depleted of fuel and pressure.  Our pressure transducers indicated that there was no pressure in fuel system 1 and only minimal (about 4 psi) in fuel system 2.  We also were very unsure of the telemetry indicator for the operation of the latch valves.  Thus the plan was to first attempt to do the maneuver with the fuel system 1 (but use latch valve C rather than A) and thrusters E and M on the other side of the spacecraft.  This would have the effect of using a different set of fuel lines but with the same fuel system tanks.  Then we would then repeat the test using fuel system 2 and the same thrusters (E,M).  If this did not work then we would open latch valve B on fuel system two and repeat with thrusters (E,M).  If that did not work then the final test would be to use latch valve D with thrusters (F,N).

The Attempt on the 9th

The results on the 9th were pretty much the same as on the 8th.  We initially made the same mistake by commanding latch valve C when the 28 volts was not on.  The results were basically as they were the day before.  Figure 7 shows the result of the first firing:

As you can see, the uppermost blue line is the accelerometer output.  It looks pretty much like the trace in figure 4.  It is opposite in sign because we used thrusters E,M, and sector 556 (the other side of the spacecraft) to start pulsing.  The pale flesh covered lines are where we pulsed (21-30) the latch valve.  There were other thruster firings done, but none of them with any effect.  This ended the pass for July 9th.

Failure Investigation

After a short bout of team depression, we got started on our failure investigation.  In learning how to be an engineer, it is just as instructive to study failure, and many times more so, than to study success.  I have read virtually every failure investigation in the last 50 years for in space failures, from the early Ranger days to GEO comsats, and Shuttle failures.  There is a common thread that not too many people write about, which is that these failure investigations almost always include people from outside of whatever organization was building or flying the system that failed, and that these guys were normally the best in the business.  Another common denominator of such failure investigations it that they take time and lots of money.  Our team has neither.

We seriously thought that there were no options going forward, but in order to understand what happened, and to see what we could learn, we decided to dig into the telemetry and see what happened and to see if we could determine the failure mechanism.  No one on our team is an experienced hydrazine expert.  My own expertise is more in communications, avionics, and power systems, with a good bit of experience in ion propulsion.  Marco Collelouri, our controls and AOCS engineer is very smart, but without a lot of experience.  Thus I felt, after receiving a few emails from people who offered suggestions on what might have happened, decided to throw the problem out to the world.  I was astonished at the response.

The Collective…..

On the 10th we threw out a few questions related to some suggestions that had been sent to us by some of our global distributed network of supporters unsolicited.  Keith put this on NASA Watch and http://www.spacecollege.org.  We immediately started getting responses.  While, as one might expect, many of them were uninformed, though enthusiastic, some were from the most qualified professionals in the world.  I am not going to name names at this time and without their permission but literally, the very top tier of experts started weighing in.

We sent them telemetry, and information on the spacecraft.  Fortunately there is a lot of information out there on ISEE-3, especially from the AIAA (see our references at the end of the article).  With this, and with our telemetry that is also public, we started to build a picture of a set of plausible failure modes and the state of the system.  I must stress that I have had multiple interactions with different groups of experts, some from industry, some from government, and some international.  I have not shared a lot of the information from one group to the other in order to get their unbiased responses, until a consensus started to emerge.  We also were able to be put in touch with (again, through a senior industry engineer), with some a retired person from TRW, the company that built the propulsion system in the first place.  Even though this person did not personally build the one from ISEE-3 he did know the general design and had extensive experience with hydrazine propulsion systems.

The Conclusions of the Ad Hoc ISEE-3 Hydrazine Propulsion System Failure Study

After just a few days of consultations with these groups, and a consensus started to form some common elements stand out.  Here is what we know about the state of the HPS system.  Please refer back to all previous figures as this explanation unfolds.

Integrity of the Propulsion System Finding 1

It is quite clear from the telemetry that both fuel system 1 and 2 downstream of the latch valves was still fully pressurized.  If this had not been the case, we would have not had a successful spin up maneuver, nor would we have had the initial thrust from both fuel systems on the 8th and the 9th.  This means that the following hardware is working:

1. All thrusters have their seals intact and the thrusters provided impulse, showing that the catalyst beds are also intact (or at least mostly so).

2. The propellant lines downstream of the latch valves are also intact and were fully pressurized.

3. Temperatures in the system are high, in some cases, in excess of what is desirable, though not dangerously so.  Temperatures in the propellant tanks are well within bounds.

Here is what we don’t know.

1. How much pressure is in the tanks (the options are failed sensors or depleted tanks).

2. The true status of the latch valves.

Procedures, Finding 2

We made a procedural mistake.  That mistake was turning the latch valves on without + 28 volts applied.  In our defense this was not in the AOCS procedure worksheet, though looking back into the old Mission Operations Plan, checking their status was part of the procedure.  We thought that when we did not have an indication of operation, that the telemetry was dead, which is also the case in the communications system.  Sometimes we imply the state of the spacecraft when telemetry is dead, by looking at whether the command that we sent was executed.  So the latch valve status we thought was dead.  It turns out this was incorrect.  We corrected this in the middle of firing on the 9th, but it did not effect the propulsion system operation as can be seen in figure 7 and right at the right edge where we initiated propulsion but got no response.  After we turned on the +28 volts and commanded the latch valves, they did successfully response (commands 21-30 as shown in figure 7).  You can see that the latch valves vibrated the spacecraft in the accelerometer data.  However, we did not have time before the end of that pass to fully investigate the issue.

Postulations

So the above is what we know.  Now we had to use our own knowledge and that of our outside experts to eliminate, one by one, possible failures.

Propellant Tanks and Upstream Propellant Lines

In examining the telemetry we know that the HPS-1 and 2 fuel systems downstream of the latch valve were fully pressurized.  This eliminates loss of fuel and nitrogen this way.  We also know, and it was our rookie mistake, that nitrogen does not dissolve in Hydrazine, more than just by fractional amounts.  This is why it is used as a pressurizing gas.  We also know that this is a blow down system (see reference 2), which means that the nitrogen gas is mixed with the hydrazine and not in a separate bladder or pressure tanks.  This is a simpler system and it eliminates failure modes.  Thus there is only three ways that we could have lost fuel and pressurizant from the system.

The first way is through the latch valves.  We already know that there was pressure downstream of the latch valves and thus that is eliminated.

The second way is that the fuel and pressurizant could have been lost through either the fill and drain valves or the fill and vent valves.  There are two reasons this is not plausible.  The first is that as far as our experts know, neither of these types of valves have ever failed in flight as they are physically capped before launch.  Also, if they had failed, it is even less likely that they would fail over ten years after launch.  Even less likely than that would those valves failed in both systems.

The third way would be a failure in the propellant lines upstream of the latch valves.  While this has happened in the past.  It is unlikely to have happened in both fuel systems, and if it had, there would be serious consequences for the spacecraft as Hydrazine eats spacecraft wiring and other hardware.  If that had happened it is likely that the spacecraft would have been lost.  Also, this would have had an effect on the attitude of the spacecraft, which we did not see when we first tabulated the telemetry after recovery.

Thus the conclusion of virtually all of our experts is that it is highly unlikely that fuel and propellant has been lost in the system.  This brings us to the next postulation.

Latch Valves

The latch valves on this spacecraft were built by Hydraulic Research (see reference 2).  These valves were popular and used on several spacecraft.  In researching the company, we found documents related to pressure testing and leak testing of the valves.  The valves do preferentially allow diffusion of nitrogen through the seals (we found this data on the NASA technical reports server).  However, since the lines downstream were pressurized, it is unlikely that this happened.  For this mechanism to operate would take further diffusion of the nitrogen through the thruster valves which is also unlikely with full pressure there.

What we did find, and I can’t be too specific here as this information is not in the public domain, is that there are different seal materials and that the type that most probably flew on this spacecraft is subject to temperature based swelling.  Since we also see very high temperatures, in excess of 62 degrees C on the upper propellant lines, which are in contact with the valve, we now have a plausible culprit for our problem.

This possibility is enhanced by our own mistake in how we operated the latch valves prior to figuring out that we had to have the +28 volts on to actuate the valve. This is somewhat mitigated by the fact that later we did actuate the valves and did see physical vibration of the spacecraft from our opening and shutting the valve (commands 21-30 in figure 7 indicate that something happened).  There is a case to be made that the valves did not actually open and we did not fully investigate this during our short Arecibo pass last week.

Temperature

The upper propellant lines near the latch valves is above their specified operating temperatures.  We found that the line heaters have been on since the last propulsion maneuver in 1987, or over 27 years.  While the temperatures are not dangerously high, the long term storage of hydrazine at elevated temperatures can cause the slow decomposition of hydrazine into ammonia and nitrogen and then eventually into nitrogen and hydrogen.  Did this happen?  We have no idea as there is no pressure transducers in the lines.  However, one of our outside experts worked on the Magellan to Venus mission and gas evolution in the propellant lines was seen there.

Since this was a mission to Venus, a planet that gets twice the radiative heat that the Earth gets, and since ISEE-3 came considerably closer to the sun every 354 days for 27 years, it is very plausible that this happened.  If it did, we would still get thrust out as we saw, but there would be gas in the propellant lines.  We did see large drops in temperature in the hot upper propellant lines and large increases in the lower propellant lines.  Rapid swings in temperature could be from gas and or hot ammonia (NH3 a decomposition product of Hydrazine) in the system.

Conclusion

Thus we have a plausible mechanism for our propulsion failures on the 8th and the 9th with the latch valves.  High temperatures expanding the seal material could have either impeded the flow, or have precluded the latch valve from opening even with the microswitch indicated to telemetry that the valve was open.  We also have high temperatures possibly evolving gas, causing a large gas bubble in the propellant lines Is this the case?  There is a way to test both cases.

Corrective Action

There is a pretty good possibility now that we have pressure and or fuel in the tanks but that it is not getting to the propellant lines and out the thrusters.  We are going to of course turn the +28 volts on this time!  We will also open both valves on one of the fuel systems, the primary and redundant.  We will also heat the tanks to see if we can see a rapid increase in temperature.  If we see a rapid rise, that would indicate no fuel in the tanks (testing for all eventualities).  There are several things we will do to test out and try propulsion to bleed all the gas out of the lines.

What we could see would be not much activity and then toward the end of the pulses from the thrusters we could see propellant flow, temperature increase, and thrust!

Cross your fingers.  We will have a pass on July 16th at Arecibo, so we will soon find out what the outcome is.

The Collective Consciousness of the InterNet

There was a great article on space.io9.com related to distributed engineering and our project, and how the people from the net came together to help us.  I first saw the term distributed engineering in the late 1980′s from the amateur radio community.  It began through using ham radio to do this, then it migrated to email before the advent of web browsers, and then to the web.  What happened with our call for help goes far beyond that as the distributed engineering meme begins with a pre organized group of people that collaborate in geographically disparate locations toward a common engineering goal.  Before our call for help last week, I knew maybe one or two of the experts that came in and helped us.  This goes well beyond distributed engineering to a collective consciousness.  I often characterize the internet as the global extension of my brain, with vast stores of knowledge that the brain organizes through the interface of the browser.

In the beginning of the net we used this to research information.  With the rise of the ubiquitous internet among the professional class and beyond in the world, we now have something never before seen on this scale in the history of mankind, a near instantaneous way to not only research information, but to rapidly organize humans to do “things”.  We now have crowd funded efforts that bring people together of like interests to fund interesting projects like ours.  We have crowdsourced collaboration in the arts, sciences, and engineering.  There is a lot of talk about singularities in the technology world, and for the most part they are marketing myths from my experience.  However, and this is what I leave the reader to ponder, we have reached a threshold where vast numbers of people can work together in a near real time manner to solve problems and do good and interesting (or evil) things.  One wonders where this will go….

For us it was great!

[1] Farquhar, R, Muhonen, D, Church, L; Trajectories and Orbital Maneuvers for the ISEE-3/ICE Comet Mission.  AIAA-84-1976, AIAA/AAS Astrodynamics Conference, Seattle, WA, August 20-22, 1984

[2] Curtis, M.S., Description and Performance of the International Sun Earth Explorer-3 Hydrazine Propulsion Subsystem, AIAA/SAE/ASME 15th Joint Propulsion Conference, Las Vegas, NV, June 18-20, 1979

=================================================================

This is really great news, because a few days ago I thought this mission was now DOA.

My approach would be to first figure out a way to cool the local section of the spacecraft, so that the “O” rings aren’t swelled. And of course, turn on the 28 volts first.

If that doesn’t work, my idea if it is a stuck valve would be to “impact hammer it”, just like a jackhammer. Pulse it with a series of 28 volt spikes if your control systems allows it. I’m sure they already thought of this, but I’d be looking for valves in every space scrap yard there is and setting up a simulation in a heated chamber.