Just after 10 am Eastern time, the SpaceX successfully launched their Dragon capsule on a second resupply mission to the International Space Station. The launch, rocket stage and spacecraft separations went perfectly, but the Dragon experienced an anomaly at about the time the solar arrays should have deployed. The SpaceX webcast announced that the spacecraft experienced a problem and then ended the webcast. NASA TV has not offered information either. We’ll provide more information as soon it becomes available.
Update: (10:44 EST) Elon Musk, SpaceX CEO just tweeted: “Issue with Dragon thruster pods. System inhibiting 3 of 4 from initializing. About to inhibit override.” Then minutes later he added, “Holding on solar array deployment until at least two thruster pods are active.”
See below for a continuation of our live-blogging of the Dragon anomaly, as it happened..
Also, here’s the launch video from launch to separation (note, the separation video is spectacular!):
Dragon carries 18 Draco thrusters for attitude control and maneuvering, so there may be an issue with those. Dragon’s thruster problem may be preventing the spacecraft from going into array deploy attitude, thus preventing array deploy.
Only time will tell if this is a software or hardware problem. The SpaceX press kit describes what needs to happen for solar array deploy:
Dragon separates from Falcon 9’s second stage, and seconds later, Dragon will reach its preliminary orbit. It then deploys its solar arrays and begins a carefully choreographed series of Draco thruster firings to reach the space station.
If Dragon can’t make it to the ISS, then it would need to be decided if and how it can return back to Earth on a good trajectory with limited thruster control.
The SpaceX controllers are obviously working to try and resolve the problem.
Update: 11:10 EST An update from NASA TV at about 11:10 am, said that part of response to problem with Dragon may be reorganizing the burn sequences in order for the spacecraft to be able to approach to the ISS. Musk just tweeted: “About to pass over Australia ground station and command inhibit override.”
Update: 11:25 EST: A statement from SpaceX says that “One thruster pod is running. Two are preferred to take the next step which is to deploy the solar arrays. We are working to bring up the other two in order to plan the next series of burns to get to station.”
Update: 11:42 EST: Elon Musk just tweeted: “Thruster pod 3 tank pressure trending positive. Preparing to deploy solar arrays.” That is good news.
Update: 11:50 EST: Another Tweet from Musk, with some short but sweet news: “Solar array deployment successful” That means they were abe to get at least two of the four thruster pods working, per the minimum requirement. Now, to see if the thruster problem causes any issues with getting to the ISS.
Update: 12:05 EST: Elon Musk continues to be the best source of info. He’s just tweeted, “Attempting bring up of thruster pods 2 and 4.”
We’re assuming that means 1 and 3 are already working, since it was going to take at least 2 pods thrusting to enable solar array deploy.
Update: 12:14 EST: The latest statement from SpaceX : “After Dragon achieved orbit, the spacecraft experienced an issue with a propellant valve. One thruster pod is running. We are trying to bring up the remaining three. We did go ahead and get the solar arrays deployed. Once we get at least 2 pods running, we will begin a series of burns to get to station.”
So, they were able to deploy the arrays with just one thruster pod working, and they are now working to get at least one more working. One question to consider is if the spacecraft and arrays were able to be in the right position to gather sunlight and produce power.
Update: 12:42 EST: According to reports on Twitter, a NASA official has said that three Dragon thruster pods are needed to fly to the ISS, and at this point, only one pod is working.
Update: 2:00 EST: This update from SpaceX: “SpaceX is still working through issues in Dragon’s propulsion system. The mission’s first rendezvous burn was delayed at least one orbit to about 2 p.m. EST.” (which is now)….stay tuned for updates. There will be a press conference sometime today, NASA has said, but has not yet set a specific time.
Update: 2:28 EST: NASA says the Dragon capsule Dragon will no longer be able to make its scheduled rendezvous with the station tomorrow. SpaceX indicated the rendezvous with the ISS will now not happen until at least Sunday morning. There will be a news briefing at 3pm EST.
Communication between the ISS crew and mission control confirms the delay for the rendezvous: “They are making progress recovering their prop system, but it’s not going to be in time to support the rendezvous and capture for tomorrow,” NASA’s spacecraft communicator told the crew. “So that is not going to happen tomorrow.”
“OK, copy, sounds like another off-duty day for us,” said ISS commander Kevin Ford. “We don’t wish that. We wish it gets fixed and gets up here to us. That’s really awesome they’re working their way through the problems. That’s what it’s all about.”
Update: 2:58 EST: Latest Tweet from Elon Musk has good news: “Pods 1 and 4 now online and thrusters engaged. Dragon transitioned from free drift to active control. Yes!!”
Update: 4:11 EST: Here’s a brief synopsis of the press briefing:
During the press conference, Elon Musk indicated there was a blockage in the helium line leading to the oxidizer pressurization system, or perhaps a stuck valve, and the pressure was not high enough to turn on the thrusters. The engineering team cycled the valves several times, using pressure hammering to loosen up and debris that might be stuck in the valves. And now, the pressure is nominal, with all the oxidizer tanks holding the target pressure on all 4 pods, and so Musk said he is hopeful that all the thrusters will be able to work.
Musk said they decided to deploy Dragon arrays before two thrusters were online because the temperature of the array actuators was dropping rapidly. But they also deployed solar arrays partly to mitigate Dragon’s attitude rotation “like a skater extending arms during a spin,” he said.
Musk mentioned the batteries on board had 12-14 hours of life, so deploying the arrays early was not a question of running out of power, but making sure solar arrays didn’t get so cold that they couldn’t deploy later.
NASA’s Mike Suffredini said that at least 3 of the 4 pods have to be operational in order to make an approach to the ISS.
They will make a decision on when to make the rendezvous when they’ve ascertained if the thrusters are working.
Ken Kremer will provide a more detailed update later.
{End of live blogging}
The Dragon is carrying 544 kilograms (1,200 lbs) of scientific experiments and supplies to the space station, and docking was scheduled for Saturday at 6:30 a.m. ET (1100 GMT). That will likely change.
Dragon’s 18 Draco thrusters permit orbital maneuvering and attitude control. They are mounted on four pods: two of the pods contain five thrusters and the other two contain four thrusters. They are powered by nitrogen tetroxide/monomethylhydrazine, and the thrust is used to control the approach to the ISS, power departure from the ISS, and control Dragon’s attitude upon reentry.
As far as the solar arrays, each array is constructed of four panels. Each array measures 6.4 meters (21 feet). The arrays can draw up to 5,000 watts of power, which is about enough power to light about 85 light bulbs. For launch, the arrays are stowed in the unpressurized trunk of the Dragon and are covered by protective fairings. When the fairings jettison, the automated deployment of the arrays is triggered. On the way back to Earth the trunk is jettison and it, along with the arrays, burn up in the atmosphere.
If the arrays do not deploy, battery life on Dragon is only a few hours 12-14 hours, which would not be enough time to keep it alive until it would reach the ISS. It will be about 20 hours from launch until it approaches the ISS, if the mission continues on its scheduled timeline.
You may recall that during the previous mission to the ISS in October 2012 (the second flight to ISS, but first operational SpaceX resupply flight) one of the rocket’s first stage engines suffered an anomaly, where a combustion chamber ruptured. That engine was shut down, and the other engines fired longer to make up for it. The Dragon cargo made it to the correct orbit and continued on the ISS, but a satellite that was launched to orbit as a secondary payload by the Falcon 9 rocket was sent into the wrong orbit as a result of “a pre-imposed safety check required by NASA,” on October 7, and the ORBCOMM OG2 satellite later deorbited and fell back to Earth.
For this launch, all nine engines appeared to work nominally and all stages separated perfectly.
But obviously the SpaceX team saw the thruster problem immediately, as that’s when the webcast was ended.
OTay then… it must be nail biting time at Space-X! Here too… I hope they get it figured out AND it is a simple fix! Of course, nothing is ‘simple’ in rocket science and finger nails regrow prettay slowly…
Thanks for the updates Nancy!
I think that fixing these problems means more expensive rockets because they need higher quality hardware and software than originally thought.
Nonsense. I am an engineer. Neither of us know the precise failure mechanism. We have no idea if it’s a simple fix or if there is a paucity of redundant mechanisms compared to other capsule designs. I doubt the latter as redundancy and reliability is the core philosophy of SpaceX as exemplified by their cross propellant feed engine out capability in the Falcon 9, something no other US launcher has today (Saturn 5 had a limited engine out to orbit capability).
3 out of 4 thrusters that fall out at a critical moment is always a freaking signal that there is a design flaw. It is not 1 out of 4 but 3 out of 4 that failed.
It might not be hardware related, but don’t think that a software related issue is any more cheaper to fix. A cross-feed propellant engine out sounds cool but is completely worthless if the computer is not redundant or the software is not built up in such a way that an error in the application corrupt memory of other even blocks other processes. A bug fix in the software must be tested and this in turn can be very expensive because it must be tested with real thrusters in the end.
Today I had a big discussion with people that designed hardware related interfaces in their design. I warned them that hardware does not always behave logically. It is a completely different way of developing than a simple database applications. Electrical noise e.g. in the wiring caused by a pump that starts could trigger false data. The position of local variables or allocated memory is also important if you develop in C(++). One part of the function could write one byte too far corrupting another critical structure…. And I am not even talking about multi threading problems.
Hardware will always behave logically as well as any other system. They do not think. People just may not know the logical reason for the failure; does not make it illogical.
It is illogical in the sense that you will get unexpected and unpredictable results that is not part of the software logic you designed. You must be prepared to handle situations you cannot predict.
And you must be prepared to deal with this unpredictable situation in real time. So no pop up with an exception showing on your screen or something logged in your log file expecting that a user restarts a service.
This is not your simple database application.
Dude it was an unexpected blockage and the technique they used to clear it is so common it has a name. It’s called a “pressure hammer” and has been used for nearly as long as humans have made used of pressurized systems.
In fact I used it earlier today when my pickup didn’t want to go into reverse. Although in my application it’s called “double clutching.”
Take a pill and wait until we have some data before jumping to wild unbased conclusions. Unless you are psychic talking about redesigns, replacing components, and rebuilding systems now would be the equivalent me jumping out in the parking lot of Wal-Mart today and proceeding to tear out my tranny and clutch.
There you go: It was an “unexpected” blockage.
This indicates that it was not part of their test plan to test such a problem.
They probably did not even realize that it could be a problem.
It also showed that redundancy failed for these thrusters. If the
redundancy would have worked then only 1 thruster would have stopped not 3. (1 out of 4 failed thrusters means bad luck, 0 out of 4 means we have probably an easily fixable problem, 1 out of 4 means design flaw)
It means that they need to step up their quality of components, and
software design. More quality tests to prevent that this is happening
next time. More tests means more resources, more time and more
expensive.
Sounds like the perfect excuse for Elon to send up his crew-ready Dragon we all know he has stowed away to send up there and fix the issue with a heroic EVA!
This is what happens when you havent spent years working out all of the possible bugs in your hardware and software. While I love the idea of private space programs, it just goes to show that to make things work right and do it almost every single time, you need to spend the big money testing hardware and software. That’s just not possible with the smaller budget these programs have.
So, while programs like this are a great step, they are bound to suffer these types of problems for at least a decade or more until they have enough launches that their hardware and software has seen every problem.
Considering the headline on the second article above this one is “Curiosity Rover Has Computer Problems”, your post is ridiculous!
It is irrelevant what problems Curiosity has. This thing is going to the ISS. One misfire of the thrusters and the ISS is damaged.
This bug/hardware failure will cost additional development costs making the rocket more expensive, heavier, more complex and tons of delays. 4 thrusters and only one working means a design flaw. And design flaws are always very bad news.
I am not saying that they made a bad product, but I do predict that this rocket will double and even triple the design cost and lower the payload capacity because of heavier redundancy systems.
The rocket performed flawlessly this time, and all 4 thruster pods on dragon are back online.
Online, but safe to operate near the ISS?
They failed at a critical moment. The last thing you need is that they fail when they need to break when it comes close to the ISS. We are not talking about 1 thruster that failed, but 3 out of 4=75%
“This bug/hardware failure will cost additional development costs making the rocket more expensive, heavier, more complex and tons of delays.”
And you know this, because?
First, some hardware fixes are very simple ones. ‘Don’t operate in ‘x’ manner again.’ for example. Or; ‘Add backup valve in line ‘y.’ Problem solved.
And software issues, though they will naturally take some degree of time and money (possibly just overtime for a few engineers), it does not follow that it’s a Manhattan Project effort. They’ve been there with a simple fix to a staging situation that prevented the third Falcon-1 from reaching orbit.
And once a software issue is identified and patched, it will weigh…nothing. (at least no more than any other set of bits)
“4 thrusters and only one working means a design flaw.”
At this point, no one can say that, either. Even potentially catastrophic events can come from simple sources. A Gemini launch once had a scary on-pad anomaly, because someone failed to remove two plugs from a specific point in the plumbing. They changed the procedure to decrease the chance of that, and moved on, they didn’t redesign the system in question. This may be as straightforward as that, or not. We’ll see.
“And you know this, because?”
It is my full-time profession to design things like they use.
And a rocket is not like an ordinary PC that you just patch and hope that it starts up. This is a design flaw. It means that the system is not redundant enough to guarantee a safe operation. It also does mean that there is a big risk that the thrusters would (mis)fire randomly near an inhabited ISS.
“And you know this, because?”
It is my full-time profession to design things like they use.
And may I assume that professionally, you would not jump to sweeping conclusions like the above, without analysis and facts?
While I agree that this problem will lead to additional development costs for SpaceX, that does not necessarily mean SpaceX will then be forced to charge more for their rockets. They haven’t made any assertions about increased costs despite the additional rocket engine development they have been doing, so why should they increase costs for changes to the thruster pods?
Also, it is a little early yet to be claiming a design flaw since we don’t know what the actual problem is yet. It may be a design flaw, but it could also be nothing more than a stuck valve.
” so why should they increase costs for changes to the thruster pods”
They need more testing and a bigger development cycle to catch these new bugs/problems they recently discovered. This in turn will make them discover a whole series of new classes of errors that they did not notice before. They will need more physical hardware tests with real thruster actions which in turn costs more. They will need more people to do it, and in turn need more project managers and have more general managers to manage these project managers.
We are not talking about a simple PC that we can jump start by going to a command line and tweak some parameter to jump start it.
“They need more testing and a bigger development cycle to catch these new
bugs/problems they recently discovered.”
Even accepting that for the moment…
“This in turn will make them
discover a whole series of new classes of errors that they did not
notice before.”
You again conclude what will (not a qualified ‘may,’ but ‘will) be found, if anything, before actually doing what you suggest.
Presupposing too much, only tends to lead investigations and analysis in the wrong directions. (possibly even to the point of missing a real, but different and more subtle defect than you assume.)
But then again, you don’t seem to be one of the SpaceX engineers who will actually have to get to the bottom of this anomaly…
However, recoverable vehicle that it is, the actual hardware in question, not merely the design, should ultimately be back for physical study.
“They need more testing and a bigger development cycle to catch these new bugs/problems they recently discovered. This in turn will make them discover a whole series of new classes of errors that they did not notice before.”
One of the first things I was taught during post-graduate training was that to “ass u me” makes an ass out of you and me. There is not enough hard data available yet to make such a claim. If you have a link to the design schematics for their thruster systems then that would be a starting point for a discussion about what exactly went wrong.
Yes there is hard data to make this claim.
When 3 out of 4 fails at a critical moment, than that means we have a design flaw. Software/Hardware or both. It clearly means that this condition has never been tested before.
You could easy point to a faulty valve or faulty sensor or maybe a computer bug, but of a critical system even with a faulty sensor, valve or software bug this problem should never have happened in the first place. Software should compensate for faulty sensors or valves. Computer systems should be ready to compensate for faulty software/computers.
Let’s assume that it is a faulty valve. Better valves means probably more mass. More mass probably means shifting components. But also more powerful motors to control those valves which in turns means more required power, and bigger batteries.
Assume that it is faulty software. Then this means that you need to revise the software. Fixing bugs might cause other bugs in other systems. The bug might even require a redesign of several modules. You need to test the whole system again, and in the end you need to test it with real thrusters.
Hmmm… how many rockets shuttles etc has NASA blown up over the years??? Some with people in them? The shuttle didn’t even achieve it’s reliability goal. And it was a $$$ pit!
Point taken, but spending “the big money” can’t be relied on to provide bullet-proofing. The Space Shuttle fleet, which consumed over $200 Billions during its lifetime, had plenty of anomalies and two of them were doozies.
Bombs away?
Everyone is a rocket scientist today… good luck dragon.. it is companies like SpaceX inspiring the imagination and motivating the next generation… not government institutions,, what a pity.
But government institution always hires contractors and private industries to build their systems. It is the same type of people that built the NASA systems that are employed by SpaceX.
Not the same type of leaders though; that makes all the difference. Private industry doesn’t have to do something simply because it serves the goals of the government or satisfies the majority of the populace. That allows them to take more risks and be more innovative which in turn allows them to develop and improve their systems faster than any government agency ever could.
It is a small company and small scale projects, that is why they can move faster. But when they grow they will face the same problems a government organisation has. A private company is not immune from bad management, too many meetings, blaming other private companies because they provided components that sucks and political interests.
At this time all systems are nominal all thrusters are green, burn completed to get the the ISS. Looked like a stuck helium valve to pressurize tanks, several open, shuts cleared problem. Dragon is on the way. Good job to all at SpaceX command center. Giter done Elon. Special thanks to Air Force Comm for tracking while in free drift and Nasa for “Not” aborting mission.
Let me get this straight.
4 thrusters, 3 failed.
How an 1 stuck pressure tank valve cause 3 to fail and 1 to operate?
Are we talking about 3 pressure valves that decided to fail and 1 that actually worked?
I am assuming that they use redundancy for those critical thrusters. So that would mean that 3×2 valves failed. and 1 or 2 actually worked correctly.
Oh yes you need 2 pressure valves minimal per thrusters and both must operate correctly. 1 for the fuel, 1 for the oxidizer.
So for redundancy you would have 2×2 per thruster pod.
Failure of a pod means that 3 out of the 2×2 failed per thruster pod failed.
Since 3 out of the 4 thruster pods failed that would mean that minimal 3×3=9 valves failed.
Do you work for one of SpaceX’s competitor’s, Olaf?
No, but I am education myself big time to get into space related contractors job if I ever got a chance. I don’t care what company would hire me.
I do not know how SpaceX designed their system but I do know that the helium tanks are used for pushing the oxidant and the fuel inside the reaction chambers. Helium because it does not react with the fuel or the oxidizer. If you have 2 helium tanks then you will have a cross-feed to even out the pressure, just in case one of the tanks builds up too much pressure. You build up pressure by heating the helium in the tank.
With a cross-feed if one valve fails then you will guaranteed to have pressure from the other helium tank. And that means that all 4 of the thrusters should have Helium pressure pushing on their fuel and oxidizer tanks.
The thing is, assuming there is no redundancy, 1 out of 4 stuck helium valves would means bad luck. But 3 out of 4 means design flaw.
One thruster working means that the helium tank gives off pressure.
But I am wondering if the pressure in the Helium tank was high enough (not a stuck valve) because that would explain it more logically that only 1 thruster operated if you want to blame it on a Helium problem.
Good question Dan: funny. Judging by Olaf2’s response here (and all his previous statements in this comment section), if I were HR in any of these companies or NASA, I would put his resume at the bottom of the pile.
And what are my responses that would put my resume at the bottom of the pile?
Hi Olaf, don’t be dissuaded by the negative comments stated by others to you. I’m sure you’d do well. If you care to know, Bill Borucki’s Kepler mission was rejected numerous times by negative folks, and it somehow managed to finally break through and is now one of the most successful space-based telescopes.
Dan.
I think the success of the Kepler mission was actually driven by negative people to make the craft even better to prove them wrong 😉 :-p :-p :-p
Am wondering how many grey hairs Elan now has? Space-X’s accelerated learning curve is probably a good thing(?) as eventually there will be crew on board. Better to work out all the wrinkles NOW! HO!