36 IEEE TRANSACTIONS O N RELIABILITY, VOL. R-33, N O . 1, APRIL 1984

Space Reliability Technology: A Historical Perspective

Haggai C o h e n N A S A H e a d q u a r t e r s , W a s h i n g t o n , D . C .

Key Words—History, Space reliability.

Abstract—This paper first discusses the earliest US space flight at­tempts and some of the reasons for their low success record. Then it ex­amines the lessons learned which fed into the Thor-Able-Star and Delta launch vehicles, followed by the ICBM family. Improvements in design, materials, fabrication, workmanship, contamination control, testing, and management are discussed. The first major breakthrough in more formal reliability efforts, which occurred in the man-rating process for manned space flight, is examined. Special efforts to foster motivation are described. Introduction of reliability prediction to the design process in Apollo is presented. Growth of the failure mode and effects and criticality analysis techniques are shown, together with the growth of hazard analyses via fault tree techniques. The identification of critical items, coupled with tender loving care (TLC) handling, hard­ware pedigree, exhaustive analysis of failed critical components to pre­vent recurrences, EEE part selection and application, use of clean rooms, improved N D E techniques, are all examined and their contribu­tion evaluated. Factoring of lessons-learned into the Space Shuttle plan­ning efforts is discussed. The areas of mission success versus pay load safety for Shuttle payloads are examined, as well as the effect of in flight repair or return of failed hardware on payload hardware design.

The distance space reliability technology has come and the pro­gress made in improvement of space hardware can perhaps best be il­lustrated by the T V coverage of the last few Shuttle missions. First, for launch, generally only the last minute or two of the countdown and the few minutes of spectacular lift-off are shown. What we then see are the space walks, the repairs, the deployments, the recoveries and the berthings — ie, the things we can do in orbit. The rest (getting there) is almost taken for granted and this type of excellence has virtually become the norm for our space activities. This is the ultimate test of ef­fective reliability efforts and we are faring well on this test.

T H E EARLY A T T E M P T S

Since the dawn of recorded history man has searched the skies trying to read the mysteries of the sun, moon, and stars. Scientists developed instruments of ever-increasing sensitivity to search the heavens, but no matter how power­ful the instruments, observations from the earth's surface were limited. To the scientist, the earth's atmosphere was a dense filter which prevented him from obtaining the knowl­edge he was seeking.

In the decade after World War II, rockets pierced this filter briefly. This was done by high altitude research rockets such as the Viking (built by the Glenn L. Martin Co.). It took the Viking about five minutes to reach altitude (158 miles) and another five minutes to fall back to earth. Only a small part of the ten minute flight was above the atmosphere. If the observation-times above the atmosphere for all vertical rockets fired through 1956 were summed, they would total no more than 10 hours. For the scientists, the orbiting

satellite provided the means to increase the time of upper air observation.

The International Council of Scientific Unions esta­blished a special committee to direct the efforts of nations who would be contributing to the intensive research that was to take place during the period called the IGY (International Geophysical Year — 1957 July 1, through 1958 December 31). In the fall of 1954, this special committee recommended that launching of satellites be considered as a means of ob­taining significant scientific data.

In 1955 July, the President announced that the USA would embark on an earth satellite program, to be ac­complished during the IGY, and to be designated project Vanguard. The Martin Company was awarded the prime contract for producing the satellite launching vehicle. Martin designed a 3-stage rocket, approximately 72 feet long, with a gross weight of approximately 22 000 pounds. The first stage was powered by a 27 000 pound thrust (at sea level) engine, fueled by kerosene and liquid oxygen. Low tank pressures were made possible by incorporation of a turbo-pump, powered by the decomposition products of hydrogen perox­ide. The second stage used hypergolic propellants: unsym-metrical dimethyl-hydrazine and white fuming nitric acid, with high pressure stainless steel tanks (no turbo-pump). The third stage consisted of the satellite, a jettisonable nose cone and a solid propellant rocket motor that was spin-stabilized on a small rocket-motor-powered turntable. The first and second stages were guided by a 3-axis reference system, consisting of three integrating rate gyroscopes, a pitch programmer, and an integrating linear accelerometer that provided velocity information for in-flight correction to the satellite injection time.

Vanguard combined cryogenics, storable propellants, and solid rocket motor technology all in one vehicle. It was designed with virtually no redundancy, very high mass ratio, minimal quality program, a high degree of complexity (for that time), and a carefully planned and orderly test program consisting of six test vehicles and six satellite launch vehicles. After three test vehicles had flown, the Soviet Union's Sputnik I burst upon the unsuspecting world on 1957 October 4, and the orderly Vanguard test program was promptly scrapped with orders to "go for broke" on the next test vehicle (designated TV-3) which was the first all-up configuration. On live television, one second after lift-off, the first stage engine lost thrust because of an improper engine start, the vehicle settled back on the launch stand and exploded. With it went a nation's hope for a quick recoup of its national prestige. This recouping was accomplished instead by an Army Ballistic Missile Agency Group that launched the western world's first satellite, Explorer I, on 1958 January 31. On 1958 February 4, Vanguard tried again, but broke up in flight at 20 200 feet after about 60 seconds in flight. Vanguard 's next attempt (designated TV-4) successfully

0018-9529/84/0400-0036$01.00 © 1984 IEEE

COHEN: SPACE RELIABILITY TECHNOLOGY: A HISTORICAL PERSPECTIVE 37

placed 57 pounds (4 pound payload and 53 pound third-stage motor case) in an orbit estimated to last at least 1000 years. In all, 11 attempts were made to achieve orbit, of which three were successful.

Reasons for Low Success Rate of Vanguard

In retrospect, in addition to the plowing of unknown fields, Vanguard systems were designed to fit into a 45-inch diameter first stage and a 30-inch diameter second stage. This made for difficult fabrication and inspection accessibility. From a reliability viewpoint, the design of all single string systems left no margin for failure of critical components. Selection of parts was sporadic and contamination control was limited primarily to hydrogen peroxide and first stage liquid oxygen systems. This latter point, in fact, caused the loss of two vehicles, when heat-treat scale from the inside of the se­cond stage tankage broke loose and clogged engine filters and lines. This failure mode resulted in the corrective action of a pickling and flushing operation of those tanks, followed by a successful launching of Vanguard II in 1959 February.

Improvements After Vanguard

The Vanguard heritage was used on the Thor-Able and then the Thor-Able-Star launch vehicles, which picked up the Vanguard second and third stage technology. The lessons learned were also incorporated into the military In­tercontinental Ballistic Missiles (ICBMs) such as the Titan. The Titan ICBM was a major step forward in sophistication and technology. Engines were much larger (300 000 pounds of thrust on first stage versus Vanguard's 27 500 pounds). Guidance became a radio inertial system followed by an all inertial guidance in later vehicles. Launch control equip­ment moved from the Vanguard manual fire switch to a Titan automatic sequencer that monitored final critical parameters and automatically fired the vehicle. The vehicle size increased to 10 feet in diameter for first stage and 8 feet for second stage. This large increase in interior volume made a vast difference in accessibility for both fabrication and inspection.

Titan also saw a widespread use of environmental ac­ceptance testing (in addition to qualification testing) which was used as a manufacturing workmanship screen. Problem reporting and corrective action had become a strong discipline as had configuration control and contamination control. Susceptibility to contamination control was reduced by attention to the problem in design via proper filtration in both ground and flight systems. Procedures were refined to a high degree for both checkout and launch preparation with incorporation of safety checks and balances during haz­ardous propellant loading, vehicle erection and transport, ordnance installation and checkout, high pressure opera­tions, etc. Welding of vehicle tanks had progressed enor­mously with automatic welding fixtures, with associated x-ray techniques to check weld integrity, and with proof loading of tanks as a final acceptance screen.

Shipment of stages from the factory to the launch site via larger military cargo aircraft also improved overall system reliability, although some early problems had to be overcome to allow sufficient tank breathing through desiccant containers to match changing altitude during transport. Lack of such breathing caused an oil-canning of tanks on one early flight.

Vehicle handling on the launch pad was accomplished by means of a complex hinged mechanism used to erect the vehicle, called the erector. It weighed hundreds of tons and was raised, with the vehicle inside, by a complex system of winches, cables and counterweights. The erector was then used for access to the various levels of the vehicle in the ver­tical position during launch preparations. In the final part of the launch countdown, the erector was lowered before the vehicle could lift off. Actually, it would be more correct to say the erector was planned to be lowered. The urgency of making this massive erection and de-erection safe had created such a host of fail-safe devices, safety interlocks, switches, relays, brakes, etc. that it turned out to be almost absolutely safe — because it hardly ever worked when called upon in the terminal part of the countdown. There turned out to be one override relay in the winch pit that could barely be reached, whose manual contacts could be closed to effect a lowering of the erector. The man doing this job had the title of "erector director". It was a good example of the point that being safe does not necessarily mean being reliable.

TITAN'S MAN-RATING PROCESS

Results of all the above efforts on Titan were reflected in improved success in flight. In the first 14 months of flight testing, 14 Titans were fired. Of these, two were destroyed at launch, two were partial successes and ten were complete successes. This was a considerable improvement over Vanguard.

When the Titan ICBM became a manned space flight launch vehicle for the Gemini Program, the vehicle design, its fabrication, its quality program, and its test program were all subjected to a unique process called man-rating. During the design phase, critical functions were examined and iden­tified analytically and then design redundancies or work­arounds provided in case of failure. This involved dual guidance systems, dual hydraulic systems for gimbaling the engines, dual power supplies, etc. In fabrication, careful training and certification procedures were used in areas such as soldering, brazing, welding, contamination control, crane operations, tubing fabrication, and waterproofing. In the quality control area, detailed inspection planning was pro­vided not only for in-house work but also for use by source inspectors at subcontractor and vendor facilities.

A program of handling hardware with tender loving care was instituted, involving transfer carts with shock-mounted tires and anti-tampering seals. Critical components were labeled inside and outside shipping containers. Air freight ramp personnel were trained and instructed on pro­per procedures for handling critical components at fre­quently used airports.

38 IEEE TRANSACTIONS ON RELIABILITY, VOL. R-33, NO. 1, APRIL 1984

Testing was performed to detailed, pre-planned pro­cedures with rigid discipline on actions to be taken upon hardware failure. At such time, a special investigative team was convened on the spot to devise a failure isolation and troubleshooting procedure to be certain that the probability of finding the precise failure mechanism would be maxi­mized and not lost in a fumbling, troubleshooting effort. The failure analyses were then followed by prompt and formal corrective action to prevent recurrence of failure. Customer (Air Force) involvement in the problem closeout process added immeasurably to the rigid discipline. A full-time engineer had the title of "Get Well Manager" and his task was to track and present satisfactory closeout actions and recurrence control on all hardware failures.

Generating Motivation

Significant efforts were expended in motivation of peo­ple. Astronauts visited the fabrication areas. Films, posters, contests, team flags, decals, special uniforms, launch team jackets, etc., were used. There is an interesting story relating to this effort:

The Program Director and the Quality Director journeyed out to Denver to award a Gemini Team Flag to a supplier of a critical component — the explosive tie-down bolts which held the entire launch vehicle to the stand while the rocket engines came up to full thrust. In a converted garage, they found a total of 17 totally dedicated people involved in the operation — from the woman president on down. Engineer­ing was done by contracting out to the University of Denver. The test site was an open field with only an outhouse on it. Sample bolts were taken out to the field, test fired, and the lot accepted. The Quality Department was one man working out of one cabinet. The top shelf contained all his procedures and standards. The next shelf contained all the in process in­spection records. The third shelf contained all the necessary gages for dimensional checks and included calibration cer­tificates traceable to the National Bureau of Standards. The fourth shelf was the material review board area for impound­ing nonconforming material. No one could find a single fault with anything they saw. The company delivered on time, on budget, with never a discrepancy and the hardware worked 100% during the life of the program. The award flag was cheerfully handed over and all watched while it was raised on the new flag pole erected for the occasion (cement was still wet) and while the local T V cameras ground away.

GEMINI : T H E $2 000 000 I N C E N T I V E

A significant feature of the Gemini Launch Vehicle con­tract between the Air Force and Martin Marietta was the attention-getting flight performance incentive. Translated in­to simple terms, the contract was for 12 launches, the first two being unmanned, followed by 10 manned flights. The first launch was not counted in the flight performance incentive calculation but the next 11 flights were. If only 10 of the 11

flights were successful, the contractor would lose $2 000 000 in incentive fee. This potential loss of $2 000 000 of profit for just one failure hung over the program and colored all of the major technical decisions in a very positive way.

A typical example would be the failure of a critical com­ponent where there was a shortage of spares and the usual ac­tion would have been for the logistics group to route the com­ponent back to the vendor for rapid repair and availability as a flight spare. Instead, with no urging from the reliability group, the component was routed to failure analysis, no mat­ter how urgent the need for a replacement spare was. Logistics personnel fully understood the need for effective failure analysis and corrective action to avoid the possibility of an in flight failure that could cost the program $2 000 000.

The program did achieve 11 out of 11 (actually 12 out of 12) successful flights and the full flight performance incentive was paid.

APOLLO: AMAZING SUCCESSES

The Apollo Program followed Gemini and with it we saw the coming of age of a full reliability program. Design effort incorporated failure mode effects and criticality analyses (FMECA) which identified for action those failure modes which could endanger crew safety or cause loss of mission. Redundancies, special testing programs, multiple ways to carry out critical functions, special inspections, and pedigree quality controls resulted from this effort.

Reliability was predicted to determine, as the design was progressing, the probability of landing on the moon and returning safely to earth. Because of the numerous unknowns involved in this pioneering effort, reliability estimates were naturally very conservative. This resulted in a success prob­ability so low that the entire reliability prediction effort was disbanded and never resurrected. The effect was so traumatic that to this day, numerical reliability analyses at NASA are usually limited only to their use as a tool in design trade-offs, where relative numbers have meaning and absolute values are unimportant.

The Apollo spacecraft fire brought a whole new area into focus — that of flammability, toxicity, off gassing, and odor testing of materials. This, in turn, spawned a large material research effort to find new materials that would not burn or that would self-extinguish in the spacecraft atmosphere. It also brought fire detection and fire suppression systems into manned spacecraft. Most importantly, there was a re-emphasized manned spaceflight safety program that carried a large clout and significantly affected program activities from that time forward.

Management of the vast undertaking of landing a man on the moon was a major challenge. Program control was dif­ficult but vital. Program control centers and their intercon­necting networks became the central nervous system of the program. Schedules, costs, changes, and technical problems had to be communicated, digested, understood, and acted upon. For assurance, an overall technical integration and evaluation contractor was added to oversee all the pieces in the

COHEN: SPACE RELIABILITY TECHNOLOGY: A HISTORICAL PERSPECTIVE 39

program. That contractor introduced, in the safety world, fault-tree analyses and sneak-circuit analyses. Formal milestone reviews became the forum for determining technical progress (or lack of it). Senior management was totally involved in the milestone review process and some In­tercenter rivalries were used as an incentive tool.

Motivation techniques were extensively used. With per­mission of Charles Schulz, Snoopy became the astronauts' mascot, and astronaut-awarded silver Snoopy pins became a coveted prize. Workers from all parts of the program were selected as *'launch honorées' ' and as a reward were brought to the Kennedy Space Center to view a launch. Contractors developed unique in-house motivation programs and rewarded outstanding employees with cash awards, prizes, etc.

In retrospect, Apollo achieved an amazing success record, successfully landing on the moon on each attempt, with the exception of Apollo 13. Apollo 13, even though un­successful, did return the crew safely (but barely), and demonstrated dramatically the importance of providing, wherever possible, alternate ways for critical functions to be carried out to keep critical single failure points from causing catastrophic failure. The Apollo 13 story, for those who do not remember, is briefly recalled.

Apollo 13, with its crew of three astronauts on board, found themselves in very harrowing circumstances on the way to the moon as a result of an explosion inside an oxygen tank in the service module attached to their command module. The explosion damaged vital systems involving power and air revitalization and unknown possible damage to the heat shield needed to withstand the searing heat of re-entry. An unprecedented team effort followed, involving flight crew and hundreds of engineers, contractor employees, flight con­trollers, doctors, technicians, etc. working around the clock for days preparing jury rig procedures to use the lunar module (which was docked to the command module) as a lifeboat. Life support systems were connected with hoses and tape and powered systems cut to an absolute minimum to conserve small power resources. Since the problem occurred on the way to the moon, the mission had to continue with a swing around the back side of the moon, including an engine burn in lunar orbit and a long trip back home. It was a true adventure story filled with courage, endurance, stamina, brilliance, and prayer.

SKYLAB DEMONSTRATES THE IMPORTANCE OF MAN

Apollo was followed by the Skylab Program which placed a Space Station in orbit, filled with scientific experiments, and equipped to handle rotating crews. The Space Station was spacious, providing a shirt-sleeve work and sleep environ­ment. Hot meals could be prepared and individual sleeping areas were available with personal hygiene facilities, including a shower. All of this great success, however, came after another harrowing experience.

The workshop had been launched on top of two stages of a Saturn V rocket, unmanned. It was planned to launch the crew to rendezvous and dock with the workshop on the following day. Instead, after what appeared to be a perfect launch, two critical events failed to occur — the meteoroid shield surrounding the workshop failed to deploy, as did the two wings containing the vital solar arrays which were to provide electrical power to the cluster of spacecrafts compris­ing Skylab. In addition, rapidly rising temperatures in the workshop verified that the meteoroid shield not only failed to deploy but was probably totally missing. During the next ten days, volunteer efforts by NASA Centers and contractors resulted in several proposed solutions to provide heat shields for the badly overheated workshop. A solution was selected and a parasol was fabricated which could be pushed through a small airlock in a folded position and then opened up to provide the critically needed shade to save the equipment from the blistering solar radiation. The parasol was made of lightweight metallized mylar film, with a spring type open­ing mechanism, similar to an ordinary umbrella. The device was tested, loaded under the crew couches of the spacecraft waiting on the adjacent launch pad, and launched with a crew trained in the parasol erection procedure. This 10-day long scenario represents a saga of superhuman achievement probably unparalleled in history. The repair crew was also able to photograph the partially deployed solar wing and point out the metal debris strap that prevented the wing from deploying. A variety of cutting tools were fabricated for the second crew to use and procedures developed to pull the wing free after the debris strap was cut. All these efforts were, in fact, successful and the released solar wing slowly unfolded and deployed. Power flowed into the Skylab systems and the workshop was restored to essentially full working capacity. It was a magnificent save!

Skylab went on to supply a home in space for three crews, the last one staying in space for 84 days. Data on solar astronomy and earth resources, manufacturing in space, and even spider web spinning in weightlessness, were collected and returned to the ground. The most important lesson that was learned from this experience was the importance of understanding and using the capability of man to view, judge, repair, and replace failed hardware. This was demonstrated repeatedly on Skylab and even more dramatically recently, on the incredible in-flight repair per­formed by the Shuttle crew on the Solar Max Satellite.

SPACE SHUTTLE: DIRECTION FOR THE FUTURE

The development of the Space Shuttle has brought together the talent, the experience, the revolutionary engineering advances that were developed in the program described above. In the reliability area, this includes failure mode and effect analysis, careful part selection, adequate margins in design, selected redundancies, alternate means to carry out critical functions, detailed designed reviews, meticulous quality programs, safety analyses, etc. Of even greater significance, it has brought reuseability of expensive

40 IEEE T R A N S A C T I O N S O N R E L I A B I L I T Y , V O L . R-33, N O . 1, A P R I L 1984

hardware to the space program and this has and will con­tinue to revolutionize all that we will be doing in space for the next several decades. Payloads will be routinely designed for in-orbit servicing and repair. They can be designed less expensively because of the capability of replacing worn out or failed modules. The entire area of mission success can be left completely to the payload developer, with the only Shut­tle imposed criterion being: Is it safe to carry on the Shuttle? (This philosophy is already in place with a published set of safety requirements for Shuttle payloads and a Payload Safe­ty Review Panel imposing and interpreting those re­quirements.)

A U T H O R

Haggai Cohen; 8072 Inverness Ridge Road; Potomac, M D 20854 USA. Hagga i Cohen is the Deputy Chief Engineer for Safety, Reliability

and Quality Assurance at NASA Headquarters and is responsible for these

functions at all NASA Centers and across all NASA programs. He was formerly the Director of Reliability, Quality and Safety for the Office of Space Transportat ion Systems of NASA Headquar ters . He also served as the Director of Reliability, Quality and Safety for the Skylab Program Of­fice and upon completion of the Skylab Program he assumed respon­sibilities as Reliability, Quality and Safety Director for the European Spacelab Program. He joined NASA in 1966 September.

Prior to his assignment, he was Quality and Reliability Assurance Manager of the Gemini Launch Vehicle with the Mart in Company in Baltimore. He was the Quality Manager on the Dyna-Soar Booster Pro­gram at Martin-Balt imore, on the Ti tan I Program at Mart in-Canaveral , and on the Vanguard Program at Martin-Baltimore. He was with Mart in from 1955 to 1966.

Mr . Cohen has an Electrical Engineering degree from the University of Mani toba in Winnipeg, Canada , and completed the curriculum for an M S degree at Case Institute of Technology.

He has taught system safety and reliability at the School of Continuing Engineering Education of the George Washington University, and has taught reliability courses for the ASQC Education and Training Institute.

He is the recipient of two NASA Exceptional Service Medals (for Apollo and Skylab) and the Outs tanding Leadership Medal (for the Space Shuttle Program). * * *