1.5 The Test as a Contract

After this section you can: name the four documents a test lives in, say what each one owes the one before it, and know when the log starts.

A test is four things in order: requirement, procedure, result, report. Each one answers to the one before it, and a reviewer reads them backward. They need not be four separate documents. Some organizations keep each as its own controlled document; others keep one document for the test and append to it as the work goes, the procedure after the requirement, then the result, then the report. Which way is the organization's choice; what matters is that each part is there and answers to the one before it. The procedure carries two things with it, a risk analysis and a review.

Three names recur in this volume. The operator runs the machine at the console. The test engineer owns the procedure and the lab's part of the plan. The lab is the organization that holds the machine. The names are jobs, not people; in a small lab one person may hold two.

The requirement says what to prove: the step on the sequence, the purpose from 1.2, the profile from 1.3, the standard from 1.4. And it says what a good test is. Pass is a definition, not a feeling. The profile was held within tolerance for the duration. The article functioned before, during, and after, and the functional checks are named. There was no visible damage on the post-test inspection, and the inspection is named. The pre-test and post-test signature surveys (low-level runs, a sine sweep or a flat random, run the same way before and after the test and compared) agree within a stated shift in their resonance frequencies (and amplitudes, where the requirement calls for them), if surveys are called for. And the requirement says which deviations are acceptable without a re-run. If it does not say what good is, the lab and the customer will decide it after the run, in disagreement, with the data already on the table. If the requirement does not say all of this, the lab asks before it schedules.

The procedure says how: which machine, which fixture, which channels and where, which one controls, what the limits and aborts are, who signs. It is the lab's document, and it is where the standard's silence gets filled. It is also where safety lives. What the article is: energetic (explosive or propellant-loaded), pressurized, hazardous, high value, one of a kind. What happens if it comes off the fixture. Who stands where, and what the exclusion zone is during the run. Which interlocks and aborts protect people and which protect hardware. How the article is lifted and mounted on the shaker. Who has the authority to stop the test; unless the procedure says otherwise, anyone at the console may stop it, and the test engineer decides whether it restarts. A procedure that has not been reviewed by someone who did not write it is a draft. The review before the run (this series calls it the readiness review; the lab may have its own name for it) is where someone other than its author checks the plan, and it is where the customer hears about the deviations that are already known.

The procedure's risk analysis is written before that review, by the procedure's author, and it says what can go wrong and what is being done about it. To people: an energetic or pressurized article, the lift, the exclusion zone. To the article: an over-test from a control fault, a fixture resonance, a loss of control (the drive running away) on a one-of-a-kind unit. To the machine: a profile near the shaker's limits, a mass near the table's limit. To the program: a failed test with no spare, a schedule with no restart window. Each risk gets a likelihood, a consequence, what the procedure does to reduce it, and who accepts what is left (the residual risk; who that should be is part of the plan, which Volume 2 treats). It is usually a page. Its value is that it forces the question the controller's end-of-test summary never asks: what happens if this goes wrong, and did we decide that on purpose. The review reads the risk analysis first and the procedure second.

The result is the data and the log, not the controller's end-of-test summary, whatever the controller calls it; that summary is one page of it.

The report is the argument that the result proves the requirement. It is the only one of the four that anyone reads a year later, and it has to carry the other three with it. At the least it states the requirement and its definition of pass, the procedure as run, the result against that definition, and every deviation with who decided it and why. Reading a report and writing one are the subject of Volume 2.

No test runs exactly to its plan. A notch, a limit, an abort, a restart, a level change: each one is recorded when it happens, with who decided and why, and each one becomes a sentence in the report. A deviation recorded on the day is a fact. One reconstructed a week later is a story. An event during the run that the plan did not foresee (quality organizations often call it a test anomaly) is recorded the same way, and whether it costs a re-run is decided afterward by the program and the customer, against the requirement's list of deviations acceptable without one; Volume 2 covers how deviations and anomalies are written up.

Witnesses and customers need to be settled before the low-level run, not on the day: who watches, who signs, what they need to see during the run and after it. Some controllers split the low-level run into a system check at a very low level and then rising levels; the deadlines in this chapter mean before the first of them.

Where this book says the plan, it means the test plan, the document above the four that ties them together; on many programs the plan and the procedure are separate controlled documents. The plan, who writes each part, who signs it, and what has to be settled before an article is scheduled, has its own chapter in Volume 2. This volume closes with the machine's side of that plan, Chapter 10: the arithmetic, the facility, the machine's health and the coordination the lab owes the plan before the article arrives.

Three ideas run through this volume. A test is a partnership between the people who designed the hardware, the lab, the customer and the analyst (who models the article's dynamics and reads the test data), and most of the time those four do not talk. The decisions about who does what are made before the test, or they are made badly on the day. And the record, including the photographs, is what settles the argument a year later.