March 15, 2004

Volokh: Writing to Think Like a Lawyer

Test Suites: A Tool for Making Your Student Article Sounder and More Persuasive

(from Academic Legal Writing: Law Review Articles, Student Notes, and Seminar Papers)

When you�re making a prescriptive proposal (whether it�s a new statute, an interpretation of a statute, a constitutional rule, a common-law rule, a regulation, or an enforcement guideline), it�s often easy to get tunnel vision: You focus on the one situation that prompted you to write the piece�usually a situation about which you feel deeply�and ignore other scenarios to which your proposal might apply. And this can lead you to make proposals that, on closer examination, prove to be unsound.

For instance, say you�re outraged by the government�s funding childbirths but not abortions. You might therefore propose a new rule that �if the government funds the nonexercise of a constitutional right, then the government must also fund the exercise of the right�; or you might simply propose that �if the government funds childbirth, it must fund abortions,� and give the more general claim as a justification. But you might not think about the consequences of this general claim�when the government funds public school education, it would also have to fund private school education (since that�s also a constitutional right), and when it funds anti-drug speech, it might also have to fund pro-drug speech.

Your argument, at least at its initial level of generality, is thus probably wrong or at least incomplete. But the focus on your one core case keeps you from seeing the error.

One way to fight these errors is a device borrowed from computer programming: the test suite. A test suite is a set of cases that programmers enter into their programs to see whether the results look right. A test suite for a calculator program might contain the following test cases, among many others:

1. Check that 2+2 yields 4.
2. Check that 3-1 yields 2.
3. Check that 1-3 yields -2 (because the program might work differently with positive numbers than with nega�tive ones).
4. Check that 1/0 yields an error message.

If all the test cases yield the correct result, then the programmer can have some confidence that the program works. If one test yields the wrong result, then the programmer sees the need to fix the program�not throw it out, but improve it. Such test suites are a fundamental part of sound software design. Before going into law, for instance, I wrote a computer program that had 140,000 lines of code, and 50,000 lines of test suites.

You can use a similar approach for testing legal proposals. Before you commit yourself to a particular proposal, you should design a test suite containing various cases to which your proposal might apply.

Assume, for instance, that you are upset by peyote bans that interfere with some American Indian religions. The government has no business, you want to argue, imposing such paternalistic laws on religious observers. You should design a set of test cases involving requests for religious exemptions from many different kinds of paternalistic laws, for instance:

1. requests for religious exemptions from assisted suicide bans, sought by a doctor who wants to help a dying patient die, or by the patient who wants a doctor�s help;
2. requests for religious exemptions from assisted suicide bans, sought by someone who wants to help physically healthy fellow cult members commit suicide;
3. requests for religious exemptions from bans on the drinking of strychnine (an example of extremely dangerous behavior);
4. requests for religious exemptions from bans on the handling of poisonous snakes (an example of less dangerous behavior);
5. requests for religious exemptions from bans on riding motorcycles without a helmet (an example of less dangerous behavior, but one that�unlike in examples 3 and 4�many nonreligious people want to engage in).

Then, once you design a proposed rule, you should test it by applying it to all these cases and seeing what results the proposal reaches.

What you might find by testing your proposals

What information can this testing provide?

1. Error: You might find that the proposal reaches results that even you yourself think are wrong. For instance, suppose that the proposal is that religious objectors should always get exemptions from paternalistic laws. Thinking about test case 2 might lead you to conclude that religious objectors should not be allowed to help physically healthy people commit suicide. The proposed rule, then, would be unsound. What can you do about this?

a. You might think that the proposal yielded the wrong result because it didn�t take into account countervailing concerns that may be present in some cases�for instance, the special need to prevent a voluntarily assumed near-certainty of death or extremely grave injury, rather than just a remote risk of harm. If this is so, you could modify the proposed test, for instance by limiting its scope (for example, by adding an exception for harms that are likely to be immediate, grave, and irreversible).
b. Another possibility is that the insight which led you to suggest the proposal�in our example, the belief that there should be a religious exemption from peyote laws�is better explained by a different rule. For instance, as you think through the test cases, you might conclude that your real objection to the peyote ban is that it�s factually unjustified (because peyote isn�t that harmful), and not that it�s paternalistic. You might then substitute a new rule: courts should allow religious exemptions from a law when they find that the religious practice doesn�t cause any harm, whether or not the law is paternalistic.

2. Vagueness: You might find that the proposal is unacceptably vague. Say that the proposal was that religious objectors should be exempted from paternalistic laws when �the objectors� interest in practicing their religion outweighs the government�s interest in protecting people against themselves.� In the peyote case, this proposal might have satisfied you, because it was clear to you that the government�s interest in protecting people against peyote abuse was so weak. But as you apply the proposal to the other cases, you might find that the proposal provides far too little guidance to courts�and might therefore lead to results you dislike. This could be a signal for you to clarify the proposal.

3. Surprise: You might find that the proposal reaches a result that you at first think is wrong, but then realize is right. For instance, before applying the proposal to the test suite, you might have assumed that religious objectors shouldn�t get exemptions from assisted suicide bans. But after you think more about this test case in light of your proposal, you might conclude that your intuition about assisted suicide was mistaken.
You should keep this finding in mind, and discuss it in the article: It may help you show the value of your claim, because it shows that the proposal yields counterintuitive but sound results.

4. Confirmation: You might find that the proposal precisely fits the results that you think are proper. This should make you more confident of the proposal�s soundness; and it would also provide some examples that you can use in the article to illustrate the proposal�s soundness.

Developing the test suite

How can you identify good items for your test suites? Here are a few suggestions:

1. Identify what needs to be tested. The test suite is supposed to test the proposed legal principle on which the claim is based. Sometimes, the claim is itself the principle: For instance, if the proposal is that �the proper rule for evaluating requests for religious exemptions from paternalistic laws is [such-and-such],� you would need a set of several cases to which this rule can be applied.
But sometimes the claim is just an application of the principle; for instance, the claim that �religious exemption requests from peyote laws should be granted� probably rests on a broader implicit principle that describes which exemption requests should be granted. If that�s so, then you should come up with a set of cases that test this underlying principle. One case should involve peyote bans but the others shouldn�t.
2. Each test case should be plausible: It should be the sort of situation that might actually happen. It�s good to base it on a real incident, whether one drawn from a reported court decision or a newspaper article. You need not precisely follow the real incident, and you may assume slightly different facts if necessary�the goal is to have the reader acknowledge that the case could happen the way it�s described, not that it necessarily has happened. But you should make sure that any alterations still leave the test case as realistic as possible.
3. The test suite should include the famous precedents in this field. This can help confirm for you and the readers that the proposal is consistent with those cases�or can help explain which famous cases would have to be reversed under the proposal.
4. At least some of the cases should be challenging for the proposal. You should identify cases where the proposal might lead to possibly unappealing results, and include them in the test suite. Skeptical readers, including your adviser, will think of these cases eventually. Identifying the hard cases early�and, if necessary, revising the proposal in light of them�is better than having to confront them later, when changing the paper will require much more work.
5. The test cases should differ from each other in relevant ways, since their role is to provide as broad a test for the claim as possible. If you are testing a claim about paternalistic laws, for instance, you shouldn�t just focus on various drug laws, or just on paternalistic laws aimed at protecting children. You should think of many different sorts of paternalistic laws, and choose one or two of each variety.
6. The cases should yield different results. For instance, if your proposed rule judges the constitutionality of a certain type of law, you should find some laws that you think should be found unconstitutional, some that you think should be found constitutional, and some whose constitutionality is a close question.
7. The cases should involve incidents or laws that appeal to as many different political perspectives as possible. Say that you are a liberal who wants to argue that the Free Speech Clause prohibits the government from funding viewpoint-based advocacy programs. You might have developed this view because you think the government shouldn�t be allowed to fund anti-abortion advocacy, and your proposal will indeed reach the result you prefer in that case.

But what about advocacy programs that liberals might favor, such as pro-recycling advocacy, or advertising campaigns promoting tolerance of homosexuals? It would help if the test suite included such cases, plus generally popular programs such as anti-drug advertising, or programs that even small-government libertarians might like, such as advocacy of respect for property rights (for instance, anti-graffiti advocacy). This wide variety of test cases will help show you whether the proposal is indeed sound across the board, or whether even you yourself would, on reflection, oppose it.