Back

When prose is free, the memo worth approving names what would change the author's mind

·19 min read

Page 191 of the Columbia Accident Investigation Board's report, published in August 2003, reproduces one briefing slide and a critique of it. The slide came from a Debris Assessment Team briefing to NASA's Mission Evaluation Room about what the foam that struck Columbia's wing might have done to the heat-shield tiles. It has a title and nested bullets that create six levels of hierarchy, and the critique counts eleven simple sentences in it. Its title says that a review of test data indicates conservatism, a phrase that refers to the choice of test models and says nothing about the predicted damage. The fact that mattered most, that one estimate of the debris was 640 times larger than the data used to calibrate the damage model, appears only at the bottom. The Board added that later analysis put the figure at 400 times. By either figure, that comparison belonged in the headline.

The Board's claim is narrower than blaming the slide for the loss of the shuttle. At many points in the investigation it was surprised to receive slides where it expected technical reports, and it read the habit as an illustration of problematic technical communication at NASA. The annotated copy printed in the report adds that the high-level NASA officials for whom the engineers' reports were written were satisfied that Columbia was not in danger, and no attempt was made to examine the threat further. Nobody can say what a six-page memo would have done in that room. What the page does show is a decision about whether to look harder at the wing, made by officials who were satisfied by engineering reports of the kind this slide exemplifies, one that buried its own most important fact.

I will argue that a written decision document does two jobs that are easy to confuse. It forces its author to commit to a position, and it leaves a record that others can audit after the people involved have moved on. We have judged such documents mostly by how well they read, a cheap and imperfect proxy for the first job. Language models have cut the cost of fluent prose to almost nothing, so the proxy has stopped working, and reviewers need to ask for the parts of a decision that a prompt cannot supply: what was chosen, what was rejected, what evidence supports the choice, and what observation would change the author's mind. I hold this for decisions that are expensive to reverse. For small ones I think it is wrong, a limit I return to below.

The Columbia slide buried its own headline

The Board's section on the slide is titled "Engineering by Viewgraphs," and the failure it describes is one of priority. Information on a slide is ranked by its position and its indentation, and the ranking can contradict the engineering. The vaguely quantitative word "significant" or "significantly" appears five times on the slide, the report points out, with meanings that range from detectable in an irrelevant calibration case to enough damage that everyone dies. The unit of volume is written three different ways. Edward Tufte, whose analysis the report reproduces, proposed a different headline: the review of test data indicates that two models are irrelevant.

The same chapter also benchmarks NASA against the Navy's submarine safety program. In the Navy system, the Board observed, the technical recommendations of contractors are documented in peer-reviewed formal written correspondence, and PowerPoint briefings are not accepted as a substitute for completed staff work. The two systems differ in what must happen before a recommendation moves. In the Navy's, the author has to finish the thought on paper and expose it to reviewers. In a briefing, as I read the contrast, the thought gets compressed, and everyone in the room can assume someone else holds the details.

Two limits apply. One accident is one case, and a memo can bury a headline as well as a slide can. The Board's remark concerns format rather than prose style, so the lesson I take from it is about what an artifact makes the author state. A format that never asks what is being decided, or which piece of evidence matters most, will let both go missing.

Writing makes the author commit, and the record outlasts the people

The first claim has a compact statement at the head of Chapter 1 of Leslie Lamport's 2002 book Specifying Systems: "Writing is nature's way of letting you know how sloppy your thinking is," credited there to Guindon. Lamport adds that mathematics does the same to writing. His argument for specifications is practical: specifying a system helps us understand it, so it is worth doing before the system is built.

Amazon turned this into a management routine. In his 2017 letter to shareholders, Jeff Bezos wrote that the company does not use slide presentations and instead writes narratively structured six-page memos, which everyone reads in silence at the start of a meeting, in what he calls a study hall. He admits that quality varies widely. His diagnosis of the weak ones is that authors misjudge the scope of the work: a memo of the standard he wants is rewritten, shared with colleagues, set aside for a couple of days and edited again, and probably takes a week or more. A secondhand briefing on Colin Bryar and Bill Carr's book Working Backwards describes the silent reading as lasting about twenty minutes and says Tufte's writing on slides influenced the decision to drop them. I did not open the book, so I treat those two details as reported and not verified.

The second job, the auditable record, is where the engineering literature is more specific. Michael Nygard proposed architecture decision records in November 2011 with an argument about newcomers. Without a record of why a decision was made, a person joining a project can only accept it blindly or reverse it blindly. Blind acceptance leaves a team afraid to touch anything, and blind reversal can destroy a property the old decision protected. His format is one or two pages with a title, the context in value-neutral language, the decision in active voice beginning "We will," a status, and all the consequences, the negative ones included. Numbers are never reused, and a reversed decision stays on file marked as superseded, because it is still relevant to know it was once the decision. He also writes, in parentheses, that bullets kill people, even PowerPoint bullets.

Malte Ubl's 2020 description of design docs at Google adds the item I would single out for reviewers. Besides context and goals, he asks for explicit non-goals and a section on alternatives considered, which he calls one of the most important in the document because it shows why the chosen design beats the others against the stated goals. He also lists what the docs are for: finding design problems while changes are cheap, building consensus, and forming an organizational memory. GitLab applies the same logic to company process. Its handbook says to write the solution down first and announce it afterward, and it keeps only 90 days of chat history on purpose, so that chat cannot become the place where projects are managed end to end.

These sources agree on the mechanism, and I found none that disagrees, but the agreement is weaker than it looks. They are descriptions by practitioners at firms that adopted the practice and survived, and Nygard's own experience report covers a few months on a few projects. GitLab's claims of efficiency are the company describing itself. I know of no controlled comparison in which similar teams decided with and without a written record. The case rests on mechanism, on the Columbia finding, and on the plain fact that a decision no one wrote down cannot be audited.

Free fluency removes the proxy that reviewers relied on

Fluent prose used to cost effort, which is why it carried information. A reader who met a clear, organized, well-edited memo could infer that its author had spent time and probably understood the subject. That inference was never reliable, but it was cheap to make, and organizations leaned on it.

There is direct evidence of the proxy failing in a neighboring setting. Anaïs Galdin and Jesse Silbert studied job proposals on the freelance platform Freelancer.com, where, before language models became widely available, employers showed a high willingness to pay for applications that were more tailored to the posting. After the models arrived, that willingness fell sharply, and in their post-adoption sample the hiring rate was flat across the whole range of the tailoring measure. In their structural model, a market where written applications carry no information about ability hires the top fifth of workers 19% less often than before and the bottom fifth 14% more often. That last result comes from a simulation, and a market for short proposals is not a design review. I still think the mechanism carries over: a signal that costs nothing to produce no longer separates the careful from the careless.

Peer review shows the same shift from the reader's side. Weixin Liang and colleagues estimated that between 6.5% and 16.9% of the review text at ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023 could have been substantially modified by a language model, with higher shares in reviews that reported low reviewer confidence or were submitted close to the deadline. In 2026 the ICML organizers ran a randomized experiment in which one group of reviewers was forbidden to use language models and another was allowed limited use. In the survey that followed, 22.5% of the forbidden group said they had used one anyway. The organizers did not use generic AI-text detectors. They hid instructions in the submission PDFs that made a model include chosen phrases in its review, flagged 795 reviews from 506 reviewers, which is about 1% of all reviews, and had a person check every case. They acknowledge the method probably caught only the careless. The experiment also found that the policy a reviewer was assigned had almost no effect on final decisions or scores. That null result says the two policies were hard to tell apart in outcomes. It does not say whether the model-assisted reviews were any good.

The productivity gain has experimental support too. In a preregistered experiment with 453 college-educated professionals reported by Shakked Noy and Whitney Zhang in 2023, access to ChatGPT cut the time spent on occupation-specific writing tasks by about 40% and raised the quality ratings given by independent evaluators by about 18%, and the gap between workers narrowed. Quality there means grades from blinded evaluators who work in the same occupations, with three evaluators per piece and, in the working-paper version I could open, an average agreement between them of only 0.44 as a correlation. The tasks were short, self-contained assignments of 20 to 30 minutes. I found no separate audit of factual accuracy in that version, so quality in the study is mostly what an informed reader perceives.

This changes the first job of a decision document. The benefit of writing, that it forces the author to think, depends on the author being the one who struggles with the sentences. A model can turn an unresolved idea into a resolved-sounding narrative while the author resolves nothing. The sloppiness that Guindon's line says writing exposes can now be covered by the tool, and the artifact still looks like evidence of the effort Bezos asks for. That is my inference, and I found no study that tests it for memos or design docs.

Reviewers should ask for what a prompt cannot invent

A model can write a persuasive account of nearly any decision, so a review has to ask for things that are checkable and that live outside the document. The template I propose has five questions, and each one is an old idea from the sources above, made to depend on a pointer to something a reviewer can open.

  1. What exactly is being decided, in one sentence beginning "We will," and what happens by default if nobody decides?
  2. Which alternatives were rejected, who proposed each, and what did each have in its favor?
  3. What is the evidence, where did it come from, and which parts were observed and which inferred?
  4. What observation, measured how and by what date, would make the author reverse the decision?
  5. What does reversal cost, and who pays if the author is wrong?

Question 1 is Nygard's decision statement, with a test attached: a sentence that cannot be written in that form means the decision has not been made. Ubl's alternatives section, with names added, becomes question 2, because a list of alternatives that no one argued for is the kind of thing a model produces easily. Observation and inference are separated in question 3, which is the line the Columbia slide blurred, since the 640-fold figure was an estimate sitting in a bullet beside the test-database numbers it was compared with. Question 4 is the one I consider most important. Plans routinely include a sentence like "we will revisit if conditions change," which commits the author to nothing. A threshold, a way of measuring it and a date give a stranger something to check a year later. Question 5 comes from Nygard's requirement to list all the consequences, plus the question of who bears them.

The pointer requirement carries most of the weight. Every answer to questions 2, 3 and 4 should link to a thread, a ticket, an experiment run or a named person who made the objection. An answer without a pointer is a claim, and a reviewer should treat it as one. A model can invent a plausible alternative, but it cannot make the alternative be the one that someone actually raised in an actual meeting, and it cannot produce a working link to an experiment that was never run.

A decision record with invented numbers shows what a reviewer can attack

This example is invented for illustration: the team, the people and every number in it. A team that classifies supplier invoices pays for a large model on every document. A draft answering the five questions might read as follows. We will send invoices from the 70% of suppliers with a known template to a smaller model and keep the large model for the rest. The default if nobody decides is a monthly bill that has risen from $9,000 to $18,000 over two quarters and will keep rising with volume. The rejected alternatives are staying on the single large model, which a finance partner argued for on the grounds of simplicity, and fine-tuning the smaller model, which an engineer proposed and which was dropped because collecting labeled data would take about two months. Evidence comes from a two-week shadow run on 1,200 invoices, linked, in which the smaller model matched the large one on 97.1% of fields. That was observed. The expectation that template invoices behave the same way next quarter was inferred. The decision reverses if, on a weekly audited sample of 400 invoices, the error rate exceeds 3% for two consecutive weeks, with a first review on 15 January. Reversal costs about a day of routing configuration, and finance bears the cost of any misposted invoice until it is caught.

A reviewer can attack every line of that. They can ask whether the shadow run covered the month-end spike, or whether the finance partner agrees the objection was fairly stated. They can also check the trigger with arithmetic, which I did using the binomial distribution. If the true error rate were exactly 3%, a weekly sample of 400 invoices would show 13 or more errors, enough to read above 3%, in about 42% of weeks, so two such weeks in a row would force a review in roughly 18% of two-week windows by chance alone. If the true rate were 4%, about 66% of two-week windows would trigger the rule, and at 2% almost none would. The threshold is usable, but the author should know it is noisy, and a reviewer who sees the number can ask for a larger sample or a longer window.

A fluent paragraph that said "we evaluated several approaches and selected the most cost-effective option" would offer none of those handles. A model can write either version in seconds. Only the first can be wrong in a way that someone can detect and correct.

Writing is expensive, and some decisions belong in the room

The strongest objection is about price. Ubl says plainly that design docs are overhead, that reviews can be a dangerous trap of overhead, and that if the solution is not ambiguous a doc has little value. GitLab concedes that handbook-first work takes more time up front because you must decide where a change belongs and fit it into existing text. Bezos puts a week or more on a good memo. Multiply by the number of decisions a team makes, add the review questions above, and the process can cost more than many of the decisions are worth. Applied to every choice of library, five questions with pointers would take longer than the choice itself.

A second objection concerns what meetings do. The Amazon practice, in both Bezos's account and the secondhand one, pairs the memo with a meeting: the document is read in the room and the discussion follows. A meeting lets disagreement surface at the speed of conversation and conveys stakes that prose flattens. It also allows an author to be questioned without notes, which bears directly on the model problem. An author who cannot answer a follow-up about the fourth question has told you something that no review of the text would. An oral defense exposes a ghostwritten position quickly, and nothing in the written practice replaces it.

A third objection is that cheaper writing is a gain, and it is. Noy and Zhang's result suggests it helps weaker writers most, and I would expect it to help people drafting in a second language, though that goes beyond what their study measured. An organization that refuses model help in order to preserve the struggle is paying a real price in uneven access to clear expression.

My answer is a sorting rule. Reserve the full format for the decisions that Nygard calls architecturally significant and that Ubl's test marks as ambiguous: hard to reverse, shared across teams, or resting on evidence that is contested. For everything else, a paragraph in a ticket is enough, and a conversation is fine. Allow model help with drafting, since it is cheap, but require that the person who signs owns the five answers and can defend them live. The rule costs something, and it assumes reviewers will actually click the links, which no policy can guarantee.

No study I found tests whether these questions catch weak memos

I could not find a study in which reviewers with a checklist of this kind were compared with reviewers without one, on memos of known quality that include memos drafted by models. The sources above support the premises: a format can hide the key fact, and in the settings studied the polish of text carries less information than it did. They do not show that the five questions catch weak memos better than a careful read. The test could be run. Take an archive of decision documents whose outcomes are known, strip the authorship, let some reviewers read for quality and others apply the five questions with pointer checks, and compare how many of the eventual failures each group flagged.

A team can run a small version of this without a study. Take the last three decisions your group approved, ask the author of each to answer the five questions, and count how many of the answers to questions 2, 3 and 4 end in something a stranger can open.

Sources

  1. Columbia Accident Investigation Board (August 2003). Report Volume I. NASA and Government Printing Office. Chapter 7, section 7.5, "Engineering by Viewgraphs" (p. 191, including the annotated slide and Tufte's analysis) and the NASA/Navy benchmarking discussion (p. 182). https://s3.amazonaws.com/akamai.netstorage/anon.nasa-global/CAIB/CAIB_lowres_full.pdf
  2. Bezos, J. (2017 letter). Amazon.com 2017 Letter to Shareholders, section "Six-Page Narratives." https://www.aboutamazon.com/news/company-news/2017-letter-to-shareholders
  3. Bryar, C., Carr, B. (2021). Working Backwards: Insights, Stories, and Secrets from Inside Amazon. St. Martin's Press. Cited only as summarized in Charterworks, "Book briefing: Working Backwards" (https://www.charterworks.com/book-briefing-working-backwards-by-colin-bryar-and-bill-carr/); I did not open the book.
  4. Nygard, M. (15 November 2011). Documenting Architecture Decisions. Cognitect blog. https://www.cognitect.com/blog/2011/11/15/documenting-architecture-decisions
  5. Ubl, M. (6 July 2020). Design Docs at Google. Industrial Empathy. https://www.industrialempathy.com/posts/design-docs-at-google/
  6. GitLab. The importance of a handbook-first approach to communication, and Handbook Usage (the passage on up-front time cost). The GitLab Handbook. https://handbook.gitlab.com/handbook/company/culture/all-remote/handbook-first/ and https://handbook.gitlab.com/handbook/about/handbook-usage/
  7. Lamport, L. (2002). Specifying Systems: The TLA+ Language and Tools for Hardware and Software Engineers. Addison-Wesley. Chapter 1, "What Is a Specification?" https://lamport.azurewebsites.net/tla/book-02-08-08.pdf
  8. Galdin, A., Silbert, J. (11 November 2025). Making Talk Cheap: Generative AI and Labor Market Signaling. arXiv:2511.08785 (v1).
  9. Liang, W. et al. (2024). Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews. Proceedings of the 41st International Conference on Machine Learning, PMLR 235; arXiv:2403.07183.
  10. Kim, S. S. Y. et al. (16 September 2026). Use and Effects of LLMs in Peer Review: A Randomized Experiment and Survey at ICML 2026. arXiv:2609.19420; and ICML 2026 organizers (18 March 2026), On Violations of LLM Review Policies. https://blog.icml.cc/2026/03/18/on-violations-of-llm-review-policies/
  11. Noy, S., Zhang, W. (2023). Experimental evidence on the productivity effects of generative artificial intelligence. Science, 14 July 2023, pp. 187-192. DOI 10.1126/science.adh2586. The 453 participants and the 40% and 18% figures are from the published version as summarized by Stanford SCALE; the evaluator details come from the March 2023 working paper (444 participants), https://economics.mit.edu/sites/default/files/inline-files/Noy_Zhang_1_0.pdf
  12. Illustrative decision record: the team, people, invoice volumes, costs, sample sizes, error thresholds and dates in the worked example are invented for this essay and describe no real team. The probabilities quoted for the 400-invoice audit (42% of weeks at a true 3% error rate, about 18% of consecutive pairs of weeks, about 66% of consecutive pairs at a true 4%) are my own binomial calculation on those invented inputs.