Back

A CTO's no to hype needs a test, a tripwire and a log, or it is only a mood

·18 min read

Bessemer Venture Partners keeps a page on its website called the Anti-Portfolio. It lists seven companies the firm passed on, each with a short account of the pass, and its introduction says the only thing worse than missing a great company is pretending it did not happen. Google is on the page. In 1999 and 2000 a college friend of the partner David Cowan rented her garage to two Stanford students who were building a search engine, and twice tried to introduce them to him. By the page's account, Cowan replied by asking how he could leave her house without going anywhere near the garage. Facebook (2004), Tesla (2006), Atlassian and Apple appear as well. The partner Byron Deeter passed on Tesla because of its negative margins and put down a deposit on the car anyway.

A technology leader's refusals leave no page like that, and this is my starting point. A CTO who says no to a hyped technology produces no invoice and no incident. The company does something else instead, and nobody can compare that world with the one in which the answer was yes. My argument is that a no is credible only when it comes with a procedure: written criteria, a cheap test with a deadline, a published tripwire that would change the answer, and a dated log of refusals that someone reviews on a schedule. The log matters most, because the other three decay without it.

A refusal without those parts is a preference, however experienced the person holding it. A CTO who cannot name the evidence that would reverse a refusal should expect to be overruled, and some of the time the overruling will be right. I also think the right rate of reversal is above zero. A leader whose refusals are never reversed has probably stopped testing them.

Wrong refusals vanish unless someone writes them down

The Anti-Portfolio is useful mainly as a record, and the individual stories matter less. Bessemer's account of Atlassian starts with a first contact in 2006, moves to a 2010 chance to invest at a valuation of about $400 million that the team found rich, and ends with the 2015 initial public offering, which the page calls the largest technology listing in Australian history. The Facebook entry records a 2004 pass after a partner told Eduardo Saverin that Friendster was finished. In a July 2006 interview on the public radio program Marketplace, Cowan said the firm does not want to claim credit for its successes, and he criticized venture websites that make portfolio companies look destined to win, since most of the credit belongs to founders and employees.

The page also has a limit that matters for anyone copying it. It lists the misses that became famous and none of the passes that turned out right, so there is no denominator, and a reader cannot compute how often the firm passes on something good. Seven entries on a web page could describe a firm that is too cautious or one with an ordinary miss rate; the page cannot tell them apart. A CTO's log has to include every refusal, the boring ones too, or it will show only the regrets that happened to be memorable.

I expect technology refusals to be harder to reconstruct than investment passes. A no is often delivered by email or in a hallway, with a reason that sounded sufficient on the day. Months later, when a competitor ships something similar, the original reasoning has usually gone, and the organization remembers the outcome and supplies a reason that fits it. A reason written down on the day, together with the evidence that would have changed it, lets a later reader tell a bad judgment from bad luck.

Technology judgments get feedback too slowly to trust confidence

Daniel Kahneman and Gary Klein spent years on opposite sides of a debate about expert intuition, and in 2009 they published a joint paper in American Psychologist on where they agreed. Their answer to when an expert's gut can be trusted has two conditions. The environment must contain cues that reliably predict outcomes, and the person must have had the chance to learn those cues, which they describe as prolonged practice with feedback that is both fast and unambiguous. Firefighters and nurses meet the conditions. Long-term forecasts of political events, they say, do not: they cite Tetlock's study of expert political forecasts, in which highly educated experts did no better than untrained newspaper readers. Kahneman adds that stock analysts, psychotherapists and intelligence analysts can receive ample feedback on short-term tasks that supports their confidence, while the feedback on their long-term judgments is delayed, sparse and ambiguous, a mix the paper calls conducive to overconfidence. Their paper states plainly that subjective confidence is not a reliable indicator of accuracy.

A CTO's adoption decisions sit uncomfortably between the two cases. Whether a vendor's agent platform handles a company's invoices correctly can be learned in weeks. Whether that vendor survives and whether the market settles on the standard it picked are questions that resolve over years, mixed with a dozen other causes, and a CTO makes perhaps a handful of such big decisions a year. That is too few feedback cycles for intuition to form.

The practical response has two parts. First, convert the judgment into questions with short feedback loops. Asking whether a technology will matter is a long-loop question; asking whether it completes forty of our own cases correctly in a month is a short-loop one. Second, attach a probability to the long-loop calls that remain and score them later. This is the habit that Philip Tetlock and Dan Gardner describe in Superforecasting (2015), drawn from a forecasting tournament that the US intelligence research agency IARPA began running in 2011, in which probabilistic forecasts were scored with Brier's squared-error rule. I take the scoring habit from that work and nothing more. The log in this essay is a small private version of it.

Both errors cost money, and only one shows up in the budget

Dan McKinley's talk Choose Boring Technology, which he published as an essay on 30 March 2015, makes the usual argument against a wrong yes. Every company, he says, gets about three innovation tokens for a long while, and each unproven choice, such as MongoDB or a service-discovery tool less than a year old, spends one. The expense is mostly in operating the thing: he writes that the long-term costs of keeping a system working vastly exceed the inconvenience of building it. He is not an absolutist, though. He writes that you should choose new technology sometimes, and he asks for a written account of what is too hard with the current stack before a token is spent. Klarna's 2025 reversal on its AI customer-service assistant, which an earlier essay on this site covers, is the example I would point to for the cost of a wrong yes.

The wrong no is harder to price, and one reason is that refusing an official tool rarely stops people using it. Microsoft and LinkedIn's 2024 Work Trend Index surveyed about 31,000 knowledge workers in 31 markets between mid-February and late March 2024, with the fieldwork done by Edelman Data & Intelligence. It found that 78 percent of the people who use AI at work bring their own tools. The figure comes from self-report in a report produced by Microsoft, which sells AI products, and I would not rest a budget on the exact number. The direction is plausible, though, and it changes what a refusal does: the demand stays, and the use moves out of sight, where the CTO cannot set data rules for it.

Whether a late yes is cheap depends on the kind of technology, and I want to be explicit that this is my judgment and not a measurement. For infrastructure, a late yes is usually cheap. The product matures, prices fall, and other companies find the sharp edges first. A wrong early yes is expensive because migrations are slow. For capabilities whose payoff comes from the organization learning to use them, such as redesigned workflows or a team's skill at evaluating model output, a late yes costs more, because that learning cannot be bought later. The first kind of decision deserves a high bar and the second a low one. A procedure should ask which kind it is facing before it asks anything else.

Eight lines on one page are enough to make a refusal auditable

The eight lines below fit on one page. Any numbers in them are mine and are meant as illustrations.

  1. State the claim, who is making it, who decides, and the date. "Vendor X's agent will resolve a third of our tier-one tickets" is a claim; "agents are the future" is not.
  2. Write the criteria before the demo: the outcome, the metric, the baseline from the current process, and the pass line. A vendor demo can then be judged against a number fixed in advance.
  3. Run the smallest test that could embarrass the claim. Use four weeks at most, your own data, a human baseline on the same cases, and repeated runs of each case.
  4. Cap the cost of a yes in advance: the scope, which innovation token it spends, and the exit plan if it fails.
  5. Publish a tripwire to the proposer: the specific evidence that reopens the question, with a recheck date. It can be a price, a measured reliability on your cases, a peer's production result or a regulatory change.
  6. Record your probability that this refusal will look wrong at the review date.
  7. Enter the decision in the log, with the date, the one-line reason, the test result, the tripwire and the review date.
  8. Review the log every quarter, score the probabilities, and record which refusals reversed.

I expect items 2 and 5 to be where refusals most often fail. Without a pass line the test becomes an argument after the fact, and without a tripwire the refusal can be reopened only by escalation, and I suspect a team that must escalate to be heard will eventually stop proposing. A tripwire tells the proposer that the answer is no until some named evidence appears, and the proposer can go and produce it.

An agent product shows why the test repeats runs and the tripwire expires

Take the recent wave of products sold as agents. Gartner predicted in June 2025 that over 40 percent of agentic AI projects would be canceled by the end of 2027, and it used the term agent washing for vendors that relabel assistants, scripts and chatbots; the release itself returned an access error when I tried to open it, so I rely on trade coverage of it. A forecast about a category says little about the one vendor in front of a CTO this week, and a test can.

The test has to measure repeated runs and not one good demo. The τ-bench benchmark, by Shunyu Yao and colleagues (arXiv, June 2024), defined pass^k as the chance that an agent solves the same customer-service task in all k separate trials. Its best 2024 agent, GPT-4o, solved about 61 percent of retail tasks in a single trial but fell to roughly 25 percent when all eight trials had to succeed. Those are old numbers for one older model, and newer models score higher by an amount I did not check. An earlier essay on this site, on when a workflow beats an agent, works through the metric in detail; here I only need its consequence for the log.

Here is a made-up example of that consequence. Suppose a vendor demo shows an agent resolving a handful of support cases, and the criteria written beforehand say that 85 percent of cases must pass all five of five repeated runs, with no failures on refunds above a set amount. The test uses 40 cases from last month's queue, each run five times. If the agent succeeds on 80 percent of runs and every case is equally hard, a case passes all five runs about 33 percent of the time (0.8 to the fifth power). If instead 20 cases are always solved and the other 20 are solved 60 percent of the time, the average is still 80 percent, but about 54 percent of cases pass all five. Either way the agent misses the pass line, and the log entry is short: refused on this date because of this measured consistency, to be reopened when the vendor shows 85 percent on our set or offers pricing per audited resolution. The vendor can read that and know how to win the decision back.

Evidence moves, so the tripwire needs an expiry date. Coding assistants show how fast. METR's randomized trial of 16 experienced open-source developers on 246 of their own tasks, published in July 2025, measured tasks taking 19 percent longer with AI, although the developers had predicted a 24 percent speedup. A CTO who refused on that evidence was following it. Seven months later METR reported on a follow-up with 57 developers, 10 of them returning from the first study. Its raw estimates leaned toward a speedup (tasks taking 18 percent less time for the returning developers and 4 percent less for the new ones), but the 95 percent intervals ran from -38 to +9 percent and from -15 to +9 percent, so both included no effect. METR called the signal unreliable, in part because 30 to 50 percent of surveyed developers said they had withheld tasks they did not want to do without AI. A refusal based on the 2025 result needed a recheck date, and the recheck would have found that the measuring instrument itself had degraded. Both facts belong in the log.

The strongest objection is that a CTO who refuses often becomes the risk

A leader with a procedure for saying no may simply have a more respectable way of saying it. The incentives lean that way, because a wrong yes is traceable to a person, with an invoice and a migration plan attached, while a wrong no is diffuse and shows up as a rival's launch. Misses do cost something, and the Anti-Portfolio is a record of them. The saying that people overestimate the effect of a technology in the short run and underestimate it in the long run is often called Amara's law, after the futurist Roy Amara, and it is used to argue that skeptics lose over time. The attribution is shakier than the saying is popular. Quote Investigator traces a related saying, about what can be done in one year against five or ten, to J. C. R. Licklider's 1965 book Libraries of the Future, where he calls it a modern maxim and disclaims credit, and the earliest explicit attribution to Amara it documents is a 1989 collection, What Futurists Believe. The saying is a heuristic that nobody has measured, and I would not call it a law.

A second version of the objection says to follow the cycle. If hype eventually collapses into a trough and recovers, a CTO could wait for the trough and buy cheaply. That is the pitch of Jackie Fenn and Mark Raskino's Mastering the Hype Cycle (2008), which extends a model that, according to a 2009 review in Strategy & Leadership, Gartner's Fenn first drew in 1995. Ozgur Dedehayir and Martin Steinert tested the model against Gartner's own technology databases in a 2016 review in Technological Forecasting and Social Change and reported incongruences with Gartner's reports; they suggested the idea of hype dynamics could survive inside other life-cycle models, but not necessarily the cycle itself. I have read only the abstract-level account of that paper, but on that account I treat waiting for the trough as a timing rule that this review gives no reason to trust.

My answer to the objection is that the log is how a CTO can find out whether the objection applies to them. Suppose, in an illustrative case, a CTO records 12 refusals in a year, each with a 20 percent stated chance of looking wrong at the review. The expected number of reversals is 2.4. If the stated probabilities were honest, zero reversals would occur about 7 percent of the time (0.8 to the twelfth power), and six or more about 2 percent of the time. A single year of zero reversals is therefore only a hint that refusals have become permanent or that nothing is being tested, and a second year of it would be harder to explain. Six or more would mean the refusals were far too confident. Neither result is a verdict, since 12 cases is a small sample, but both would be invisible without the log.

Two other measurements belong next to the reversal rate. One is the number of tests run per quarter: a CTO who has run none is a blocker whatever the quality of the reasoning. The other is the time from proposal to answer, since a no that takes six months arrives after the shadow users have already decided.

The log also has limits that I cannot remove. It records refusals someone proposed and says nothing about ideas nobody dared to raise. It weights every refusal equally, while one wrong no on a decisive technology can cost more than a hundred correct ones, so each entry should carry a rough estimate of the upside if the refusal proves wrong. And a CTO reviewing their own log is grading their own work. Handing the quarterly review to a peer or to the proposers whose ideas were declined is cheap, and it is the part of the procedure I would least like to skip.

The Thoughtworks Technology Radar, whose April 2026 edition (volume 34) has 118 entries by my count of the lists on its page, shows the institutional version of this. A Technology Advisory Board of about 20 senior technologists debates each candidate and assigns it a ring. The fourth ring, called Caution on the current site, is for things that are widely accepted but that Thoughtworks has had poor experiences with. The volumes are public and dated, so anyone can look back at what was said. I did not find any published scoring of past placements against outcomes, and that missing step is the one I am proposing for a single company.

A concrete place to begin is the email trail from the last six months. List every proposal that was declined or deferred, write the reason from what the thread says, assign a probability that each will look wrong, and put a review date on the page. The first review, for a log started this week, falls on 14 January 2027.

Sources

  1. Bessemer Venture Partners. The Anti-Portfolio. bvp.com/anti-portfolio (opened October 2026); and Marketplace, American Public Media (July 2006). Segment on the anti-portfolio, interview with David Cowan. marketplace.org/story/2006/07/28/anti-portfolio.
  2. Kahneman, D. and Klein, G. (2009). Conditions for Intuitive Expertise: A Failure to Disagree. American Psychologist 64(6), 515-526. DOI 10.1037/a0016755.
  3. McKinley, D. (30 March 2015). Choose Boring Technology. mcfunley.com/choose-boring-technology.
  4. Tetlock, P. E. and Gardner, D. (2015). Superforecasting: The Art and Science of Prediction. Crown. The IARPA tournament and Brier scoring are described in secondary summaries (including the Wikipedia entry on the Good Judgment Project, which gives a July 2011 start); I did not open the book.
  5. Microsoft and LinkedIn (2024). 2024 Work Trend Index: AI at Work Is Here. Now Comes the Hard Part. Survey of about 31,000 workers in 31 markets, fielded 15 February to 28 March 2024 by Edelman Data & Intelligence.
  6. Gartner (25 June 2025). Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027. Press release; details taken from trade-press coverage (CRN India, dated 25 June 2025, and others) because the release itself was blocked when I tried to open it.
  7. Yao, S., Shinn, N., Razavi, P., Narasimhan, K. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045.
  8. METR (10 July 2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study; and METR (24 February 2026). We are Changing our Developer Productivity Experiment Design, metr.org/blog/2026-02-24-uplift-update.
  9. Quote Investigator (3 January 2019, updated 2024). People Tend To Overestimate What Can Be Done In One Year And To Underestimate What Can Be Done In Five Or Ten Years. quoteinvestigator.com/2019/01/03/estimate.
  10. Fenn, J. and Raskino, M. (2008). Mastering the Hype Cycle: How to Choose the Right Innovation at the Right Time. Harvard Business Press; Davidson, A. (2009). A model for picking technology winners? Strategy & Leadership (review of Fenn and Raskino), DOI 10.1108/sl.2009.26137bae.001; Dedehayir, O. and Steinert, M. (2016). The hype cycle model: A review and future directions. Technological Forecasting and Social Change 108, 28-41. DOI 10.1016/j.techfore.2016.04.005.
  11. Thoughtworks (April 2026). Technology Radar, volume 34, and the Radar FAQ on rings and the Technology Advisory Board. thoughtworks.com/radar.
  12. Illustrative arithmetic, made up for this essay: 40 cases run 5 times with an 85 percent pass line; per-run success of 80 percent with equal difficulty (0.8^5 = 33 percent) or with half the cases always solved and half solved 60 percent of the time (0.5 + 0.5 x 0.6^5 = 54 percent); 12 refusals each given a 20 percent chance of reversal (expected 2.4; zero reversals 6.9 percent; six or more about 1.9 percent; zero reversals in two successive years under the same assumptions, about 0.5 percent).