Your AI system will copy your org chart, and the eval is where the copy shows

In 2008 Nachiappan Nagappan, Brendan Murphy and Victor Basili published a study of Windows Vista covering 3,404 binaries and more than 50 million lines of code. They tried to predict which binaries would fail after release. A model built on eight organizational measures identified failure-prone binaries with 86.2% precision and 84.0% recall. The measures included how many engineers had edited a binary, how many of them had since left the company, and how far down the reporting tree sat the manager whose organization made more than three quarters of the edits. Code churn reached 78.6% and 79.9%, and code complexity 79.3% and 66.0%. Dependencies (74.4% and 69.9%), code coverage (83.8% and 54.4%) and counts of bugs found before release (73.8% and 62.9%) all scored lower still. For this one system, the org chart said more about where the product would break than any of the code-based models.

Melvin Conway had claimed in 1968 that systems end up shaped like the communication structure of the group that builds them, and the Vista result is a large measurement in that direction. My argument is that AI products inherit this structure at least as strictly as conventional software, and probably more strictly, because they add artifacts that nobody had to own before: an eval set, a retrieval index, a prompt, a pinned model version. Those artifacts sit on the seams between the teams most companies create first, which are a data team, an ML platform team and a product team. The eval set is the one artifact that can say whether the parts work together, and in the usual split it belongs to nobody.

ThoughtWorks writers named the remedy the inverse Conway maneuver: decide the system boundaries you want, then shape the teams to match. For AI products I would put the eval set inside a stream-aligned product team, run the platform as a service with paved roads, and keep scarce specialists in a small enabling group. A later section separates the evidence for this from the heuristic, because the topology is more popular than it is tested.

The evidence shows mirroring as a strong default, though it falls short of a law

Conway's paper "How Do Committees Invent?" ran in Datamation in April 1968. Its best-known sentence says that organizations designing systems are constrained to produce designs that are "copies of the communication structures" of those organizations. The reasoning behind it is more specific than the slogan. A large design is divided into subsystems, each subsystem goes to a group, and each interface between subsystems becomes a negotiation between groups. Conway added that organizing a design team is itself a design act, since the first split of the work fixes choices before anyone has studied the problem. He also noted that the number of possible communication paths is approximately half the square of the number of people, which is why organizations restrict who talks to whom. His examples reached beyond software, to governments among others.

Three bodies of research test the claim, each in a different way. The first is by Alan MacCormack, Carliss Baldwin and John Rusnak, who compared five matched pairs of software products with the same function, one from a commercial firm whose people are tightly coupled in goals and structure and one from an open source community whose people are loosely coupled. In every pair they examined, the open source product was more modular, and the gap reached a factor of eight in how far a change to one component could spread to others. The comparison is observational and small. Commercial firms and open source projects differ in many ways besides communication structure, the authors themselves call five pairs a limited sample, and the pairs were not developed at the same time, so some of the gap could be learning between release dates. One open source product, the Gnumeric spreadsheet, had a much higher propagation cost than the other open source products. The authors looked at its contributor records and found that one person accounted for almost 40% of logged changes, which they read as a tightly coupled organization, so the outlier fits their hypothesis. That is a post hoc explanation, offered after seeing the anomaly.

The second is the Vista study from the opening. It is unusually large, and its authors name its limits: all the data come from one system, they say the results cannot be assumed to generalize beyond Windows, and the correlation says nothing by itself about whether reorganizing would have prevented any failures. They also note that the Windows engineers did not know the study was running, so nobody was adjusting behavior to look good on the measures.

The third and broadest is the review by Lyra Colfer and Carliss Baldwin (2016) of 142 empirical studies. They conclude that mirroring is a common pattern and a long way from universal. Studies of open collaborative projects, mostly software, mostly did not support it, and the authors argue that digital tools enable coordination modes that let groups depart from the classic pattern. In technologically dynamic industries they suggest that partial mirroring, with knowledge boundaries drawn more broadly than operational ones, may be the better strategy. The reading I find defensible is that organizational structure exerts a strong default pull on system structure. Planners have to account for that pull, and deliberate choices can still bend it.

The usual AI split puts the seams where the failures happen

A common way to staff a first serious AI product runs like this. A data engineering team owns the pipelines and the index. An ML platform team owns model access, training and serving. A product team owns the application and the prompts. Evaluation lives in a data scientist's notebook, a QA group, or an annual red-team exercise. Each team can report green on its own measures.

Research on machine learning in software organizations suggests this split is riskier than the same split for conventional services. Saleema Amershi and colleagues at Microsoft interviewed 14 practitioners and surveyed 551 software engineers (a 13.6% response rate). They observe that in conventional architecture, separate modules are often assigned to separate teams, with APIs defining the interface so that little communication is needed. For machine-learned components, they report, strict module boundaries are hard to maintain, for two reasons. Models are not easily extended or recombined, and models interact in non-obvious ways, so that even if separate teams built each model, they would have to collaborate closely to train and maintain the full system. The practices that several engineers in their sample recommended point the same way: version ML and non-ML code in common repositories, and tightly couple the ML and non-ML sprints and standups. The sample is Microsoft only and self-reported, and the authors say some findings may be specific to those teams.

Miryung Kim and colleagues interviewed 16 data scientists from eight Microsoft organizations and, in a 2016 paper, described five arrangements: a triangle of data scientists, data stewards and instrumentation engineers; a hub and spoke model with a central platform team and product-specific spoke teams; a consulting model; the lone data scientist inside a product team; and a virtual team of such individuals. The paper describes itself as a qualitative study based on 16 interviews, and the section I read lists the arrangements without ranking them. What carries over here is that where to place the specialist was already an open question for data scientists, before AI products added more specialist roles.

An invented example shows how the seam fails. Suppose a claims-triage assistant is built by four groups: a data team that owns document ingestion and the retrieval index, a platform team that owns the model gateway, a product team that owns the interface and the prompt, and a group of claims reviewers who label cases but report into operations. The data team halves the chunk size from 800 to 400 tokens because its own offline measure, recall at ten, rises from 0.71 to 0.78. Meanwhile the prompt had been tuned on the larger chunks, which carried enough surrounding context for the model to answer correctly. End-to-end accuracy on the product team's 200-case spot check falls from 0.84 to 0.77. No dashboard shows this, because the only end-to-end measure is a spot check done when someone remembers, and the reviewers who could notice are two reporting lines away. Customers see the drop before anyone on the four teams does. These numbers are illustrative, made up for the example, and I list them as such in the sources.

The coordination arithmetic points the same way. A group of n people has n(n-1)/2 possible pairwise communication paths, the formula Fred Brooks used and Nagappan's team cited. One team of 8 has 28. A flat group of 24 has 276, and a change that touches four separate teams needs six team-to-team agreements before anyone writes code. These count possible links and say nothing about how often people talk. They do bound the problem: a team boundary that cuts through an interface that changes every week turns each change into a negotiation.

Draw team boundaries around slices of user work, with the eval set inside

A ThoughtWorks podcast credits the inverse Conway maneuver to Jonny LeRoy and Matt Simons, who wrote about it in 2010. Their idea is to treat the force as given and use it: choose the architecture you want, then organize the teams so that communication lines run where the interfaces should be. For an AI product I would draw the boundaries around slices of user-facing work, such as handling a claim, answering a customer's billing question or drafting a contract summary. Technology layers like retrieval, model access and interface make poorer boundaries. A slice contains the prompt, the retrieval configuration, the tool definitions, the eval set and the monitoring, so the parts that interact most sit inside one team.

Matthew Skelton and Manuel Pais, in Team Topologies (IT Revolution, 2019), name four team types and three modes of interaction among them: collaboration, X-as-a-service and facilitation. Applied to an AI product organization, the structure I would draw has these parts.

1. Stream-aligned product teams, each owning one slice of user work end to end, including the prompt, the retrieval setup, the tool integrations, the eval set, the release gate and the on-call rota. A domain expert, such as a claims reviewer or a support lead, writes and labels eval cases as a named member of the team, with that work on the team's own backlog.
2. A platform team that offers paved roads as a service: a model gateway with routing, budgets and logging, an eval runner that plugs into continuous integration, tracing, prompt and index versioning, and permissioned data access. It is judged by how quickly a product team reaches its first working eval and how rarely teams need to ask for help.
3. An enabling group of the scarce specialists, in evaluation methodology, safety testing and fine-tuning. It works with a product team for a few weeks, leaves behind a skill or a tool, and moves on, so the specialists do not turn into an approval queue.
4. A complicated-subsystem team only where a component needs deep specialist work that a product team cannot reasonably carry, such as a proprietary ranking model or a speech stack. Many companies will need none.

Eval ownership in the first item is the part I am most willing to argue for. In a January 2026 engineering post, Anthropic said it had experimented with various approaches to eval maintenance. What proved most effective, it said, was a dedicated evals team owning the core infrastructure while domain experts and product teams contribute most of the eval tasks and run the evals themselves. The post says product teams should treat owning and updating evals like maintaining unit tests, and suggests that product managers, customer success staff and salespeople, who are closest to what users need, can submit eval tasks as pull requests. That is a vendor drawing on its internal work and on customers, with no comparison group, but it matches the topology above: central tooling, local ownership of what counts as correct. I have written elsewhere on this site that the eval set is the specification. If that is true, the team that owns the specification has to be the team that answers for user outcomes.

Where the evidence stops and the heuristic begins

Part of the argument rests on evidence of moderate strength, all of it observational. Mirroring is a common tendency, per the 142-study review. Organizational measures predicted post-release failures better than code measures in one very large system. The DORA research program, which Google Cloud runs, describes loosely coupled teams as ones that can test, deploy and change their systems without depending on other teams, links such structures to better delivery performance, and suggests tracking the share of design changes that need approval from outside the team.

DORA's 2025 report on AI-assisted software development draws on survey responses from nearly 5,000 technology professionals plus qualitative data, collected between 13 June and 21 July 2025. A survey of this kind can show association and cannot establish cause. It finds that 90% of organizations have adopted at least one internal platform and that 76% have at least one dedicated platform team. Most relevant here, it reports that AI adoption has a negligible effect on organizational performance when internal platform quality is low and a strong positive effect when it is high. The same report adds that AI adoption still raised delivery instability. One case in the report, from an Adidas executive describing a generative AI pilot, says teams in loosely coupled architectures with fast feedback loops saw productivity gains of 20% to 30% on commits, pull requests and delivery velocity, while teams tightly coupled to ERP systems, with slower feedback loops, saw little or no AI benefit. That is a quoted pilot result, which I treat as an anecdote.

The scope of that report limits what it can support. The AI in it is mostly coding assistance used by software teams, which differs from building an AI feature for customers. Applying its platform finding to teams that ship LLM products is an extension, and I make it as a judgment: a team with a poor internal platform cannot absorb a new capability cheaply, whatever the capability is.

The remaining pieces are heuristics. The proposal that a stream-aligned team should own its eval set has one practice report behind it, plus a December 2025 preprint by Melissa Pan and colleagues on agents in production, based on 20 case studies and a survey of 306 practitioners, of whom 86 described deployed systems. That paper reports that 74% of the practitioners who answered its question on evaluation methods for deployed agents relied on human-in-the-loop checks (the figure's base is 31 deployed systems, so a small base; my arithmetic puts the count at 23), and it describes development teams working with domain experts to build benchmarks. That supports keeping the domain expert in the loop. It does not say which reporting line produces the best evals. For the enabling group I found no evidence at all beyond the logic that scarce skills should be spread by teaching and not by queueing. Team Topologies is a practitioner book, and I did not find a controlled comparison of team topologies for AI products or for anything else.

The proposal also has costs. Each product team needs some eval skill, so work will be duplicated until the platform's eval runner matures. A platform team can drift into gatekeeping, which is why DORA's advice to treat the platform as an internal product matters. A small company with one AI team gains nothing from the vocabulary, because its structure is already one team. The topology starts to pay when there are several slices of work, and the count at which it does is a number I cannot give you.

Two objections to Conway's law, and why neither fully holds

The strongest objection comes in two versions. One says the law is trivially true: people who need to talk will talk, so interfaces appear where conversations happen, and nothing follows from that. The other says whatever force the law once had has weakened, because shared repositories, chat, video calls and now AI coding assistants let anyone coordinate with anyone.

The weakening version has partial support, and it comes from the reviewers of the evidence. Colfer and Baldwin found that open collaborative projects mostly did not mirror their structure and suggest digital coordination tools may explain that. Martin Fowler, in the ThoughtWorks podcast, said the force of the law is always present. He added that remote work leads to less rich communication than working in close proximity, and that it is probably too early to tell how the trade-offs will play out. The best direct evidence on tooling comes from Longqi Yang and colleagues, who analyzed email, calendar, chat and call data on 61,182 US Microsoft employees in the first half of 2020. When the whole company moved to remote work, the collaboration network became more static and more siloed, with fewer bridges between groups, and a Microsoft Research write-up puts the drop in cross-group collaboration time at about 25% of its pre-pandemic level. The shift was forced by the pandemic and was no considered choice of tooling, so it does not prove that better tools could not do the job. It does show that an open channel to everyone does not make people use it. The law depends on the gap between who could talk and who does.

On triviality, I concede that "structure follows communication" is close to a tautology in its weakest form. It says something useful in two places. It predicts where defects cluster, as the Vista study found, and it implies that a reorganization is an architecture decision, which Conway himself said. Where it falls short is direction, and that is where the inverse maneuver takes a risk. MacCormack's matched pairs compare organizations of different kinds, and the Vista study observes organizations as they were. Neither shows that taking an existing company and redrawing its teams will redraw its software. I believe the direction runs that way for AI systems, because the artifacts are cheap to create and move, so the people who own them tend to define the interfaces. I hold that belief on the strength of the mechanism and of Amershi's findings about entangled components, not because anyone has run the experiment.

Before the next reorganization, count the teams each regression needed

Draw the AI feature's dependency graph on one page: the model and its pinned version, the prompt, the retrieval index, the tool APIs, the data feeding the index, the eval set and the online monitor. Next to each node write the name of the team that can change it without asking anyone. Then take the last five regressions or incidents and count how many teams each one needed to diagnose and fix. DORA's suggested measure, the share of design changes that need outside approval, is a reasonable way to keep the count honest over time.

Two further checks are quick. Can the owner of the eval set block a release? And does that owner sit in the same reporting line as the person who answers for user outcomes? If both answers are no, you have found the seam, and the cheapest first move is to put one domain expert and the eval set inside the product team, before touching anything else.

The question I cannot settle is what happens when a dozen product teams share one model gateway. A dozen locally owned eval sets give each team control at the price of inconsistent standards. A pooled set owned by a central group gives consistency at the price of the original problem, an owner far from the user. Anthropic's split, central infrastructure and local tasks, is one answer, and I found no published data comparing it with the alternatives.

Sources

1. Nagappan, N., Murphy, B., Basili, V. R. (2008). The Influence of Organizational Structure on Software Quality: An Empirical Case Study. Proceedings of the 30th International Conference on Software Engineering (ICSE), pp. 521-530; Microsoft Research technical report MSR-TR-2008-11. Precision and recall figures are from Table 4 of the report.
2. Conway, M. E. (April 1968). How Do Committees Invent? Datamation 14(4). melconway.com/Home/Committees_Paper.html.
3. MacCormack, A., Baldwin, C. Y., Rusnak, J. (2012). Exploring the Duality between Product and Organizational Architectures: A Test of the "Mirroring" Hypothesis. Research Policy 41(8), 1309-1324; author-accepted full text read via Harvard DASH; Harvard Business School Working Paper 08-039 is the earlier version.
4. Colfer, L. J., Baldwin, C. Y. (2016). The Mirroring Hypothesis: Theory, Evidence, and Exceptions. Industrial and Corporate Change 25(5), 709-738 (abstract read; full text not read).
5. Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T. (2019). Software Engineering for Machine Learning: A Case Study. ICSE Software Engineering in Practice track.
6. Kim, M., Zimmermann, T., DeLine, R., Begel, A. (2016). The Emerging Role of Data Scientists on Software Development Teams. ICSE 2016.
7. Skelton, M., Pais, M. (2019). Team Topologies: Organizing Business and Technology Teams for Fast Flow. IT Revolution Press; team types and interaction modes as listed at teamtopologies.com/key-concepts.
8. ThoughtWorks Technology Podcast. Reckoning with the force of Conway's Law (attribution of the inverse Conway maneuver to Jonny LeRoy and Matt Simons, 2010, spelled Johnny Leroy in the transcript; comments by Martin Fowler on remote work). thoughtworks.com/insights/podcasts/technology-podcasts/reckoning-with-the-force-conways-law.
9. DORA (Google Cloud). Capabilities: Loosely coupled teams. dora.dev/capabilities/loosely-coupled-teams/.
10. DORA (2025). State of AI-assisted Software Development. Google Cloud; survey fielded 13 June to 21 July 2025.
11. Anthropic (9 January 2026). Demystifying evals for AI agents. anthropic.com/engineering/demystifying-evals-for-ai-agents.
12. Pan, M. Z. et al. (December 2025, revised 2026). Measuring Agents in Production. arXiv:2512.04123.
13. Yang, L., Holtz, D., Jaffe, S., Suri, S., et al. (2021). The effects of remote work on collaboration among information workers. Nature Human Behaviour, doi:10.1038/s41562-021-01196-4; the 25% figure is from the Microsoft Research write-up of the paper.
14. Illustrative claims-triage example: chunk size 800 to 400 tokens, recall at ten 0.71 to 0.78, end-to-end accuracy 0.84 to 0.77 on 200 cases. All made up for this essay.
15. Illustrative communication-path arithmetic: n(n-1)/2 for n = 8 (28) and n = 24 (276), the formula Brooks used and Nagappan et al. cite; counts of possible links, not data.