The Business Value of Critical Thinking When AI Does the Work
AI agents can produce analyses and recommendations in seconds. Their business value also depends on whether employees ask useful questions, assess evidence, and learn from the results. Intelligence Architecture therefore needs to develop people's capabilities. How can everyday work support that development, and how can an organization assess its value?

AI can help a company produce more analysis, answer more customer inquiries, and ship more software. Whether that amounts to a business gain depends on whether the work is needed and the results are good. It also depends on what employees learn along the way. Are they becoming better at judging difficult cases, or mainly getting faster at passing on completed answers?
In Critical thinking when agents do the work, Addy Osmani describes a change in how that learning happens. When agents return plausible results quickly, working through a task no longer necessarily makes a person examine the problem. Delegating work can gradually become accepting an answer without forming an independent judgment.1
Osmani is describing his experience, not establishing a general decline in human ability. His observation nevertheless raises a design question. Intelligence Architecture should help employees tackle harder questions, make better decisions, and learn from their consequences. Developing those capabilities belongs in the business case for AI.
Find the decision that better thinking could change
Critical thinking here means framing a question clearly, testing a claim against evidence, and revising a judgment when the evidence fails to support it. It includes noticing missing information and recognizing when an apparently strong answer addresses the wrong problem. The number of objections someone raises would be a poor measure of this ability.
Consider a hypothetical customer service team. An agent classifies complaints and drafts replies, and cases are closed more quickly. An employee notices that several differently worded inquiries may concern the same product problem. She examines the original messages, compares affected customer groups, and proposes fixing the product. The existing performance measure would mainly have rewarded faster case handling. Her investigation could prevent further complaints.
The potential business contribution is a better choice of task. The agent was asked to answer inquiries efficiently. The company might gain more by removing the reason for those inquiries. AI could help identify the pattern and challenge the proposed explanation. People still have to evaluate it and decide whether a product change would justify its cost and possible side effects.
An investment decision offers another setting. A polished market analysis may address the wrong customer group. Someone who notices that assumption early can request different research or propose a limited trial before substantial resources are committed. Critical thinking can prevent costs and reveal opportunities. Its value has to be assessed through the decision that follows and what actually happens as a result.
Performance and learning are separate outcomes
There is evidence that AI can help employees perform better. Erik Brynjolfsson, Danielle Li, and Lindsey Raymond studied the staggered introduction of an AI assistant in customer support. The November 2023 NBER version of their study covered 5,179 employees. Issues resolved per hour increased, with particularly large gains among less experienced or previously lower-performing workers. The researchers also found evidence suggestive of successful practices spreading and of worker learning.2
Those findings are encouraging. They do not mean that every improvement during AI use becomes a lasting human capability. Someone can complete a difficult task with an assistant’s help without understanding it better afterward.
Judy Hanwen Shen and Alex Tamkin examined this distinction in a randomized experiment with 52 software developers, most in junior roles. Participants worked with a Python library they had not used before. On a comprehension test immediately afterward, the AI-assisted group averaged 50 percent and the group without AI 67 percent, a difference of 17 percentage points. The AI group’s modest time advantage was not statistically significant.3
This was a short learning task followed by an immediate assessment. It establishes neither long-term skill loss nor a general effect of autonomous agents. A subsequent qualitative analysis found stronger learning outcomes among participants who sought explanations or asked conceptual questions. Participants were not randomly assigned to those interaction styles. The association suggests directions for design, but does not establish that a particular teaching method caused better learning.
For an employer, the distinction is practical. Better work and more capable employees are two possible returns from AI-assisted activity. Both need to be observed; the second cannot simply be inferred from the first.
Use AI to extend what employees can investigate
In the hypothetical service team, an agent could explain how experienced staff approach difficult cases. It could propose an alternative explanation for the suspected product fault, identify missing customer information, or suggest an observation that would disprove the hypothesis. An employee could use that help to plan an investigation that previously required expertise she did not have.
For her own capability to grow, she needs to understand and apply those distinctions. Work can be organized to support this. For selected difficult cases, the responsible employee first writes down a provisional explanation. She then compares it with the agent’s account and checks one consequential assumption against the original records. When the outcome becomes known, it comes back to her. She can see which reasoning held up and what she should change in a later case.
The agent can research, calculate, explain, and challenge. Much of the verification may also be automated. The question is whether employees finish with a better understanding of the case and its limits, and can apply that understanding when the situation changes. An additional AI explanation that nobody understands would not achieve that.
Domain knowledge matters. People who do not know relevant customer requirements, technical dependencies, or legal limits will struggle to notice missing premises. General instructions to think critically cannot replace professional education or conversation with experienced colleagues. AI can prepare those conversations and make unfamiliar questions easier to explore. Junior staff still need opportunities to reason through work rather than merely approve completed recommendations.
Give learning time and authority in the workflow
The employee needs permission to pursue the suspected product issue. Without the original complaints, access to the product team, or a way to escalate the problem, her better understanding may change nothing. If she is assessed only on closed tickets, careful investigation could even hurt her own performance rating.
This is part of the work of Intelligence Architecture. It connects access to information with assignments, responsibilities, and the power to change decisions. Design therefore includes choosing which cases offer learning opportunities, who discusses them together, and when that work happens. Capability development cannot be treated solely as a private activity employees should fit in after hours.
Every routine task need not become a lesson. Full delegation may be appropriate when the consequences of error are minor and results are easy to verify. More consequential decisions or unfamiliar subject matter offer stronger reasons for intentional learning. Which skills the team needs to retain, and which tasks no longer need manual practice, are choices about its future work. They deserve an explicit decision.
Make the business case include the cost of learning
A count of critical questions cannot establish economic value. Numerous objections may expose poor recommendations, create unnecessary friction, or prepare a better decision. Their consequences distinguish those possibilities.
For the service team, repeat inquiries, later corrections, and product problems actually resolved would be more informative than case volume alone. The accounting also needs to include research, review, coordination, and learning time. Avoided rework cannot be counted once as time saved and again as the full associated labor cost. Freed time becomes a realized benefit when it is put to worthwhile use or an expense is actually avoided.
Learning requires its own observations. Can an employee explain a new case of similar difficulty? Can they recognize an important exception missing from the recommendation? Does that ability persist after several weeks? Both AI-assisted tasks and selected tasks without a ready-made AI answer can help assess this. The purpose is to examine the application of knowledge, not to stage a competition against the machine.
A company could try different working arrangements in comparable teams or introduce them in stages. The mix of cases and the standard of quality would need to remain comparable. It could then investigate whether additional learning time leads to fewer corrections, more independent decisions, or better solutions later on. Satisfaction surveys or a brief knowledge test alone would not establish that business return.
More capable AI can increase immediate throughput. Good Intelligence Architecture should also help employees understand unfamiliar problems and handle them responsibly. Part of its business value lies in whether a team becomes able to do work that currently defeats it. Building that capability is a goal of AI deployment in its own right.
Footnotes
-
Addy Osmani, Critical thinking when agents do the work. A practitioner account of agent-assisted work, delegation, and reviewing results. The economic and organizational argument developed here goes beyond Osmani’s contribution. ↩
-
Erik Brynjolfsson, Danielle Li, and Lindsey R. Raymond, Generative AI at Work, NBER Working Paper 31161, November 2023 revision. A field study of a staggered rollout in one company. The authors report suggestive evidence of learning, not general proof of durable skill development. A revised version appeared in the Quarterly Journal of Economics in 2025. ↩
-
Judy Hanwen Shen and Alex Tamkin, How AI Impacts Skill Formation, 2026; see also the authors’ research report of January 29, 2026. A randomized experiment with 52 participants and a subsequent qualitative analysis of AI use. The assessment measured immediate understanding of a new library, not long-term skill development. ↩

