Monday, February 16, 2026

What Your RTO Data Isn't Telling You—And Why It Matters for Your People

Let's talk about something that's probably been nagging at you if you're in HR: those badge swipe reports everyone's using to justify return-to-office decisions? They might be telling you a story that isn't quite true.

Here's the thing—when we looked at how traditional analytics measure office attendance, we found they overestimate how much remote-preferring employees actually want to come in by about 33 percentage points. That's huge. Your dashboard might say someone's 45% office-inclined when they're really closer to 12%.

This isn't about blaming anyone or pointing fingers at your data team. It's about understanding a hidden bias that affects everyone—and more importantly, figuring out how to actually support your people based on what they truly need.

Why This Happens: The Sarah and Marcus Problem

Let me paint a picture that probably sounds familiar.

Sarah is one of those people who genuinely loves the office. She lives nearby, enjoys the energy of being around colleagues, and comes in four days a week. Over six months, you've collected tons of data about her—which desk she prefers, what days work best, how she uses shared spaces.

Marcus is a senior engineer with a longer commute, two kids at home, and a really nice home office setup. He comes in maybe twice a month, usually for important meetings.

Here's the problem: you have five times more data about Sarah than Marcus. And that's not random—it's because Marcus simply isn't there to be observed.

This is what statisticians call "Missing Not At Random" (MNAR)¹, and it creates a sneaky bias [Rubin, 1976; Little & Rubin, 2019]. The people who don't love the office... don't come to the office. When they don't come, we don't capture their preferences. So our data ends up dominated by people like Sarah.

When traditional tools try to fill in the gaps for Marcus, they use patterns from all that Sarah-heavy data—a technique known as collaborative filtering [Koren et al., 2009]. The result? They predict Marcus wants to come in way more than he actually does.

From an HR perspective, this matters enormously. If we're building policies, designing spaces, or making decisions based on data that doesn't accurately represent over half our workforce, we're setting ourselves up to create experiences that don't serve our people well.

What We Actually Found When We Corrected for This Bias

We analyzed data from 500 employees over 26 weeks using a more sophisticated approach based on Causal Matrix Completion [Agarwal et al., 2023]. The findings were eye-opening—and honestly, they should change how we think about hybrid work.

The Real Distribution of What People Want

Who They Are	% of Your Team	What They Actually Prefer	What Standard Reports Say
Remote-First Folks	~54%	Less than 30% office time	~45% (way overestimated)
Hybrid-Regulars	~23%	30-60% office time	Roughly accurate
Office Enthusiasts	~10%	70%+ office time	Slightly underestimated
Work-Life Balancers	~6%	It varies	Often misclassified
Flexibility Seekers	~4%	Depends on context	Averaged incorrectly
Collaboration Seekers	~3%	High when their team is there	Overgeneralized

The headline? Over half of your knowledge workers are remote-first. They genuinely prefer being in the office less than 30% of the time. But standard analytics report them as wanting nearly half-and-half.

Why This Happens: The Missing Data Gap

Employee Type	How Much Data We Actually Capture	What's Missing
Office Enthusiasts	82.5%	17.5%
Middle-of-the-Road	67.3%	32.7%
Remote-Preferring	45.2%	54.8%

Remote-first employees have three times more missing data than office enthusiasts. Every decision we make using this data inherits that imbalance—unless we correct for it. This observation pattern is consistent with research on selection bias in observational data [Heckman, 1979; Imbens & Rubin, 2015].

A Better Way to Understand Your People: Causal Matrix Completion

There's a technique called Causal Matrix Completion [Agarwal, Dahleh, Shah, & Shen, 2023] that helps us see through this noise. I won't bore you with all the math, but here's the intuition:

Instead of filling in missing data using patterns from people who are observed (which skews toward office enthusiasts), this approach asks: "What would we see if everyone had an equal chance of showing up?"¹

It works by giving more weight to rare observations—a technique called Inverse Propensity Weighting (IPW) [Rosenbaum & Rubin, 1983]. If someone who almost never comes in does show up, that tells us something meaningful—so we weight that information more heavily.

How Causal Matrix Completion Works

The technical foundation combines insights from three fields:

Matrix Completion [Candès & Recht, 2009]: The idea that we can recover missing entries in a matrix if the underlying data has low-rank structure (i.e., people's preferences cluster into a few underlying types).
Propensity Score Methods [Rosenbaum & Rubin, 1983]: Weighting observations by the inverse probability of being observed to correct for selection bias.
Synthetic Control Methods [Abadie et al., 2010]: Using similar individuals to estimate counterfactual outcomes.

Agarwal et al. (2023) unified these approaches for panel data settings where both the outcome and the treatment/observation probability can be correlated with unobserved factors—exactly our office attendance scenario.¹

The Difference It Makes

What We're Measuring	Traditional Approach	Corrected Approach	Improvement
Accuracy for remote-first employees	Pretty poor	Much better	43% improvement
Bias for remote-first employees	Overestimates by 33 points	Overestimates by 8 points	75% reduction

What this means in practical terms: Traditional analytics might tell you to plan for 45% office attendance from your remote-first folks. The corrected approach predicts closer to 20%. That's the difference between designing for 60% occupancy and 40% occupancy—which has real implications for real estate costs, space design, and how your people experience the workplace.

The Four Things That Actually Drive Office Behavior

Once we could measure preferences accurately, clear patterns emerged. Understanding these can help you support different employees more effectively. This analysis builds on research in behavioral propensity modeling [Ajzen, 1991; Davis, 1989].

1. Their Baseline Preference (Office vs. Remote)

This is someone's natural tendency—do they gravitate toward the office or toward working from home? In statistical terms, this is their propensity for office attendance [Austin, 2011].

What influences it most:

Commute distance is the biggest factor [Bloom et al., 2015]. Every additional kilometer of commute reduces office preference. A 30km commute basically cancels out any baseline preference for the office.
Role matters. Sales folks and managers tend to prefer the office more; engineers and data scientists tend to prefer remote [Choudhury et al., 2021].
Home office quality makes a 29 percentage point difference between people with poor setups and those with great ones.
Having kids at home reduces office preference (flexibility needs are real) [Barrero et al., 2021].

One surprise: Age wasn't a significant factor. Contrary to stereotypes, millennials aren't necessarily more remote-preferring than Gen X or Boomers once you account for other things [Bloom, 2020].

2. How Much They Value In-Person Collaboration

Some people genuinely need face-to-face interaction to do their best work. Others thrive with async tools like Slack or email [Yang et al., 2022].

Here's something interesting: true collaboration-seekers make up only about 3% of the workforce, but they're force multipliers. When they're in the office, others tend to follow—a phenomenon related to social influence in networks [Aral & Walker, 2012]. Consider them your "anchor" employees for team sync days.

3. What They Think About Office Amenities

Here's a counterintuitive finding: amenities are retention tools, not attraction tools.

That fancy cafeteria and new gym? They keep your office enthusiasts happy, but they won't convince someone with a 90-minute round-trip commute to come in more often. Free lunch is lovely, but it doesn't offset a long, stressful commute or competing caregiving responsibilities. This aligns with research on workplace satisfaction factors [Herzberg, 1959; Judge et al., 2001].

Don't invest in amenities expecting them to increase attendance. Invest in them to support the people who are already choosing to be there.

4. How Much Flexibility Matters to Them

This one has an almost perfect inverse relationship with office preference [Barrero et al., 2021]. People who highly value flexibility almost by definition avoid fixed office schedules.

HR heads-up: Your flexibility-seekers (about 4% of the workforce) are the hardest to plan around. They won't commit to regular schedules—and forcing them to increases turnover risk [Mas & Pallais, 2017]. These are often your high performers who've earned that autonomy. Think carefully before creating one-size-fits-all policies.

The Feedback Loop We Need to Talk About

Here's something that gets philosophically interesting—and has real ethical implications for HR.

Office recommendation systems don't just predict behavior. They change it. This phenomenon is well-documented in the recommender systems literature [Adomavicius & Tuzhilin, 2005; Ricci et al., 2015].

How This Works

Exposure Effect: Someone who never comes on Mondays might assume the office is chaotic that day. If we recommend they try Monday (knowing it's actually quiet and productive), they might discover they like it [Zajonc, 1968]. Now their Monday preference increases, and future Monday recommendations are more likely accepted.

Social Influence: Recommend the same day to an entire team, everyone shows up, collaboration happens, everyone's preference for that day increases [Cialdini & Goldstein, 2004]. The effect compounds.

Habit Formation: Behavioral research tells us habits form after about 66 days of repetition [Lally et al., 2010]. If we nudge someone into a pattern for two months, that pattern tends to stick.

Norm Formation: Aggregate enough individual nudges and you've created culture. "Tuesday-Wednesday are office days" can emerge not from policy but from reinforced recommendations. New hires adopt it automatically—a process studied in organizational culture research [Schein, 2010].

The Questions We Should Be Asking Ourselves

This creates some uncomfortable territory related to the ethics of algorithmic nudging [Thaler & Sunstein, 2008; Yeung, 2017]:

If someone's preference gradually changes through recommendation-driven exposure, did we help them discover something about themselves? Or did we manipulate them?
At what point does a helpful recommendation system become a subtle compliance mechanism?

There's no perfect answer here. But as HR professionals, we have a responsibility to consider:

Transparency: Do people understand how recommendations are generated? [Diakopoulos, 2016]
Control: Can they opt out?
Whose interests are we serving? Do recommendations help employees, or just make scheduling easier for management?

What You Can Do Tomorrow Morning

For HR Leaders and People Ops

1. Take badge data with a grain of salt. It's systematically biased toward people who already like the office [Agarwal et al., 2023]. At minimum, look at different employee segments separately instead of treating everyone as one group.

2. Build flexibility into your policies. A blanket "3 days per week" mandate will get maybe 30% compliance from your remote-first majority—and probably some resentment along with it [Barrero et al., 2021]. Consider:

1 day/week for remote-first folks (critical meetings and collaboration)
2-3 days for hybrid workers
Whatever works for your office enthusiasts

3. Watch trends, not just attendance numbers. If overall office preference is declining, find out why. Is the office experience getting worse? Are people setting up better home offices? Is manager pressure backfiring?

4. Have honest conversations. Survey your people. Ask them what they actually need. The data can tell you patterns, but only your people can tell you why.

For Facilities and Real Estate Decisions

1. Plan for 40-50% peak attendance, not 100%. Our data shows about 37% average attendance with significant day-to-day variation (Tuesday-Wednesday peaks, Monday-Friday valleys).

2. Design for different needs. Not everyone uses the office the same way. A rough breakdown:

~15% permanent desks (for your office enthusiasts)
~35% hoteling/flexible workstations (for hybrid workers)
~25% meeting and collaboration spaces
~15% quiet pods for focused work
~10% amenities and social spaces

3. Watch for crowding problems. If everyone's being pushed toward Tuesday-Wednesday, you'll create miserable peak days and ghost-town off-days. That's bad for everyone's experience.

For Managers Leading Hybrid Teams

1. Coordinate, don't mandate. "Tuesday is our team sync day—consider joining if it works for you" respects individual circumstances while signaling when in-person connection is most valuable.

2. Protect your remote-first people from proximity bias. Make sure they're evaluated on what they produce, not how often you see them [Choudhury et al., 2021]. Document decisions; don't rely on hallway conversations they can't be part of.

3. Default to async. Design meetings to accommodate remote participants as first-class citizens. And honestly, if it can be an email or a Slack message, make it one. Your people will thank you.

The Bigger Picture

The hybrid work conversation has been full of strong opinions and not enough good data. Executives point to badge reports showing 40% attendance and conclude people want to return. Employees feel like mandates are disconnected from reality.

Both sides are working with incomplete information. The data has been biased—but it's fixable.

Getting this right isn't just about technical accuracy. It's about honesty. It's about acknowledging that most knowledge workers, when given genuine choice, prefer more flexibility than we've traditionally offered [Bloom et al., 2015; Barrero et al., 2021]. That's not a cultural failure or a problem to solve. It's information to incorporate into how we support our people.

Organizations that build policies on accurate data will make better decisions, see lower turnover, and create more nuanced approaches to collaboration. Those that don't will keep investing in things that don't move the needle and creating mandates that don't improve engagement.

The data exists. The methods exist. The real question is: are we ready to listen to what it's actually saying—and to design workplaces that genuinely serve the people in them?

A Quick Note on the Technical Stuff

If you're curious about the methodology behind these findings, the approach is called Causal Matrix Completion with Inverse Propensity Weighting [Agarwal et al., 2023]. In plain English: it corrects for the fact that we have more data about some people than others, and weights observations accordingly.

The key insight is simple: when data is "missing not at random"—meaning the people we don't observe are systematically different from those we do [Little & Rubin, 2019]—traditional analytics will mislead us. Causal approaches help us see a more complete picture.

You don't need to become a data scientist to apply these insights. But if you're working with analytics teams on workforce planning, sharing this perspective might spark some valuable conversations about how to get better, more equitable insights from your data.

The goal isn't to prove that remote work is "better" or office work is "wrong." It's to understand what your people actually need—and to build policies that support them in doing their best work, wherever that happens.

References

Causal Matrix Completion & Missing Data

Agarwal, A., Dahleh, M., Shah, D., & Shen, D. (2023). Causal matrix completion. Proceedings of The 34th International Conference on Algorithmic Learning Theory (ALT 2023), PMLR 195, 3-36. https://proceedings.mlr.press/v195/agarwal23c

Agarwal, A., Dahleh, M., Shah, D., & Shen, D. (2021). Causal matrix completion. arXiv preprint arXiv:2109.15154. https://arxiv.org/abs/2109.15154

Little, R. J. A., & Rubin, D. B. (2019). Statistical analysis with missing data (3rd ed.). Wiley.

Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581-592.

Propensity Scores & Causal Inference

Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46(3), 399-424.

Heckman, J. J. (1979). Sample selection bias as a specification error. Econometrica, 47(1), 153-161.

Imbens, G. W., & Rubin, D. B. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge University Press.

Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41-55.

Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688-701.

Matrix Completion & Recommender Systems

Adomavicius, G., & Tuzhilin, A. (2005). Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Transactions on Knowledge and Data Engineering, 17(6), 734-749.

Candès, E. J., & Recht, B. (2009). Exact matrix completion via convex optimization. Foundations of Computational Mathematics, 9(6), 717-772.

Koren, Y., Bell, R., & Volinsky, C. (2009). Matrix factorization techniques for recommender systems. Computer, 42(8), 30-37.

Ricci, F., Rokach, L., & Shapira, B. (Eds.). (2015). Recommender systems handbook (2nd ed.). Springer.

Synthetic Controls & Panel Data

Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic control methods for comparative case studies: Estimating the effect of California's tobacco control program. Journal of the American Statistical Association, 105(490), 493-505.

Athey, S., Bayati, M., Doudchenko, N., Imbens, G., & Khosravi, K. (2021). Matrix completion methods for causal panel data models. Journal of the American Statistical Association, 116(536), 1716-1730.

Remote Work & Workplace Research

Barrero, J. M., Bloom, N., & Davis, S. J. (2021). Why working from home will stick. NBER Working Paper No. 28731. https://www.nber.org/papers/w28731

Bloom, N. (2020). How working from home works out. Stanford Institute for Economic Policy Research (SIEPR) Policy Brief. https://siepr.stanford.edu/publications/policy-brief/how-working-home-works-out

Bloom, N., Liang, J., Roberts, J., & Ying, Z. J. (2015). Does working from home work? Evidence from a Chinese experiment. The Quarterly Journal of Economics, 130(1), 165-218.

Choudhury, P., Foroughi, C., & Larson, B. (2021). Work‐from‐anywhere: The productivity effects of geographic flexibility. Strategic Management Journal, 42(4), 655-683.

Mas, A., & Pallais, A. (2017). Valuing alternative work arrangements. American Economic Review, 107(12), 3722-3759.

Yang, L., Holtz, D., Jaffe, S., Suri, S., Sinha, S., Weston, J., ... & Teevan, J. (2022). The effects of remote work on collaboration among information workers. Nature Human Behaviour, 6(1), 43-54.

Behavioral Science & Organizational Psychology

Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179-211.

Aral, S., & Walker, D. (2012). Identifying influential and susceptible members of social networks. Science, 337(6092), 337-341.

Cialdini, R. B., & Goldstein, N. J. (2004). Social influence: Compliance and conformity. Annual Review of Psychology, 55, 591-621.

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319-340.

Herzberg, F. (1959). The motivation to work. Wiley.

Judge, T. A., Thoresen, C. J., Bono, J. E., & Patton, G. K. (2001). The job satisfaction–job performance relationship: A qualitative and quantitative review. Psychological Bulletin, 127(3), 376-407.

Lally, P., Van Jaarsveld, C. H., Potts, H. W., & Wardle, J. (2010). How are habits formed: Modelling habit formation in the real world. European Journal of Social Psychology, 40(6), 998-1009.

Schein, E. H. (2010). Organizational culture and leadership (4th ed.). Jossey-Bass.

Zajonc, R. B. (1968). Attitudinal effects of mere exposure. Journal of Personality and Social Psychology, 9(2, Pt. 2), 1-27.

Ethics of Algorithms & Nudging

Diakopoulos, N. (2016). Accountability in algorithmic decision making. Communications of the ACM, 59(2), 56-62.

Thaler, R. H., & Sunstein, C. R. (2008). Nudge: Improving decisions about health, wealth, and happiness. Yale University Press.

Yeung, K. (2017). 'Hypernudge': Big Data as a mode of regulation by design. Information, Communication & Society, 20(1), 118-136.

Sunday, November 16, 2025

From Insight to Action: The Architectural Blueprint for a Self-Healing, AI-Driven IT Operations Platform

For the past decade, IT operations teams have been promised a revolution by AIOps. We were told it would save us from "alert fatigue" and the overwhelming complexity of modern, hybrid-cloud environments. We've deployed an arsenal of tools—Splunk, AppDynamics, ThousandEyes, ServiceNow—yet the result is often the same: We are drowning in data.

First-generation AIOps platforms are sophisticated at correlation. They sift through millions of events and reduce the noise, but they stop at the most critical juncture. They present an "insight" on a dashboard and leave the "now what?" to a human.

This is the Operational Chasm: the gap between insight and action. To close it, we must evolve beyond simple AIOps toward a Unified Automation & Intelligence Platform—a framework that doesn't just find the problem, but intelligently fixes it.

The Ops Chasm: Why MTTR Is Still Broken

For any SRE or Ops leader, the goal is to minimize the time and disruption of incidents. However, the traditional anatomy of an incident reveals a painfully manual process:

Failure & Detection (MTTD): Something breaks; we wait to find out it’s broken.
Investigation & Triage (MTTI): The "manual war room." Teams scramble to figure out why, sifting through siloed data.
Assignment & Remediation (MTTR): Getting the right people involved. This stage is frequently throttled by limited personnel and slow ticket responses.

The core problem is that our tools are siloed. An AI's insight is useless if it cannot be programmatically translated into action.

The 5-Stage Closed-Loop Remediation Architecture

A true self-healing system operates on a continuous, closed-loop feedback model. This architecture moves from observation to action and back again across five stages:

Event Ingestion & Normalization: The platform ingests data from any source (logs, metrics, APM) and normalizes it into a standardized schema, decoupling the engine from specific tool implementations.
Context Enrichment: The engine automatically queries specialized systems—pulling logs from Ansible, dependency info from a CMDB, or policies from security platforms—to provide the "why" behind the alert.
AI-Powered Decision: The enriched alert is fed into an AI/ML model that performs Root Cause Analysis (RCA) and generates a recommended remediation plan.
Automated Action: The engine executes the plan through outbound connectors, such as triggering an Ansible Playbook or creating a ServiceNow ticket.
Feedback Loop: Monitoring tools observe the outcome. This data is fed back into the system, allowing the AI to learn from the success or failure of its recommendation.

Deep Dive 1: The Model Context Protocol (MCP) Server

The "central nervous system" of this workflow is the Model Context Protocol (MCP) Server. It acts as a universal integration hub built on a plug-in model:

Inbound Connectors: Responsible for listening for events and querying for context (e.g., Monitoring or CMDB connectors).
Outbound Connectors: Responsible for executing actions (e.g., Ansible or ServiceNow connectors).

The MCP Server operates on principles of Universal Abstraction, Context-Aware Routing, and Transactional Integrity, ensuring that every command is delivered reliably and securely with a full audit trail.

Deep Dive 2: The AI Generation Pipeline (SLM + RAG + LLM)

The decision-making "brain" is not a single black box, but a multi-step pipeline:

Step 1: SLM Triage: A Small Language Model (SLM) handles high-volume alerts cost-effectively, generating a structured prompt for the next stage.
Step 2: RAG Orchestration: Retrieval-Augmented Generation (RAG) pulls factual, domain-specific documents (past tickets, runbooks) to ground the AI in your specific reality.
Step 3: LLM Generation: A Large Language Model (LLM) performs complex reasoning to propose a raw remediation plan.
Step 4: The Guardrails (Critical): Before execution, a safety module validates the plan for syntax, safety (e.g., ensuring no "delete" commands on production), and policy compliance.

Beyond Firefighting: The Automation Maturity Model

This architecture provides a clear path from reactive "fixing" to true business transformation:

Level 1: IT Automation: Executing discrete tasks (e.g., "Restart service").
Level 2: IT Orchestration: Connecting tasks into end-to-end processes.
Level 3: Digital Transformation: Orchestration becomes the foundation for DevOps and "Infrastructure as Code."
Level 4: Business Transformation: IT becomes an innovation engine, reacting to market demands with unprecedented speed.

The Bottom Line: The future of IT operations isn't about better dashboards. It’s about building a brain (the AI pipeline), a nervous system (the MCP server), and hands (automation tools) that can act intelligently. It is time to stop observing our infrastructure and start building one that can heal itself.

Monday, October 27, 2025

Beyond the Demo: Engineering AI Agents for Production Scale

The promise of agentic AI systems has captivated the technology industry, with demonstrations showcasing unprecedented capabilities in autonomous reasoning and task execution. Yet beneath the surface of these compelling demos lies a stark reality: the journey from prototype to production represents a fundamental engineering challenge that goes far beyond simple scaling. The path forward requires abandoning attractive but flawed architectural assumptions in favor of pragmatic design patterns that prioritize reliability, observability, and incremental deployment.

The Case Against Multi-Agent Complexity

Perhaps the most counterintuitive lesson emerging from production deployments is that more agents do not equal better performance. The industry is witnessing a decisive shift away from complex, hierarchical multi-agent systems toward architectures centered on a single capable model that orchestrates a rich ecosystem of tools and components. This isn't merely a preference—it's a response to hard-won experience. In one documented financial advisory prototype, critical context was lost after just three agent handoffs, triggering cascading failures that compromised the entire system. The lesson is clear: agents do not require human-like organizational charts to be effective. Instead, engineering efforts should focus on creating robust environments of tools and context that a single orchestrator can leverage effectively.

This architectural simplification doesn't imply building monolithic systems. Rather, successful implementations employ modular, composable designs where a primary orchestrator makes high-level decisions while delegating specific tasks to smaller, specialized models or deterministic tools. The data supports this approach: a 7-billion-parameter specialist model paired with a 34-billion-parameter planner can outperform a single 70-billion-parameter model on certain tasks while simultaneously reducing token consumption. The engineering challenge shifts from managing complex inter-agent communication protocols to designing robust APIs and implementing graceful error handling.

The Reliability Imperative

The reliability challenges facing agentic systems are fundamentally different from traditional software engineering. In multi-step agentic workflows, reliability compounds negatively in ways that can be mathematically devastating. Consider a three-step process where each individual step achieves 80-85% reliability—seemingly reasonable performance thresholds. The compound effect, however, yields a system that succeeds only 42-66% of the time. This arithmetic explains why many early agentic systems exhibited unacceptably high failure rates despite using state-of-the-art models.

Improving individual model performance, while important, cannot solve this architectural challenge. The solution lies in implementing redundancy at multiple levels: circuit breakers that prevent cascade failures, intelligent retry logic that distinguishes between transient and permanent errors, and human-in-the-loop validation for critical decision points. Some teams have achieved reliability improvements through multi-model quorum voting, though this approach carries higher computational costs. The trade-off is often worthwhile in production environments where failure costs exceed infrastructure expenses.

Progressive Autonomy as a Production Strategy

The tension between automation and reliability need not be an either-or proposition. A progressive autonomy framework offers a pragmatic middle path by graduating an agent's independence based on measured performance. This framework typically includes three levels: manual supervision, conditional automation, and full autonomy. Most production systems currently operate at levels one and two, where agents successfully handle 60-75% of routine tasks. This approach significantly accelerates the path to production compared to attempts at immediate, full automation.

The incremental deployment strategy extends beyond autonomy levels. While prototypes can be built in hours, production deployment is a months-long engineering effort that fundamentally reshapes the system. This gap reflects the challenge of transforming an experimental model into a reliable service that integrates with legacy systems and meets governance standards. The most effective strategy employs shadow-mode validation initially, then gradually migrates traffic while maintaining legacy fallbacks and enabling features progressively.

Observability and the Evaluation Paradigm Shift

Traditional monitoring tools prove insufficient for agentic systems, necessitating a new discipline of "agentic observability" that combines real-time guardrails for safety with offline analytics for optimization. The most critical component is reasoning traceability—the ability to capture and inspect the complete chain of decisions, tool calls, and confidence scores that led to any outcome. This capability makes non-deterministic systems debuggable in ways that were previously impossible. Teams implementing full reasoning-chain analysis report both faster incident resolution and improved user trust.

The evaluation paradigm is shifting from abstract benchmarks toward production-oriented metrics. What matters in production are performance under real-world conditions, latency at scale, token consumption per task, and tool success rates. This requires adapting continuous integration and deployment frameworks for non-deterministic systems, treating prompts and system configurations as versioned code. Successful teams build "golden test suites" from production logs to run nightly regressions, catching subtle performance degradations before they impact users.

Conclusion

The journey from agentic AI prototype to production system is not a scaling problem—it's a fundamental engineering challenge that requires rethinking architecture, reliability, and deployment strategies. The patterns that succeed share common characteristics: they favor simplicity over complexity, embrace incremental deployment over big-bang launches, engineer for compounding reliability, and establish observability that matches the non-deterministic nature of these systems. Most importantly, they treat large language models as reasoning engines rather than autonomous entities, focusing engineering effort on the environment of tools, context, and safeguards that enable reliable operation. As the field matures, success will increasingly belong to teams that master these production realities rather than those captivated by demonstration capabilities alone.

References:

Galileo - "A Guide to AI Agent Reliability for Mission Critical Systems" (2025)
UiPath - "Why orchestration matters: Common challenges in deploying AI agents" (May 2025)
Microsoft Azure - "Agent Factory: Top 5 agent observability best practices" (September 2025)
Gartner - Reports on AI project failure rates and cost limitations
McKinsey - "One year of agentic AI: Six lessons from the people doing the work" (September 2025)
OpenTelemetry - "AI Agent Observability - Evolving Standards and Best Practices" (March 2025)
VentureBeat - "Beyond single-model AI: How architectural design drives reliable multi-agent orchestration" (May 2025)
arXiv - "Security Challenges in AI Agent Deployment" (July 2025) and "The Landscape of Emerging AI Agent Architectures" (July 2025)

Thursday, August 25, 2022

Training with Microsoft DeepSpeed

I just started using this, and the review is not ready yet since I just started testing (training first, but you get the picture). The installation and configuration are very straightforward, with good integration with HuggingFace Transformers and Pytorch Lightning. My training models so far are in a multi-node configuration compatible with OpenMPI. More later! #training #testing #pytorch #deepspeed #huggingface #datascience #ai #ml

GitHub Link:

https://github.com/microsoft/DeepSpeed

Thursday, May 5, 2022

What is Linear Regression?

This is a "newish" series I will certainly be sharing from the beginning, and it's basically about different AI models and their ramifications. I will also attempt to simplify as long as feasible to make it easier for newer readers to comprehend. I have not done this yet, so please give me any feedback needed!

Linear regression designs are utilized to show or predict the partnership between 2 variables or variables. When two (2) or even more independent variables are used in a regression analysis, the version is no longer an easy linear design. When greater than one independent variable is utilized to forecast the value of a numeric dependent variable, this natural regression formula is called multiple straight regression.

Fitting a straight regression model can be used to identify the partnership between one predictor x j and the action variable y when all other forecasters in the model are "dealt with". Linear regression efforts to design the relationship between a scalar variable and also several independent variables by installing a direct equation to the observed information. Direct regression devices produce a simple design for estimating values or connections between variables based upon direct relationships.

Generalized linear regression develops a design of the variable or process that you are trying to understand or forecast, which you can make use of to discover and evaluate relationships between attributes. Features containing missing values independent or informative variables will certainly be left out from the analysis; nonetheless, you can utilize the Fill out Missing Values device to complete the dataset before running the Generalized Linear Regression tool.

from stats.stackexchange.com

This simple direct regression calculator utilizes the least-squares method to locate the line of ideal fit for a combined data collection, allowing you to approximate the value of the reliant variable (Y) from an offered independent variable (X). Unadjusted linear regression develops linear versions that minimize the number of settled mistakes between the actual and forecasted values of the training data target variable.

Basic direct regression is made use of to locate one input variable (predictor variable, independent variable, input function, input parameter) and one result variable (forecaster variable, dependent variable, result feature, result specification) as well as one input variable (forecaster variable, independent variable) the best partnership in between variables, input variables). features, input specifications) result variables (prediction, dependent variable, result functions, output criteria), supplied both variables are continual. Any type of econometric design that considers several variables can be a multiple regression.

Several regression versions are intricate, and it ends up being extra complex when extra variables are included in the version or the amount of data to be evaluated boosts. When all other predictors are considered continuous, i.e., for our traditional height-weight example, if x1 differs by one device, and x2 and x3 stay the same, Y will certainly vary generally by b1 units.

Quote of the day 5 May 2022

Have a specu-taco-ular Cinco de Mayo!

Now check out my article on ->. Linear Regression!

Wednesday, May 4, 2022

Quote of the Day - 4 May 2022

This holiday is yours, where we all share with you the hope that this day brings us closer to freedom and to harmony and to peace. No matter how different we appear, we’re all the same in our struggle against the powers of evil and darkness.

– Princess Leia

Tuesday, May 3, 2022

Quote of the day, 3 May 2022

“Whoever fights monsters should see to it that in the process he does not become a monster. And if you gaze long enough into an abyss, the abyss will gaze back into you.” ― Friedrich Nietzsche

Monday, May 2, 2022

"Talking nonsense is man’s only privilege that distinguishes him from all other organisms." -Dostoyevsky

Saturday, November 20, 2021

16 Things that make a Developer scream, "NO!"

Babcock's Laws of IS Development

This comes from a colleague in the IT sector from long ago (could have been in a galaxy far away, but that's not important right now).

1. If you automate a mess you end up with an automatic mess.

This gem was one of the first principles learned in an IS career that now spans three decades. As far as I know, it is original with me. It seems that the first problem most of us encounter as we seek to apply IS automation to any process is gaining an understanding of that process. We often find that the current process is both inconsistent and incoherent. Automation applies a rigor that generally forces the existing processes to be recast in order to achieve the required coherence. Unfortunately, this step is often overlooked because it is costly or unpleasant. If we don't take the time to straighten out the existing "manual" processes prior to automating them, we will end up with a mess that is now automated. This has been the downfall of many IS projects. We rush right in with a technical solution before fixing the underlying logical/organizational problems.

The developer is in the unenviable position of being the messenger bearing bad news. Nevertheless, if he is true to his craft, he must have the fortitude to "tell it like it is." They do, on occasion, shoot the messenger. You don't really want to work for those clients anyway. If the customer's processes are logically flawed they will not be helped by mere automation. Adding automation can actually result in performance that is WORSE than that prior to automation.

If the customer is adamant and you don't mind the risk to your integrity and reputation, then at least work on an hourly basis. You'll come out ahead financially, albeit with ulcers.

2. Those who don't plan to fail surely will.

I believe that I coined this one as well. However, in all fairness, I must credit my early training (both explicit and incidental) in this topic to Exxon. In the late 70's they were very high on sending their engineers to courses put on by Kepner-Tregoe that addressed this concept.

It has been my observation that most project plans are put together with a sort of linear optimism which reflects Gantt-chart thinking as each task moves to the next in stair-step order. Almost never do I see plans that actually include FAILURE as a task to be planned for. This is most often reflected in IS plans which show the stages of development neatly followed by a testing phase (almost always too short) which in turn is followed by a roll out to production.

I seldom see a formally published plan that actually shows a task (or the time allotment required) for reworking a project to fix the problems encountered during testing. One exception I recall was done by a fellow that asked me to review his plan and applied this law as noted. He actually added tasks to account for failure. His plan was a success, by the way. I suppose that it is politically incorrect to suggest that our efforts are less than perfect on the first try. My experience suggests otherwise. How many programs run without error on the first compile?

On the other hand, those who diligently PLAN for failure will often be pleasantly surprised when that part of their plan is not needed. They, at least, have a bonafide chance to come in ahead of schedule. If they DO have to make repairs, at least they have planned for it.

3. If it is an R & D project, plan to do it twice.

What is an R & D project? Basically, anything that has never been done before or which significantly depends upon component technologies with which there is little or no prior experience. This kind of endeavor typically requires that at least one version be built to completion as a prototype. Only in the rarest of cases will the prototype be satisfactory. More often than not, the first really viable version will be the second try. This law is related to law number two in the sense that most project managers don't have the fortitude to plan to fail. This is simply silly in the face of an obvious R & D project. I suppose this stems from the unwillingness of project managers/leaders to ask their bosses for the resources to "waste" on learning how to actually do some new thing. The bosses are not without blame as they often don't receive such requests with good sense either.

The sad result is that often the prototype is often foisted upon the unwilling victims as the production project with the assurances that all "minor" deficiencies will be fixed "in the next release." Instead, we should plan to build one to learn how to build one and then throw it away prior to building the "real" version. It may seem hard to ask for resources to build a "throwaway" but isn't it more honest? In reality, this is what happens anyway. If we were honest about it in our planning then so many of the recriminations and ulcers that we get would disappear. In addition, the TRUE cost of an endeavor would be available to managers "up front" before the resources are committed. This can (we hope) result in better decision-making.

4. Let costs remain where they are incurred.

I suppose this "law" has been forming for years but it really has stood the test of time in every industry in which I've had the pleasure of working from aviation to utilities. It is hard to capture the idea in a brief, pithy statement so some amplification is in order. An analogy may help.

Consider that as you walk from the parking lot to your cube, there is one piece of sidewalk that is raised so that you stub your toe PAINFULLY upon it. I'll wager that the next time you come in, you studiously avoid repeating the incident. But suppose that, through some mix-up in your nervous system, when you stubbed your toe, you felt the pain in your knee. Now based upon the erroneous location of the pain you conclude that the pain you are suffering on your daily walk to work must result from the jogging that you have been doing as a form of regular exercise. So you quit jogging, get fat, and die from a heart attack.

Organizations are not unlike the human body. However, the management/nervous system is much less sophisticated. It has relatively few pain receptors and these are most concentrated in the areas of budgets, schedules, and resources (time, money, materiel, and personnel.) Management has been described as "Achieving the desired end(s) within the available resources." In our analogy, management directs the activities of the organization in order to "minimize pain." I realize that, at face value, this statement may cause some of you to lose me at this point but hang in there.

What typically happens is that some problem or obstacle arises which impedes progress. This is the "pain." Instead of dealing with the cost of working through the problem, a shortcut or alternate route is taken. This alternate route has its own "pains" associated with it. These pains, however, may not be felt by the immediate group that chooses the alternative and so they don't feel any local pain. However, some other part of the organization usually does. The problem is that, now that the pain has been dislocated, management perceives the pain as originating in the other part of the organization and therefore makes ill-advised plans for remedies.

A recent case study will serve to illustrate the (proper) application of this principle. Up to now, we have been using a form of disk cloning to mass-produce workstations of a given configuration. This practice dates back to the earliest days of DOS and the availability of disk cloning hardware/software. It is considerably faster and easier than individual installations of software using the manufacturer's installation programs. It seemed to work pretty well with DOS. The advent of more sophisticated 32 bit operating systems has made this practice increasingly questionable. Although it has worked reasonably well with OS/2. It will not with NT.

Originally, we planned to "clone" NT configurations as we have with DOS and OS/2. However, we found that Microsoft would not agree to support these configurations if they were cloned. They insist upon individual builds. There are, in fact, reasonable technical arguments for this. However, it is much more COSTLY in terms of manufacturing and "upfront" configuration development. With the architecture as it stands and our intended use of it, it was tempting to ease the "pain" of manufacturing and development by cloning. However, this merely shifts that cost "downstream" to support personnel and to future rework, if Microsoft's architecture takes a turn that makes their present restriction of more effect. If we took the shortcut, the costs and hence the organizational "pain" would be borne in the support area instead of in manufacturing. Management might well try to apply fixes to the support structure when, in fact, it was manufacturing that really needed the attention. Proper manufacturing would reduce the support costs. Since those in the support world rarely get the chance to voice their opinions in the early phases of a project, it would be easy for those concerned with manufacturing to try and "save a buck." Besides, manufacturing (since it is outsourced) is "real" money whereas the consumption of support staff is "funny money", right? Wrong!

Compounding this is the fact that costs that are dealt with head-on are often very identifiable whereas costs that are dislocated often become obscured. It is easy for management to feel the pain of an expensive manufacturing invoice. By the time costs have been dislocated downstream to the support staff, the "pain" becomes harder to locate with the same degree of precision.

Fortunately, we made the "right" decision and have elected to adhere to the Microsoft guidelines. This will mean higher upfront costs and effort but by meeting these costs "where they are incurred" we will undoubtedly avoid much more difficult "pain" in the future.

Finally, having been a manager myself for a number of years, may I say that it is simply unfair to the poor creatures to distort the little data they have to work with by obscuring and dislocating costs. Give them a break. Let costs remain where they are incurred. If you HAVE to move them, make sure that the pointers to the original "pain" are well documented and understood by all. At least this gives a manager a chance at a good decision.

5. Don't encode independent information in database keys (or names for that matter.)

This is really just an application of database normalization. When a database is in third normal form, the relationship between table records and their keys may be summed up by saying that the information in the record is related to the key, the whole key, and nothing but the key. I've seen numerous database designs that did not follow this dictum and they almost always ran into difficulties later on. There are broader applications outside of mere database discussion.

Entity-Relationship (ER) modeling applies to more than just database design. I find that it often arises when discussing network-naming standards. More generally, it will apply anytime naming is in view. Consider the name as the key and the thing named as the record. To be "normalized" the attributes of the thing named should be associated only with the name, the whole name, and nothing but the name.

Take the case of your checkbook. Suppose you designated check numbers ending (or prefixed with) in "1" for a mortgage, "2" for utilities, "3" for auto expenses, "4" for medical bills, etc. Well, you only pay the mortgage once per month so if you pay that with check 1 and get to check 11 before the next month you have to "skip" it or discard it. Invariably, you end up painting yourself into some kind of corner when the underlying attribute of the entity whose key you are trying to encode changes. Or, you end up having distributions of attribute occurrences that conflict with the more or less even distribution of any numeric encoding scheme. The check number should ID the check, period. It should NOT ID what the check was written for or where it was applied.

As a real-world example of the problems that can arise when this is not followed, consider the case of having a participant ID "scheme" in conducting clinical trials that is more than just a simple number. In a typical multi-center trial I've seen the participant ID assigned so that the leading digits correspond to a given center. But what happens when that participant in a long trial moves into another region and starts reporting to a different center? Now you have an anomaly that you live with or you have to go refactor all of the participant records with a new ID.

The bottom line is that any time you try to encode some information into a key or name for an object that is not dependent solely and wholly upon that object but has outside dependencies, you will inevitably encounter the corruption of the naming scheme when any of those outside dependencies change. It is tempting to try and encode locations, topologies, and the like into names so that the very name conveys additional meaning. Invariably, it paints the "schemers" into a corner that later proves difficult to correct. Don't do it.

6. Never try to swim with too much chain.

At first, glance, what I'm about to describe may sound like good old-fashioned scope creep. In a way, it is. However, the genesis is much different. With conventional scope creep, you typically can easily identify the source of accretions to your project. Usually, your customer is simply adding to his expectations. The beast I'm going to describe is much more sinister and hard to recognize.

An organization in many ways can be thought of as a body. The body has so-called "voluntary" and involuntary systems. For example, your heartbeat is an involuntary system. We don't have to consciously command the heart to beat. Eating is a voluntary system. Breathing can be both. Similarly, the organization has formal and informal management systems. The formal management systems are easy to recognize (they are called bosses.) The informal management systems that exist generally arise from organizational needs that are not met by the formal management systems. They can be harder to recognize but can actually be more powerful (to a point) than the formal systems. They are harder to recognize because they are created on an ad hoc basis by staff trying to get something done. They won't appear on any organizational chart or in any policy manual. For the most part, the players seem to intuitively understand this and manage to discover those that are essential to getting what needs to be done, done.

As I said, these "involuntary" systems can be extremely powerful. If you starve your body for protein, it will begin to rob protein from muscles in order to survive. Ultimately, this will result in death if not corrected. In the same way, there are strong forces at work in any organization to meet its unmet needs. The difficult part is that these are often hard to recognize. Failure to recognize and deal with these properly can literally spell death for an endeavor.

Here's what typically happens: First, you have an organization that has a number of unmet needs. This is particularly likely in organizations that are "lean." Someone in the organization begins to address one of these needs. Someone doing something to correct one problem attracts the attention of other problem holders who then try to find ways of getting their needs met by the new initiative. To visualize how this happens, imagine a group of swimmers in a rough sea all trying to tread water. Suddenly, one swimmer begins to make progress toward the shore. Immediately, the others try to attach themselves to the strong swimmer, and thus overloaded and distracted, all drown.

We have a saying down in Louisiana when some miscreant is discovered drowned in the Mississippi river, "Poor boy just tried to swim with too much chain." I've seen otherwise worthy initiatives sink rapidly out of sight when they tried to take on "too much chain." It is essential that project managers keep their focus clearly upon the "shore" and not allow other "swimmers" to jeopardize their success, no matter how worthy the plea. It is often very hard to say no. It takes maturity and discipline. Better than a few make it to shore than for all to drown.

7. Thou shalt not admit vagrant processes.

A vagrant is defined as one with "no visible means of support." As we grapple with providing solutions to the problems that confront us, no doubt many good suggestions will be placed on the table. Before putting any of these into effect, it is necessary to clearly identify how the new processes will be supported. Generally, this means that a process "owner" must be clearly identified. It is unusual to find processes that don't require some "care and feeding" during their useful life. The "caretakers" must be identified.

Unfortunately, it seems more often the case that some new process/solution will be created and placed into service to meet an immediate need. With the crisis past, we forget to go back and add the infrastructure necessary to continue the process. This often results in illogical and awkward assignments of duties within an organization. One group may end up "owning" the solution, not because they are the logical place for it but merely because they were "there."

It is, therefore, better to require that such proposed solutions be required to present their "means of support" before they move to any stage of the actual implementation. Or, to put it another way, don't waste time and effort wrestling with a solution, no matter how attractive, that has no identified means for support. Ultimately, you will end up wasting effort and creating a larger problem. There is a real danger that if sufficient effort is expended, it will become too "embarrassing" to discontinue the unwise course and the solution/process will take on an unwarranted life of its own.

This is all somewhat general so a couple of clear examples from experience are in order. In one company plagued with a complicated and arcane procurement process, one of the proposed new processes was an online catalog that would allow end-users to directly view and select items for purchase from it. It was a wonderful idea. Unfortunately, the catalog was implemented and considerable effort was expended on the technology with little or no thought given to who would ultimately maintain the thing. When it came time to put the technology into effect, there was no catalog caretaker to be found. The effort though considerable, was stillborn. The real and present danger is that the potential for "embarrassment" may cause management to force the caretaker role on some unwilling victim instead of "cutting their losses."

In another company, we were considering bringing in Microsoft's SMS to manage our workstation environment. This is truly a comprehensive product. However, even Microsoft points out that it requires an organization to take care of the product and administer it. It may be the greatest thing technically since sliced bread but we should not take one step toward implementing it until the support means has been explicitly identified. To do so would invite failure, embarrassment, recriminations, and anger. It is always so.

The proper approach is to never allow any proposal or idea that does not come with an identified and explicit means of support to remain for very long on the table of ideas. It is certainly proper to bring forth suggestions but before those suggestions are taken very far and before much thought and effort are expended on their behalf, we must insist that the "hard" questions concerning who, what, when, why, and how are answered or have a reasonable (and not just wishful thinking) expectation of being answered.

8. Thou shalt not covet useless data.

By definition, useless data is data for which there is no specific and defined use. One of the more common tasks in developing various information processes and systems is the creation of paper or automated data collection forms. During the design phase of these forms, it is exceedingly difficult to keep squarely focused on the task at hand as those involved suggest all manner of "nice to have" items that might be solicited. Fundamentally, no data should be solicited without also specifying the destination or consumer of the data and the associated downstream process. If this requirement is kept, it will go a long way towards keeping the data entry and collection tasks under control. The architect should insist that no information be included for which there is no specifically defined consumer process.

9. Honor thy parser for then it will be well with you and then you shall have good success.

When you boil information processing down to its essentials, you can describe it by saying that we take data in one place and move it to another and perhaps manipulate it on the way. In over a quarter of a century of programming in numerous languages, I've found many similarities and only a few salient features that put some languages at the top of my "most useful" list. All of them have assignment operators, branching operators, logical operators, math operators, etc. Only a few have robust built-in parsing capability.

Moving data to and fro is not difficult. When you have to take that data apart and put it back together in new forms, you must invariably parse it. Logic and math operations are straightforward but you can spend hours of coding to get the parsing "right" if you don't have a language tool that provides robust parsing operators. Parsing fixed format data is not difficult but as the data becomes more "conversational" in structure, parsing takes on additional challenges.

Whether you want to have glitzy GUI's or wow them with your website, you will always need to parse. Choose your weapons wisely. An appropriate choice can make you look like a productive genius.

10. It is easier to get criticism than criteria.

It seems to be hard to get customers to tell you what they want. Sometimes this is because they don't know what they want. Sometimes they don't have the "language" with which to communicate their needs (programmers have been known to speak in a strange tongue.) Sometimes it is just hard to get them to take the time out of their busy schedules and focus on the problem long enough to render a usable specification. However, you can be sure that when you've built whatever it is that you are building for them, they will be quick to tell you what they really wanted it to do and even quicker to point out what is just plain wrong.

If you have the stomach for it (and management that is clued-in to the approach) you can often get design criteria much faster by building a "throwaway" (the politically correct term is "prototype".) In the face of ever-shortening deadlines and scarce resources, it is often hard to get "buy-in" to this approach but in the hands of a seasoned developer, it can actually get you where you need to go faster and with less expense.

For this to work, you must establish a minimum level of credibility with your client. This is because they have a natural fear that the prototype will be the production model. Until you demonstrate that you really will make good on your promise to build the "real one" after the prototype, you may take a lot of "heat." However, if you tell them from the start that this is the approach you will take and then follow through, subsequent projects can become much easier as a spirit of true collaboration is developed.

So, if you are having a tough time nailing down what is required, consider taking your best shot and presenting it. You will then get all of the design criteria you would ever want and then some!

Original post: https://www.babcock.org/is_laws.htm

Disclaimers for techgeezer.io

All the information on this website/blog is published in good faith and for general information purpose only, and does not make any warranties about the completeness, reliability and accuracy of this information. Any action you take upon the information you find on this website, is strictly at your own risk. Techgeezer.io will not be liable for any losses and/or damages in connection with the use of our website. From our website, you can visit other websites by following hyperlinks to such external sites. While we strive to provide only quality links to useful and ethical websites, we have no control over the content and nature of these sites. These links to other websites do not imply a recommendation for all the content found on these sites. Please be also aware that when you leave our website, other sites may have different privacy policies and terms which are beyond our control. Consent By using our website, you hereby consent to our disclaimer and agree to its terms. Update Should we update, amend or make any changes to this document, those changes will be prominently posted here.