The Pattern I Keep Seeing

Here’s how every AI agent project I audit goes:

Month 1: Beautiful demo. The agent works. The board is impressed. The founder thinks they’ll ship in 6 weeks.

Month 2: The agent entered an infinite loop and burned $4,000 overnight. OpenAI went down and took the entire product with it. Duplicate requests are processing twice because nobody implemented idempotency.

Month 3: The sprint board is full of tickets like “Implement circuit breaker for LLM calls” and “Add Redis caching for embeddings.” Zero tickets that deliver value to users.

Month 4: The team realizes they’ve accidentally built 80% of LangChain, badly. Or they’re so deep in custom infrastructure that the two engineers who understood it have quit. The agent barely works, and nobody knows why.

Sound familiar?

But the infrastructure mess is a symptom. The root cause is a dangerous assumption baked into how most teams think about agents.

Assumption Zero: Your Agent Is Not Trustworthy

This is the assumption that 90% of agent frameworks refuse to say out loud, but 100% of production systems are forced to confront: the LLM is an unreliable brain.

It will hallucinate. It will misinterpret intent. It will confidently propose actions that are catastrophically wrong. Not because it’s “bad”—because that’s what probabilistic systems do.

LLM output is always a proposal, never an instruction.

Once you accept this, everything changes. Every design decision in Kite—policy enforcement, audit trails, human-in-the-loop checkpoints, explicit execution boundaries—is not a “feature.” It’s a logical consequence of taking this assumption seriously.

If you remove this assumption, Kite loses its reason to exist. And your production system loses its safety net.

The Three Separations Most Frameworks Refuse to Make

Most agent frameworks blend three fundamentally different things into one messy layer: cognition (thinking), decision (choosing), and execution (acting). Kite separates them deliberately.

1. The Agent Has No Authority

The LLM lives exclusively at the cognition layer. It can think, reason, suggest. But it cannot decide and it cannot act. All authority lives in code and in humans. The agent proposes; the system disposes.

This is kernel-level thinking, not app-level thinking. The LLM is an unprivileged process. It can make syscalls (tool requests), but the kernel (Kite’s enforcement layer) decides whether to grant them.

2. Safety Does Not Live in the Prompt

Prompt engineering is not security. Alignment is not safety. A jailbreak isn’t a bug—it’s a characteristic of the medium. If your safety strategy is “we told the LLM to be careful,” you don’t have a safety strategy.

Kite’s safety is enforced in code: circuit breakers, idempotency keys, kill switches, policy validators, boundary checks. These are deterministic walls that the LLM cannot talk its way through.

3. Boring Beats Smart

Production AI is 99% plumbing, 1% magic. Your competitive advantage isn’t your agent framework—it’s your domain expertise, your data, your business logic. Kite handles the boring 99% so you can focus on the 1% that matters.

Is Kite Strong Enough to Build Real Apps?

Yes—precisely because it doesn’t pretend the LLM is reliable.

The question people actually want answered is: “Where does the logic live? Who’s responsible when things go wrong?”

Logic does not live in the prompt. In Kite, the prompt is untrusted input. The LLM output is a proposal. The reasoning is a suggestion, not a fact. Business logic lives in code—testable, auditable, rollbackable code.

When the LLM proposes “delete instance X,” Kite asks deterministic questions: Does the policy allow this? Is the instance in the allowlist? Does a human need to approve? Is this within budget? The prompt never makes the call.

If you put logic in your prompt, you are building something you cannot test, cannot audit, and cannot roll back.

Kite does not assume the agent is correct. The opposite. Kite’s foundational assumption is that the LLM will be wrong, and it will be wrong in dangerous ways. Therefore: the agent is not trusted, the agent has no authority, and the agent bears no responsibility. It’s an advisor, not an actor.

The framework does not “cover for you” when the LLM hallucinates. Kite doesn’t try to fix hallucination with better prompts. It doesn’t pretend agent output is reliable truth. Instead, it limits the blast radius. Every action passes through enforcement boundaries. Every step is traced: what the agent said, where the framework refused, who approved what.

The framework is responsible for architecture. It is not responsible for the LLM’s thoughts. That’s the correct boundary.

So who’s responsible when a bug happens? The system is. Not the LLM. If a bug occurs in a Kite-based application, the root cause should never be “the LLM was wrong” or “the prompt wasn’t good enough.” It should be: the policy was incomplete, the validator was weak, the boundary had a gap, or the human approved something they shouldn’t have. This is production thinking.

What Kite Actually Gives You

Stop rebuilding the same infrastructure. Here’s what’s in the box:

Safety Layer. Circuit breakers, idempotency, kill switches, rate limiting. An agent that works 99% of the time and burns $10K the other 1% is a liability, not an asset.

@ai.circuit_breaker.protected
def process_refund(order_id, amount):
    return stripe.Refund.create(charge=order_id, amount=amount)

# 3 consecutive failures → circuit opens → blocks for 60s
# No infinite loops. No $15K mistakes. Deterministic.

result = ai.idempotency.execute(
    operation_id="user_123_payment",
    func=process_payment,
    args=(user_id, amount)
)
# Run this 10 times. It executes once. The rest return cached results.

Memory Systems. Vector memory (FAISS/ChromaDB), Graph RAG, session memory. Semantic search and multi-hop reasoning out of the box. Lazy-loaded—you only pay for what you use.

Agent Patterns. ReAct, Plan-Execute, ReWOO, Tree-of-Thoughts. Best practices, not boilerplate.

Pipeline System. Checkpoints for human approval, intervention points, state persistence. Real human-in-the-loop, not polling loops.

Provider Agnostic. One API across OpenAI, Anthropic, Groq, Ollama. Your business logic stays the same. The provider is just a config variable.

# Switch providers in one line. Business logic unchanged.
ai = Kite(config={"llm_provider": "openai"})
ai = Kite(config={"llm_provider": "anthropic"})
ai = Kite(config={"llm_provider": "groq"})
ai = Kite(config={"llm_provider": "ollama"})

Observability. Metrics, circuit breaker stats, cost tracking, structured logging. When something goes wrong, you need to know exactly what the agent proposed, where the framework blocked it, and who approved the rest.

What Kite Isn’t

Not production-ready. This is v0.1.0 (alpha). The framework works. We use it internally. But it has no comprehensive test suite, limited tooling, and APIs that might change.

Not a LangChain replacement. LangChain has 1000+ integrations. Kite has 10 core components. If you need every obscure tool connector, use LangChain.

Not an enterprise platform. If you need 24/7 support and compliance certifications, use AWS Bedrock.

Kite is for small teams (1–20 engineers) who want to move fast without spending months on plumbing—and who understand that “the agent is always right” is a dangerous fantasy.

Conclusion: Your Users Don’t Care About Your Infrastructure

They care if your agent works. And more importantly, they care if your agent doesn’t destroy things when it doesn’t work.

Kite doesn’t make the agent smarter. Kite makes the agent less dangerous when it’s wrong. If you’re building infra automation, financial workflows, data pipelines, or anything with compliance requirements—that’s not a premium feature. That’s the minimum.

Kite is open-source (MIT): → GitHub: github.com/thienzz/Kite

Want to learn the architecture philosophy behind it? → Read: Designing Agentic AI Systems

Questions? Reach me at thien@beevr.ai or open an issue on GitHub.

For a long time, we treated AI like a librarian. Its job was to walk into the stacks (your database), read a book, and summarize it. If the librarian hallucinated, the user got bad advice. It was embarrassing, but the failure was contained. The database remained intact. The bank account was untouched.

That era is ending. Stakeholders are no longer impressed that an AI can summarize an invoice. They are asking a question that sounds simple but carries terrifying engineering implications: “Why can’t the AI just pay the invoice?”.

This is the shift from Read-Only AI to Write-Access AI. And when you hand a stochastic, probabilistic model the keys to your deterministic infrastructure, things can go wrong very quickly.

The Anatomy of a $15,000 Mistake

To understand the risk, let’s look at a real scenario from a fintech startup I audited recently. They wanted to upgrade their support bot from “answering questions” to “handling refunds”.

The setup seemed standard. They gave the Agent access to the Stripe API and a system prompt: “If a customer has a valid complaint… you are authorized to issue a full refund.”.

Then came Black Friday.

1. A user messaged: “My package is late, I want my money back.”.
2. The Agent correctly identified the intent and called the refund_transaction tool.
3. The Glitch: The Stripe API was under heavy load. The refund succeeded on the backend, but the HTTP response timed out before reaching the Agent.

In a traditional Python script, we would handle this with a specific error code. But the Agent isn’t a script; it’s a reasoning engine.

The Agent received a generic TimeoutError. It “thought” to itself: “Oh, the tool failed. The user is still angry. I must try again to fulfill my goal.”.

It hit retry. Again. And again. It entered a tight loop, hammering the API.

By the time the engineering team noticed the anomaly in the logs—about 45 seconds later—the Agent had successfully issued 50 duplicate refunds for the same order.

It didn’t just refund the purchase price; it drained the merchant account of nearly $15,000 in duplicate transactions and non-refundable processing fees.

Probabilistic Software vs. Deterministic World

This story illustrates the fundamental conflict in Agentic Engineering.

For fifty years, software engineering has been built on a bedrock of certainty. If you write if x > 5, it will always be true when x is 6. If it isn’t, it’s a bug. You find the line of code, you fix it, and it stays fixed.

AI Agents operate on a different physics entirely. They are Probabilistic Software.

• Run 1: The model sees “I am disappointed” and offers an apology.
• Run 2: It sees “I am disappointed” and offers a refund.
• Run 3: It decides “disappointed” is a threat and flags the user.

If you build Agentic systems using the same “vibes-based” engineering you used for your RAG chatbot—relying on prompt engineering and hope—you will create disasters.

You cannot prompt your way out of a race condition. You cannot “fine-tune” away a network timeout loop.

The Solution: The Deterministic Shell

If we want to build safe agents for the enterprise, we must stop treating them like magic and start treating them like untrusted components.

We need to build a Deterministic Shell around the Probabilistic Core.

The architecture requires us to assume the LLM will make mistakes, and design the system so that those mistakes cannot burn down the building. This means implementing hard engineering guardrails—written in code, not prompts—that the AI cannot override.

Here is what that “Shell” looks like in Python. It wraps the AI’s tool call with Idempotency (so duplicate requests are ignored) and Circuit Breakers (to stop runaway loops).

from functools import wraps
import hashlib
import time
# import stripe # Giả định đã import thư viện stripe

class RefundCircuitBreaker:
    def __init__(self, max_failures=3, timeout=60):
        self.failures = {}
        self.max_failures = max_failures
        self.timeout = timeout

    def __call__(self, func):
        @wraps(func)
        def wrapper(order_id, amount, **kwargs):
            # 1. GENERATE IDEMPOTENCY KEY
            key_data = f"{order_id}:{amount}:{kwargs.get('reason','')}"
            idempotency_key = hashlib.sha256(key_data.encode()).hexdigest()
            
            # 2. CHECK CIRCUIT BREAKER
            if idempotency_key in self.failures:
                last_failure, count = self.failures[idempotency_key]
                if time.time() - last_failure < self.timeout:
                    if count >= self.max_failures:
                        raise ValueError(f"Circuit breaker open for {idempotency_key}")

            try:
                # 3. CALL THE API
                result = stripe.Refund.create(
                    charge=order_id,
                    amount=amount,
                    idempotency_key=idempotency_key  # Critical!
                )
                
                # Clear failures on success
                if idempotency_key in self.failures:
                    del self.failures[idempotency_key]
                
                return result

            except stripe.error.APIConnectionError as e:
                # 4. HANDLE FAILURE DETERMINISTICALLY
                if idempotency_key in self.failures:
                    last_time, count = self.failures[idempotency_key]
                    self.failures[idempotency_key] = (time.time(), count + 1)
                else:
                    self.failures[idempotency_key] = (time.time(), 1)
                
                if self.failures[idempotency_key][1] >= self.max_failures:
                    raise ValueError("Circuit breaker activated - too many failures")
                
                raise 
                
        return wrapper

@RefundCircuitBreaker(max_failures=2, timeout=300)
def process_refund(order_id, amount, reason):
    return stripe.Refund.create(
        charge=order_id,
        amount=amount
    )

Why This Code Matters

Notice what is happening here:

• Idempotency: Even if the LLM enters a panic loop and calls the function 50 times, the idempotency_key ensures the downstream API handles it as one transaction.
• Circuit Breaker: If the API fails 3 times, the Python code throws a ValueError and stops the execution. The LLM is not asked for permission to stop. It is forced to stop.

This is not “Prompt Engineering.” This is Software Engineering.

Conclusion: Autonomy is a Bug, Not a Feature

When stakeholders ask for an “Agent,” they usually imagine a digital employee that figures everything out on its own. This vision is what sells venture capital rounds. It is also what kills production systems.

In my view, autonomy is a bug, not a feature.

Every degree of freedom you give an Agent increases the surface area for errors. If a task can be done with a linear script, do not build an Agent just to look cool.

We are moving from the library (Read-Only) to the real world (Write-Access). The “Intern” now has the company credit card. It’s time we started engineering the safeguards to match that reality.

A new client comes to us, frustrated. They just paid a ‘modern’ tech agency for a platform that’s completely unmaintainable.

We pop the hood, and it’s magnificent.

It’s a state-of-the-art “Cloud-Native,” “AI-Powered,” “Event-Driven,” “Serverless” system. A stunning monument to modern engineering, designed to handle 10 million concurrent users for a B2B app that has 500.

The previous agency didn’t solve the client’s problem. They solved their own problem: how to get “GenAI” “Kubernetes” and “VectorDB” onto their developers’ resumes.

Welcome to the most toxic trend in our industry: Resume-Driven Development as a Service (RDDaaS).

The RDDaaS Playbook (How to Scam Your Client)

It’s a brilliant, cynical business model, and it works like this:

Step 1: The Future-Proof Pitch You show the non-technical client a beautiful slide deck. You blind them with charts showing massive ROI, impressive KPIs, and tech buzzwords like “infinitely scalable,” “AI-powered,” and “Future GenAI-ready.”

Step 2: The Training Ground Your mid-level developers, who have only read about “RAG Pipelines” and “Kubernetes,” now get to learn them right on the client’s dime.

Step 3: The Over-Engineering Phase The project, which should have been a 3-month simple CRUD app, now takes 9 months. They’re not solving the client’s problem. They’re solving Google’s problem and Meta’s problem.

Step 4: The Successful Handoff The system is delivered. The chatbot confidently hallucinates the wrong phone number. The vendor’s developers proudly update their LinkedIn profiles. The vendor gets paid.

Step 5: The Victim’s New Life

The client is now the proud owner of an intelligent thing that requires a team of ex-FAANG SREs just to add a new form field.

Their new life includes:

1. The Hiring Nightmare: Their in-house “IT guy,” John, quits after seeing a terraform script that spawns 15 AWS services (EKS, Lambda, VectorDBs…). The first real candidate who understands this mess demands $250k.

2. The Wrong Web-Scale Performance: The app is slow. A simple request now makes 5 HTTP calls through a service mesh and a $0.02 call to gpt-4-turbo just to say “Hello.”

3. The “WTF” Cloud Bill: The first few bills look okay (thanks, AWS Free Tier). Then, Month 4 hits. (Trust me, you really don’t want this.) The real bill arrives, full of NAT Gateways, EKS Control Planes, Managed Vector DBs (99% empty), and OpenAI API fees. Their plumbing costs 20x more than their app.

4. The Simple Change Request: The client asks: “Can we just add a normal search bar? The AI one is too expensive.”

   • In the old monolith: 1 hour.
   • In this modern system: “Uh, that’s not how the RAG pipeline works. We’d have to re-architect the whole data flow. That’ll be a new 2 sprints.”

The Root Cause: Why Did This Happen?

Why does this scam always work? It takes two:

1. The Vendor’s Hidden Agenda: The client isn’t the customer. They’re the training ground. The agency didn’t solve the client’s problem “I need a reliable app”. They solved their own problem “Our developers need ‘GenAI’ and ‘K8s’ on their resumes”.

2. The Client’s Blinders: The client lets this happen. They get blinded by the ROI slides and tech buzzwords. They’re so terrified of being “legacy” that they actively forget to ask the two boring, critical questions:

• “What is my actual problem?”
• “What’s the Year 2 maintenance and cloud bill going to look like?”

The architecture was optimized for Imaginary Scale and Imaginary Intelligence, not Current Maintainability.

How to Not Get Scammed

This isn’t just a rant. This is a pattern I see many times. It’s the entire reason my philosophy is built on Boring Technology

The antidote to RDDaaS is to stop being impressed by buzzwords and start asking the questions that actually matter.

Next time a vendor pitches you a “Cloud-Native AI-Powered” solution, just ask them these things:

• “Can you justify why we need Kubernetes for this?”
• “Walk me through the full development process—from ticket to deployment—for adding one new database field to the ‘User’ model.”
• “What is the fallback mechanism when the AI/RAG pipeline fails or hallucinates?
• “Show me the Year 2 cloud bill for this architecture.
• “What kind of engineer do I need to hire to maintain this after you leave?”

You don’t need FAANG-scale complexity. You need Product-Market Fit.

Start buying maintainable solutions that actually let you find it.

We’re all a bit drunk on the hype. We’re treating AI like magic dust we can just sprinkle on any problem.

Clients want a Youtube or Netflix level recommendation engine on day one. Devs, quite reasonably, are excited to put the shiniest new Vector DB on their resumes.

We’re starting backward. We’re trying to build the penthouse while the foundation is still a sketch on a napkin.

The reason this “penthouse-first” approach fails is that we’re ignoring the reality of what an AI actually is. We treat it like a magic box we can just plug in, but it’s not.

You can’t “set-it-and-forget-it.” You have to manage it.

• The world changes, there is a new trend?? Your AI employee will be confused because it has never seen this. It starts to drift right from Day 2.
• It makes a mistake? It doesn’t self-correct. You have to build a feedback loop to fix it.
• You want it to be smarter? You have to re-train it, and that is a continuous operational cost (OpEx).

When you understand AI is something you must maintain and not something you own, your entire architecture changes.

My Take: 99% Boring, 1% AI

So if AI is this high-maintenance “penthouse,” what’s the “foundation”?

My personal philosophy is simple: Never use expensive AI to do a job a good old SQL query can do.

The value isn’t in a single “black box” AI. The value is in a hybrid system where boring rules and code do 99% of the heavy lifting. The AI is just the 1% of spice you add at the very end.

A Practical 3-Phase “Boring” Roadmap (My Small Advice)

If I’m building a “smart” matching system, I will not start with AI. I build it in phases, layering complexity only when necessary.

Let’s walk through this with one single example: building a “Smart Candidate Search” for a recruitment platform.

Phase 1: The Foundation of Correctness (SQL Rules)

This is the 99% of the work. The goal here is 100% Correctness, enforcing the non-negotiable rules of the business. An “AI-first” system might think a candidate in Ho Chi Minh City is a “great match” for a job in Hanoi, but your business rules say that’s unacceptable. This layer is the “bouncer” at the door.

• In Practice: WHERE location = 'Hanoi' AND salary_request < 5000 AND years_experience >= 3.

This filters on facts, not suggestions. location and years_experience are binary facts. We must do this first to avoid wasting expensive AI compute on candidates who are an immediate “no.” This is your blazing fast, dirt cheap foundation.

Phase 2: The Layer of Relevance (Full-Text Search)

This is still the 99% of the work. Now that we have a list of correct candidates (e.g., in Hanoi, >3 years exp), we solve for Relevance. Our SQL filter was correct, but “dumb” about human language. A recruiter searching “programmer” won’t find “developer.”

• In Practice: Use Elasticsearch or BM25 (built into Postgres) on the resume text so “java programmer” matches “java developer.”

This 20-year-old boring tech is built specifically to solve the synonym problem without the cost or “black box” nature of AI. Critically, it’s explainable—you know why that resume showed up.

Phase 3: The 1% Layer of “Nuance” (The AI)

Only now, after our 10 million candidates have been filtered down to 1,000 correct (Phase 1) and relevant (Phase 2) candidates, do we add the final 1%: Nuance. Full-Text Search is great with words, but not ideas. This is the 1% problem AI is actually good at.

• In Practice: The AI’s job is to know that a recruiter searching for “Senior Java” is conceptually similar to candidates strong in “Spring Boot” or “Scala” (even if they didn’t type those words).

This is the key to performance: an AI ranking 10 million items is impossibly slow and expensive. An AI ranking 1,000 correct and relevant items is near-instantaneous and cost-effective.

The AI is not the filter; it’s the re-ranker. We contain the expensive, unpredictable AI. We let it play only within the 1,000 safe results our boring filters found. This gives you the correctness of SQL plus the nuance of AI, but your costs drop 99% and your speed goes up 1000x.

Final Thought

The hype will pass. The core of good engineering isn’t using the shiniest tool. It’s the wisdom of knowing how and when to solve a problem.

Your “AI Strategy” shouldn’t be about AI. It should be about the system.

Stop building the penthouse first. Build your boring, indestructible foundation.

Perfect headshots. Studio lighting, crazy sharp, precise smiles, not a hair out of place. Perfect posts and comments. Flawless grammar, zero typos.

It’s all clean, polished and soulless.

This is the Great AI Flood. The cost of looking competent, of sounding smart, has just dropped to zero.

And this is where the problem begins.

The Collapse of Signal

In economics, when you flood a market, the asset’s value collapses. For many years, polished content was a signal of professionalism. Now that AI can produce it instantly, polish has just become noise.

The burden of effort has shifted from the creator to the consumer.

My mental energy as a reader is no longer spent understanding your idea. It’s spent on a exhausting calculation: Is this real?

• Is this a real photo, or Stable Diffusion, or Gemini?
• Is this a real insight, or a ChatGPT] remix of the top 10 blog posts?
• Is this a real comment, or a bot?

This is the collapse of the signal-to-noise ratio. And it’s eroding the one thing that matters: Trust.

The Return of “Rough Edges”

When a signal becomes cheap, it’s no longer a reliable signal.

For years, a professional headshot was a signal: “I care enough about my career to spend $200.” Today, a perfect AI headshot is a signal: “I care enough to spend 30 seconds on a prompt.”

When “polish” is cheap, rough edges become the new status symbol.

A slightly blurry selfie from your office? I think it’s real. It’s Proof of Effort. A post with a small typo or an awkward sentence? It’s Proof of Thought.

But here’s the deeper signal, the one that really proves expertise: The Bumps and Bruises.

An AI-generated case study is perfect: “We increased ROI by 400%.” It’s clean. It’s also unbelievable.

The real signal of human expertise isn’t perfection; it’s the messy story. It’s the Proof of Experience.

“This was a tough project. We chose the wrong database and had to migrate for 3 weeks. Here’s what we learned…”

An AI can generate a plausible-sounding failure story. It can say, ‘we chose the wrong database,’ and even invent a ‘stubborn VP of Engineering’ or a ‘4 AM call.’ It’s a perfect remix of the thousands of ‘war stories’ it was trained on.

But that’s a script, not a memory.

The real signal isn’t hearing the story anymore; it’s interrogating it. Ask why. Ask for the specific details. ‘What exact query failed?’ ‘Why Postgres and not MSSQL?’ An AI’s story is perfect most of the time, but it collapses under deep, specific questioning.

And even then, where is the proof of history? An AI can’t show you the messy GitHub commit history. It doesn’t have three real former colleagues you can call for verification.

An AI can talk about the mess. It can’t prove the mess. It has no verifiable bumps and bruises.

What’s Next?

I keep thinking about what’s next. Honestly, I don’t think it’s going to be a better AI. I bet it’s going to be the mess of tools and habits we come up with to deal with this AI flood.

For example, it seems like we’ll go back to trusting things that are hard to fake. Why do I feel like a podcast is more real than a blog post lately? Because it takes effort. You can’t just generate one in 10 seconds. We’ll instinctively trust formats that are expensive in time and energy to produce.

This also means those hard-to-join communities suddenly become more valuable. Their annoying rules accidentally make for the best bot filters.

In the end, I guess it all comes down to origin. I find myself caring less about how polished something looks and more about where it came from. It wouldn’t be surprising if we start seeing tools that can actually certify that a real human wrote this or this photo was taken by a Sony Camera.

Conclusion

In the Age of AI, polish is no longer the signal. It’s just the baseline.

A polished surface with no proof is the new noise. A messy truth with no polish is just sloppy work.

The real signal of trust isn’t about being messy.

It’s about proving that your polish is earned.

• A database for users, products, and orders. (Easy: Postgres)
• A way to send confirmation emails when an order is placed. (Okay, add a RabbitMQ job queue).
• A cache for the homepage’s “Top 10 Products.” (Fine, add Redis).
• A full-text search bar. (Ugh. Add Elasticsearch).
• A nightly job to aggregate sales reports. (Spin up a Cron server).
• A new AI feature to find “similar” products. (The VCs will love this! Add Pinecone).

Before writing a single feature, our architecture diagram is a mess. We have six different systems to provision, monitor, secure, and scale. We have what the team at Supabase calls “dotted line complexity”—the invisible, brittle connections between these systems that will inevitably break in production.

This is the case for collapsing your stack. Not by returning to a monolith, but by realizing that the “boring” tool you started with, PostgreSQL, can replace almost all of it.

Let’s rebuild the “SimpleStore” and see how.

A Cohesive Example: Building “SimpleStore” with Just Postgres

Instead of a sprawl, we’ll build each feature by extending Postgres. The magic isn’t just in replacing tools; it’s in how the connections between them become simple and atomic.

1. The Core: The Order and the Email (The ACID Test)

This is the most critical link. In a sprawled stack, you’d have this dreaded code:

# The "Dotted Line" Failure Point
try:
    order = db.create_order(...)  # Step 1: Postgres COMMIT
    queue.send_email_job(...)     # Step 2: RabbitMQ PUSH
except:
    # What if Step 2 fails? The user paid but gets no email.
    # What if Step 1 fails? The code is a mess.
    handle_complex_rollback()

With Postgres, this is one atomic unit. We’ll use the FOR UPDATE SKIP LOCKED pattern to create a powerful job queue inside the database.

Step 1: Create the tables

CREATE TABLE orders (
    id SERIAL PRIMARY KEY,
    product_id INT,
    user_id INT,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE jobs (
    id SERIAL PRIMARY KEY,
    queue TEXT DEFAULT 'default',
    payload JSONB,
    status TEXT DEFAULT 'queued', -- queued, running, failed
    run_at TIMESTAMPTZ DEFAULT NOW()
);

Step 2: The Magic (One Transaction)

Now, our application logic becomes beautifully simple:

BEGIN;

-- Insert the order
INSERT INTO orders (product_id, user_id) VALUES (123, 456)
RETURNING id;

-- Use the returned ID to create a job IN THE SAME TRANSACTION
INSERT INTO jobs (queue, payload)
VALUES ('emails', '{"type": "order_confirmation", "order_id": 1}');

COMMIT;

This entire block either succeeds or fails together. It is impossible to create an order without its corresponding email job. We have just achieved perfect data integrity, something that is incredibly difficult with separate systems.

A Go or Python worker can now pull from this queue with a simple, highly-concurrent query:

SELECT id, payload FROM jobs WHERE status = 'queued' ORDER BY run_at LIMIT 1 FOR UPDATE SKIP LOCKED;

2. The Homepage: Caching Top Products (Replacing Redis)

Our homepage needs to show the Top 10 products. This query is slow, so we need a cache. Instead of adding Redis, we’ll use an UNLOGGED TABLE.

CREATE UNLOGGED TABLE cache_top_products (
    id INT PRIMARY KEY,
    name TEXT,
    sales_count BIGINT,
    cached_at TIMESTAMPTZ
);

An UNLOGGED table does not write to the Write-Ahead Log (WAL). This makes writes incredibly fast. The catch? If the server crashes, the table is automatically truncated.

This sounds just like the durability model of Redis! It’s the perfect, high-speed, non-durable store for transient data.

3. The Search Bar & Reports: (Replacing Elasticsearch & Cron)

We can continue this pattern for our other features:

• Search Bar: Instead of Elasticsearch, we use Postgres’s built-in Full-Text Search.

-- Add a tsvector column for product descriptions
ALTER TABLE products ADD COLUMN search_vector tsvector;

-- Keep it updated with a trigger
UPDATE products SET search_vector = to_tsvector('english', description) ...

-- Search is now a simple, fast, indexed query
SELECT * FROM products WHERE search_vector @@ to_tsquery('english', 'shiny & leather');

• Nightly Reports: Instead of a cron server (a single point of failure), we use the pg_cron extension.

-- Run a job every night at 3 AM
SELECT cron.schedule('nightly-sales-report', '0 3 * * *', 
  $$ CALL generate_sales_report(); $$
);

The best part? If you have a High-Availability (HA) Postgres setup, your cron job is *also* HA. It's no longer a fragile script on one server.

The All-Important Caveat: The “Good Enough” Trap

This approach is pragmatic, not dogmatic. The “Postgres for Everything” mindset doesn’t mean never using another tool. It means you add a new tool only when Postgres is no longer “good enough.”

The “Boring Technology” choice wins when it’s 80% as good as the “Best” tool, but 10x simpler to operate.

A perfect, modern example is pgvector vs. a Dedicated Vector DB (like Pinecone or Milvus).

• When pgvector is “Good Enough”: You have 50,000 product vectors for your “similar items” feature. pgvector will handle this beautifully. You can JOIN user data with vector data in one query. The simplicity is a massive win.
• When pgvector Breaks Down: You are building the next ChatGPT and need to query 100 million vectors with 99.9% recall and 20ms latency. pgvector will fail. It wasn’t built for this. Its HNSW index isn’t as optimized, and its query planner wasn’t designed for vector-first workloads. At this scale, the “dotted line” to a dedicated, purpose-built Vector DB is not a liability; it is a necessity.

Conclusion: Your Job Is to Fight Complexity

We collapsed our “SimpleStore” stack from six complex services into one robust database.

The benefits are not just theoretical; they are immediate:

1. Atomic Integrity: You gain ACID guarantees across your entire workflow (e..g., Orders + Jobs).
2. Reduced Cognitive Load: A new developer doesn’t need to learn six systems. They just need to know SQL.
3. Lower Operational Cost: You monitor, back up, and secure one thing.
4. Faster Development: You can stop writing “glue code” for the dotted lines and start building features.

“PostgreSQL for Everything” isn’t a silver bullet. It’s a maxim against premature optimization and over-engineering. It’s a reminder that your primary job isn’t just to add technology, but to cull complexity. And Postgres is the best tool for that job.

“Microservices Hell” is real, and the rent is high. It’s the state you reach when your plumbing is infinitely more complex than the business logic it’s supposed to support.

Based on my experience, here’s what that journey into hell really looks like.

1. The “Eventual Consistency” Headache (aka The Death of ACID)

The first thing that hits you is the database.

I remember a project where we had a critical flow: Create Order → Update Inventory → Process Payment. In a monolith, this is a single, beautiful database transaction. It’s atomic. It’s safe. It just works.

In microservices, this is now 3 services, probably with 3 separate databases. You can’t have a transaction. You are now forced to write compensating logic (also known as a Saga, but it’s basically “code to undo code”).

This “compensating logic” is a massive source of bugs. You’re now living in the land of “eventual consistency,” which is just a polite way of saying, “Your data is currently wrong, but we’ll probably fix it… eventually.” For any FinTech or HealthTech system, this is a non-starter.

Many teams try to “hack” this by using a shared database. Don’t do that. You’ve just created a monster: a hidden coupling… a single-point-of-failure.

2. The Network Tax (aka “My Function Call is Now a Bug”)

When you trade reliable in-memory function calls for unreliable HTTP calls, you pay a heavy tax. Every developer on your team must now become a distributed systems expert, whether they like it or not.

Every. Single. Call. must handle:

• Timeouts: What happens when the UserService just… doesn’t answer?
• Retries: If you retry, was the request idempotent? Congrats, you just charged the customer twice.
• Circuit Breakers: You must implement these to stop one dead service (InventoryService) from killing every other service that calls it in a “cascading failure”.

Cognitive load skyrockets. And it gets worse when teams get “Service-Mania”. I’ve seen teams of 10 engineers trying to maintain 40 services. A new feature? “Create a new service!”. And now a simple feature requires deploying 3 services at the same time. You’ve just rebuilt your monolith, but over a slow network link.

3. The Observability Tax

This is the part that kills velocity. Remember when you had one log file? Good times.

Now, a single user click might touch 10 different services. When it fails, you’re not debugging; you’re playing a distributed game of Clue.

Before you can even start writing features, you’re forced to pay the “Observability Tax”:

1. Distributed Tracing (e.g., Jaeger) to follow a request.
2. Centralized Logging (e.g., ELK Stack) to find the logs.
3. Metrics Aggregation (e.g., Prometheus) to see what’s on fire.

And this isn’t just a production problem; it destroys your development environments. How can you run 40 services on a developer’s laptop? How do you test E2E (End-to-End) when you can’t even be sure which version of a service is running?

4. The People & Management Hell

But the worst tax isn’t technical. It’s the people tax.

• An Engineer-to-Service Ratio Gone Wild: I’ve seen teams with 4-5 services per engineer. This isn’t “autonomy”; it’s “burnout.” One person is now the operator, debugger, and on-call for half a dozen systems.
• “Resume-Driven Development”: When “autonomy” means “anarchy,” you get a service in Kotlin, one in Go, and one in Rust that only one person understands. When that person leaves, you’ve “orphaned” a part of your system.
• Architecture That Mirrors the Org Chart: Your architecture will inevitably look like your company’s org chart. This is fine… until the reorg. And the company always reorgs. Suddenly, the “Payments” team is split in two, but all the infrastructure, namespaces, and IAM policies are still tangled together. You’ve just signed yourself up for a painful, long-term migration project that delivers zero value to the customer.

So… When is the “Boring” Monolith (Done Right) Just Better?

A well-structured Modular Monolith (decoupled modules in a single codebase) isn’t “legacy.” It’s a pragmatic, often superior, choice. In my experience, the monolith wins hands-down in these cases:

1. When Transactional Integrity (ACID) is King: If you’re building FinTech, HealthTech, or a complex ERP, your business must be 100% consistent. The simplicity and reliability of a real database transaction are non-negotiable. Don’t trade this for compensating logic.

2. When You Are an Early-Stage Product (Speed-to-Market is King): Your biggest risk isn’t scale; it’s building the wrong thing. A Modular Monolith lets you move incredibly fast. Refactoring a module inside a monolith is 100x easier than refactoring 10 microservices that you defined incorrectly.

3. When You Are a Small-to-Medium Team (1-20 Engineers): Microservices are a tool to solve people scaling. If you’re one team, microservices will kill your velocity with meetings about API contracts. A monolith lets your team just… code.

4. When You Don’t Have a Dedicated Platform Team: Choosing microservices without a dedicated SRE/Platform team is like buying a Formula 1 car to go grocery shopping. It’s expensive, incredibly hard to drive, and you’re going to crash.

My Parting Advice

Start with a Modular Monolith.

Design it with clean boundaries, communicate between modules via interfaces, and do not share database tables between modules. This gives you 90% of the benefits of microservices (decoupling) with 10% of the operational cost.

Only extract a module into its own microservice when you have a clear, painful, and obvious reason (like asymmetric scaling needs or a new tech stack). Microservices are a refactoring step, not a starting point.

“Look,” he said, skipping the pleasantries. “Your deck says ‘HIPAA compliant’ and ‘Security is in Our DNA’. Every vendor says that. My real concern isn’t a hacker from outside; it’s an employee. Someone curious, or someone careless.”

He leaned in. “How do you actually stop a logged-in, authenticated doctor from getting curious and pulling the record of another doctor’s patient?”

He was right to ask. That’s the real question.

He wasn’t asking about a checklist. He was asking about architecture. His concern is the single biggest failure point I see in “compliant” systems, and it stems from a fundamental, “context-blind” design.

Here’s the deep dive on the problem and the specific architecture we use to solve it.

The “Compliant” Flaw: Context-Blind Architecture

Most systems fail this test because they are built “happy-path” first. The “happy path” assumes a Doctor is a trusted entity. The architecture then follows the path of least resistance.

The “compliant” checklist only requires:

1. Encryption: AES-256 at-rest, TLS in-transit. Check.
2. Authentication: The user is logged in via OAuth/SAML. Check.
3. Role (RBAC): The user’s JWT token has Role: Doctor. Check.
4. Logging: The access is written to a log file. Check.

The resulting API is predictable: GET /api/v1/patients/{patientId}

The code’s “security” logic, often in a single middleware, just checks: “Does this user’s token have the Doctor role?” If yes, access is granted.

This is context-blind. It confirms the user’s role, but not their relationship to the data. It can’t tell the difference between “their” patient and “any” patient. This isn’t just limited to patient files. What about:

GET /api/v1/doctors/{doctorId}/schedule

What stops dr_smith from querying dr_jones’s schedule to see all his patient names for the day? This simple IDOR (Insecure Direct Object Reference) vulnerability is a direct result of a lazy, role-based architecture.

The Architectural Fix: Context-Aware Security

You cannot solve this by adding “more compliance.” You must fix the architecture.

1. From RBAC to ABAC: The “Who” vs. “Why”

RBAC (Role-Based Access Control) fails because it’s static. A role doesn’t understand relationships.

We enforce ABAC (Attribute-Based Access Control). This model is “context-aware” and dynamic.

• RBAC asks: “Is this user a Doctor?”
• ABAC asks: “Is this Doctor currently treating this Patient for an active case?”

The security policy isn’t just Allow if Role == Doctor. The policy, enforced in code, becomes:

'Allow 'Read' IF: Subject.Role == 'Doctor' AND Resource.PatientID is IN Subject.ActivePatientList AND Action.Purpose == 'ActiveTreatment'

Implementation Detail: This policy isn’t just documentation. It’s code, ideally enforced at the API Gateway (like Kong or Apigee) or in a service mesh sidecar (like Istio), before the request even hits the application.

And to answer the next logical question—performance—that ActivePatientList isn’t a live JOIN query on every API call. That would be a database killer. It’s a denormalized, read-optimized cache (e.g., a Redis set) that is populated by the “Admissions” or “Scheduling” service. The cache is updated via events (e.g., PatientAdmitted, PatientDischarged) with a clear TTL. Security must be performant, or developers will find a way to bypass it.

2. From “Audit Logs” to “Forensic-Ready Observability”

Most “compliant” audit logs are a data lake of noise, built only to satisfy an auditor. They say: [Timestamp] User 'dr_smith' accessed Patient '12345'.

This is useless for security. It’s indistinguishable from a million other valid events. You can’t build anomaly detection on it.

A secure log must capture “access decisions” for true observability.

The “Denial” Log: When that curious doctor tries to access 12346 and our ABAC policy blocks it, this is what we log:

[Timestamp] Subject 'dr_smith' (IP: x.x.x.x) attempted 'Read' on Resource '12346'. Decision: DENIED. Reason: 'Policy Violation: Resource.PatientID not found in Subject.ActivePatientList'.

This log doesn’t just go to a file. This is a high-priority event piped directly to a SIEM (like Splunk or Sentinel). You can now write a simple rule: ALERT if Subject.UserID > 10 'Decision: DENIED' events in 1 minute. You’ve just caught an active insider threat.

The “Break-the-Glass” Log: What about emergencies? A doctor needs to access a file outside their normal context. A secure system must allow this via an explicit “break-glass” function. But logging this is even more critical:

[Timestamp] Subject 'dr_smith' accessed Resource '12347'. Decision: GRANTED (EMERGENCY OVERRIDE). Purpose: 'User-provided reason: Cardiac arrest in ER'.

This also triggers an alert, but a different one: P2 - Post-Incident Review Required. The system is secure, usable, and—most importantly—accountable.

Key Technical Questions

Compliance is the floor, not the ceiling.

Don’t ask your partner if they can “pass a checklist.” You’re not buying a checklist. You’re buying an architecture that protects you when the checklist fails.

Ask them these questions instead:

1. “Show me your data model for authorization. Is it role-based or attribute/relationship-based?”
2. “How do you log an authorization failure versus an authentication failure?”
3. “How do you handle ‘break-the-glass’ scenarios in your logging and alerting pipeline?”