Any solution, AI or not, needs a good foundation. Data and architecture are the things you can't neglect if you want something functional, efficient, and reliable.
Architects have to think about everything. Software engineers start at data structures, DevOps at infrastructure and security, UX at the end user's experience. The architect's job is to make the foundation manageable, scalable, reliable, operationally efficient, secure, and affordable, and balance the business aspects as well as the technical ones.
A system that needs compliance, high security, encryption, and third-party integrations is complex to start with. Make it a multi-tenant SaaS product, and the complexity grows. Under that pressure we forget things, skip them, and create technical debt. Left unaddressed, that debt keeps the solution from producing the outcomes it was built for.
The pattern is familiar. Someone with a strong vision and a deadline says, "We'll clean it up after we prove it works." The demo works, and the demo becomes the product. The cleanup never happens, because now there are customers.
Here is what it looked like when I refused to make that trade.
The platform had to be multi-tenant, HIPAA-compliant, and secure enough for PHI and PII. It also had to sync with third-party systems such as EHRs, which meant long-running, failure-prone work that couldn't share a process with anything a user was waiting on.
One of the first decisions was about boundaries. I split the platform into separate services:
The models and enumerators, all services need, live in one shared package, so an entity means the same thing everywhere.
Underneath, I extracted authentication, RBAC, encryption, multi-tenant database access, connection pool management, and infrastructure integration into a family of libraries: paper-core, paper-auth, paper-db, paper-infra, and paper-email. They are published in lockstep, so one version applies to the whole family.
This is the step people skip to save time. Authentication, encryption, and tenancy are not features. If each service implements its own, you end up with slightly different security models and no way to fix them all at once. When I later changed how connections were pooled, I changed it in one library, and every service inherited the fix.
The obvious fix for opening a connection per request is a connection pool. Multi-tenancy changes the arithmetic. With a database per customer, you need a pool per tenant:
db connections = pool size × tenants served × running tasks
That multiplies quickly, and Postgres runs out of connections long before it runs out of memory.
Each tenant gets a pool of four, with tripled burst headroom that closes after use. Idle pools are disposed of after five minutes, so a task that touches every tenant once doesn't hold them all open. I didn't guess those numbers; I measured them. At burst, I held 240 persistent and 720 database connections. Requests took about 30ms, versus up to 56ms before pooling, and acquiring a pooled connection cost was around 4ms.
My current setup, for testing, is 15 tenants, 10 locations per tenant, and 6 users per location, so 900 users. With 4 pools per container instance, each tenant needs about 33 persistent and up to 146 burst-time connections. Across 15 tenants, that's 495 persistent and up to 2,190 burst-time connections. A large Postgres instance tops out around 900.
PgBouncer sits between your applications and the database and lets many of them share a small set of connections. Think of a receptionist handing out a few desks to a large team instead of giving everyone an office.
There's a trap, too. If each application also keeps a private stash of connections, it hoards the very connections PgBouncer is trying to share. So, I built one switch into the shared library that turns the private stash off, and I flip it in every service at once. Creating a new customer database is the one exception, because that operation needs a direct line to Postgres.
PgBouncer's transaction mode has trade-offs, so I handled them once, in the shared library, behind a single flag. It disables prepared-statement caching, which breaks when a transaction lands on a different server connection. It also turns off local pooling, which would hoard the connections PgBouncer is meant to share. Session state was a non-issue: tenants are separate databases, not schemas.
In the end-to-end run, peak Postgres connections dropped significantly, from 67 to 33, at a cost of about 9% in wall time, with an average wait for a server connection of 41µs and no queued clients.
The bigger gain came from changing the sizing model. Instead of counting pools per service, I calculated connections from throughput. My starting assumptions are 1 request every 5 seconds per user, 3 transactions per request, 14ms per transaction (measured after the initial pooling work):
900 users × 0.2 req/sec × 3 transactions × 0.014 sec = 7.6 ≈ 8 connections
Persistent Connections: 495 Per-Service Pools | ~8 PgBouncer+Throughput Sizing
Burst-Time Connections: 2,190 Per-Service Pools | ~24 (triple burst) PgBouncer+Throughput Sizing
Reduction: 62x persistent, ~90x burst-time
The throughput figure is an assumption, and it's easy to update as I collect more production data. But it's a far more efficient way to model the system, and it lets it scale.
Making a system "secure" isn't a single decision. It's a stack of small ones, each assuming the layer above it has failed.
One public door
Only the frontend sits behind nginx. The other services have no public route and send no CORS headers. A vulnerability in a backend service isn't reachable from the internet, because there's no path to it.
The "Least Privilege", per service and per tenant
No service ever connects as the superuser. Each service has its own database role with only the grants it needs. Each tenant has an owner role that can't log in, and a runtime role that owns nothing and can run no DDL (Data Definition Language). Only the migration role can change a schema. If a bug becomes SQL injection, the attacker gets row access, not the ability to drop a table or reach another tenant's database. The audit log is "append-only"—ledger-like—enforced at the database level via a revoked UPDATE/DELETE mechanism.
Sessions die on the next request
Every service checks the session against the control plane on every request. A logout, password reset, or deactivated account takes effect immediately, not whenever a token happens to expire.
Encryption you can rotate
Sensitive values are encrypted with a data key per account, wrapped by a KMS key. Rotation is a command, not a project: it creates new keys, re-encrypts what needs it, and reports what is encrypted under which key. Token-signing keys rotate the same way, without signing anyone out.
Developer's experience gets the least credit; all the focus goes to the end user. I built scripts for the flows that hurt:
That last one is the difference between "I think the pooling works" and "I measured it."
Teams building AI products are especially prone to this. The model is the exciting part; the demos are impressive, and the pressure to ship is intense. But the model sits on top of the same things every system needs: identity, access control, data isolation, encryption, observability, and the ability to change things safely.
Moving fast and building the foundation aren't opposites. The foundation is what lets you keep moving fast in year three. A few things I'd hold to:
So, would it work for 10,000 clients? 100,000? Decide in the first month, not the third year.
