Tag: quality

Your Framework Should Not Own Your Business Logic

We have a habit of calling things non-functional requirements when we don’t want to think too hard about them.

Transactions. Retries. Rollbacks. These are often delegated to frameworks through annotations, configuration, or other runtime mechanisms. We treat them as infrastructure concerns, separate from the application itself.

I think this is a mistake.

Not because frameworks are bad, or because annotations are inherently wrong. The problem is that, by using them, we can unknowingly delegate decisions that belong to our application to a framework that has no business making them.

A transaction is not just an infrastructure concern

Consider a Spring application:

@Transactional
public void processOrder(Order order) {
    inventory.reserve(order);
    payment.charge(order);
    shipping.schedule(order);
}

This looks reasonable. The method performs a business operation, and Spring takes care of the transaction.

But what does taking care of the transaction actually mean?

  • Which operations must succeed or fail together for the order to remain consistent?
  • What happens if inventory is reserved but the payment fails?
  • Can we roll back the payment? Can we release the inventory?
  • What happens if shipping has already been scheduled when a later operation fails?
  • Which failures should trigger a rollback, and which should not?

These are not merely database questions. They are business questions.

A transaction defines a consistency boundary. It determines which state changes must be treated as one logical operation and what happens when that operation cannot be completed.

Putting @Transactional on a method does not answer these questions. It delegates the execution of a transaction to Spring, while the actual consistency requirements remain the application’s responsibility.

Worse, the annotation can make us believe that we have handled consistency when we have only configured one particular mechanism for it.

A database rollback, for example, cannot undo a payment processed by an external provider. The framework cannot infer the correct business response to that situation.

The application must own the decision. The framework should only help implement it.

The same problem applies to retries

Consider another familiar annotation:

@Retryable(maxAttempts = 3)
public void processPayment(Order order) {
    paymentGateway.charge(order);
}

Three attempts. Sounds reasonable.

But should we retry every failure? What if the payment provider processed the charge but the response was lost? Could another attempt charge the customer twice? Is the operation idempotent? Should a timeout be treated differently from a declined payment?

The correct retry policy depends on the operation’s semantics, its failure modes, and the guarantees provided by the external system.

Those are application-level concerns.

A retry annotation can execute a policy, but it cannot determine whether that policy is correct for the business operation. Even when the annotation is configured carefully, the retry mechanism can obscure the actual control flow and make it harder to reason about the application’s behavior.

This does not mean that retry mechanisms should be implemented from scratch. It means that the decision to retry, the conditions under which retries are allowed, and the consequences of repeated execution should be explicit and testable application behavior.

The framework may provide the mechanism. It should not silently become the owner of the policy.

Why do we call these non-functional requirements?

I suspect the terminology contributes to the problem.

We call performance, reliability, consistency, transaction management, and retry behavior non-functional because they do not describe the primary business outcome of an operation.

But this classification can be misleading.

Some concerns that are commonly grouped under non-functional requirements have direct consequences for functional correctness.

A payment processed twice is not merely a reliability issue. An order whose inventory has been reserved but whose payment has failed is not merely a transaction-management issue. A retry that repeats an irreversible operation is not merely an infrastructure issue.

These are failures of application behavior.

Of course, not every non-functional requirement is business logic. HTTP transport, database drivers, memory consumption, and deployment configuration are genuine infrastructure or operational concerns. But we should not use the label non-functional as an excuse to remove important decisions from the application code.

If a decision changes what the application does when something succeeds, fails, or is repeated, it deserves the same scrutiny as any other business rule.

Make the application testable without its framework

The application should be testable using plain Java tests, without starting Spring, a CDI container, or a database just to verify its decisions.

Consider a payment operation that should be retried only when a temporary failure occurs. Instead of delegating the entire policy to a @Retryable annotation, we can express it through a separate, reusable retry operation:

public static <T> Either<RuntimeException, T> retry(
        Supplier<T> operation,
        Class<? extends RuntimeException> retryOn,
        int maxAttempts) {

    RuntimeException lastError =
        new IllegalArgumentException("maxAttempts must be positive");

    for (int attempt = 0; maxAttempts > 0 && attempt < maxAttempts; attempt++) {
        Try<T> result = Try.of(operation::get);

        if (result.isSuccess()) {
            return Either.right(result.get());
        }

        Throwable cause = result.getCause();

        lastError = cause instanceof RuntimeException
            ? (RuntimeException) cause
            : new RuntimeException(cause);

        if (!retryOn.isInstance(lastError)) {
            break;
        }
    }

    return Either.left(lastError);
}

The business operation remains independent of the retry mechanism:

public Either<RuntimeException, PaymentResult> charge(Order order) {
    return retry(
        () -> paymentGateway.charge(order),
        TemporaryFailure.class,
        3
    );
}

Now we can test the retry policy independently, using a plain unit test and a fake payment gateway. We can verify that a temporary failure is retried, that success stops further attempts, and that the final failure is returned.

No Spring context. No proxies. No application server.

The example is deliberately simple; a real implementation must also consider idempotency, timeouts, backoff, and which failures are safe to retry. Those decisions depend on the operation and belong in its policy, not in a generic annotation applied without considering its semantics.

The point is that we should be able to test the application’s decisions independently of the mechanism that executes them.

Testing through the execution mechanism adds complexity on both sides: tests become harder to write and maintain, while the application inherits additional runtime behavior that can introduce overhead, hidden interactions, and bugs.

That complexity has a cost: development time, maintenance effort, and, ultimately, correctness. We should pay that cost only when the mechanism provides real value, not simply because our framework makes it convenient.

The same principle applies to transaction boundaries. We should be able to test which operations must succeed together and how failures are handled, while using integration tests to verify that the real database adapter correctly implements commits and rollbacks.

Does this mean we should eliminate frameworks?

Absolutely not.

An application does not run in a vacuum. It needs an environment, and that environment provides essential capabilities.

We are not going to write our own HTTP server just to receive user requests. We are not going to write a database driver. We do not need to implement dependency injection ourselves when a suitable container already exists.

Using Spring Boot, CDI, an application server, or another runtime is perfectly reasonable. So is using framework-specific adapters to connect the application to databases, message brokers, HTTP endpoints, and external services.

The distinction is where we draw the boundary.

I want the application to depend on abstractions that express the capabilities it needs, while the runtime supplies the implementations.

For example:

  • An HTTP adapter translates incoming requests into application calls.
  • A persistence adapter translates application operations into database queries.
  • A transaction adapter implements the transaction boundaries requested by the application.
  • A retry mechanism executes a retry policy that the application has explicitly defined.

The application should not need to know how an HTTP request is received or how a database connection is managed. But it should know which operations must be consistent, which failures are recoverable, and what should happen when an operation cannot be completed.

This is not about eliminating all framework dependencies from every line of code. It is about keeping framework-specific concerns at the edges, rather than allowing them to define the application’s behavior.

Framework independence is also about avoiding unnecessary complexity

There is another reason to care about this distinction: frameworks can introduce behavior that is not immediately visible in the code.

A method annotated with @Transactional may be intercepted by a proxy. Its behavior can depend on how the object was created, how the method was invoked, whether the invocation crossed the relevant proxy boundary, and how transaction propagation and rollback rules are configured.

Self-invocation, for example, can bypass proxy-based transaction interception in common Spring configurations. A method may look transactional in the source code while its actual execution does not behave as expected.

The issue is not that Spring is defective. These mechanisms have legitimate uses, and the framework documents their constraints.

The issue is that an implicit runtime mechanism introduces another layer of behavior that developers must understand and verify.

Sometimes that layer is justified. Sometimes a small, explicit piece of application code would be easier to reason about, easier to test, and less likely to conceal a mistake.

The more important a decision is to correctness, the less comfortable I am leaving it implicit.

Portability is a consequence, not the main goal

An application whose core behavior can be tested independently of its runtime is generally easier to move to another environment.

Perhaps today we use Spring Boot and tomorrow we need an application server. Perhaps we replace a database, change a message broker, or introduce a different external service.

If our application is tightly coupled to framework-specific annotations and runtime behavior, these changes can affect far more code than they should.

If the application expresses its requirements through ordinary code and clear interfaces, we can often replace the relevant adapters without rewriting the business logic.

But portability is not the primary goal.

I am not suggesting that every application should be designed to support five different frameworks or databases. That can become abstraction for abstraction’s sake.

The primary goal is to make application behavior explicit, understandable, and independently testable. Portability follows naturally from that discipline.

A practical rule

When deciding whether something belongs in the application or in a framework, I ask a simple question:

Can I verify that this behavior is correct using ordinary tests of the application itself, without starting its runtime?

If the answer is no, I investigate why.

Sometimes the behavior genuinely belongs to infrastructure, and an integration test is the appropriate way to verify it.

But sometimes the application has delegated a decision it should own to an annotation, a proxy, a container, or a framework default.

In that case, I try to bring the decision back into explicit application code and leave the framework responsible for providing the underlying mechanism.

This is not a ban on annotations. It is not a demand for hand-written infrastructure. And it is certainly not an argument for implementing everything ourselves.

It is a way of keeping responsibility where it belongs.

Your framework should help your application do its job. It should not quietly decide what that job is.

When Software Turns Into a Swamp: The Natural Evolution of Complex Systems

Why software systems become harder to change over time — and why that doesn’t necessarily mean they were badly built.

A software system can work perfectly well for years and still become increasingly difficult to maintain, extend, and understand.

Adding a seemingly simple feature starts requiring changes across multiple modules. Developers hesitate to touch certain parts of the codebase because the consequences are difficult to predict. A change that once took a day now takes a week, and every release seems to carry a little more risk than the previous one.

Eventually, someone looks at the system and asks the obvious question:

How did this become such a mess? Was it badly designed from the beginning?

Sometimes, the answer is yes. But often, the reality is more interesting — and more difficult to address.

Software doesn’t evolve in a vacuum

When a system is first built, its scope is relatively well understood. The business has a set of requirements, the team designs an architecture around them, and the resulting software reflects a particular understanding of how the business operates.

But businesses don’t stand still.

Users change their expectations. New revenue streams emerge. Regulations introduce new constraints. Integrations with external systems become necessary. Priorities shift, deadlines tighten, and features that once seemed unimportant suddenly become critical.

The software must adapt.

Meanwhile, the original architecture was designed around yesterday’s understanding of the problem. New requirements don’t always fit naturally into existing abstractions, so developers introduce exceptions, add new dependencies, or extend components beyond their original responsibilities.

Individually, these decisions can be entirely reasonable. The alternative might be delaying an important feature, breaking existing behavior, or spending months redesigning a system that the business needs today.

The difficulty is that every decision changes the environment in which future decisions will be made.

Software architecture is not a blueprint that can simply be followed forever. It is a set of assumptions about a business that keeps changing.

The cost of many reasonable decisions

There is another factor: software is rarely built and maintained by one person, following one consistent vision, for its entire lifetime.

Developers join and leave. Teams reorganize. Different people make different trade-offs, often with limited knowledge of the decisions made before them. Documentation becomes outdated, and some of the reasoning behind the original architecture disappears.

Even when everyone involved is competent, maintaining architectural consistency becomes increasingly difficult.

Consider a component originally designed to handle one specific responsibility. Over several years, it acquires additional responsibilities because it is already the most convenient place to implement related functionality.

Other components begin depending on it. Eventually, changing it requires understanding interactions that were never part of its original design.

Nobody necessarily made one catastrophic mistake. Instead, a series of locally sensible decisions gradually produced a globally complicated system.

This is one of the most misleading aspects of software maintenance: a system can become difficult to work with without having been obviously badly engineered at any particular point in its history.

Of course, poor architectural decisions, insufficient testing, excessive coupling, and inadequate technical discipline can accelerate this process. Complexity is not an excuse for bad engineering. But neither is complexity, by itself, proof of it.

Enter the swamp

Imagine walking through a field.

At first, the ground is firm. You know where you’re going, and every step takes you forward. You can change direction easily because the terrain is predictable.

Now imagine that the field gradually becomes a swamp.

The destination hasn’t changed, but moving toward it becomes harder. Each step requires more effort. You need to understand the ground before putting your weight on it. A route that worked yesterday may no longer be the safest one today.

Mature software systems can develop the same characteristics.

The code still runs. The business still depends on it. New features are still being delivered. But the effort required to make each change increases because the relationships between components have become harder to understand and predict.

The problem is no longer just the complexity of individual modules. It is the complexity of their interactions.

A small change can affect an unexpected part of the system. A component cannot easily be replaced because too many other components depend on its behavior. A seemingly isolated improvement requires coordinating changes across multiple teams or services.

The system becomes a swamp: progress is still possible, but increasingly expensive.

And the larger the system becomes, the more costly it is to ignore the problem.

Why adding more architecture doesn’t always help

When a system becomes difficult to maintain, the instinctive response is often to introduce a new architectural pattern, split components into services, or begin planning a complete rewrite.

Sometimes these are the right decisions. Sometimes they make the situation worse.

Microservices, for example, can help isolate responsibilities and allow teams to evolve components independently. But splitting a system into more services also introduces communication overhead, distributed failure modes, deployment complexity, and additional operational work.

If several services are tightly coupled, change together, and exist mainly to call one another, distributing them across a network may simply make the original complexity more expensive.

Conversely, a monolithic application is not inherently a problem. A well-structured monolith can be easier to understand, test, deploy, and modify than a collection of poorly separated services.

The goal is not to adopt a particular architecture. It is to find an architecture that matches the system’s current responsibilities, constraints, and business needs.

That match can change over time.

Realigning the architecture

If a system naturally accumulates complexity as the business evolves, the answer is not to expect its original architecture to remain appropriate forever.

The answer is to periodically realign the architecture with the system as it actually exists.

This doesn’t necessarily mean rebuilding everything. Often, targeted interventions can produce significant improvements.

1. Isolate the parts that are difficult to change.

Some components become deeply embedded in the business and expensive to modify safely. Instead of repeatedly spreading their complexity throughout the codebase, we can isolate them behind a well-defined interface or communication layer.

The rest of the system interacts with that interface rather than depending directly on the component’s internal implementation.

This creates room to improve, replace, or eventually retire the component without requiring the entire system to change with it.

The component may remain complex internally. The important difference is that its complexity becomes contained rather than contagious.

2. Reconsider service boundaries.

A system that has grown into dozens of microservices may no longer benefit from having so many independently deployed units.

If several services have strongly overlapping responsibilities, change together frequently, or require constant coordination, consolidating them into a larger, more cohesive service can reduce network communication, operational overhead, and the cognitive burden of making changes.

This is not a step backward. It is a correction to an architecture that may no longer reflect the system’s actual needs.

3. Restore clear boundaries and responsibilities.

Over time, components often accumulate responsibilities that do not naturally belong together. Carefully separating these responsibilities, reducing unnecessary dependencies, and establishing explicit contracts can make the system easier to reason about.

The objective is not architectural purity. It is to make future changes more predictable and less expensive.

4. Treat architectural improvement as ongoing work.

Architectural realignment is most effective when it is not postponed until the system becomes almost impossible to change.

Regularly identifying the parts of the system where complexity is growing fastest allows teams to make focused improvements before the cost of intervention becomes prohibitive.

This requires engineering judgment, an understanding of business priorities, and the willingness to invest in changes that may not deliver an immediate user-facing feature.

It also requires recognizing that keeping software operational and keeping it easy to evolve are two different achievements.

The goal is not to eliminate complexity

A mature business is inherently more complex than it was when it started. Its software will often reflect that reality.

Trying to force a growing business into an architecture designed for its earliest stage can be just as damaging as allowing the system to evolve without any architectural direction.

Some complexity is essential. Some is the accumulated cost of past decisions. Some comes from requirements that no longer exist but whose implementation remains. And some comes from architectural boundaries that no longer make sense.

The challenge is distinguishing between them.

There is no universal architecture that remains optimal for every system throughout its lifetime. There is no single refactoring technique that can solve every maintenance problem, and no rewrite that guarantees a better outcome.

What matters is understanding where the system is becoming difficult to change, why that difficulty exists, and which architectural adjustments will provide the greatest benefit.

Keep the system moving

Software becoming more complex over time is not, in itself, a failure. It is a natural consequence of software serving a business that continues to evolve.

But complexity left unattended accumulates interest.

Eventually, the system becomes a swamp in which every step requires more effort, more caution, and more resources than before.

The answer is not necessarily to drain the entire swamp and start over. It may be enough to build a few solid paths, isolate the unstable ground, and redirect traffic around the areas that have become too difficult to navigate.

In software terms, that means making deliberate architectural adjustments: introducing boundaries where they are needed, removing them where they no longer help, and reshaping the system to reflect its current reality.

Good software engineering is not about building a system that never becomes complex. It is about preventing that complexity from making the system unnecessarily difficult to change.

At bitGloss, we work with software systems that have evolved over years of real business use. Our focus is not on replacing everything simply because it has become complicated, but on identifying the sources of friction and finding practical ways to make the system easier to maintain, faster to evolve, and better aligned with the business it serves.

Is Your Tech Stack Appropriate or Are You Just Trying to Be Fancy?

When a software project starts struggling, the first instinct is often to blame the code.

But in many cases, the real problem sits one level higher: the architecture and technology choices themselves.

We’ve seen systems built around microservices long before they had enough scale to justify them. We’ve seen multiple frontend frameworks living in the same application. We’ve seen message queues, event buses, service meshes, and orchestration layers solving problems that didn’t actually exist.

The result is predictable: development slows down, debugging becomes harder, onboarding takes longer, and every new feature costs more than it should.

The challenge is that it’s difficult to evaluate an architecture from diagrams and presentations alone.

We Don’t Start With Opinions

Before we assess your technology stack, we need to understand it.

That’s why we don’t begin with a slide deck full of recommendations. We start by working inside your system.

We fix bugs. We implement features. We deal with the same constraints your team deals with every day.

This hands-on involvement gives us a realistic understanding of how your application behaves in production, how your development process works, and where complexity is helping versus hurting.

Only after we’ve worked within your codebase do we form conclusions.

The Questions We Try to Answer

Is your Spring microservices architecture genuinely supporting your scale and organizational needs?

Or would a simpler modular monolith allow your team to move faster?

Are those message queues providing real business value?

Or are they introducing operational complexity that could be replaced by straightforward API communication?

Do you actually need multiple frontend frameworks?

Or are they creating fragmentation, duplicated knowledge, and inconsistent user experiences?

Most importantly:

What complexity is essential, and what complexity is self-inflicted?

An Honest Technical Assessment

After gaining real experience with your system, we’ll give you a straightforward assessment.

If the current architecture makes sense, we’ll tell you.

If parts of it are over-engineered, we’ll explain exactly why.

We’ll identify components that can be consolidated, technologies that can be removed, and areas where complexity can be reduced without sacrificing business capabilities.

Our recommendations are practical and grounded in experience with your codebase—not architectural fashion trends.

Simplicity Is Often a Competitive Advantage

Technology teams rarely suffer because they have too little complexity.

More often, they suffer because complexity accumulates faster than business value.

Every service, framework, integration point, and infrastructure component carries a maintenance cost. Over time, those costs compound.

The best architecture is rarely the most sophisticated one.

It’s the one that solves today’s problems while remaining easy to understand, maintain, and evolve tomorrow.

That’s what we help our clients achieve.

Not more technology.

The right amount of technology.

Every architecture decision has a cost.

If you’d like an honest assessment based on real hands-on work – not diagrams – we’d love to hear about your project.

bitgloss.ro

How We Stopped a Legacy JBoss System from Slowing Down Delivery

At bitGloss, we often work with systems that businesses still depend on heavily, but that very few teams still fully understand.

One common pattern we encounter is enterprise infrastructure that was designed for a very different era of software engineering: large application servers, highly abstracted deployment models, and operational complexity that slowly becomes impossible to justify.

Recently, we helped a client modernize an aging distributed JBoss environment running on AWS.

The Problem: Enterprise Complexity Without Enterprise Benefits

The client’s platform relied on a clustered JBoss architecture that had gradually turned into a maintenance bottleneck.

The environment:

  • Complex XML-based server configuration
  • Custom classloader hierarchies
  • Specialized deployment descriptors
  • Distributed session replication setup
  • Cluster coordination management
  • JBoss-specific operational knowledge

In theory, these application server capabilities offered flexibility and scalability. In practice, most of the advanced enterprise features were barely used.

What remained was mostly operational overhead.

Deployments were slow and fragile. Memory usage was unnecessarily high. Troubleshooting required specialized knowledge that fewer engineers still possess today. Even simple upgrades became risky because the platform depended on behaviors tightly coupled to the JBoss runtime.

The infrastructure had effectively become legacy middleware.

The Real Cost of Legacy Middleware

This is a pattern we see frequently in older enterprise systems.

The issue usually isn’t that the application itself is obsolete. Often, the business logic is still valuable and stable.

The real problem is the operational surface area surrounding it.

Older enterprise stacks tend to accumulate:

  • Vendor-specific deployment models
  • Overengineered runtime abstractions
  • Fragile configuration layers
  • Tooling that fewer engineers understand each year
  • Infrastructure dependencies that slow down delivery

Over time, the organization ends up spending more effort maintaining the platform than improving the product.

The Migration Strategy

Rather than attempting a risky full rewrite, we focused on reducing infrastructure complexity while preserving application behavior.

We migrated the platform from JBoss to lightweight Tomcat instances.

The migration included:

  • Refactoring deployment descriptors to standard servlet specifications
  • Eliminating complex EAR packaging
  • Simplifying deployments to standard WAR artifacts
  • Removing unnecessary application server dependencies
  • Reducing runtime-specific configuration requirements

The goal was not simply “moving servers.”

The goal was operational simplification.

The Outcome

The results were immediate and measurable:

  • Deployment times reduced by 75%
  • Memory footprint reduced by 40%
  • Operational troubleshooting became dramatically simpler
  • Infrastructure became easier to automate and maintain
  • The client was no longer dependent on increasingly rare JBoss expertise

Most importantly, the operations team could stop fighting middleware and focus again on delivering business value.

Legacy Modernization Does Not Always Mean Rewriting Everything

Many organizations assume modernization requires rebuilding entire systems from scratch.

In reality, some of the highest-impact improvements come from simplifying the infrastructure layers around existing applications.

Replacing heavyweight legacy middleware with lightweight, standardized components can significantly reduce operational risk, infrastructure cost, and maintenance burden — without disrupting core business functionality.

That is often where the fastest return on investment exists.

At bitGloss, we specialize in exactly these kinds of legacy modernization projects: untangling complex systems, reducing operational friction, and helping businesses regain control over infrastructure that has become difficult to evolve.

Is Your Website Going Down Right When You Need It Most?

There are few things more damaging to a business than an application that becomes unresponsive exactly when traffic peaks. Not during quiet hours, not in the middle of the night — but during a product launch, a marketing campaign, or the busiest trading period of the year. The timing feels cruel, but it’s not a coincidence. Peak load is precisely when hidden architectural problems surface.

This is the story of one such system, and what was actually causing it to fail.

The architecture that looked fine on paper

The infrastructure in question was built on Azure and, by most measures, looked solid. Redis was in place as a caching layer to reduce database pressure. The database itself was properly indexed. Servers had comfortable headroom on both CPU and memory — no resource exhaustion, no obvious bottlenecks. On a quiet afternoon, everything worked beautifully. Response times sat around 200ms, well within acceptable range.

Then peak traffic would arrive. Response times would climb from 200ms to 2 seconds, then 5, then 10. Eventually the application would stop responding altogether. And then, as traffic subsided, it would recover on its own — as if nothing had happened.

This pattern is particularly disorienting for engineering teams. The system heals itself, so there’s no crash to investigate, no error log with a clear smoking gun. Just a recurring window of failure that’s hard to reproduce and even harder to explain.

Why the obvious suspects weren’t guilty

The natural instinct when an application slows under load is to look at the most visible resources: CPU, memory, database query times. All of them were fine. Redis, the caching layer specifically designed to handle this kind of load, was responding in microseconds. The database wasn’t under unusual pressure. The servers weren’t breaking a sweat.

This is where many investigations stall. If the database is fine, Redis is fine, and the servers have headroom — what’s left?

The answer was in a place most teams don’t think to look: thread pool metrics and connection pool utilization.

The actual problem: threads waiting for nothing

Modern web applications handle concurrent requests using thread pools — a fixed set of worker threads that process incoming requests. When a request comes in, it gets assigned to an available thread. If all threads are busy, the request waits in a queue.

The application was running with default thread pool settings. Those defaults are reasonable for low-to-moderate traffic, but they set a relatively low ceiling on how many threads are available at any given moment. Under normal load, there were always enough threads to go around. Under peak load, every thread was occupied — not doing heavy work, but waiting. Waiting to make a call to Redis. Waiting for a Redis response that would arrive in microseconds.

Here’s the paradox: Redis was fast. The problem wasn’t Redis performance. The problem was that the threads making Redis calls were blocking while they waited, even for those microseconds, and there weren’t enough of them to keep up with the incoming request volume. Requests piled up in the queue. Response times climbed. Eventually the queue filled and the application became unresponsive.

The infrastructure was fine. The bottleneck was a configuration default that nobody had revisited since the system was first deployed.

What the fix looked like

The solution involved three targeted changes, none of which required re-architecting the system.

Thread pool reconfiguration. We analyzed the expected concurrent load and pre-allocated a sufficient number of worker threads to handle peak traffic without queuing. This meant the application could process many more simultaneous requests without threads blocking each other.

Proper connection pooling for Redis. Related to the thread problem was how connections to Redis were being managed. Without a proper connection pool, the application was creating and tearing down Redis connections more frequently than necessary, adding latency and overhead to every cache interaction. A well-configured connection pool meant connections were reused efficiently, and Redis calls became as fast as they should have been all along.

Monitoring for thread pool utilization. Perhaps as importantly as fixing the immediate problem, we added visibility into thread pool metrics going forward. CPU and memory graphs are standard in most monitoring setups. Thread pool saturation almost never is — which is exactly why this problem had gone undetected for so long. If thread pool utilization starts climbing toward its ceiling, the team now knows before users feel it.

The results

Response times stabilized at under 300ms even during peak traffic periods. The infrastructure was able to handle five times the previous concurrent load without degradation. The underlying hardware, the database, and Redis itself didn’t change. Only the configuration did.

What this means for your team

If your application behaves well under normal conditions but degrades or fails under peak load, the problem is almost certainly not the thing you’re measuring most. CPU and memory are easy to monitor, so teams watch them closely. Thread pools, connection pools, and queue depths are harder to instrument, so they go unmonitored — and that’s precisely where these failure modes hide.

Before reaching for more servers or a bigger cache layer, it’s worth asking: do we actually know what’s happening inside our application at the thread level during peak load? In most cases, the answer is no. And in many cases, that’s where the answer is.

Infrastructure problems are rarely about infrastructure. They’re about configuration, visibility, and knowing where to look.

At Bitgloss, we help engineering teams find the real cause of performance failures — not just the obvious suspects. If your application is struggling under load, get in touch.

Is Your Database Slow? Probably Not.

Nine times out of ten, when an engineering team starts complaining about “slow database performance,” the database itself is perfectly fine. The real culprit is almost always hiding somewhere else — in the way queries are written, in how the application accesses data, or in decisions that made sense at the time but were never revisited as the system grew.

This distinction matters more than it might seem. If you assume the database is the problem, you start looking at hardware upgrades, migration to a “faster” database engine, or expensive infrastructure changes. All of that costs time and money — and probably none of it will fix the actual issue.

The symptom everyone misreads

A slow application feels like a slow database. A page that takes five seconds to load, a report that times out, an API endpoint that keeps failing under load — these all point fingers at the database layer. But the database is usually just doing exactly what it was asked to do. The problem is what it was asked to do.

We’ve seen queries that took literal minutes to complete, not because the server was underpowered, but because they were missing proper indexes, performing full table scans on millions of rows, or doing complex joins in the wrong order. In one real case, a reporting query was joining five tables with no indexes on any of the join columns — and then passing the entire result set back to the application to be filtered in code. The database was faithfully returning millions of rows that would eventually be narrowed down to a few hundred. It was working extremely hard to accomplish something that should have been trivial.

What’s actually going wrong

There are a handful of patterns that cause the vast majority of database performance problems:

Missing indexes on join and filter columns. When a query runs a WHERE clause or a JOIN on a column with no index, the database has no choice but to scan the entire table to find matching rows. On a table with a few thousand rows, this is fast enough that nobody notices. On a table with a few million rows, it becomes a bottleneck that grows worse every week as data accumulates.

Filtering in application code instead of in the database. This happens when developers pull a large dataset and then loop through it in the application to find what they need. The database does more work than necessary, more data travels across the network, and the application spends time on logic the database could handle in a fraction of a second.

N+1 query patterns. This is one of the most common and most damaging patterns, especially in applications that use ORMs. Instead of fetching related data in a single query, the application fires one query to get a list of records, then fires a separate query for each record to get its related data. Fetch a list of 200 orders? That’s 201 database queries. At scale, this silently destroys performance.

Joins ordered by convenience rather than efficiency. The order in which tables are joined can dramatically affect how much work the database needs to do. Starting a join with a large, unfiltered table and narrowing down later means the database is carrying a heavy load through most of the operation. Reordering joins so that the most selective conditions are applied early can cut execution time significantly.

What a proper fix looks like

When we analyzed the execution plans on the slow queries described above, the path forward became clear quickly. Execution plans show you exactly how the database is processing a query — which indexes it’s using (or not using), how many rows it’s scanning at each step, and where the time is actually being spent. Most teams never look at them.

The changes we made were not dramatic. We added indexes strategically on the columns used in WHERE clauses and JOIN conditions. We rewrote the join order to match the actual selectivity of each table. We moved filtering logic from application code into proper WHERE clauses. And we replaced N+1 patterns with proper JOINs or batch fetches.

The results were. Queries that previously took two to three minutes now complete in under 100 milliseconds. The database hardware didn’t change. The database engine didn’t change. Only the way the queries were written changed.

What this means for your team

If your application is feeling sluggish and the database is getting the blame, the first step is not to panic and not to reach for the infrastructure budget. The first step is to look at what your application is actually asking the database to do.

Pull the slowest queries from your logs. Look at their execution plans. Check whether the columns you’re filtering and joining on have indexes. Look for N+1 patterns in your ORM queries. These are not exotic problems — they are extremely common, and they are fixable without rewriting your application or migrating to a new database.

The database is rarely the bottleneck. It’s just being asked to do things inefficiently.

At bitGloss, we help engineering teams diagnose and fix exactly these kinds of problems — turning slow, expensive queries into fast, predictable ones without unnecessary infrastructure changes. If your application is struggling with database performance, get in touch.

Quality in software development

What does this truly mean? Somebody actually told me, during a business talk, that quality was just a buzzword. It felt wrong, but in a way it felt real too. I now understand where he was coming from and I can explain it. The only way he experienced quality in software development, was through the word itself: quality. The context around it was usually messy (code, defects, unhappy clients etc.). No wonder he started hating the word and sarcastically referring to it as a buzzword.

So back to the question: what does quality in software development really mean? There is more than one point of view that we should explore, so here we go:

The big picture

Ultimately, everyone wants to get what they ask for, without any hassle and within their budget. The holy grail is getting the best thing, for free, right away. Don’t try to fight this. It’s true. Think of the best version of anything you want and tell me you wouldn’t like it materialising in front of you, right now. So from this point of view, quantification of quality is done on the scale of “this is not what I wanted and it’s too expensive and it takes too long to get” to “this is exactly what I wanted and it’s free and I got it“.

Of course, there are some variations, one of which is heavily encountered in the industry: “this is not quite what I wanted, but it’s cheap, yet it took too long to get“. These are the murky waters many clients jump into when having to do trade-offs. In my experience, these are usually budget trade-offs, that end in an unfortunate state, where over-budget gets spent in the same bucket, but for patch work.

This is the scenario I like best:

Client: This is my problem. Can you solve it?

Provider: Yes.

Client: What will it cost me, in time and money?

Provider: I estimate around 6 months and 500.000$

Client: Great! Let’s do it!

(every couple of weeks)

Provider: Is this what you had in mind?

Client: Almost. Here, I’d like a little more blue.

Provider: Done!

(around 6 months later)

Client: It looks like we’re done. Great job! Thanks!

Provider: Anytime.

There are lots of details behind this scenario, but this is the big picture. When I read it, it gives me a peaceful feeling of “yes, this is how business should be“.

Processes

These are blueprints of behaviour that help when dealing with certain problem patterns. They also help the entities that participate, have some (illusion of) predictability of the future. It’s a sort of guarantee that the future will happen in a certain way. This helps people feel in control and reduces the level of stress. BUT! Since we derive a level of comfort from having the illusion of a certain future, we tend to equate the process with the future itself. This is a mistake that leads to process rigidity and reluctance to step outside its bounds.

I like to look at processes as checkpoints and guidelines along the road. The main focus should always be the product. Engaging my expert skills right now, while actively building it, IS the best guarantee I can have of the future I want. The future is built from a constant stream of present 🙂

I always look at the reason for the existence of the process, so it gives me a better understanding of why it was set up in the first place. This way I can work along the lines of that reason, rather than blindly following a process I cannot understand (and sometimes come to hate).

One thing I noticed is that the fuzzier the goal, the more processes there are. The more processes there are, the more people tend to make a goal out of them. And so the wheel spins…

Craft

This is, surprisingly, not the most looked at factor in the software development industry. In my experience, if a client sees processes and pays an amount that suits her, THE CODE (which is ultimately the product) doesn’t matter as much, as long as it… exists.

WHAT??!! This is outrageous! But wait… is this something the client should even be aware of? Why would it be so? If I buy a pair of shoes, I’m not interested in the way they are built, what I’m really interested in is how they look, how they feel and how long they last.

The craft IS really important, inside the guild. Because we’re talking about code, things should be simple, really: it should work as required (without maintenance, preferably) and it should be easy to modify (if needed).

These are two seemingly straightforward characteristics, but to achieve them, takes not only time and experience, but continuous learning too. A craftsman should be able to easily understand the request (even help with it) and pick the most appropriate tools for the job. A craftsman should be able to read code to business people so that they hear a narrative. A craftsman should be able to write such code.

I want my client to understand where their product stands at any time and what it can do. I do NOT want my client to be forced to listen to what kind of refactoring and how many unit tests I wrote yesterday. Or even worse, she shouldn’t be forced to decide if I need to write them tomorrow.

Also, I don’t want to hear that “we are thinkers, don’t force us to estimate“. Well, if you’re thinkers, then think about how long it takes you to do something you should already know how to do. Nobody said estimates are deadlines (did you?!!), but clients need to have a rough idea about resources they need to allocate. Your GPS doesn’t say “you know what… we’ll get there when we get there! The important thing is the magic I constantly use to display these awesome maps.“

Trust

I don’t want to be cheesy and say that trust is earned, because of course it is. The difficult thing is to trust someone I don’t know. Looking at references and traces of past work helps, yes, but I also place a huge weight on the quality of the first discussion. It always served me well in assessing my future collaboration with anyone. Everyone is an intuitive psychologist by nature (not mentioning special health conditions).

The thing is, we almost always know (trained people are an exception, hence the “almost”) when someone is hiding important details in a conversation and we can also “feel” when what they say is what they think.

Obviously, the higher the level of trust, the higher the expected product quality. I’ll stop here and save the psychology magic for another time 🙂

Conclusion

These are my most valuable points of view, when looking at quality in software development. I also look for all of the above in business partners and also, I’m happy to say that for us, at bitGloss, quality is not just a buzzword.