The Nightmare Begins

At the start of the year, my team inherited three full-stack projects from a third-party software development services provider. That meant three full-stack projects complete with frontend & backend, became our responsibility.

And the person who was going to inherit them? I, because I met the technical criteria, I don’t remember being particularly happy about it as I’d dealt with this third party’s work before. Still, I accepted the challenge.

To give you a sneak peek of what their previous work looked like, here’s an example I had come across in the past:

foreach (var item in list)
{
    try
    {
        // NETWORK CALLS - UPDATING DATA
    }
    catch (Exception ex)
    {
        continue;
    }
}

Yes, network calls inside a foreach with swallowed exceptions. When I asked one of their devs about it, he just said something along the lines of:

“Honestly, I don’t remember why we did this…”

So you can imagine my expectations going in. And the reality turned out to be worse than I expected: the projects were riddled with bugs. In one case, there was a ticket that had been open maybe for half a year and they still hadn’t managed to fix it. In another, they had made a promise to the client that a certain feature could be developed when it literally isn’t possible even with AI as we know it today. (Maybe I’ll write about this in another post.)

When I finally started working on the projects, I quickly discovered the real nightmare: overengineered abstractions and code that made no sense even in the simplest places. One of my first tasks was a seemingly straightforward change request, which, of course, required touching both the frontend and backend.

Code That Makes You Go ‘Why?’

Here’s another gem I found while in the entry point of the backend:

var methods = "GET, POST, OPTIONS, PUT, DELETE".Split(',').Select(s => s.Trim()).ToArray();

Be honest, if “WHY?”, “WHAT?”, or “USELESS” weren’t the first words that came to mind, you might want to reflect a little. This was one of many examples of unnecessary overengineering scattered throughout the codebase.

If this level of unnecessary complexity existed even in something as basic as defining a list of HTTP methods, imagine what the rest of the codebase looked like.

Here’s a much simpler alternative:

var methods = new[] { "GET", "POST", "OPTIONS", "PUT", "DELETE" };

The Abstraction Problem: When DRY Goes Too Far

Abstraction is a common theme in software development. Write reusable code. Don’t repeat yourself. Think ahead. That’s what we’re always told. And most of the time, that’s good advice; abstraction can make code cleaner, easier to maintain, and more adaptable.

However here’s the catch, when taken too far, it stops helping and starts hurting. If you’re not careful, you’ll end up with code that’s technically reusable but practically impossible to read, change, or debug.

I’ve also been a victim of the abstraction rabbit hole. It usually starts with good intentions. Maybe you have two functions that look almost the same, you think:

“These are pretty similar. Let’s merge them into one function to avoid repetition.”

So you generalize. You add a parameter or two to handle the small differences. Great, now you have one function instead of two.

Then a new use case appears, slightly different again. You tweak the function to support that too. Then another comes along, and another tweak. Pretty soon, your once-simple helper is full of conditionals, flags, and edge cases. Now it’s harder to understand and harder to change than if you’d just kept two small, clear functions.

Worse, if you ever need to update just one of those original behaviors, you risk breaking the others.

This is what was done to this backend, except that they had written multiple layers of wrappers to an already well-established library, one that I was already familiar with.

Tracing a simple function from start to finish meant jumping from function to function. Despite knowing the library well I couldn’t make sense of the system, and months later I still haven’t untangled it all.

Bug and crash reports were coming in left and right, auth was breaking frequently, production server was crashing nearly every day and pressure was building up, higher ups were getting involved and fingers started pointing in every direction, and most importantly I was trying to make sense of it all, developing new features, debugging auth, stress testing the api and analyzing both backend and frontend.

This is a clear example of abstraction going wrong. The goal isn’t to write “clever” or “reusable” code, but it’s to make it understandable, maintainable, and easily changeable.

But the real disaster is yet to come.

The Real Disaster - Misusing Async & Await

The root cause for the failing of production turned out to be deadlocks.

Deadlocks are situations in computing where two or more processes cannot proceed because each is waiting for the other to release resources. This typically occurs when processes hold resources while waiting for others, making a cycle of dependencies that prevents any of them from continuing.

In our case, it was due to asynchronous code being blocked by synchronous calls like .Result or .Wait() (.NET) causing tasks to wait indefinitely for each other to finish. As a result the system struggled to handle even 100 concurrent requests in a test environment before it would freeze and crash, this is because each request ended up waiting for another, making a chain reaction that brought the server to a standstill.

I ended up analyzing the codebase and came to the conclusion that in order to fix this problem, all the synchronous functions had to be converted into asynchronous using async await, and that’s easier said than done because if Function A calls Function B, and Function B becomes asynchronous, then Function A must also become asynchronous. This builds a cascade effect that propagates all the way up the call stack, meaning the entire backend had to be rewritten.

One of the most used synchronous functions was one that fetched the current user’s data, and it was basically used in every endpoint and every function, so converting it to async/await required changing 90+ references and all of the functions all the way up the call stack.

And with one of the pain points defined, we could proceed on to the fix.

The Fix

Once the root causes were clear, the question became: who was going to fix all of this?

It was a very time-sensitive topic, the app being B2B didn’t help, emails were getting exchanged, and I was involved in the midst of all to clarify the technical problems and to bring actual proof that the provided codebase had extreme performance problems, and so I did. (I plan on writing another blog on how I identified the problems)

After reviewing the situation, the third-party team agreed to implement some immediate fixes:

1. Server-side caching for frequently used queries like the current user’s data
2. Improving the production server instance on Azure.

While these steps helped alleviate some of the immediate pain, they weren’t a true solution to the underlying problem: the backend’s synchronous code and tangled abstractions. Throwing more RAM and CPU at the server is only a patch, if the load increases the problems will still be present.

Whether we like it or not, this is how the game is played. Code breaks, deadlines loom, and responsibility…well, is nowhere to be found. Patches pile up, quick fixes hide the real problems, and somehow, all of it lands squarely on your desk. You just grit your teeth and grind, because the finish line isn’t going to cross itself.

Takeaways

In the end, surviving these nightmare projects wasn’t just about diagnosing broken code it was about navigating it while still delivering value. As I worked to understand and document the deep-rooted issues, I was also building and shipping new features on both the frontend and backend, often on top of unstable foundations. Exposing the problems clearly, helping guide the technical direction, and still pushing the product forward, built a lot of trust. It’s what ultimately earned me recognition and positioned me to take on even more complex systems in the future. Messy codebases might be undeterminable, but so is the growth that comes from untangling them while still moving forward.

Here are a few lessons and practical tips you can take from my experience:

• Readable code beats clever code every time: Abstractions should make your code easier to understand, not harder. Avoid over-engineering small things.

• Don’t be afraid to refactor: Sometimes a messy codebase can’t be patched, it needs thoughtful restructuring. Start small, but aim for clarity.

• Communicate clearly with stakeholders: Document problems, explain technical limitations, and set realistic expectations. It saves time and stress later.

• Learn from mistakes, yours and others’: Seeing what not to do can be as valuable as seeing what works. Use messy code as a learning tool.

• Celebrate small wins: Every bug fixed, every function untangled, is a step forward.