Software development Teams make many decisions while building systems. They make these decisions with the information and knowledge available to them at the time. There are some decisions that Teams realize in retrospect were not good ones, and they wish they had made a different decision. I call these decisions regrettable because not only do these decisions create problems, but they are also costly to fix. A system may need to live with these issues, preventing it from realizing its full potential. In hindsight, in these cases, it's better to take the more reliable, secure, and performant path, even if it takes more time to complete. I present eight decisions that, in the course of my career, I have seen Teams regret later.

One database schema for transactions and reports

Creating a database view or a procedure to implement a report can be a tempting solution over implementing an ETL process. Developers underestimate how much users love reports! Users will run reports more often than what the Team planned for, and they will also tend to run the more powerful reports at their disposal to get all the relevant data. As a result, reports tend to squeeze out the database server resources, starving the transactions and the applications that depend on them. Another disadvantage is the stored procedures used to implement the reports are hard to maintain.

Use separate transactional and reporting schemas. Initially, the Team can implement ETL with stored procedures and replace them with out-of-process solutions if needed. However, stored procedures can go a long way if they decouple the transactional and reporting schemas using data files. This decoupling allows the transactional and reporting schemas to run on different database servers when needed.

Merging incomplete migrations

For any software system maintained over three years, migrations are necessary to keep technical debts down. The biggest mistake while undertaking a migration is merging an incomplete migration to the main or trunk code. Migrations need to be executed in one shot - start to finish. If a migration needs to be suspended for a compelling reason, it can persist on a separate branch, ready to resume when conditions are favorable again. Otherwise, this increases the technical debt of the Team because they now have to maintain multiple technologies in the same code base. I have seen a Team grappling with four different web technologies because of this - JSP, Google Web Toolkit, Angular JS, and Angular!

Adopting proprietary web technology

Adobe Flash, Microsoft Silverlight, Java Applets, COM components - an almost endless list of proprietary web technologies that were enthusiastically adopted, resulting in massive migration costs to open standards. Unfortunately, in some cases, CTOs and Architects were let go because they recommended these technology choices. Some effects of forgoing open web standards:

• When Adobe Flash was unsupported, a 40-person Team was reduced to a 6-person Team. The Team lost clients who moved on to other products.

• A 20-person Team has been removing Silverlight for the last two years while the Product has stood functionally still and become uncompetitive.

• A 20-person Team stuck with a front-end that has four different web technologies. Any change now requires tip-toeing around these technologies or facing the prospect of multiple costly migrations.

Choose open web standards. My rule of thumb is to use a web technology only when W3C ratifies it as a standard and Mozilla Firefox supports it.

Laxity on modular architecture

At a scale of more than 30 developers, a lack of modularity will considerably slow the Team down. Some symptoms of a non-modular system -

• Unrelated modules in a single code structure
• God objects referenced across modules
• Deep code hierarchy

A monolith application is perfect at the beginning for a startup or a small product. It avoids immature optimizations while keeping the operational view simple for Teams. However, modularity is essential even in this early stage. Modularity leads to systems that are better aligned with the business concepts. Modular systems are more testable because of one-way dependencies. They also do a better job of adhering to the single responsibility principle. When the time comes for one or more modules to scale differently, have a different security model, or need to be isolated for compliance needs, it will be much simpler to separate those modules into independent services (say hello to microservices!).

Running separate concerns as threads instead of processes

Processing web requests and executing other tasks concurrently within the same process means when the process is interrupted for any reason, such as a redeployment, everything gets interrupted. Let's say the system is set to process reports on a scheduled day and time, and the process is stopped for routine maintenance simultaneously. This stoppage will mean that the system will not generate those reports, and customers or stakeholders depending on those reports will be disappointed. I know a Team that fell into this trap and ended up with many customer support tickets for failed reports. The Team had to implement a store and resume reporting functionality because it was too costly to refactor it into a reporting service.

Single tenant database

Systems may start with the premise of an on-prem or single-tenant deployment. It may be tempting to forgo the organization or customer entity in the system's design. This decision would eliminate database joins across different queries and simplify the application security. However, this decision will forever stand in the way of this Product becoming a SaaS solution. Irrespective of the deployment decision, every business has a customer concept, which must be part of the logical design.

Ignoring automation

Whether in development environments, tests, or deployments, ignoring automation results in quality and deployment issues.

When it comes to development environments, some Teams write up the steps so that any new person can follow them. Even when these instructions are kept up to date, manually executing steps can be error-prone and take weeks before someone's environment is up and running. A 40-member Team with a 5% quarterly attrition rate will have at least two people onboarding every quarter. If four weeks are needed to set up the development environment, then 32 weeks are lost in a year! Surely, this loss is a good enough justification to automate the environment setup, especially if it takes less than 32 weeks.

Similarly, we should not put off QA automation until we have a "big enough system. " Any test you need to execute on subsequent builds is a suitable candidate for automation.

The biggest difference, however, comes from deployment automation. With today's cloud deployments, which involve multiple roles, systems, networks, load balancers, queues, databases, containers, cloud functions, etc., even the simplest systems can be a challenge to repeatedly set up. Ignoring automation in favor of runbooks is a bad idea from day one.

Hub & spoke architecture

You may have read this warning elsewhere as well - "use dumb pipes and smart end-points, " "avoid the know-all ESB, " "prefer bus over broken architecture," and so on. They are all saying don't put business logic in the integration systems or layers, which is what the hub and spoke architecture does.

The first challenge is building a Team to integrate systems. If two systems need to integrate over a hub, do we create a new Team? What happens when two other systems need integration? Next, we run into prioritization issues. How does the hub Team prioritize the problems faced by integrating Teams? Then we get to testing. Integration testing is hard enough as it is; extending the chain over distributed systems makes it even harder.

I don't know of a silver bullet for this. For some cases, adopting a schema-aware data transport protocol like Avro may be sufficient and keep messages flowing to different systems where they are processed. In other cases, transactions may need to be implemented with a Saga pattern, while for the rest of the cases, adopting a service-based architecture and moving the aggregator upstream may solve the problem.

How do we avoid making these decisions?

There is no way to avoid making mistakes. That said, having more eyeballs on the options, understanding the business considerations and the tradeoffs involved, and discussing the proposal before making a decision serve most Teams well. Such decisions are captured with Architecture Decision Records (ADRs), which you may want to know more about.

If you have recently made any of the above decisions and are not fully committed to them, I suggest rethinking them as soon as possible (today!).