In the world of Data and Analytics, a well-crafted design document isn’t just a formality. It’s the blueprint that transforms raw ideas into salable solutions. Clarity in design leads to clarity in execution.

Agile and DevOps mean no design and no documentation. Now don’t get shocked. This is one of the biggest myths I have seen in the last couple of years in quite a lot of companies and also in my discussion with Engineers.And I would like to put a rest to it by saying “Don’t make the mistake of not designing things properly before you start developing” whether it be a Data Engineering pipeline or creating a Data visualization or creating that fancy Machine learning pipeline. Every single development you do needs a design and not doing so is the biggest mistake we do in any kind of project. By design, I don’t mean spending all your time till eternity to achieve perfection in your design. The first and foremost thing you need to do is to make sure that your design and architecture are flexible, agile, and can evolve with changes as a result of learnings during implementation.

In this article I will try to share my experience as an Architect and also share what I believe are the things that we should capture in a solution design document before we take it to Architecture Review Boards for approval and deal with that pain in ass Enterprise Architect for approval (I am one of them). I will also share a markdown file as a template which you can use to create your own design document. The github repo contains a few examples of how a design document must be written.

Following are the things, in my experience, your design document should be able to answer. Now I am talking purely from my experience in the world of Data and Analytics, but I can assume that these must also apply to any other domain as well.

Why? What is the problem you want to solve?

Now, this is what should be your first point in any design document. We must try to clear this one out even before we want to jump into the actual solution. The design document should be able to clearly state why you would like to implement this design and what is the current problem you would like to solve. I am a proponent of the YAGNI (You ain’t gonna need it) design principle of software engineering but I also believe that each design should be value driven. You should have a forward-thinking approach while creating your design but on the other hand focus on the problems you are trying to solve. Don’t base your design on hypothetical scenarios.

What is your ask (when seeking advice/approval) from the review board?

Be very clear about what you expect from the members of the review board if you are presenting your design to an audience that needs to approve it. That helps keep the audience focussed and not propel them into unnecessary discussions and also provides them a clear idea of what to look for or observe in the design.

Options to solve a problem

Now every problem can be solved in many ways and no one option is right or wrong. As a solution architect, you need to ensure that you are not biased towards a solution or tool but present all the options on the table with the pros and cons of each option. What is also important, with your experience, is to clearly explain what option you recommend as the best option and why.

What, Why, and How are the three most important aspects which should be covered by any good solution design document. Apart from these three things, there are a few other things that should always be considered when you create such a document. Let’s talk about them.

Privacy and Security

Your design document should clearly highlight how the security aspects will be taken care of if it is a new design. For a change in the existing design, you must clearly specify the aspects which might get impacted from a security standpoint and how you intend to take care of them. In a nutshell, follow the principle of “Security by Design”. You must explain platform security, data security, authentication, and authorization mechanisms.

Another really important aspect that the design must cater to is the classification of the data that will be processed and how will your access management framework cater to it. Also please do extra care in case GDPR or Personally Sensitive information is part of the data. Always specify in your design the measures you have put in place to make sure that such information is not compromised and all the procedures are followed.

Availability

In production, solutions catering to critical business requirements are expected to have high uptime and availability. You must clearly design how you would take care of this requirement and what would happen if your application suffers an outage due to platform issues. For instance, what would happen if there is an outage in the data center where your database is hosted? Will there be a failover to a database in a different data center in a different region?

Resiliency — Restartability and Disaster Recovery

Trust me, over the years I have seen these two things to be the most underrated functionalities in design documents when it comes to the Data domain. Always explain how the way your application would be restarted if there is a failure and what would happen when it is restarted. Is there a checkpoint from where it has to be started or does it have to start from scratch? Also important is to explain what would happen if the database crashes or get corrupts. Will there be daily backups and how will you restore the database to the point in time where it crashed? Always make sure you consider this scenario in your design.

Scalability

Scalability is the ability of a system to handle increased. Your design must be very scalable and can respond to the increase of decrease in demand. Hence your design must consider scalability both in terms of the platform as well as an increase in data loads. For example, if your daily loads are deltas and not very large but month-end loads are a full large load you could consider autoscaling where the platform can have more computational power on month-end and automatically scale back to a lower configuration for the rest of the month. Scalability automatically impacts the operational efficiency and cost impact of your design.

Observability

It is very important to detail how your design will support observability in terms of monitoring, logging, and auditing. This part of the solution design should also detail the kind of alerts which will be configured and the corresponding actions against an alert. As part of observability, I also consider Data Quality. If your design is about loading data into a data platform you need to ensure that all critical data elements have been identified and that

Extensibility

This section is relevant mostly in designs that have to deal with data engineering or code development. For instance, if your design is about extracting data from source systems into your data platform you must highlight how would your code look like and adhere to the principles of software engineering, and will be modular and reusable.

Deployment

This section of the design document comes into play in almost all aspects whether it is Platform, Data Engineering, Data Modeling, or Data Visualization. Whatever is being built or developed has to be deployed and your design should have clear instructions on how it should be done and must take into account how automated testing, deployment, CI/CD, deployment of the database, and deployment of reports would look like.

Latency and Orchestration

As part of your solution design, it is very important to highlight the consistency of your data loads. The latency of data needed by the business users in the end product and the frequency of data loads into the platform must be clearly defined so that the development team can configure orchestration for the loads.

Performance considerations

Again, performance is one of the most underrated aspects in a design document and that bites in the end when you have deployed your solution in production and is ready for the real world. You must clearly highlight expected performance bottlenecks and ways development teams must solve them. Details about the volume of data loads, network bandwidth, partitions, and indexes as part of the design come in handy when the solution is being deployed in production.

Cost Considerations

Cost is THE MOST important aspect of any solution design document and something which I have seen in the world of Cloud not being considered. A big mistake and you should absolutely avoid it. Review boards must reject any solution design which does not have a cost-impact analysis. You must clearly highlight the cost impact of every new component being introduced into the architecture and the impact on the cost due to any proposed changes to the existing design. Consider the volume of data loads, the computation aspects, data loading frequencies, movement of data outside and inside your network, operational cost, etc. into account when you create any design. Again never ever miss this part of your design. Also would be advisable to make part of the design the ways in which costs can be optimized.

Governance

The design document must clearly highlight the ownership aspect of the platform, the application, and the data residing within the application. Clear ownership must be defined in terms of who owns the application, who owns the data, who owns the platform, and what should be the standard operating protocol when there is an issue in either the application, data, or platform. This is something that I have seen not be considered well in advance as part of the design and we end up with orphan applications in production.

Please note that with these points I don’t mean to go back to the days when we used to write 200+ pages of High level and low-level designs which were dumped somewhere on a SharePoint that no one ever cared to read. Trust me all my design knowledge of any project came from running the application and reading the code myself.

Technical Debts

Many times implementation of a solution design may lead to a situation where you are introducing technical debts either due to delivery timelines or due to a lack of functionalities at that point in time. It happens that we sometimes tend to move in a different direction than the target. Please make sure that you highlight all the technical debts in your design document and also provide a commitment and a plan to fix the technical debt.

Piece of Advice

• Don’t get into the trap of “bigger the better”. My advice is to keep the design document brief, very clear, and must have only relevant information.
• Refrain from writing designs using power points and word documents. In my experience using markdown files as part of Github, Azure Wiki, or tools like Notion and Confluence is better where the design is presented more like a blog page rather than a multi-pager boring word document or PowerPoint presentation.
• A picture speaks more than a thousand words. Always trying to express your thought in a solution design rather than lots of text. Trust me now one reads a text busy design.
• Leverage the power of versioning tools like Github and Azure Repo to maintain your designs. GitHub pages, readme files, or code wikis are really very handy where you can commit your markdown files to the repository branches and review them using pull requests. It really instills discipline within an architecture team.

Please click here to visit my GitHub repository to access a Solution design template and some examples of design documents that you can use as a reference for yourself. Please feel free to send me pull requests if you want to contribute to the design templates.