In the past few months, our team has learned the hard way the importance of meticulously testing our infrastructure as we prepare for deployment.
This all seems trivial… at first. We have a strong foundation for our setup. We have tools to measure performance and we have tools to profile the software. But then we start testing, there are numbers everywhere, mountains of logs, and it can all become very overwhelming, very fast.
It turns out, some of the greatest challenges when testing for deployment is figuring out what you should be looking at, understanding what you are seeing, while the whole time correlating this information across the entire infrastructure. Common questions are not only focused on the software side either. Rather than just wondering whether this function is optimized, questions quickly start focusing on the infrastructure itself. Does our database have enough CPU? Does it have enough RAM? Is our data partitioned well enough?
Performance testing is a long and arduous process, but fascinating at the same time.