DevOps, continuous integration, and continuous deployment are often talked about like silver bullets to solve the woes of development and operations when it comes to software development. Anyone who’s tried to implement these innovations in software development can attest that it’s not magic, and it takes a lot of work. However, the benefits cannot be understated and often leave you with the question: “How do I get it?”

Unfortunately, DevOps has a relatively nebulous definition. It can be loosely described as a culture or philosophy of collaboration between teams or individuals to increase efficiency and quality. What that looks like is different from organization to organization, but will generally rely on things like agile development principles, process automation, continuous integration, and continuous deployment. Despite the amount of literature on how to implement each of these pieces, like DevOps, there is no silver bullet. At Synergex, we have been working on achieving DevOps (whether we knew it or not) since March 2018, and it’s still a work in progress. Continuous integration, the practice of making and testing incremental changes frequently, has been the driving force behind our shift to a DevOps culture, and this is some of how we’re making that transition.

Identifying impediments

In any software development team or department, there’s always a process, task, or piece of the tech stack that stands out as needing improvement. Maybe your source control model always causes a headache at release time. Or maybe final build validation takes longer than it should. Making small, incremental changes to improve pain points is preferable to large, sweeping changes, and reduces potential negative impacts to workflows. As they say: “the squeaky wheel gets the grease,” so keep an ear out and some grease handy!

For Synergex, the loudest squeak we had when we started was the UI testing for Synergy DBL Integration for Visual Studio (SDI). We had been improving the tests for years, but weren’t making enough progress fast enough. The total run time was over 24 hours, they were hard to maintain if something broke, and they broke regularly due to changes in Visual Studio and SDI. We weren’t exactly sure where we’d end up when we were done, but we knew that if we ever wanted to achieve a faster development cycle, we needed to make the tests more consistent, easier to maintain, and faster to run.

Over the course of a year, we rewrote the tests from the ground up, utilizing the test team and some members of the SDI team, and the results speak for themselves. Test run time is down to about five hours, review went from four man-days to four man-hours, and the pass rate went from about 75% to 90%. Also, test maintenance has improved drastically due to the infrastructure to execute test operations being more resilient than before (resulting in fewer tests breaking when changes occur), and instead of making the same fix in multiple tests, we now make a single change that fixes multiple tests. Review time and pass rate continue to improve (to one man-hour and 95%, respectively). Additionally, we have plans to reduce run times to an hour and execute full test runs for every bug fix and feature addition.

Interesting developments

By improving or implementing one piece of your continuous integration strategy, you may find there were additional benefits to the task that hadn’t been expected. For example, the SDI test improvement project resulted in additional development resources by utilizing test engineers as developers. It also resulted in the SDI team gaining the ability to confidently test experimental fixes for difficult problems or changes that could cause other issues. You may also discover that there’s still more to do: bits of infrastructure or process that you never thought about before now require changes.

As the SDI test improvement project progressed, we realized our virtual machine maintenance and deployment strategy was not going to keep up with what we wanted to do. Maintenance and deployment of virtual machines for testing was a manual effort that could take more than a day to complete, and the machines were bloated due to being around for years accumulating changes and updates to software, the OS, and the like. We determined that the personnel and hardware requirements for the existing maintenance and deployment process would not scale well with an increase to virtual machine count or execution frequency of the process. This spawned a project to automate virtual machine creation and deployment, rather than maintenance and deployment. After a little bit of work, we came up with a solution that creates new VMs based on a premade user-specified configuration, is infinitely scalable for system and machine counts, is easy to modify, runs unattended in just over an hour, and can run on any schedule we want.

Surprisingly, the change in virtual machine creation and deployment has been the single biggest catalyst for additional process improvement since we started making improvements with an eye to continuous integration. Because we can configure the virtual machines any way we want, Azure DevOps automation agents were added to the SDI test machine configurations to allow us to run tests through Azure DevOps rather than kicking them off manually on each virtual machine. We still needed to collect the results off the machines after the tests were done, so we figured out how to automatically publish test results to Azure DevOps and display the data on a monitor in the development department. To get full continuous integration for SDI, we still need to get it building through Azure DevOps and automatically running the tests on changes staged for a merge into the main development branch, but even in our current state, we’re far better off than we were when we started two years ago. Similar integration for other products is forthcoming, and additional automation efforts have been inspired by the example that these pieces have provided.

Continuous improvement

We obviously have a way to go before we achieve full continuous integration, but we’re already seeing a DevOps culture starting to take shape. There’s major work in the pipeline, and we’re going to continue to take it one piece at a time. At the end of the day, it’s a change in mentality about developing and delivering software. The key is accepting that processes and infrastructure could be better and making desired changes one piece at a time.