Test-Driven DevOps
The tortoise became the hare. The longest path is now the shortest path.
Test-Driven DevOps is for DevOps engineers and everyone else delivering software (developers, testers, and operators who code and develop infrastructure, servers, applications, clients.) It’s leaner than regular DevOps. And now with generative AI, it’s actually a lot faster right out of the gate.
Test Driven-DevOps extends Test-Driven Development (TDD) principles across the entire software delivery pipeline end-to-end, including application code, infrastructure as code (IaC), and deployment processes. While traditional DevOps focuses on automating and integrating development and operations, Test-Driven DevOps emphasizes writing tests before any changes are made, ensuring that every component — from code to infrastructure — is validated before deployment. This approach provides a more rigorous and consistent way to maintain reliability, especially in environments where automation and AI-generated code are prevalent.
By the way it might be worth mentioning that DevOps engineers can write programs other than scripts. In Test-Driven DevOps Design, tools and even platforms are designed and implemented by DevOps engineers to improve quality and velocity of the overall SDLC for the user. Also, DevOps is for everyone in the software delivery pipeline, not just DevOps engineers. One time I had to write a massively concurrent PaaS for bootstrapping applications, servers, and distributed systems. I was thinking of just using the existing solutions, but they couldn’t handle the volume. Rock-solid code (written by a DevOps engineer) came out of that, even if the architecture is perhaps more interesting than than the code.
Test-Driven DevOps replaces the silos that once existed between development and operations (including Site-Reliability-Engineering) through automated testing. I cry inside every time I find a lone DevOp or a group of DevOps engineers who are in a different world from SRE. Hyperspecialization is the enemy and generalists may very well inherit the earth. Taking a lean approach to DevOps which invites and encourages LLM / NLU based coding, integrates CI/CD with SRE and automates QA is the right thing to do for the right reasons.
When I was building out Gitstrapped at Elastic Provisioner, there were some contrarian assumptions underlying the platform. The first thing is this: code is cheap, but consistent behavior could be a lot less expensive. So we started testing the whole SDLC pipeline end-to-end using client-side tests. Back then there was selenium, now we have puppeteer, etc. Back then we had to write the tests by hand. Now we have generative AI to write the puppeteer tests and the code for application logic. It’s so easy.
Quality of reliable scalability was the sole motivator of end-to-end client-side automated testing in the 2010s when Test-Driven DevOps started. How do we know the service or distributed system can handle the massive concurrent user requests or transactions? There’s an DevOps-written app for that. And it’s a client-side test harness to prove the concept every time you integrate your changes. There was the tangible need to solve problems of unpredictable scale that you can’t solve by capacity planning since you don’t know how much capacity you may need. It meant that you needed to quickly be able to stress test the runtime environment including all infrastructure at the service level. And you had to lobby for the resources to build that quality in early on. Now the tortoise is the hare. This is now the faster way.
Test-driven DevOps design is a leaner form of DevOps than your typical heavy duty DevOps as seen at velocity conferences and maybe in Gene Kim books. (I think. To be honest, I didn’t read any DevOps books, but that doesn’t mean you shouldn’t). Version control in Lean DevOps approaches such as (Test-driven DevOps) is the implementation of Change Control.
Look how much faster you can get stuff done with Test-driven DevOps compared to just writing code or just having LLMs write code. As generative AI tools like LLMs become more prevalent, and it becomes harder and harder to run away from LLMs and silo them off from your DevOps life, they introduce a new layer of complexity in software development. Take the first-prompt fallacy for example. The first prompt you give your code-generator LLM might not be the best prompt. Even if you give your code-gen LLM the same prompt it may not give the best response first. Imagine what we could do with a client-side test harness generated first? I wrote this OpenAI API PoC in under an hour by code-generating the puppeteer behavior validation first.
While these code-copiolot and code-gen LLM tools can generate code rapidly, the consistency and reliability of the generated code are often questionable. This is where Test-Driven DevOps shines — it ensures that all code, whether human-written or AI-generated, undergoes rigorous testing throughout the entire SDLC. By applying these practices, organizations can mitigate the risks associated with AI-generated code, ensuring that it meets higher and higher quality, reliability, and now even velocity standards.
Incorporating Test-Driven DevOps methodology into AI-driven environments facilitates software and system reliability, scalability, and responsiveness to change, even as the pace of innovation accelerates. Test-driven DevOps was once tortoise, but now it’s a hare; leveraging the full potential of prompt-based LLM code-generation while maintaining the integrity and performance of the SDLC end-to-end.
