Richard Walker

Monolithic Pipelines?

Published on 23 May 2021 4 min min read
Monolithic Pipelines?

It's fair to say that developers have a consensus on cloud-native application modernisation. They follow the twelve-factor application methodology and promote microservices, which are loosely coupled and self-contained. Communication between services takes place using a well-defined interface, an API. This approach has many benefits, and an application is the sum of many parts. The result is a resilient application. One microservice does not break an entire application, and teams can work independently, on different services, and in parallel. Changes to the implementation of a microservice get done in isolation from all the others. Moreover, the scaling of individual services equals cloud resource efficiency.

An essential aspect in today's DevOps practices includes pipelines. Build pipeline, continuous integration, continuous development, continuous delivery. Again, from a software development perspective, such pipelines are bread and butter for building, testing and deploying code.

However, the focus here is on the Ops "side" of DevOps. It is essential to point out that using the word "side" echos of poor culture and should not be thought of a being on one side or the other. That said, the role of developers and the often ambiguous role of DevOps engineering seem to imply a divide.

The point here is that when it inevitably comes to including infrastructure-as-code for provisioning cloud infrastructure and platform, things can become clunky. The most popular approach is to dive into using Terraform. Configuration management tools such as Ansible and Puppet are other popular choices. However, the point is that HashiCorp sticks to the Unix philosophy and make each program do one thing well. Terraform is an open-source infrastructure-as-code software tool, full stop. It makes rational sense to its popularity.

Let us be clear, there many permutations, solutions and approaches to building pipelines. Here, I'll keep this generic and high level, but one common theme is the tooling. Infrastructure-as-code tools make perfect sense in their own right, as does the configuration management tooling, as does the software development build and deploy. When it comes to combining all those stages in a pipeline, it risks becoming complex, fragmented and troublesome. In a pipeline definition, often, you will see a cacophony of stages. Stages, running ad-hoc shell commands, Terraform commands, Ansible commands, cloud CLI commands, Maven commands, and the list goes on.


The next logical evolution is to modularise everything. Terraform, Puppet and Ansible are all mature and cater for sensible structuring of reusable code. Terraform, and Puppet calls them modules, Ansible has roles. Whatever you name it, it comes back to the decoupling of services. One module or role gets designed to implement one thing and do it well. They all adopt lessons from coding languages, such as classes and inheritance, not repeating yourself.

So why do pipelines get so clunky? It occurred to me that while the infrastructure and platform provisioning and configuration tools are mature and work well in their own right, it's the pipeline that tries to pull everything together that makes things ugly. Pipelines risk becoming monolithic. Why are many good aspects and best practices adopted in tooling, yet one key design pattern gets overlooked?

Although not at the time, it also occurred to me that one of the most successful pipelines I helped deliver for a customer did use an approach akin to microservice software architecture. For those who are reading this and paying attention (and you might already have guessed), but I stated at the beginning that application development includes communication between services that take place using a well-defined interface, an API.

Does it not make perfect sense, then, to only have API calls in pipeline stages? I believe it does, and I've seen it work well in the real world, and I believe it's not taking this approach that leads to cumbersome pipelines that become challenging to maintain.

The best tool for implementing APIs for infrastructure and platform is, without any doubt, Ansible Tower. There is a community role for managing Terraform deployments and plans. Ansible itself is an incredible tool for provisioning and configuring. Most importantly, Ansible Tower provides the APIs to call the templates and workflows you define. It's possible to decouple specific tasks, cherry-picked and daisy chain into pipelines, crucially, in a standard way.

I've managed to get this far without mentioning containers or OpenShift, but this topic has to conclude with this theme. Any problems experienced using one cloud provider and cloud-native infrastructure only get amplified when working with multiple cloud providers. It becomes even more challenging to maintain standardisation. Implementing well-defined interface APIs is one thing. Adopting Kubernetes or an OpenShift container platform reduces the underlying cloud infrastructure's scope to manage and deliver a common platform in multi-cloud environments.

Maybe it's time to have a rethink, are your pipelines feeling monolithic? Would a well-defined API approach work better throughout all aspects of your DevOps pipelines? Would reducing the dependency and scope of cloud-specific infrastructure and services help simplify and standardise?

Ansible Tower can provide the tools for well-defined infrastructure as code APIs, and OpenShift can standardise the platform reducing cloud vendor dependencies and scope.

Required for comment verification
Back to the post list  ⋅  0 comments have been posted.

Want to read more?

You can view all my blog posts here:

View Blog