I would like to start a tradition - I am going to gather all the discussions I had last year with customers, and will craft my predictions on how DevOps and related technologies will evolve and impact business in 2021. As most of my discussions with customers are always focused around Infrastructure as Code (IaC), this is where I feel I can most accurately speculate.Â
The world of automation has changed in the past year, remote work, support of remote business, the new digital era, and pandemic constraints will force infrastructure and code automations to level up in 2021. All in all, it is clear that the direction is more automation and less manual work.
So here is a sample of what we are about to see in the upcoming year:Â
1. More code, less ClickOps
About a year ago I asked so many people one simple question: “Are you already using or planning to use IaC soon?”. The answer used to be either “we are” or “it is on our short roadmap”. Now, the answer is almost always “we are” or “sure, we started a few years ago” and only on rare occasions “it is on our short roadmap”. I even hear often “what do you think that I am crazy?? Of course we are using IaC! How else do you think I provision cloud resources??”. And when I try to understand what caused those customers to use IaC, I feel that the answer is a combination of other previous trends - the shift to microservices and the shift to cloud-native. Environments became much more complicated and the only way to provision those in scale is by using code. So we see less organizations that click buttons in the AWS (and other vendors’) consoles, and instead - writing, maintaining and executing IaC.Â
2. More IaC, less configuration management tools for provisioning
Sometimes people ask me something like “what is the difference between Ansible and Terraform?”. My answer usually involves talking about the concept of Terraform State files and that IaC like Terraform understands your current state in the cloud account - which resources are already deployed so you can have a dry run (a plan) of things that you need to add, change and destroy, before you actually apply those changes. In Ansible and other configuration management tools, it is basically a recipe (that is where Chef got its name) of things to do, in a certain order. That is why IaC is ideal for provisioning infrastructure whereas configuration management is ideal to configure the application itself (like your pom.xml file if you have a Java application) in different environments.
3. More Ansible, less Chef and Puppet
Actually I am not that sure why, but we often hear one of the following:
“I used Chef but now I use Ansible” or “I used Puppet but now I use Ansible”. We never hear the other way around. Maybe due to Ansible’s Python style syntax which is very common for DevOps engineers. Maybe due to its simplicity. But the trend is clear. From Flexera’s state of the cloud report, the numbers are clear. Maybe this is the reason that Chef was sold for only $220M to Progress, followed by layoffs. Chef used to be much bigger a few years ago. The question is where Puppet is going and what is their agenda. They need to do something soon.
4. More Terraform and Pulumi, less Cloudformation and ARM
Another question that I always ask is “which IaC framework do you use?”. The answer two years ago was “we debated between Cloudformation and Terraform and chose Cloudformation”. A year ago the answer changed to “we debated between Cloudformation and Terraform and chose Terraform”. But now, I hear “we debated between Pulumi and Terraform and chose Terraform”... By the way, obviously answers were exactly the same but with ARM instead of Cloudformation for Azure customers.
So for me it is clear that the community/engineers prefer a multi-vendor, open-source, open-approach IaC framework and currently there are 2 great options - Terraform and Pulumi - each one with its pros and cons. Terraform is more mature and has a vast ecosystem of 3rd party providers, whereas pulumi with its imperative approach is easier for developers.Â
Another important advantage of a multi-vendor open-source IaC framework is that it is easier to integrate those with another solution. One example is Terragrunt which is built on top of Terraform. Another great example is Open Policy Agent (OPA) and its Terraform Plan support for policy-as-code. And more and more such integrations are being built every day.
5. More IaC management dedicated solutions, less abuse of CI for that
As more and more companies start to use Terraform and Pulumi, they need to manage the workflows. The journey starts when some engineer executes “terraform apply” locally on his/her laptop. That is good as long as you have 1 or maybe a small and super synchronized team. As they grow, they realize this is not an option anymore.
You want to see the logs of what ran yesterday. You want to force execution of “terraform init” and make sure you use the same modules as you should. That is when you usually just move the terraform “init”, “plan” and “apply” to your Jenkins/CircleCI/GitHub actions/etc. It is much better than running it locally obviously and this is a very simple (and right) decision to do. But then you face the next challenges - you want to run “terraform plan” on pull requests. You want to be notified on changes before they occur with Slack or MS Teams.
You want an approval workflow for production but automatic deployment in staging. And then you want RBAC, Policy-as-Code (like OPA), cost visibility and governance, self-service environments and many more. We see companies building in-house solutions built on their CI and abuse it and make a lot of effort and only reach a mediocre solution.
That is why env0 and other vendors focus on that - how to manage your IaC workflows with a solution that is dedicated for that. Jenkins/other CI are good for building binaries (docker images) from code and make sure those are in good quality (running tests, coverage, etc). But they are not built for the era of IaC and its workflows. That is why a new layer is needed here, in addition to CI.Â
Summary
All of those predictions mean one thing - Moving to Code and Shifting Left: code instead of GUI, several layers of code (IaC vs Configuration Management), etc. However, shifting left means providing more power and control to developers, so in my view, the future is all about managing code in order to improve developer’s efficiency, while creating policies and guardrails for the organization.
Terraform fanatic Anton Babenko has a lot to say about Infrastructure as Code, but it all boils down to users wanting simplicity, velocity, and maintainability. When it comes to the future of IaC frameworks, he says, “The one who avoids accidental complexity will win.”
I would like to start a tradition - I am going to gather all the discussions I had last year with customers, and will craft my predictions on how DevOps and related technologies will evolve and impact business in 2021. As most of my discussions with customers are always focused around Infrastructure as Code (IaC), this is where I feel I can most accurately speculate.Â
The world of automation has changed in the past year, remote work, support of remote business, the new digital era, and pandemic constraints will force infrastructure and code automations to level up in 2021. All in all, it is clear that the direction is more automation and less manual work.
So here is a sample of what we are about to see in the upcoming year:Â
1. More code, less ClickOps
About a year ago I asked so many people one simple question: “Are you already using or planning to use IaC soon?”. The answer used to be either “we are” or “it is on our short roadmap”. Now, the answer is almost always “we are” or “sure, we started a few years ago” and only on rare occasions “it is on our short roadmap”. I even hear often “what do you think that I am crazy?? Of course we are using IaC! How else do you think I provision cloud resources??”. And when I try to understand what caused those customers to use IaC, I feel that the answer is a combination of other previous trends - the shift to microservices and the shift to cloud-native. Environments became much more complicated and the only way to provision those in scale is by using code. So we see less organizations that click buttons in the AWS (and other vendors’) consoles, and instead - writing, maintaining and executing IaC.Â
2. More IaC, less configuration management tools for provisioning
Sometimes people ask me something like “what is the difference between Ansible and Terraform?”. My answer usually involves talking about the concept of Terraform State files and that IaC like Terraform understands your current state in the cloud account - which resources are already deployed so you can have a dry run (a plan) of things that you need to add, change and destroy, before you actually apply those changes. In Ansible and other configuration management tools, it is basically a recipe (that is where Chef got its name) of things to do, in a certain order. That is why IaC is ideal for provisioning infrastructure whereas configuration management is ideal to configure the application itself (like your pom.xml file if you have a Java application) in different environments.
3. More Ansible, less Chef and Puppet
Actually I am not that sure why, but we often hear one of the following:
“I used Chef but now I use Ansible” or “I used Puppet but now I use Ansible”. We never hear the other way around. Maybe due to Ansible’s Python style syntax which is very common for DevOps engineers. Maybe due to its simplicity. But the trend is clear. From Flexera’s state of the cloud report, the numbers are clear. Maybe this is the reason that Chef was sold for only $220M to Progress, followed by layoffs. Chef used to be much bigger a few years ago. The question is where Puppet is going and what is their agenda. They need to do something soon.
4. More Terraform and Pulumi, less Cloudformation and ARM
Another question that I always ask is “which IaC framework do you use?”. The answer two years ago was “we debated between Cloudformation and Terraform and chose Cloudformation”. A year ago the answer changed to “we debated between Cloudformation and Terraform and chose Terraform”. But now, I hear “we debated between Pulumi and Terraform and chose Terraform”... By the way, obviously answers were exactly the same but with ARM instead of Cloudformation for Azure customers.
So for me it is clear that the community/engineers prefer a multi-vendor, open-source, open-approach IaC framework and currently there are 2 great options - Terraform and Pulumi - each one with its pros and cons. Terraform is more mature and has a vast ecosystem of 3rd party providers, whereas pulumi with its imperative approach is easier for developers.Â
Another important advantage of a multi-vendor open-source IaC framework is that it is easier to integrate those with another solution. One example is Terragrunt which is built on top of Terraform. Another great example is Open Policy Agent (OPA) and its Terraform Plan support for policy-as-code. And more and more such integrations are being built every day.
5. More IaC management dedicated solutions, less abuse of CI for that
As more and more companies start to use Terraform and Pulumi, they need to manage the workflows. The journey starts when some engineer executes “terraform apply” locally on his/her laptop. That is good as long as you have 1 or maybe a small and super synchronized team. As they grow, they realize this is not an option anymore.
You want to see the logs of what ran yesterday. You want to force execution of “terraform init” and make sure you use the same modules as you should. That is when you usually just move the terraform “init”, “plan” and “apply” to your Jenkins/CircleCI/GitHub actions/etc. It is much better than running it locally obviously and this is a very simple (and right) decision to do. But then you face the next challenges - you want to run “terraform plan” on pull requests. You want to be notified on changes before they occur with Slack or MS Teams.
You want an approval workflow for production but automatic deployment in staging. And then you want RBAC, Policy-as-Code (like OPA), cost visibility and governance, self-service environments and many more. We see companies building in-house solutions built on their CI and abuse it and make a lot of effort and only reach a mediocre solution.
That is why env0 and other vendors focus on that - how to manage your IaC workflows with a solution that is dedicated for that. Jenkins/other CI are good for building binaries (docker images) from code and make sure those are in good quality (running tests, coverage, etc). But they are not built for the era of IaC and its workflows. That is why a new layer is needed here, in addition to CI.Â
Summary
All of those predictions mean one thing - Moving to Code and Shifting Left: code instead of GUI, several layers of code (IaC vs Configuration Management), etc. However, shifting left means providing more power and control to developers, so in my view, the future is all about managing code in order to improve developer’s efficiency, while creating policies and guardrails for the organization.
Terraform fanatic Anton Babenko has a lot to say about Infrastructure as Code, but it all boils down to users wanting simplicity, velocity, and maintainability. When it comes to the future of IaC frameworks, he says, “The one who avoids accidental complexity will win.”
.png)
