As a developer, I’ve spent many hours working with infrastructure-as-code tools, helping automate and manage cloud environments. Over the years, I’ve grown fond of Terraform in combination with Terragrunt. Terragrunt serves as a thin wrapper over Terraform, providing additional functionality to help automate complex workflows and encapsulate Terraform modules.

When working with Terragrunt, it does not take long to find out about the the run-all feature (see: https://terragrunt.gruntwork.io/docs/reference/cli-options/#run-all), which allows me to run multiple Terraform modules in a single call. This functionality for me has proved invaluable in CI/CD pipelines.

The Problem

While the run-all feature is a powerful tool for CI/CD, it comes with a downside. Each module that Terragrunt runs requires initialization, and unfortunately, Terraform’s built-in provider caching does not work as expected with the run-all feature (see: https://terragrunt.gruntwork.io/docs/features/provider-cache-server/#why-opentofuterraforms-built-in-provider-caching-doesnt-work).

As a result, each module initializes its providers individually, making repeated calls to download and install them. This is a waste of time, bandwidth and storage, especially if many of the Terraform modules in your project require the same providers. The time spent initializing the same providers for each module can quickly add up, particularly when running multiple modules in a single CI/CD pipeline.

This issue becomes even more apparent as your project scales. Imagine a project with a large number of Terraform modules. The time spent on downloading and initializing providers for each module could require significant resources, and at some point, you’d need bigger and bigger machines capable of handling all the initialization overhead.

To tackle this issue, Terragrunt offers an experimental provider caching feature to solve this problem (see https://terragrunt.gruntwork.io/docs/features/provider-cache-server/). When used correctly, it can provide a nice performance and efficiency improvements. Let’s try it out!

The Setup

In my environment, I’m running my CI/CD pipeline within a Kubernetes cluster. Each Terragrunt run is processed in an ephemeral pod, created via a Kubernetes Job. This approach ensures that my CI/CD runs are isolated and stateless, but it poses a challenge for caching.

The ephemeral nature of the pods means that any provider cache created during one run will be lost once the pod is terminated. While it’s possible to set up permanent hosts (via Virtual Machines, or Kubernetes deployments, or similar) to retain the cache, this is no option for me, since it does not meet certain requirements.

For the sake of simplicity, let’s assume that my CI/CD pipeline runs contain five Terraform modules. This allows me to experiment with the provider caching and its effect on performance without getting too complex.

The Implementation

Luckily, the Terragrunt Devs have already done most of the heavy lifting for us. The provider caching feature is well-documented (see: https://terragrunt.gruntwork.io/docs/features/provider-cache-server/). The implementation is simple:

# set terragrunt provider caching
TERRAGRUNT_PROVIDER_CACHE=1
TERRAGRUNT_PROVIDER_CACHE_DIR=/tmp/terragrunt-provider-cache
# terraform plan
terragrunt run-all plan
# terraform apply
terragrunt run-all apply --terragrunt-non-interactive

By default, Terragrunt will initialize the providers during the plan and apply phases. However, you can opt to initialize providers separately by using the — terragrunt-no-auto-init flag to disable automatic initialization. This allows you to perform initialization as a pre-step before running your plan or apply commands.

What happens behind the scenes is that Terragrunt starts a local provider cache server on your localhost. You can see in in the INFO Logs:

13:09:11.593 INFO Terragrunt Cache server is listening on 127.0.0.1:46203
13:09:11.596 INFO Start Terragrunt Cache server

The first time each module initializes, providers are fetched, cached, and installed:

13:09:14.292 DEBUG Fetching provider registry.terraform.io/hashicorp/random v3.6.3
13:09:15.161 INFO Cached registry.terraform.io/hashicorp/random v3.6.3 (signed by HashiCorp)
13:09:18.523 STDOUT [module-1-of-5] terraform: - Reusing previous version of hashicorp/random from the dependency lock file
13:09:18.678 STDOUT [module-1-of-5] terraform: - Installing hashicorp/random v3.6.3...
13:09:18.702 STDOUT [module-1-of-5] terraform: - Installed hashicorp/random v3.6.3 (unauthenticated)

Subsequent modules won’t need to fetch and cache the providers again, resulting in reduced steps to be taken:

13:09:21.135 STDOUT [module-2-of-5] terraform: - Installing hashicorp/random v3.6.3...
13:09:21.147 STDOUT [module-2-of-5] terraform: - Installed hashicorp/random v3.6.3 (unauthenticated)

However, they will still go through the installation process, which is (at least for me) acceptable, since it is only about milliseconds as you see on the Screenshot above.

Comparing multiple runs showed, that the Provider Caching has resulted in a ~50% reduction in the time spent for each CI/CD run. Obviously this number is strongly related to my setup. The reduction will be influenced by multiple factors like the power of your hosting machine, your network bandwith and latency, your terraform module amount and sizes and your CI/CD pipeline implementation.

Why not simply retain the cache?

As described earlier, in my setup, the cache won’t be retained beyond an individual pipeline run. There are ways to circumvent this behavior. I will explain three of them to you:

Don’t Use Ephemeral Hosts

Obviously, one option would be to abandon the idea of ephemeral hosts entirely. You could use permanent virtual machines or Kubernetes Deployments (yes its Pods will restart from time to time, you may need to live with some re-caching in these cases) that persist the cache. It will make the initial downloading of providers in each new CI/CD run obsolete. This will save you some time and resources. If you are enabled to use this approach, do it!

Create an External Provider Cache Server

Finally, you could set up a long-living external provider cache server (you find the documentation here: https://terragrunt.gruntwork.io/docs/features/provider-cache-server/#configure-the-terragrunt-cache-provider). By specifying the cache server’s host, port, and secret token, you can have Terragrunt use a persistent external server to store and retrieve provider caches. This would decouple the caching mechanism from your ephemeral infrastructure and ensure that cached providers persist across runs.

Use Network Storage

Another approach is to attach a network storage solution to the ephemeral hosts. This would persist the cache across runs, even if the host is terminated.

Using the Network Storage (Super?) Power

I wanted to try out the third option immediatelly since living in the cloud era, leveraging cloud-based storage solutions sounds like a no-brainer. Azure, AWS, and other cloud providers offer a variety of storage options that are both scalable and secure. Given this, why not take advantage of the cloud and use network storage for Terragrunt’s provider caching? In theory, this approach would allow us to persist the cache across CI/CD runs, ensuring that our providers aren’t fetched repeatedly and that we can make the most of the Terragrunt caching mechanism.

Let’s implement and test this solution. Just a few steps I’ve been through…

  1. I created a network storage solution in Azure (don’t judge me!). I opted for Premium FileStorage to enable read-write-many and to benefit from SSD performance, which would ideally offer faster read/write speeds compared to standard storage solutions.
  2. To minimize latency and ensure that my network storage was as close to my Azure Kubernetes Service (again, don’t judge me!) environment as possible, I set up a private endpoint. This connected my Cluster to the storage within a private network. The idea here was to ensure that all traffic between the AKS pods and the storage was isolated and as low-latency as possible, which would improve performance.
  3. With the static network storage in place, I created the necessary Kubernetes StorageClass, PersistentVolume, and PersistentVolumeClaim. By defining the PVC in my Kubernetes job, I ensured that each ephemeral pod could mount the storage during the CI/CD pipeline run, giving it access to the cached provider files.
  4. Finally, I configured Terragrunt’s provider cache directory (TERRAGRUNT_PROVIDER_CACHE_DIR) to point to the mounted network storage. This way, every time Terragrunt initialized a provider, it would attempt to use the cache stored in the network location.

The Performance Reality

At this point, you might be wondering: why am I not providing more code or details about the exact setup? The answer lies in the fact that replicating this approach for your own CI/CD setup might not be worth it. Despite all my efforts, I found that using network storage for caching actually performed worse than not using provider caching at all.

The reason is, that the Terraform providers still need to be downloaded from the Network Storage to the local pod. This rendered the caching useless because the time spent downloading providers was still significant, and the advantage of not having to fetch them repeatedly was nullified by the (private) network. In fact, private networks may sometimes have lower performance than public internet connections. Of course, the download server, in our case the Terraform Public Provider Registry, has to have a good performance. Which it has (Respect!).

After evaluating this storage network solution, I was not following the external provider cache server solution anymore. Maybe I will do this in the future. If I do so and the performance is great, you will find it here!

Conclusion

If you’re using Terragrunt on top of Terraform and your CI/CD pipelines contain the run-all mechanism, the Terragrunt provider caching feature is great! By caching providers, you’ll reduce bandwidth and storage usage and improve speed, even if you’re using ephemeral hosts. For larger projects with many modules, your CI/CD pipelines might not even run without the feature, because your storage could overflow. Even for smaller projects, the benefits of provider caching are undeniable. It allows your CI/CD pipeline to run more efficiently without overloading your machine or wasting resources.