As this externally published newsletter is still relatively new, it’s worth reiterating what we’re going on about here. This newsletter is traditionally published by the Field CTO team here at HashiCorp and focuses on looking over news of the previous week(s) and giving general commentary about what we’ve observed and how this relates to parts of industry HashiCorp is concerned with. And sometimes we just riff on whatever tickles our fancy.
It’s been quite a week. We have witnessed the passing of the longest reigning monarch in history. Adobe acquired Figma for $20 Billion in what many consider a defensive move for the Goliath. Tennis legend, Roger Federer, winner of 20 Grand Slam events, announced his retirement. And maybe not quite as impactful, there’s another HashiCorp FCTO newsletter out ;-).
Developer Experience and Platforms
I’ve seen more than a few of my colleagues taking on positions building out Developer Experience (DevX) for various large enterprises. Organizations are putting a priority on engineering productivity. These teams address the challenges that come from a disparate number of tools, services and information sources - who owns that Settlement API anyway? Much of the focus of developer experience has coalesced around internal development platforms and/or platform engineering.
Platform engineering is looking like the next stage of the industry’s evolution. Like DevOps, it enables developer self-service. Like SRE, it reduces errors and increases reliability.
With all the hype around platform engineering, there’s a lot of confusion about what it is and, perhaps more importantly, how it is different from more established disciplines like site reliability engineering (SRE) and DevOps. In fact, with many SREs and DevOps professionals moving into platform engineering roles, it’s easy to mistake them all as one and the same. - https://internaldeveloperplatform.org/
Platform engineering teams strive to build a layer on top of the tech and tooling an enterprise already has in place. It helps Ops teams structure and set up and enable developer self-service. The platform team can define governance by specifying what resources start up with what environment or API request. The goal is to automate recurring tasks such as spinning up environments and resources and makes their setup easier to maintain by enforcing policies and standards. Development teams gain autonomy by gaining the ability to change configurations, deploying, spinning up fully provisioned environment and rolling them back. You can learn a lot more about how to structure a platform team in HashiCorp’s white paper “Scale Your Cloud Operating Model with a Platform Team.” While all the pieces may not be in place, you can see how HashiCorp is directionally headed toward making cloud a simple and easily managed reality for the enterprise.
Platforms and Evolution
The early days of cloud were focused on Virtual Machines (VMs) as the abstraction layer, forcing developers to manage operating systems and manage packaging and deployment of applications at a low level. The introduction of containers has allowed the operating system to be abstracted, and to focus on just packing the application and requirements into containers. Increasingly, serverless and PaaS offerings are moving that abstraction further into just focusing on the source code and dependencies, hiding containers, operating systems, and underlying deployment platforms.
Luckily, HashiCorp can help with many of these deployment targets with a consistent workflow and tools. HashiCorp Packer hast the ability to generate VMs, AMIs and containers . HashiCorp Cloud Platform (HCP) offers a Packer registry to allow you to manage and understand what is deployed where within your enterprise. It uses metadata to track images, their artifacts, their iterations, as well as their build artifacts across clouds. Get a CVE for your base image? No problem, you can just search which services are dependent upon it.
The site internaldeveloperplatform.org identifies the five major concepts of an IDP as: Application Configuration Management, Environment Management, Infrastructure Orchestration, Deployment Management and Role-Based Access Control. The HashiCorp stack can help with all of these concerns.
Application Configuration Management
Application Configuration Management enables you to manage application configuration in a similar way to how you manage and version source code. This has a significant impact on maintaining, debugging, and governing application configuration. There are several ways the HashiCorp stack can help.
Terraform Variable Sets - Terraform Cloud variable sets let you manage reusable variables in a centralized way, so you can set the variable values once and use them in multiple workspaces. When you update the values, the changes automatically propagate to all associated workspaces, making it easier to update credentials and infrastructure settings. Terraform Cloud also lets you overwrite variables defined in variable sets on a per-workspace basis, which lets you modify the workspace's configuration without affecting other workspaces that use the variable set. Follow this tutorial for a great walkthrough on how to manage variable sets.
.
Consul is not only used for service discovery but also for application configuration management. Check out Consul KV and Consul Templates for static and dynamic configuration management. There are some great tutorials on HashiCorp’s website as well as articles published on DevOps sites. Happen to be running a Spring Boot application or a Quarkus microservice? There is already integrations out there and some decent tutorials.
Secrets Management - Your pipeline needs to access a lot of important secrets in order to provision and deploy, which could lead to the potential risk of credential leaks. HashiCorp Vault enables organizations to automatically rotate passwords for existing database users, applications, and services. Easily integrate existing applications with Vault, and improve secrets management. Check out the webinar “Securing Your CI Pipeline with Vault” for additional details
Environment Management and Infrastructure Orchestration
Environment Management is defined as allowing developers to self-serve new environments on demand. This removes a lot of bottlenecks and enables faster delivery. Each new environment is provisioned as defined by the DevOps team.
From a developer perspective, Environment Management is one of the most interesting components of an Internal Developer Platform. In many existing setups, setting up a new environment typically involves waiting for someone else (most likely the DevOps team) to create and configure the new environment. This is annoying for everybody involved and also costly.
At HashiCorp, we like to think we’ve addressed this concern. With HashiCorp Terraform, provisioning and security can be automated based on infrastructure as code and policy as code. Infrastructure and policies are codified, shared, managed, and executed within a workflow that is consistent across all infrastructure. Terraform allows enterprises to spin up and destroy environments on demand reusing best practices and company standards, allowing for app dev team self-service. You can get started with a Terraform Cloud account and some getting started tutorials within minutes.
IDP defines Infrastructure Orchestration as the ability to integrate with your existing infrastructure to enable Continuous Delivery or even Continuous Deployment (CD) processes.
With Terraform Cloud Run Tasks, your teams can integrate third-party tools into the Terraform Cloud workflow between any plan and apply. By opening up the workflow this way, you can set up conditions for runs to pass in minutes, all without having to write Sentinel policies yourself. This reduces manual code review and speeds up provisioning. The ecosystem of third-party is growing pretty quickly. You can plug in security systems, cost analysis and compliance integrations today. Check out the documentation to see how to set up run task integrations via the UI or API, or get started quickly with this hands-on integration tutorial with Snyk.
Deployment Management
Deployment Management within Internal Developer Platforms (IDPs) enables teams to move to a Continuous Deployment (CD) process. It also provides a clear record of each deployment ever made which is great for audits and similar processes.
Nomad and Waypoint can help simplify Deployment Management. Nomad is a simple and flexible scheduler and orchestrator to deploy and manage containers and non-containerized applications across on-premise, the edge, and public cloud.
Waypoint is an application deployment tool that aims to deliver a Platform-as-a-Service (PaaS) experience for Kubernetes, Amazon ECS, HashiCorp Nomad, and other platforms. Waypoint 0.10 was released earlier this month, and adds several features, including the tech preview of custom pipelines, a new CLI command to tear down existing projects and their created resources. The tech preview lets users define pipelines that can run in whichever order is required to deliver an application. Custom pipelines allow for user-defined execution steps that will spawn any kind of process as part of the pipeline execution for such operations as running tests or security scanning.
Reducing Cognitive Load with a Unified Experience
Consistent documentation, easy access to tutorials and centralized resources definitely make finding out how to work in this new world a lot easier. HashiCorp has been hard at work building out a new developer portal. Opened as a public beta in June, you can take advantage of the site today to make your developer experience that much more smooth.
It’s great to see platform engineering and developer experience teams getting some attention and focus within the industry. It’s even more promising to see how many places HashiCorp can help automate, configure, integrate and reduce the amount of toil application development and DevOps teams have to endure.
Until next time,
Ray Ploski