HashiCorp FCTO Weekly Supplement: Wait, Identity Brokering for Machines? But Why?

When dealing with Secrets Management, one of the most commonly asked questions I get is: What is the difference between Hashicorp Vault and a Key store, something like AWS Secrets Manager or Azure Key Vault, or something cloud agnostic, when it comes to static Secrets Management?

More alike

Static key stores generally allow secure secrets storage and allow for API based retrieval. All of the major Cloud Service Providers provide their own flavor of Key storage and there are several agnostic solutions on the market. In the case of CSP key vaults, authentication is commonly based on the native identity provided by the cloud, such as a Machine Identity in Azure or an IAM role in AWS. 

On the other hand, cloud agnostic key stores frequently leverage a predefined set of credentials, or challenges, including IP bindings, to authenticate and frequently authorize a user in a very transactional manner. 

Or different

While Vault certainly has the capability to act as a secrets manager and manage one to one relationships, the major differentiator for Vault is the ability to broker identity. The advantage of an identity broker is to ensure that a native identity can be leveraged across all environments, whether on premise or across multiple clouds, which doesn't rely on a specific IP or establishing an initial challenge.

By leveraging identity brokerage, we have more flexibility, secure authentication on all levels of our application stack, and better observability across systems. Because brokering allows a many-to-many relationship, the operational complexity is reduced as a result of managing fewer one-to-one relationships, resulting in added value due to more focus on declarative, relational authentication and authorization.

What is Identity Brokerage, anyway?

Brokerage is a fairly common concept in the business world. Commonly, individuals may use a financial broker in order to access a portfolio of financial assets or health insurance claims will be adjusted and aggregated through a broker on behalf of an insurer so the insurer can consolidate payments. The concept of identity brokerage follows a similar pattern. 

The most familiar use of identity brokerage is Single Sign On (SSO) which gives human users the possibility of signing on to a single point which then federates that access across multiple systems for authentication. By leveraging SSO, a user can authenticate to a single point which then propagates that identity across multiple systems ensuring proper authentication and authorization are given to the end user, much like an individual approaching a financial broker with a request for a diversified portfolio of products. 

Frequently we overlook that there is also a necessity to broker machine identities, in the same manner we use SSO for human identity. As microservices continue to drive architectural design, the requirement for applications to access multiple resources is more common. For example, an application that needs access to access a database, SSH, and DataDog, can leverage an identity broker to ensure these resources can be managed by a single authentication mechanism. In the case of shared assets, a centralized solution becomes more critical for observability and control in order to monitor authenticated resources.

Is Identity Brokerage for a single Cloud worth it?

The move to Cloud is complicated and realistically rarely results in a single cloud strategy; this could be due to strategy change, regulation, difficulty migrating from legacy infrastructure, or even acquisition. Additionally, microservice architectures are becoming more ubiquitous and as a result, shared resources. Credentials shared across email or messaging services happen far more frequently than I would care to witness, which in addition to representing a potential credential leak, circulating credentials represent a real risk around credential sprawl. 

Not only are credentials unmanaged and uncontrolled in the case of sprawl, but also present an availability risk, particularly as we adopt more complex architectures where there is a higher possibility of shared services. As a result, should Jane, a developer, rotate the password for the database credentials to Application A, without notifying Application B, Application B may suffer an outage as a result. Conversely, if we broker the permissions to the database based on identity, we can rotate the database password without any concern for degradation of service.

Still not convinced?

Currently, over 50% of organizations are using some flavor of Kubernetes, which doesn't include the companies using other types of containerization, like Rancher or Nomad. Despite being "cloud native", the usage of cloud managed Kubernetes poses its own challenges regarding identity management. 

If we look at managed Kubernetes, for example, there is a vast difference of maturity between cloud vendors at the time of writing this. In AWS it is possible to manage clusters with native AWS IAM, but as with all of the major cloud providers, identity remains at the cluster level, making identity management on the pod level almost impossible without the use of a CSI driver or a third party tool. In Azure, Machine Identity isn't available for AKS, the end result being that Cluster identity management is tied to a User identity, which is far from ideal if non repudiation is a concern. In the case of GKE, if leveraging workload identity when spinning up a new pod, "the first few seconds of a Pod's life might fail". The end result is an inconsistent management across workloads.

We require SSO for people, why not machines?

SSO for users has been pertinent for the better part of three decades in an attempt to unify the human identity and authentication gap. It's easy to speculate why application identity federation hasn't become more prominent whether it be the static nature of infrastructure, until more recently or the pervasive use of monolithic architectures or a combination; regardless, the current state of application identity leaves us with a fragmented, decentralized, highly manual approach. 

While most cloud providers have native identities for their resources, unifying these into a single, auditable, controlled workflow without a centralized brokerage is complex, not human readable, and not scalable due to the complexity required for automation. By leveraging a brokerage mechanism, we can guarantee a unified workflow over all the required application layers, from network to application across multiple resources. 

If best practice dictates that users should authenticate via SSO, whether for the VPN or a company application, why should machines be any different?