Recently, Ron Harnik, our Product Marketing Manager, and Sebastian Stadil, Scalr’s CEO, hosted a webinar on how enterprises manage multiple AWS accounts. Migrating to the cloud is never a smooth transition - by the time the company starts a CIO mandated move to the cloud, there are already more than a few AWS accounts floating around. Beyond the proliferation of accounts, access to these accounts presents a challenge due to complex IAM policies, reactive policies and multiple points-of-contact.
Part of this conversation focused on the newly announced AWS Organizations and EC2 System Manager, which have been a step in the right direction in helping users manage services across accounts and organizational hierarchies. Ron and Sebastian talked about the challenges of managing multiple AWS accounts and how enterprises are using Scalr to regain control of their infrastructure and streamline self-service in the public cloud.
We started with two quick polls.
The results? Over half of the attendees have multiple AWS accounts, and 9% of have hundreds of accounts. The scariest note is that 14% of surveyors don’t know how many accounts they have - and thus are unable to keep track of overspend and have little to no visibility on these accounts. This isn’t a one-in-a-million horror story, many organizations struggle with keeping track of their AWS accounts, and the real pain point is visibility, or lack thereof.
Ron introduced Sebastian Stadil, our CEO, to talk about his experiences with the frustration of managing multiple AWS accounts before creating Scalr.
“As soon as you start having multiple accounts, visibility really becomes a nightmare. Being in central IT, we found that it was really hard for IT to take ownership over applications and provide additional services when we had zero visibility over what was going on in the organization.”
Challenge: Credentials & “View Cycling” through resources and information
“When you’re trying to get visibility over multiple accounts, you realize that not all accounts are the same. Some are well maintained, with IAM groups and users, and everything is tied to a corporate directory. But over a large amount of Amazon accounts, that’s only happening for 1 out of 5, where the others have their root keys distributed, with no hookup to any central directory or CloudTrail, so you have no idea what keys are being used.”
Typical for bigger organizations, central IT becomes a kitchen sink of applications that they inherit over time. When one application is finished, a team moves on to another product, and so on and so on. Dependencies get deprecated, API keys get lost, developers leave and organizations get moved around. You get abandoned application fleets. And as we all know, documentation doesn’t always get finished. When central IT needs to access each of these applications across dozens of accounts, unnecessary workloads are being created. We call this issue view cycling, because you have to manually go through tons of different views to get information and answer basic questions about your applications and services. It’s not ideal for anyone, and while cloud management platforms like Scalr have risen to solve problems like this across cloud platforms, Amazon has released a new product to make things a little bit easier inside AWS.
Organizations is made up of a few key pieces:
- Organizational Units
- Organization Control Policies
Organizations are a collection of AWS accounts. At this level, one AWS Master Account is assigned, where all billing goes through.
Organizational Units are groups of accounts inside this Organization. This is the key component of Organizations, because it helps companies create hierarchies inside AWS. Accounts can belong to multiple Organizational Units, and you can nest Organizational Units inside one another.
Organizational Control Policies are the policies you can apply to OUs or individual Accounts. You can apply a OCP to an Organizational Unit, and it will apply to all accounts in that unit. The only Organizational Control Policies that are currently available enable administrators to black/whitelist particular AWS Services. For example, accounts belonging to a ‘Healthcare’ Organizational Unit of your organization wouldn’t need access to SQS or EC2, while central IT or DevOps would. Technical Support accounts would have access to CloudTrail, while an Operations Information Management division working on filtering data from one silo to another silo to help Sales would need access to RDS.
While it’s great that Amazon has finally created Organizations, many enterprise customers that have been dealing with multiple accounts for years have either consolidated them using their own tools or use cloud management platforms.
AWS Organizations is still in Technical Preview, with an expected release date sometime in 2017.
AWS EC2 System Manager
In the heap of news from reInvent also came EC2 System Manager. When you’ve uploaded your infrastructure and applications to the cloud, the task of OS patches, updates, and fixes in physical data centers now happens inside Amazon. EC2 System Manager helps you do this, similar in a way to Microsoft System Center. It’s not a completely smooth process, you’ll still need to install agents on each node and that adds another level of complication. While it’s another solution to help the enterprise market, it’s helpful only for a specific use cases.
Neither of these two solutions, EC2 System Manager and AWS Organizations, facilitates self-service for end-users. While you’re reducing workload for administrators (in AWS), you still need to streamline the solution for end-users through a self-service storefront or web application.
Challenge: Tracking Billing & Overspend
“We all know Amazon is customer driven...Amazon is perfectly aware that it’s very hard to manage many Amazon accounts. Amazon itself is one of the biggest users of [AWS], and so they have this problem internally, so it’s really good to see new functionality coming out to help with these problems. In 2010 we saw consolidated billing come out, and I’m sure everyone on this call uses it now, and so AWS Organizations is another incremental step forward.”
A lot of tools, including Amazon’s proprietary billing tool, are based on individually tagging applications and resources. It’s helpful but depends on users manually tagging their resources every time they boot up an instance. Untagged resources (sometimes up to 40% of an enterprise’s infrastructure), still remains part of the billing problem and leads administrators down the rabbit hole of searching for abandoned applications. Typically these are forgotten sandboxes, depreciated tools, unused staging servers, and so on.
Reducing unnecessary expenses is one of the reasons administrators look at solutions like Organizations and cloud management platforms like Scalr. Apart from tagging resources, the best way to accomplish that are through preventative and reactive policies that apply across all the users, environments, and cloud resources in your organization. We use Reclamation policies that determine the lifecycle of an application, based on the user creating it and the teams and divisions that they belong to. If an application’s lifecycle needs to be extended, users can make a request to the administrators. In other words, reclamation policies are another way to prevent wasteful overspend and gives administrators better visibility on their costs. As an additional solution, Scalr tool that we’ve found to be very popular is the Garbage Collector. It enables you to search through unused instances and EBS volumes to eliminate them, reducing spend.
Challenge: Policy Management & Enforcement
How can you control what end-users and developers do across a system? When there’s multiple AWS users, teams, divisions, and accounts, it’s easier said than done. Setting IAM policies that restrict what users are able to do is the key in maintaining control across accounts, but sometimes you need more fine-tuning in defining their responsibilities and freedom. Organizations has take on some of this approach, but cloud management platforms like Scalr have a more in-depth, and cross-divisional approach on policy enforcement. We call our solution the Policy Engine.
“The Scalr Policy Engine is a platform that can be used to create, manage and enforce policies, allowing your users to continue to take advantage of cloud services without creating more risk and needless spend for your organization. Scalr lets you enforce policies for an entire Account, such as restricting instance types, restricting networks or mandating security requirements. Each policy is defined by its type, which describes the goal of the policy, optional conditions that limit the scope this policy will apply to, and for some policy types that need it, additional configuration settings.”
The difference between Organizational Control Policies in AWS and one of the strengths of Scalr, is that the Policy Engine can be preventative and reactive. Reactive policies fix a problem by reverting a failing application or an instance to its ideal state, while preventative policies keep administrators and users ‘on-rails’ in a sense, keeping them from making mistakes and enabling them to move faster. You can specify policy groups by cloud providers as well - like restricting security groups for instances created in AWS. If you use Chef, in-coordination with OpsWorks or otherwise, you can set restrictions on what servers can be used. As a multi-cloud solution, you’re not trying to replicate your organizational hierarchy inside the dashboards of Azure and GCE.
A lot of enterprises end up building their own internal applications to try manage isolated problems. That’s how it starts - you have one small local problem that you solve with a homegrown tool. Then as more little issues appear in AWS on an enterprise scale, you’re building more and more applications on your own until you have your own awkward skeleton of a cloud management platform that’ll depreciate over time. But there’s a better solution that doesn’t have to be built internally over years - a cloud management platform like Enterprise Scalr.
This is just an overview on the webinar, so if you’d like to learn more and specifically how Scalr solves the issues we mentioned above, watch our on-demand webinar - Managing Multiple AWS Accounts. The first half covers an overview on how enterprises manage multiple AWS account, then at the thirty-five minute mark we jump into solutions for the challenges discussed above.