TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Cloud Cost Visibility Is Within Reach With These 3 Steps

Don't be caught off-guard when you get your cloud provider's bill at the end of the month. Follow these three best practices and raise the visibility of your cloud spending.

• By Asim Razzaq
• December 6, 2021

Cloud cost visibility (or the lack thereof) is a growing concern for enterprises worldwide, resulting in unexpected costs that frustrate finance and engineering teams, stifle optimization, and create massive amounts of resource waste.

For Further Reading: How to Implement Containers and Virtual Machines on the Same Bare Metal Cloud for Performance and Cost-Efficiency The True Cost of Moving to the Cloud 4 Reasons Data Analytics Projects Fail

A cloud spending report from Virtana (published in September, 2021) showed that 86 percent of respondents cannot get a global view of cloud costs within minutes, and 40 percent cannot get it within hours, creating delays and potentially reducing agility. Seventy-one percent of respondents agreed that limited visibility across the hybrid cloud environment hinders their ability to maximize value, creates inefficiencies, and wastes time.

It's true that the first step in taking control of your cloud costs is knowing what resources are in use, but that only answers half of your governance question. Identifying who is responsible for those resources helps create better coordination between teams and products and aligns cloud costs with business value.

Organizations need tools that deliver cloud spending reports and create forecasts by team and project so they are aware of what investments they are making in cloud resources. These tools should also help teams identify drains on cloud budgets.

Attribute Worthwhile Investments

Cost visibility allows you to calculate the real cost of various efforts to ensure proper attribution and bookkeeping. Cloud cost management solutions can help improve tag hygiene by setting up tagging policies consisting of mandatory flags for tag keys and regular expression validation for tag key values that identify and remediate past inconsistencies.

There are multiple ways you can categorize resources with tags, matching the hierarchy of the tags to your business needs. These include by deployment type (dev test, staging, QA, production), by team, or by project. Once your resources have been tagged, you can create a prototype cost analysis, analyze the budget at a team and project level, and estimate the cost of a full deployment.

For AWS users, AWS supports tagging, which, when configured with a consistent scheme and clear namespaces, provides general information about resource usage. Tags are key-value pairs and are a fundamental tool for interacting with AWS resources. The key to effective tagging is uniformity. Everything from capitalization to naming strategy must be documented and standardized. This way, when manually or automatically tagging resources, the tags will be usable in future efforts to pull and analyze costs.

Other Amazon tools can increase the visibility of cloud costs such as AWS Cost Explorer, which breaks down its bills by service and suggests generalized savings plans; or when using a managed solution such as AWS Elastic Beanstalk, AWS Cost Explorer breaks down expenses into their component services. This helps users determine what resources they have spent money on.

Eliminate Unnecessary Spending

With complete cost visibility attributed to teams, it becomes clearer which portions of spending are unnecessary. There are two different types of resources in cloud environments that cost more than they deliver: idle resources and over-provisioned resources. Idle resources involve servers that are spun up, used briefly, and left idle or doing useless calculations. Over-provisioned resources are those powerful compute units running minor tasks.

In cloud environments, virtual machines, instances, databases, load balancers, and containers, among others, may be idle resources if they are no longer needed. Idle resources are difficult to track down in resource-based reporting. Any resource may end up idle eventually, and they range from inconsequential to burdensome.

Over-provisioning is easier to solve, as what scales up can scale down. Automated tools can scan for instances that need to be right-sized. Watch out for over-provisioning when working with reserved compute instances as these are harder to adjust once provisioned. (Yotascale, the company I work for, can make right-size recommendations based on autoscaling groups or tag groups, making it easier to assess right-size recommendations at scale versus individual instances.)

Guard Against the Unexpected

Vigilance combined with cost visibility prevents sticker shock at the end of the month or accidental runaway processes racking up unanticipated bills. Engineers should give unusual spending immediate and urgent cost visibility with a machine learning-based runaway cost detection process that detects cost anomalies and sends notifications to your team by Slack, Microsoft Teams, or email to integrate directly with teams' communication patterns.

By tracking complex architectures and reporting directly to people generating the costs, companies can avoid wasting engineering time (and salaries) digging through all this information.

Everything starts with reporting cloud costs directly to each resource's owner and to anyone else who needs the information. Organizations have tamed cloud costs by bringing their finance and engineering teams closer together and keeping cloud-platform officials in the loop on the company's forecasts. However, cloud cost visibility isn't just about what information is available; it's also about how quick and easy it is to access the right information, at the right time.