AWS is not about paying for what you use, it’s about paying for what you forgot to turn off.Tweet by Michael Krakovskiy
Any individual or organization who is using public cloud services (irrespective of the provider) often struggles to recall what they used which resulted in $xyz in their monthly billing invoice. The above tweet summarizes a general feeling of cloud users, that along with “pay-for-what-you-use”, there is an axiom “pay-for-what-you-forget”.
There are many tips, best practices floating on the Internet on reducing cost so I would not mention them in detail here (AWS has a dedicated page for this topic). Instead, sharing the ones that I observed and found to be a common source of higher cost.
If you are starting into AWS or planning to expand your cloud services repertoire then use this as a checklist.
- Files uploaded in buckets having version enabled and no lifecycle policy.
- Lots of services use S3 to store temporary data or output but don’t clean it up when you delete those services.
- Manual backups of database which is not required anymore but still lying in S3.
- Athena – Running queries on large CSV data set when you could have used Parquet.
- Too many shards created
- Not using aggregation when ingesting data.
- Elastic IPs are allocated but not released after use.
- Using a high amount of on-demand EC2s instead of using reserve instances.
- Not using Spot instances for non-critical, fault tolerant tasks
- Flow logs enabled for the entire subnet or network (should use an appropriate filter)
- Running jobs on a badly partitioned data set
- Using unoptimized Spark code
- Using long running cluster for scheduled batch jobs
- Teams used a bunch of services for proofs-of-concepts but forgot to delete them post demo.
- Lack of IaC tools like CloudFormation or Terraform, to spin up AWS infrastructure resulting in forgotten & unused services such as SNS topic, Cloudwatch Logs group, Custom Images in ECR repo
- Long running clusters or databases that are not used 24/7. Do you have identified the access pattern or is there even a process in org which mandates it?
- Leaving up to individuals to keep track of services they are using or have used in past, but forgot to delete because well people from other teams requested to use it and now no one knows who owns it.
My answer and tip to all of these are unless you have a cost monitoring and optimization tool (3rd party or offered by AWS), it is not feasible to know the source of high cost and track your usage consistently over a period of time. Instead of once a quarter action item of analyzing cost and asking leads to work with their respective team to reduce cost, why not have a unified view of cost and usage pattern shared with each team (login using SSO).
AWS is like a Lego box, you need to assemble pieces to build to your application. It grows with your business and then you can’t afford to investigate each service or use another costly service to analyze it. My recommendation is to decide on a tool that provides a single picture of your entire AWS infrastructure, followed by the ability to drill down to each service, calculating its cost and most important “guidelines to save cost based on analyzing historical data” e.g. S3 Intelligent Tier or Trusted Advisor from AWS.
There is always a trade-off between you paying a cloud provider to offload operations vs hiring engineers to handle operations so that development teams can focus more on building software. I have also observed a hybrid approach by organizations where they are using AWS primarily as IaaS, paying for core compute, storage and networking services while installing and maintaining platforms and/or software on top by themselves. For specialized services, it is on a per need basis or those which have very high operating cost (e.g. Hadoop, Kubernetes).
In general, your expenditure on cloud providers doesn’t always have to be frown upon. It must reflect the strategic investment you made to have your business grow faster and scale better. So next time when you see your cloud invoice, consider these trade-offs and your business priorities.