Anyone tasked with setting up IT Services be they in the cloud public or private, virtual or physical will need to manage those resources .To be able to do that you need to be able to monitor stuff and use those results to provide proactive management.
Amazon AWS is primarily an IaaS cloud service (although they are fast morphing into providing PaaS) and as such you would be expected to be able to monitor typical metrics. You can do this by accessing pef mon stats in windows or the equivalent on Linux but Amazon AWS offers a neater more secure way of obtaining this type of data .
Amazon AWS exposes a number of metrics via a web service called Cloudwatch . The use of these metrics enables Autoscaling which is a nice reason for using cloud services. You only incur Amazon CloudWatch fees for Amazon EC2 instances you choose to monitor. There are no additional charges for monitoring Amazon EBS volumes, Elastic Load Balancers or RDS Database Instances.
It’s a simple checkbox via the AWS console to switch on Cloudwatch monitoring for instances . You can also indicate as part of the command line start-up commands that you want monitoring turned on . Monitoring for EBS and RDS is set up automatically.
My initial explorations into monitoring on AWS took an interesting route as I explored using the AWS console, an app for my phone , the Command line tools and a plug in for a popular monitoring solution. I did start looking at rolling my own wrapper using java but that was just a little too far out of my comfort zone and something I didn’t have time for especially as all the examples I looked at were using the deprecated Java Library for AWS rather than the AWS SDK for Java. I’d have to spend time getting to know he SDK and attempting to write something from scratch or refactoring an example but we have a team of Java Devs who can do that sort of thing better than I ever will be able to .
There is a .NET SDK and at some point when I get a chance I’ll probably have a look at using that . The projects I was looking at this for were Java/open source based and I wanted to stay within that arena so straying into .NET was off limits.
The times are in UTC so for us in the UK fairly easy to work out . You can view data from the last hour through to the previous 2 weeks . So if you need data for a longer period than 2 weeks to assist with capacity planning then you’ll need to archive it
Once you have monitoring switched on your instances you are able to use the AWS console to access graphs of the metrics. You can view data up to 2 weeks , and
You are then able to drill down on any particular graph:
This is fine for initial set up/ development but not really practical as a way to monitor your AWS estate . It’s cumbersome , and there is no way to set alerting from here . As nice as it is to have pretty graphs you will not be watching them 24 x 7 via this route .
Command Line Tools
If you’ve downloaded the API tools you can Use the command line tools to see the list of metrics exposed via this. It exposes metrics for EC2 instances, RDS and EBS. This entails setting up the API tools so you can run them from the command lines ( setting up env variables, telling it where Java is , where the command tools are, your authentication details etc ) you then run fairly ugly commands as the example below where I’m grabbing the CPU stats from my EC2 estate ( In this case a single instance)
mon-get-stats CPUUtilization –start-time 2010-07-11T23:00:00.000Z –end-time 2010-07-12T23:00:00.000Z –period 3600 –statistics "Average,Minimum,Maximum" –namespace "AWS/EC2"
Which spits out the metrics:
Not pretty but quick and dirty and gets the job done. You can pretty much understand why wrapping this stuff up is a must do really.
Monitoring from my phone (Decaff)
Most people have their phone close by most of the time so using your phone as an integral part of your monitoring solution makes sense. There are a couple of solutions for Android phones but I’d read good things about Decaff so thought I’d try that one. So I paid my money downloaded and installed the app. Decaff allows you to monitor and manage many aspects of the aws services . It also includes the ability to set up a watch on specific ports. The images below illustrate a few of the features.
I found the interface very usable and responsive and soon opted for that to launch instances, stop and terminate them as well as using it to provide an at a glance over view of how the systems were doing. Check out the 9Apps blog which has a great description of how they approached creating the application .
3rd party Solutions
There is a whole eco system that has built up around the AWS services and as such there are a number of 3rd party solutions available either hosted or you can incorporate into your own environment. I started by looking at PRTG as I have used it before and its still a popular monitoring solution. The fact they had a sensor for cloudwatch I admit heavily influenced me at wanting to look at this as an example rather than any alternatives. Unfortunately I was unable to get the sensor to pick up any data. There are only a few things to set up and so far I have no idea why it’s not working and to date their support have been unable to help either. If it ever gets resolved I’ll update this post . A review of 3rd party solutions would deserve a post all of its own and would mean I’d never get this post actually posted so forgive the lack of in depth analysis and I promise to revisit another day.
Cloudwatch exposes the core metrics you would expect and the fact it can be accessed in a variety of ways relatively easily should be another tick on the list for those Operational staff tasked with making sure they can manage their AWS estate as easily as their current infrastructure. I love the fact you don’t have to install agents to obtain these metrics and thus no extra ports required to be opened .The use of cloud watch to enable auto scaling is what makes the use of cloud watch in my opinion a no brainer. Yes it does cost $0.015 an hour which I guess soon adds up but that is money well spent to allow you to provide a pro active solution for monitoring and managing your AWS estate with very little effort . The issue with the cost of monitoring using Cloudwatch which may seem out of step with the actual cost of running an instance but that is a subject for another day .
A safe solution is to make use of 3rd party tools such as Cacti or PRTG ( I’m assuming if you were a paying customer any teething issues would be sorted out quickly) or possible a hosted solution, together with a mobile solution like Decaff. I’m not that comfortable with writing Java and using eclipse ( remember I banned myself from using the .NET AWS SDK) but if you have access to some Java Developers then it won’t be a problem for them to write a wrapper to use the Cloudwatch API (which is what I’d do given the time) . What suits you will obviously depend on what you use today and what your requirements are in terms of managing the estate.
|Update your feed preferences|