Storage Informer
Storage Informer

Monitoring your AWS estate using Cloudwatch

by on Jul.18, 2010, under Storage

Monitoring your AWS estate using Cloudwatch

EMC logo

Anyone tasked with setting up  IT Services be they in the cloud public or private, virtual or  physical will need to manage those resources .To be able to do that you need to be able to monitor stuff and use those results to provide proactive management.

Amazon AWS is primarily an IaaS cloud service (although they are fast morphing into providing PaaS)  and as such you would be expected to be able to monitor typical  metrics. You can do this by accessing pef mon stats in windows or the equivalent on Linux but Amazon AWS offers a neater more secure way of obtaining this  type of data .

Amazon AWS exposes a number of  metrics  via  a web service called  Cloudwatch . The use of these metrics enables Autoscaling which is a nice reason for using cloud services. You only incur Amazon CloudWatch fees for Amazon EC2 instances you choose to monitor. There are no additional charges for monitoring Amazon EBS volumes, Elastic Load Balancers or RDS Database Instances.

It’s a simple checkbox via the AWS console to switch on Cloudwatch monitoring for instances . You can also indicate as part of the command line start-up commands that you want monitoring turned on . Monitoring for EBS and RDS is set up automatically. 

My initial explorations into monitoring on AWS  took an interesting route as I explored using the AWS console, an app for my phone , the Command line tools and a plug in for a popular monitoring solution. I did start looking at rolling my own wrapper using java but that was just a little too far out of my comfort zone and  something I didn’t have time for especially as all the examples I looked at were using the deprecated Java Library for AWS rather than the AWS SDK for Java. I’d have to spend time getting to know he SDK  and attempting to write something from scratch or refactoring an example but we have a team of Java Devs who can do that sort of thing better than I ever will be able to  :-) .

There is a .NET SDK and at  some point when I get a chance I’ll probably have a look at using that . The projects I was looking at this for were Java/open source  based  and I wanted to stay within that arena  so straying into .NET was off limits.

The times are in UTC so for us in the UK fairly easy to work out . You can view data from the last hour through to the previous 2 weeks . So if you need  data for a longer period than 2 weeks to assist with capacity planning then you’ll need to archive it

AWS Console

Once you have monitoring switched on your instances you are able to use the AWS console to access graphs of the metrics. You can view data up to 2 weeks , and

Instance  Metrics:

image

You are then able to drill down on any particular graph:

image

 

RDS Metrics:

image

EBS Metrics:

image 

This is fine for initial set up/ development  but not really practical as a  way to monitor your AWS estate . It’s cumbersome , and there is no way to set alerting from here .  As nice as it is to have pretty graphs you will not be watching them 24 x 7 via this route .

Command Line Tools

If you’ve downloaded the API tools you can  Use the command line tools to see the list of metrics exposed via this. It exposes metrics for EC2 instances, RDS and EBS.  This entails setting up the API tools so you can run them from the command lines ( setting up env variables, telling it where Java is , where the command tools are, your authentication details etc )  you then run fairly ugly commands as the example below where I’m grabbing the CPU stats from my EC2 estate ( In this case a single instance)

mon-get-stats CPUUtilization –start-time 2010-07-11T23:00:00.000Z –end-time 2010-07-12T23:00:00.000Z –period 3600 –statistics "Average,Minimum,Maximum" –namespace "AWS/EC2"

Which spits out the metrics:

image

 

Not pretty but quick and dirty and gets the job done. You can pretty much understand why wrapping this stuff up is a must do really.

Monitoring from my phone (Decaff)

Most people have their phone close by most of the time so using your phone as  an integral part of your monitoring solution makes sense. There are a couple of solutions for Android phones but I’d read good things about  Decaff so thought I’d try that one. So I paid my money downloaded and installed the app.  Decaff  allows you to monitor and manage many aspects of the aws services . It also includes the ability to set up a watch on specific ports.  The images below illustrate  a few of the features.

Exif_JPEG_PICTURE                                                       Exif_JPEG_PICTURE                                                  Exif_JPEG_PICTURE                                              Exif_JPEG_PICTURE                                              Exif_JPEG_PICTURE                                               

I found the interface very usable and responsive and soon opted for that to launch instances, stop and terminate them as well as using it to provide an at a glance over view of how the systems were doing.   Check out the 9Apps blog which has a great description of how they  approached creating the application .

3rd party Solutions

There is a whole eco system that has built up around the AWS services and as such there are a number of 3rd party solutions available either hosted or you can incorporate into your own environment. I  started by looking  at PRTG as I have used it before and its still  a popular monitoring solution. The fact they had a sensor for cloudwatch I admit  heavily influenced me  at wanting to  look at this as an example rather than any alternatives. Unfortunately  I was unable to get the sensor to pick up any data. There are only a  few things to set up  and so far I have no idea why it’s not working  and to date their support have been unable to help either. If it ever gets resolved I’ll update this post . A review of 3rd party solutions would deserve a post all of its own and would mean I’d never get this post actually posted so forgive the lack of in depth analysis and I promise to revisit another day.

Conclusions:

Cloudwatch exposes the core metrics you would expect and the fact it can be accessed in a variety of ways relatively easily should be another tick on the list for those Operational staff tasked with making sure they can manage their AWS estate as easily as their current infrastructure.  I love the fact you don’t have to install agents to obtain these metrics  and thus no extra ports required to be opened .The use of cloud watch to  enable auto scaling is what makes the use of  cloud watch in my opinion a no brainer. Yes it does  cost  $0.015  an hour which I guess soon adds up  but that is money well spent to allow you to provide a pro active solution for monitoring and managing your AWS estate with very little effort . The issue with the  cost of monitoring using Cloudwatch which may seem out of step with the actual cost of running an instance  but that is a subject for another day .

A safe  solution is to make use of  3rd party tools  such as Cacti or PRTG ( I’m assuming if you were a  paying customer any teething issues would be sorted out quickly)  or possible  a  hosted solution, together with a mobile solution like Decaff.  I’m not that comfortable with writing Java and using eclipse  ( remember I banned myself from using the .NET AWS SDK)  but   if you have access to some Java Developers then it won’t be a problem for them to  write a wrapper to use  the Cloudwatch API (which is what I’d do given the time)  . What suits you will obviously depend on what you use today and what your requirements are in terms of managing the estate.

Update your feed preferences

URL: http://emcfeeds.emc.com/l?s=100003s2f6pa6831qks&r=rss2email&he=68747470253341253246253246636f6e73756c74696e67626c6f67732e656d632e636f6d25324667726163656d6f6c6c69736f6e2532466172636869766525324632303130253246303725324631382532466d6f6e69746f72696e672d796f75722d6177732d6573746174652d7573696e672d636c6f756477617463682e61737078&i=70726f78793a65383437633065372d333864392d343563302d623539332d3536373437333033653038383a3137343935


Leave a Reply

Powered by WP Hashcash

Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

Visit our friends!

A few highly recommended friends...