To learn how to configure cloud watch agent, lets first know a little about AWS CloudWatch . The main job of CloudWatch is to collect monitoring and operational data in the form of logs, metrics, and events, providing you with a unified view of AWS resources, applications, and services that run on AWS and on-premises servers. You can use CloudWatch to detect anomalous behaviour in your environments, set alarms, visualize logs and metrics side by side, take automated actions, troubleshoot issues, and discover insights to keep your applications running smoothly.
By default, Cloudwatch provides various metrics that we can monitor in AWS for eg: CPU Utilization Average, Disk Read Bytes Average, Disk Read Ops Average, etc. However, there are certain metrics that Cloudwatch doesn’t provide like memory utilization and more detailed metrics, and to get benefits of these additional metrics from the system we will need to install Cloudwatch Agent.
Cloudwatch Agent or Cloudwatch Unified Agent thus will allow us to do the following additional tasks:
- Collect internal system-level metrics from Amazon EC2 instances across operating systems. The metrics can include in-guest metrics, in addition to the metrics for EC2 instances.
- Collect system-level metrics from on-premises servers. These can include servers in a hybrid environment as well as servers not managed by AWS.
- Retrieve custom metrics from your applications or services using the
StatsDandcollectdprotocols.StatsDis supported on both Linux servers and servers running Windows Server.collectdis supported only on Linux servers. - Collect logs from Amazon EC2 instances and on-premises servers, running either Linux or Windows Server.
In this post, we will show step-by-step methods on install Cloudwatch Agent to capture error logs and access logs in AWS and also inject logs into Cloudwatch and Cloudwatch logs.
How to install Cloudwatch Agent in AWS EC2?
First, connect to the EC2 instance where you want the Cloudwatch Agent to be installed and download the Cloudwatch Agent using the following command:
wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
After the download is complete install the Cloudwatch agent using following command:
sudo rpm -U ./amazon-cloudwatch-agent.rpm
This will install the agent but it won’t start it because it won’t have the necessary config file. We will have to configure a config file to be used by this Cloudwatch agent and future Cloudwatch Agents.
Before preparing the config file we have to give EC2 instance permissions to interact with the Cloudwatch logs we need to provide an IAM role to this EC2 instance.
How to attach IAM Role to an EC2 Instance to allow permissions?
Let’s go to the AWS console and find IAM from the services and go to IAM dashboard.
Go to Roles => Create Role => AWS Servce Role => Select EC2 => Next Permissions
Then now you need to create the policy or attach the existing policy. For this demo purposes, we will select CloudwatchAgentServerPolicy and AmazonSSMFFullAccess => next tags =>next review
Give the name of the role as per your naming standards policy, we will call it “CloudwatchRole”, give some description for the role and Create Role.
Now we have to attach this role to the EC2 instance that we want for that go to EC2 dashboard => instances => right-click the instance => Security => modify IAM role => from the dropdown select the role you created earlier which is “CloudwatchRole” in our case.
Cloudwatch Agent Configuration Wizard
Now let’s go back and connect to the EC2 instance and start Cloudwatch Agent Configuration wizard by using following command:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
Press Enter and accept the default = for linux
Press Enter = for root
Press Enter = to turn on StatusD daemon
Press Enter = for Status D Port
Press Enter = for interval
Press Enter = for aggregration interval
Press Enter = to monitor metrics from CollectD
Press Enter = to monitor host metrics? e.g. CPU, memory etc
Press Enter = to monitor CPU metrics per core
Press Enter = for additional dimensions
Press Enter = for default resolutions
Press 3= for advanced config
Press Enter = for default confirmation
Press Enter = for default No existing cloud watch Log Agent Config
Press Enter = for default to monitor log files
Log File Path = var/log/secure (this is the path that we are going to monitor for this demo)
Log Group name: var/log/secure (same as before)
Press Enter = For default Log Stream name that will be instance ID
Press Enter = To specify additional log files to monitor (we want more to monitor so)
Log File Path: /var/log/httpd/access_log (we will be monitoring our webserver access_log)
Log Group name: /var/log/httpd/access_log (same as before)
Press Enter = For default Log Stream name that will be instance ID
Press Enter = To specify additional log files to monitor (we want more to monitor so)
Log File Path: /var/log/httpd/error_log
Log Group name: /var/log/httpd/error_log (same as before)
Press Enter = For default Log Stream name that will be instance ID
Press 2= To finish adding any more log files
Press Enter = For default to make sure the config is correct and to store the config file in the parameter store which is always the best practice.
Press Enter = For default parameter store name which is AmazonCloudWatch-linux
Press Enter = For default region where the instance is.
Press Enter = For default required credentials that we attached for the instance IAM role earlier. (It will use the IAM role to store the config file in the parameter store.
Now the configuration wizard is complete and will exit.
We can go and check the Parameter Store if the config we saved now exists there or not. To check parameter store we can go AWS Console ASM service (AWS System Manager) => Parameter Store. Here we can see the config we saved earlier which we can edit and modify its json file here as well.
Before we proceed we need to look into a bug fix that is required for Cloudwatch agent to run in Linux environment.
It requires software called collectD which is not installed in linux by default so for that we need to:
Make a directory called collectd by this command:
sudo mkdir -p /usr/share/collectd/
Then create a database file in the collectd folder by issuing the command:
sudo touch /usr/share/collectd/types.db
Then startup the cloudwatch agent and pull the config from the parameter store , make sure the agent is running and inject the logs in cloudwatch logs by issuing following command:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:AmazonCloudWatch-linux -s
Now the cloudwatch agent is installed and started successfully, we can check the outputs in the cloudwatch logs.
If we go to Cloudwatch Dashboard from the AWS console and check Log groups we can see our log group names. Here we can see /var/log/secure and cloudwatch agent has started logging in to this log group, other two log groups are not visible as we have not set up any web server yet.
So now if you are just testing or doing a demo you can detach cloudwatchrole by right click the instance ==> Security ==> Modify IAM role and remove cloudwatchRole from the list and type detach to detach it. Then from services go to IAM => Roles and find CloudwatchRole and delete it. You can also delete the parameter that we created earlier named “AmazonCloudwatch-Linux” from the AWS systems Manager.