How to install Cloudwatch Agent to capture logs?

To install Cloudwatch Agent to capture logs and other system metrices follow the following steps.

1. 1. Login to AWS EC2 terminal using the IAM credentials  

2. Download the Cloud watch agent from AWS using the link below:

https://amazoncloudwatch-agent.s3.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm

3. Install Cloudwatch Agent using the following command in the terminal:

sudo rpm -U ./amazon-cloudwatch-agent.rpm

4. Create and assign IAM Role to the EC2 Instance

Go to Services –> IAM Console

Inside IAM Console go to Roles –> Create role
Next set the correct permission, for that go to Services –> EC2 –> Next Permissions
Attach 2 Managed policies —> CloudWatchAgentServerPolicy and AmazonSSMFullAccess
Non the next steps provide the tags or skip and go to review. Finally

Name the role and press Create Role

5. On the EC2 console assign role to EC2 instance created above

Right click EC2Instance select –> Security and select –> Modify IAM role.
Select the role created above from the dropdown and save it.

6. Login to the EC2 instance in the terminal and to start cloudwatch agent configuration wizard enter following command:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

Press “Enter” to most of the default values, however, feel free to change the configurations as per the needs and remember some of these configurations can include paid services.

Next Provide the Log File Path
Enter the path of the log files to monitor for example: /VAR/LOG/SECURE
The path group name will be defaulted to the same value /VAR/LOG/SECURE
Keep Log Stream name as default by typing “Yes”.

The next prompt will ask for another log file path to monitor, do this until all the log files are monitored and press 2 to exit.

7 Install collectd and db for cloudwatch agent

Collectd is a daemon collecting system and application performance metrics periodically and provides mechanisms to store the values in a variety of ways. Cloudwatch Agent to capture logs utilizes colletD to gathers metrics from various sources, e.g. the operating system, applications, logfiles and external devices, and stores this information or makes it available over the network.

Create a folder for collectd and db.

In the terminal enter following command for making directory
sudo mkdir -p /usr/share/collectd/

Following command for making DB
sudo touch /usr/share/collectd/types.db

8 Start the Cloudwatch Agent

Final Step is to start up the Agent, for that the latest configuration has to be retrieved from the parameter store by using following command (Note: Parameter name should be the one that has been created in the parameter store)

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:AmazonCloudWatch-linux -s

(Sample of configuration file called AmazonCloudWatch-Linux is given below)

Now the logs will be captured by cloudwatch, go to Cloudwatch Console and check the logs in appropriate LogGroups. 

When the Cloudwatch Agent is started Metrices are also accessible like metrics for CPU, memory utilization etc are visible, otherwise were not available. 

Sample of: AmazonCloudWatch-Linux

{
    "agent": {
        "metrics_collection_interval": 60,
        "run_as_user": "root"
    },
    "logs": {
        "logs_collected": {
            "files": {
                "collect_list": [
                    {
                        "file_path": "/VAR/LOG/SECURE",
                        "log_group_name": "/VAR/LOG/SECURE",
                        "log_stream_name": "{instance_id}"
                    },
                    {
                        "file_path": "",
                        "log_group_name": "/var/log/httpd/access_log",
                        "log_stream_name": "{instance_id}"
                    },
                    {
                        "file_path": "/var/log/httpd/error_log",
                        "log_group_name": "error_log",
                        "log_stream_name": "{instance_id}"
                    },
                    {
                        "file_path": "",
                        "log_group_name": ".",
                        "log_stream_name": "{instance_id}"
                    },
                    {
                        "file_path": "",
                        "log_group_name": ".",
                        "log_stream_name": "{instance_id}"
                    }
                ]
            }
        }
    },
    "metrics": {
        "append_dimensions": {
            "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
            "ImageId": "${aws:ImageId}",
            "InstanceId": "${aws:InstanceId}",
            "InstanceType": "${aws:InstanceType}"
        },
        "metrics_collected": {
            "collectd": {
                "metrics_aggregation_interval": 60
            },
            "disk": {
                "measurement": [
                    "used_percent"
                ],
                "metrics_collection_interval": 60,
                "resources": [
                    "*"
                ]
            },
            "mem": {
                "measurement": [
                    "mem_used_percent"
                ],
                "metrics_collection_interval": 60
            },
            "statsd": {
                "metrics_aggregation_interval": 60,
                "metrics_collection_interval": 10,
                "service_address": ":8125"
            }
        }
    }
}