AWS: Linking EC2 logs to Cloudwatch
Quick and simple post on playing around with Cloudwatch on AWS EC2.
Install an IAM role to govern the transfer of the logs
Then go to the EC2 dashboard https://console.aws.amazon.com/iam
Choose ‘Create new IAM role’, then
Create Role
EC2 Instance, ignore all other settings, click next
Create policy
Paste in the following JSON policy:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "logs:DescribeLogStreams" ], "Resource": [ "arn:aws:logs:*:*:*" ] } ] }
Finally, name the role for ease of identification later.
Install AWSLogs
SSH into your EC2 instance and issue the following commands:
sudo yum update -y sudo yum install -y awslogs
The latest version of Redhat uses systemctl as opposed to legacy init.d services:
systemctl start awslogsd sudo systemctl enable awslogsd.service
on older versions of the ami the traditional commands are:
/etc/init.d/awslogs start chkconfig -add awslogs chkconfig awslogs on
Configuring Logging
By default only /var/log/messages is logged, but adding extra logs is dead easy, simply edit the following file /etc/awslogs/awslogs.conf:
[/var/log/messages] datetime_format = %b %d %H:%M:%S file = /var/log/messages buffer_duration = 5000 log_stream_name = {instance_id} initial_position = start_of_file log_group_name = /var/log/messages [/var/log/secure] datetime_format = %b %d %H:%M:%S file = /var/log/secure buffer_duration = 5000 log_stream_name = {instance_id} initial_position = start_of_file log_group_name = ssh
Also the default location for awslogs is us-east-1, you may want to change this? Edit the file /etc/awslogs/awscli.conf:
[plugins] cwlogs = cwlogs [default] region = us-east-1 Cloudwatch
Hopefully, you should then be collecting logs from the EC2 instance(s) that have the role applied.
Then the fun begins, playing with filter, metrics, alarms, and Lambda rules to shift the data into an elastic search platform.
Share on: