Quick about Logs: Organizations using AWS cloud platform and services mostly logs info in Amazon Simple Storage Service (Amazon S3) which is then shipped to an external monitoring and analysis solution (Kibana/Grafana/AWS Quicksight etc).
As of today, we follow some of the other time taking processes like provisioning a VM or installing data shippers for pushing data (logs) directly from AWS to Elastic.
Now AWS users can quickly ingest logs stored in Amazon S3 with the new Elastic Serverless Forwarder, an AWS Lambda application, and view them in the Elastic Stack alongside other logs and metrics for centralized analytics.
Elastic Serverless Forwarder is an AWS Lambda function that ships logs from your AWS environment to Elastic. The function can forward data to Elastic self-managed or Elastic cloud environments. It supports trigger/input from S3/SQS Events.
Elastic Serverless Forwarder — Is published in the AWS Serverless Application Repository (SAR ) — to simplify this design and send logs to ElasticSearch.
The Elastic serverless forwarder Lambda application supports pushing logs from the AWS S3 bucket to Elastic. The SQS queue event notification on AWS S3 serves as a trigger for the Lambda function. When a new log file gets written to an AWS S3 bucket, internally triggers the Lambda function.
Below is a high-level view to set up the SQS function trigger on their S3 bucket and provide Elastic connection information to let the logs flow and use the prebuilt dashboards and full analytics features of Kibana to bring logs data to life.
For detailed coverage and steps behind, please check the link.
Just for your reference: Most AWS Logs go to AWS S3.
Hope this helps.
Arun Manglick
