Firelens demystified

AWS Firelens is a log routing agent for Amazon Elastic Container Service (ECS) containers. Applications on ECS run as docker containers. Containers can be run on a serverless infrastructure that is managed by ECS using the Fargate launch type. For more control over the infrastructure, containers can be hosted on a cluster of Amazon Elastic Compute Cloud (Amazon EC2) instances. In both of these scenarios, AWS manages networking, storage, security, IAM, and other necessary services required to run the containers.

FireLens for Amazon ECS enables the administrator to use task definition attributes to route logs to external log aggregators. It unifies the data collection across the ECS cluster. Its pluggable architecture allows adding data sources, parsers, filter/buffering, and output plugins.

The biggest advantage with Firelens is that you can connect almost any service endpoint as long as data sinks can process general-purpose JSON over HTTP, FluentFoward or TCP protocols. The Firelens magic is all about transforming the log output of ECS containers and generating the necessary routing configuration for sending logs to the logging service.

For using Firelens, define the log collector and sink. The sink can be any log aggregation provider like LOGIQ Log Insights. Let us now see how to put this together. We need the following:

  • A log router container with FireLens configuration marked as essential.
  • Application containers specifying the ”awsfirelens” log driver.
  • A task IAM role ARN for permissions needed to route logs

Below you will find a few “logConfiguration” examples that can be for your task definition. Note how the “logDriver” is set to “awsfirelens”. The “options” contain additional attributes for the log sink where the log data will be sent.

"logConfiguration": {
        "logDriver": "awsfirelens",
       "options": {
                 "Name": "forward"
                 "Port": "24224",
                 "Host": "logiq.example.com"
}
}

The ”awsfirelens” log driver allows you to specify Fluentd or Fluent Bit output plugin configuration. Your application container logs are routed to a sidecar or independent Firelens container inside your cluster that further routes your container log to its destination as defined in your task “logConfiguration”. Additionally, you can use the options field of the FireLensConfiguration object in the task definition to serve any advanced use case.

"firelensConfiguration" : {
      "type" : "fluentbit",
      "essential":true,
      "options" : {
         "config-file-value" : "arn:aws:s3:::mybucket/myFile.conf",
         "config-file-type" : "s3"
      }
   }

The diagram above shows how Firelens works. Container logs are sent to the Firelens container using the docker Log Driver. When the ECS Agent launches a Task that uses Firelens, it constructs a Fluent configuration file:

  • A specification of log source for how to gather the logs from the container

  • An ECS Metadata record transformer

  • Optional User-provided configuration. If you specify your own configuration file, firelens will use the ”include” directive to import it in the generated configuration file.

  • Log destinations or sinks derived from the Task Definition

The following snippet shows a configuration for including ECS metadata like container and cluster details.

{
   "containerDefinitions" : [
      {
         "image" : "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:latest",
         "firelensConfiguration" : {
            "options" : {
               "enable-ecs-log-metadata" : "true"
            },
            "type" : "fluentbit"
         },
         "name" : "log_router",
         "essential" : true
      }
   ]
}

To demonstrate how Firelens works end to end, the below is a task definition example containing an HTTP web server and a Firelens sidecar container to route logs to the LOGIQ server. Also, replace the task execution role if it is named other than the default “executionRoleArn” and populate the account id shown in XXXXXXXXXXXX in the following example:

{
   "family" : "firelens-logiq",
   "executionRoleArn" : "arn:aws:iam::XXXXXXXXXXXX:role/ecs_task_execution_role",
   "taskRoleArn" : "arn:aws:iam::XXXXXXXXXXXX:role/ecs_task_iam_role",
   "containerDefinitions" : [
      {
         "memoryReservation" : 50,
         "essential" : true,
         "firelensConfiguration" : {
            "type" : "fluentbit",
            "options" : {
               "enable-ecs-log-metadata" : "true"
            }
         },
         "image" : "amazon/aws-for-fluent-bit:latest",
         "name" : "log_router_logiq",
         "logConfiguration" : {
            "logDriver" : "awsfirelens",
            "options" : {
                "Host" : "logiq.example.com", 
"Name" : "forward",
"Port" : "24224" } } }, { "essential" : true, "memoryReservation" : 100, "logConfiguration" : { "options" : { "Host" : "logiq.example.com", "Name" : "forward", "Port" : "24224" }, "logDriver" : "awsfirelens" }, "name" : "app", "image" : "httpd" } ], }
kubernetes entity hierarchy

Need For Search

We are all accustomed to search. We search to find answers. In the log analytics world, those answers have improved customer retention, aided better decisions. But the search is a tedious process. 

Well, what if we don’t have to search, what if the information has arranged in such a way that we can discover it. Remember the old search engines like excite where they divide the web into directories, everything arranged in a hierarchy. What if we could similarly discover our logs.

In our infrastructure, there could be thousands of containers running. If we have to root cause an issue specific to a container, the usual way to do this is to go to a logs search page, apply filters to drill down to the specific container we have to look into. The filters depend on how we have segregated the logs from our infrastructure. We need to apply multiple filters like the namespace, StatefulSet or pod identifiers etc..

Now let’s think for a moment. This need not be a haystack. In the case of Kubernetes based deployments, a hierarchy exists. Each Kubernetes cluster has namespaces. Namespace has multiple Kubernetes entities like StatefulSets or Deployments. Each of these has one or more pods.

kubernetes entity hierarchy

Ultimately what we need is the logs from these individual containers. Instead of search what if we could just discover it the way you could discover files from directories.

BYOK.-1

LOGIQ Insights Powers Kubernetes Observability with Its 1.2 Software Release

LOGIQ, Creator of LOGIQ Insights: a complete observability platform for monitoring, log aggregation, and analytics with infinite storage scale, has released General Availability of LOGIQ Insights 1.2.

LOGIQ’s 1.2 release brings self-service observability on the Kubernetes platform along with many unique features built for Kubernetes environments and workloads

LOGIQ is making Kubernetes observability zero-configuration, zero-click. We empower developers and administrators to help keep their infrastructure and applications always running. With the LOGIQ stack, in 5 minutes you are up and running," 

Tito George, LOGIQ's Co-founder Tweet

Highlights in this release:

  • 5-minute deployment: LOGIQ Insights can be deployed on any Kubernetes using a HELM chart in 5 minutes, enabling true self-service observability.
  • Prometheus connector: Connect your Prometheus instances and gather metrics along with logs, for complete observability.
  • Real-time log tailing: In Kubernetes environments, log-tailing can filter logs by namespace, application, Kubernetes labels, or pod name making it easy to build, deploy, debug, and manage distributed applications.
  • Log time machine: Users can go back in time and view logs. This helps root-causing issues while comparing old vs new logs.
  • Namespace and application log isolation: With the adoption of service mesh and microservices-based applications running within a namespace boundary, the observability boundaries are also redefined. LOGIQ allows zero-configuration isolation of logs from Kubernetes namespaces for applications.
  • Logiqctl: LOGIQ’s command-line toolkit offers capabilities such as application discovery, search, query, and log tailing. Alongside the UI, logiqctl is one more window of user’s choice to gather insights into their data.
We've chosen LOGIQ as our native cloud observability, SIEM/SOAR solution because of its ability to visualize the "needle in a haystack" data. In our case, we found a number of misconfigurations and anomalistic patterns in services in minutes, which could have lingered for months or years. Combined with CHAaSM AI and Falco, LOGIQ gives our hardened cloud platform real-time security visibility and monitoring no other solution can
Play Video

Availability
LOGIQ Insights 1.2
 is generally available today.

  1. Details of deploying on the Kubernetes platform can be found in our K8S Quickstart guide.
  2. HELM charts are available on GithubHELM Hub and Artifact Hub
  3. Details of deploying on AWS can be found in our AWS Quickstart guide

About LOGIQ
LOGIQ is a leading provider of S3 powered monitoring, logging, and analytics software. For more about LOGIQ, please visit our website at – https://logiq.ai or YouTube – https://www.youtube.com/channel/UCknndR4L6qUZdHq0L7H3Zlg

LOGIQ is a member of the Cloud Native Computing Foundation (CNCF)