Understanding log formats and protocols – Part 1

Introduction

In this multi-part series, we are going to explore log formats and protocols for sending them: two of the most fundamental pieces of any logging system. In Part 1, we will go over log formats and how metadata is encoded. Very little has changed as far as these core fundamental pieces are concerned, even as technology and computing has evolved.

We will start the journey with an application. This could be from the system, written by a user, etc. Applications write out human-readable messages to convey information. These messages help, both the user and developer understand indirectly, the state of the application. Two additional critical pieces of information are needed  – the when and the how good or bad the information is for a reader who is looking at it. In essence, we have uncovered 3 mandatory parts of any log message for any logging system: the information, the when and the how good/bad it is. These are typically referred to as the message, the timestamp, and the severity.

Applications run within an operating system or nowadays what we call the cloud operating system. For an operating system dealing with many applications, it is natural to separate log messages from different applications and group them together. So here we see the fourth mandatory piece of any logging system – the application name itself, typically referred to as the application name

As is universally known, applications are written by humans and humans have a tendency to make mistakes and like it or not we see applications, for no real fault of their own, sometimes, die horrible deaths, sometimes are forced to restart, etc.. Due to this, the operating system now has to deal with multiple incarnations of an application and in order for the intended reader of the log message to understand this discontinuity, it is critical to include the 5th vital piece of any logging system, the process identifier or the incarnation identifier or the instance identifier. This is typically referred to as the process Id or the Instance Id.

So far we have a good set of mandatory fields that are essential for an operating system to organize the logs created by an application so a human reading it can make sense of it.multi

RFC3164 - BSD Syslog protocol

Armed with this information, I think we are ready to segway into the first real standardized log format that came from the BSD folks – RFC 3164 – The BSD Syslog protocol. Let us look at a raw RFC3164 message

<14>Mar 29 06:15:28 customer-tooling postfix/anvil[17483]: statistics: max cache size 1 at Mar 29 06:12:05

Looking at the fields, it’s a quick guess about the mandatory fields we just talked about. Let’s see if you spotted the same as well

  • Message – “statistics: max cache size 1 at Mar 29 06:12:05”
  • Timestamp – “Mar 29 06:15:28”
  • Severity – ???
  • Application Name – “postfix/anvil”
  • Process Identifier/ Instance Identifier – “17483”

It’s not quite obvious what the severity is. So, let us look at that. RFC3164 encodes the severity in the beginning up to the first 5 characters of the message. Ref: https://tools.ietf.org/html/rfc3164#section-4.1.1

So let us decode what “<14>” means and what the severity of the message is. The BSD Syslog protocol calls this field the priority field.  The priority field encodes two pieces of information – the facility and the severity. So we have now encountered a new log message construct – the facility.

As you recall, we discussed the application name in the message. Operating systems group application processes into subsystems also referred to as the facility For e.g. all kernel messages/processes get assigned the kernel facility. Messages coming from the mail subsystem get grouped under a separate facility: mail system.

Facility and Severity are represented by integer codes in the BSD Syslog specification. Ref:  https://tools.ietf.org/html/rfc3164#section-4.1.1

Getting back to how to decode the severity of the message, the way the priority field is calculated is to take the facility integer value, multiply it by 8 and add the severity integer value. So

Priority_Code = Facility_Code * 8 + Severity_Code

So, to find the severity, we just divide the Priority_Code by 8 and use the remainder e.g. with a priority value above 14, the Facility is the quotient –  14/8. = 1 and severity code is the remainder 14-8*1 = 6

Now let’s decode what these two numbers mean. Let us look at the BSD protocol encodings for Severity and Facility below. The two tables below are reproduced from the section 4.1.1 referenced above in the BSD Syslog protocol document.

Looking at the two tables we can now conclude that the message was sent with the severity of Informational messages so nothing serious to report! The facility was user-level messages.


     Numerical Code         Severity
       
           0       Emergency: system is unusable
           1       Alert: action must be taken immediately
           2       Critical: critical conditions
           3       Error: error conditions
           4       Warning: warning conditions
           5       Notice: normal but significant condition
           6       Informational: informational messages
           7       Debug: debug-level messages
 
     Numerical Code         Facility
0 kernel messages 1 user-level messages 2 mail system 3 system daemons 4 security/authorization messages (note 1) 5 messages generated internally by syslogd 6 line printer subsystem 7 network news subsystem 8 UUCP subsystem 9 clock daemon (note 2) 10 security/authorization messages (note 1) 11 FTP daemon 12 NTP subsystem 13 log audit (note 1) 14 log alert (note 1) 15 clock daemon (note 2) 16 local use 0 (local0) 17 local use 1 (local1) 18 local use 2 (local2) 19 local use 3 (local3) 20 local use 4 (local4) 21 local use 5 (local5) 22 local use 6 (local6) 23 local use 7 (local7)

Summary

In summary, as we conclude part 1 of the “Understanding log formats and protocols” series, we have seen what the mandatory fields of a log message are and also looked at the BSD Syslog protocol encoding for a log message, also commonly referred to as an RFC3164 log message.

BYOK.-1

LOGIQ Insights Powers Kubernetes Observability with Its 1.2 Software Release

LOGIQ, Creator of LOGIQ Insights: a complete observability platform for monitoring, log aggregation, and analytics with infinite storage scale, has released General Availability of LOGIQ Insights 1.2.

LOGIQ’s 1.2 release brings self-service observability on the Kubernetes platform along with many unique features built for Kubernetes environments and workloads

LOGIQ is making Kubernetes observability zero-configuration, zero-click. We empower developers and administrators to help keep their infrastructure and applications always running. With the LOGIQ stack, in 5 minutes you are up and running," 

Tito George, LOGIQ's Co-founder Tweet

Highlights in this release:

  • 5-minute deployment: LOGIQ Insights can be deployed on any Kubernetes using a HELM chart in 5 minutes, enabling true self-service observability.
  • Prometheus connector: Connect your Prometheus instances and gather metrics along with logs, for complete observability.
  • Real-time log tailing: In Kubernetes environments, log-tailing can filter logs by namespace, application, Kubernetes labels, or pod name making it easy to build, deploy, debug, and manage distributed applications.
  • Log time machine: Users can go back in time and view logs. This helps root-causing issues while comparing old vs new logs.
  • Namespace and application log isolation: With the adoption of service mesh and microservices-based applications running within a namespace boundary, the observability boundaries are also redefined. LOGIQ allows zero-configuration isolation of logs from Kubernetes namespaces for applications.
  • Logiqctl: LOGIQ’s command-line toolkit offers capabilities such as application discovery, search, query, and log tailing. Alongside the UI, logiqctl is one more window of user’s choice to gather insights into their data.
We've chosen LOGIQ as our native cloud observability, SIEM/SOAR solution because of its ability to visualize the "needle in a haystack" data. In our case, we found a number of misconfigurations and anomalistic patterns in services in minutes, which could have lingered for months or years. Combined with CHAaSM AI and Falco, LOGIQ gives our hardened cloud platform real-time security visibility and monitoring no other solution can
Play Video

Availability
LOGIQ Insights 1.2
 is generally available today.

  1. Details of deploying on the Kubernetes platform can be found in our K8S Quickstart guide.
  2. HELM charts are available on GithubHELM Hub and Artifact Hub
  3. Details of deploying on AWS can be found in our AWS Quickstart guide

About LOGIQ
LOGIQ is a leading provider of S3 powered monitoring, logging, and analytics software. For more about LOGIQ, please visit our website at – https://logiq.ai or YouTube – https://www.youtube.com/channel/UCknndR4L6qUZdHq0L7H3Zlg

LOGIQ is a member of the Cloud Native Computing Foundation (CNCF)

LOGIQ is now a CNCF member

The LOGIQ team is super excited to join CNCF. The Cloud-native computing foundation is a great community of cloud builders and projects that is shaping how the enterprise cloud of the future looks like.

Logiq

Log Analytics finally got wings!

Log Analytics got wings with S3

Logiq

By LOGIQ team

Right, a few months back, few of us came together with one mission: To give log analytics its much-desired wings. BTW It’s not just AWS S3; you can use any S3 compatible store.

Logging is cumbersome, so we built the LOGIQ platform to bring easy logging to the masses. A platform that is so easy you can be up and running in less than 15 minutes. Send logs from Kubernetes or on-prem servers or cloud virtual machines with ease.

What If

  • you can retain your logs forever
  • you don’t have to worry about how many gigabytes of data you produce
  • you can get all crucial events out of logs without a single query
  • you can access all of it using a simple CLI or a set of API’s
  • you can build cool real-time dashboards

And what if all of this comes with a fixed cost.

How Do We Do It?

We use S3 (or S3 compatible storage) as our primary storage for data at rest. LOGIQ platform exposes protocol endpoints – Syslog, RELP, MQTT. You can use your favorite log aggregation agent (Syslog/ rsyslog/ Fluentd/Fluent-bit/ Logstash) to send the data to LOGIQ protocol endpoints. (oh we have REST endpoints too). Our platform indexes all the data as we ingest. Knowing where the data is, makes it easier to serve your queries. Use our CLI/REST/GRPC endpoints to get data, and it is served directly from S3. Yes, “The Wings”

Dashboards, Covered!

If you think it’s just about the CLI and API’s, we have cool customizable dashboards too and a powerful SQL interface to dig deep into your logs.

Real-time, Covered!

Want to know what is happening in your servers real-time, yeah use our CLI to tail logs from servers in real-time. (just like you do a tail -f)

> logiqbox tail -h 10.123.123.24 -a apache

What we solve?

Most of the solutions out there hold data in proprietary databases that use disks and volumes to store their data. What’s wrong with that? Well, disks and volumes need to be monitored, managed, replicated, and sized. Put clustering in the mix, and your log analytics project is now managing someone else’s software and storage nightmare.

With S3 compatible storage, you abstract out your storage with an API. 10G, 100G, 1TB, 10TB, it’s no different.

The race to “zero” is on

Object storage does away with the tree-like hierarchical file system of NAS and replaces it with a flat structure in which all objects have a unique identifier, with rich metadata that allows indexing, search, and analytics.

  •  
Ranjan

Ranjan

Founder, logiq.ai

All Posts

Object-based storage is becoming the de-facto data at rest option for unstructured data. Here is a quick cost comparison from some of the leading object storage vendors.

  • AWS S3 – 0.023$ per GB per month for the first 50TB
  • IBM Cloud object storage, cross-region standard – 0.0242$ per GB per month for the first 500TB
  • Azure Blob Storage (LRS) – 0.015$ per GB per month for the first 50TB

What does this mean for you? As an enterprise building and running  applications that use storage, it is time to look into S3 compatible storage as a first -class, close to “zero” cost, infinitely scalable storage layer

  • Use your S3 Bucket and Own your data
  • Bucket level data isolation and policy management
  • Freedom from retention limits
  • Lowest cost per GB
  • Durability and availability guarantees e.g. AWS S3 99.999999999% durability and 99.99% availability

Spread the word