Welcome to the MALT Stack

The MALT stack is a set of principles on applying four key strategies:  Metrics, Alerting, Logging, and Tracing.  Each of these principles is key to monitoring and debugging an application.  By themselves, each of these principles only offer a small portion of insight, but together they open a breadth of runtime knowledge.

The main premise behind the MALT stack is centralizing and aggregating all data points into a single stream.  In most cases, this is applied using Kafka or Kinesis.  The MALT stack otherwise does not enforce any particular technology.  In fact, there are several available technologies that utilize the foundation of MALT.

MALT classifies the following key terms:

Collector : Collectors collect data from running systems such as logs, metrics, traces, etc and push onto streams.  Examples collectors include Logstash, CollectD, FluentD, Telegraf, etc.

Streaming : Streams provide a single, centralized facility for capturing data

Publisher : Publishers consume data from the streams and publish to varying data systems such as Elasticsearch (ELK stack), Splunk, InfluxDB (TICK stack), Zipkin, etc.

MALT provides the following key strategies:

Metrics

Metrics are provided by several available solutions in a variety of languages.  Within Java, one of the most popular is Dropwizard Metrics (fka Codahale Metrics).  Metrics provide key data structures such as timers, counters, gauges, etc.  Metrics also provide key facets or tags that provide grouping and filtering of data.

Metrics contain application metrics, system-level metrics, container-level metrics, infrastructure-level metrics, etc.  These metrics are all pushed centrally to the streaming broker, such as Kafka.  This allows a single flow of data that can be produced to a number of other servers.

Alerting

Alerting is used to trigger notifications of key events.  Often times this is used to indicate issues or concerns before becoming more critical issues.  Alerting may be driven off logs or metrics.  Because information flows centrally in Kafka, generating alerts in a common aspect becomes trivial.  One example of alerting, based on the ELK stack, is a product from Yelp called ElastAlert.

Logging

Logging is key to any architecture as it provides a first stop for debugging especially as it relates to critical failures.  Logging includes both application logs as well as hit logs from the application or other routing or gateway appliances.  

Similar to metrics, MALT adheres to pushing logging to centralized streams such as Kafka.  The logs and data pushed into Kafka are structured outputs.  This allows the logs to maintain context allowing publishers to generically act upon and index those fields.

One of the main provides of logging within Java is Logback.  Logback, together with Logstash, can easily route logs into Kafka centrally.  Once inside Kafka, the data can be produced into other systems or directly tailed from the console.

Tracing

Tracing is the final key component to MALT.  Two of the main tracing frameworks are Dapper and Zipkin with the latter being inspired by the former at Google.  The key principle behind tracing is linking together services to show the entire depth of how a single call is impacted by all other sub-calls.  With the move towards microservices, tracing becomes paramount.

As with all other key components in the MALT stack, all tracing and span data flows centrally through Kafka.  This allows the underlying framework to be easily changed without impacting the applications directly.

Stay tuned for more information and frameworks as it relates to setting up a MALT stack.

Comments

  1. Hi, are you going to use pigeons for alert delivery? I have a coop filled with homer pigeons that I can sell you at very cheap rates with guarenteed deliveries! Atleat once unless a HAWK gets them.

    ReplyDelete

Post a Comment

Popular posts from this blog

Setting up TICK Stack on Docker

Setting up the ELK Stack on MALT

Todo Application Demo