Image for post
Image for post

When I first began writing the ASP.NET WebAPI services for our vehicle telemetry tracking, it was a small project with only a few controllers. The log data that was generated was small in volume and I would query the logs with relative ease. Fast forward to today (over 3 years later), and the ASP.NET WebAPI project now contains services that are consumed by our mobile app, web apps, licence and MOT checking apps etc. Basically, the project has grown substantially since I first created it.

At the time that I first wrote the ASP.NET WebAPI services, I investigated many of the logging frameworks that were targeted for the the .NET Framework at the time. I looked at Log4Net, NLog and Elmah. They all looked feature rich and customisable, but in the end I rolled my own logging framework. My requirements at the time were quite simple as were the sizes of the log data that would be generated. So I rolled a simple logging framework that would meet my requirements.

The log data that is generated today can get very large, and querying it can be time consuming. A single build can load 10k rows of log data. We have console apps, mobile apps and web apps all running on a daily basis that also contribute to the volume of log data that is generated. It is not uncommon to have 100s of thousands of log entries created each day. Trying to diagnose a problem with so much data to query is not always a simple task. This is where structured logging comes into play.

What is structured logging?

Log data is often comprised of log messages. These log messages are unstructured strings of text. Diagnosing an error entails searching through these text strings looking for the information that we need.

Typical log messages might look something like this.

Hide Copy Code

INFORMATION - 2019-12-24 - This is a log message
ERROR - 2019-12-24 - Something went wrong
INFORMATION - 2019-12-24 - This is another log message

Such unstructured data makes it difficult to query for useful information. As a developer it would be really helpful to be able to query and filter your log data. For example to query log data by a specific date, customer, log entry type etc. The goal behind structured logging is to solve this problem. The aim of structured logging is to bring structure to unstructured data to allow it be usefully queried and filtered to find the information you need in a timely manner. For log data to be queried and filtered requires the data to be mapped into a format that allows this e.g. XML and JSON would be ideal candidates.

What was clear was that my initial logging framework was no longer fit for purpose. The volumes of log data being created each day were getting increasingly larger, and I was no longer able to query the data in a sensible and timely manner. That’s when I came across the notion of structured logging. After much reading around and looking at other frameworks that supported this, I decided that I would update my own logging framework rather than employ one of the .NET logging frameworks (again). Although the frameworks mentioned earlier all now provide support for structured logging, SeriLog seems to be the preferred choice due it having provided support from the very beginning (whereas the others have provided support after the fact). The reason for updating my own logging framework rather than using one of the 3rd party ones was that I had already invested a lot of time and effort into my own framework, and it would probably be more straight forward to update what was already there than to start again with a new logging framework.

How have I implemented structured logging?

I have added a new column to my log table. Previously my log table consisted of a log message column that would contain text strings (as in the examples given previously). All the key information would be contained within the log message as in the following example.

Hide Copy Code

INFORMATION - 2019-12-24 - User Joe has logged onto the system

The user name ‘Joe’ is what I would be looking for in the log message. As can be seen it is part of the log message itself. With structured logging all such information is now stored in JSON (other formats are available). My log table is composed of the following columns — with the Properties column storing the structured logging data in JSON format.

Hide Copy Code

LogType
INFORMATION

N.B. one added benefit of rolling my own logging framework is that I can implement as many log types (the column LogTypes in the above example) as I need (I currently have ERROR, INFORMATION, EXCEPTION, DEBUG). I have a .config setting that allows me to switch the DEBUG log data on or off. Currently I have this set to ON for my unit test projects and OFF for production.

The biggest challenge was not so much updating my logging framework to implement structured logging, but to update all the areas of the code where I am sending output to the log (which is everywhere). I am still currently in the process of doing this.

The signature for my log message method looks like this.

Hide Copy Code

protected void LogMessage(string message, LogLevelTypes loglevel, Dictionary<string, string> properties = null)
{
//write the log entry to the log table
}

The LogLevelTypes is an enum containing the values ERROR, EXCEPTION, INFORMATION and DEBUG. The structured log data is passed to my method as a Dictionary of key-value pairs. The Dictionary is then serialised before being sent to the log table.

With your log data now stored in a structured format, you are able to utilise one of the many different tools that are available for viewing structured logging data e.g. Prefix from Stackify[^]

I haven’t yet got round to investigating any of the tools available for viewing structured data, but when I do I will be sure to write an article about it on here. Until then, have a Merry Christmas and a Happy New Year.

A father, cyclist, vegetarian, atheist, geek and multiple award winning technical author. Loves real ale, fine wine and good music. All round decent chap.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store