Sign upSign InSign upSign InMember-only storyTeodor J. Podobnik, @dorkamotorkaFollowLevel Up Coding--ListenShareIn this post, I aim to delve into an advanced topic that engineers often encounter when setting up a monitoring system for production. Debugging complex applications, particularly those comprising multiple microservices, can be challenging. However, having a comprehensive system like Grafana in place, encompassing logging, metric export, and tracing, can greatly facilitate the process. But to truly unlock its value, integration of these three components is crucial. For instance, imagine being able to trace a log message in Loki and seamlessly find the corresponding reference in Tempo. This level of debugging granularity enables pinpointing errors within the microservice architecture. In my opinion, this functionality is invaluable.I believe it’s important to mention one of my previous posts — Setting up the Grafana Monitoring Stack that received significant attention, indicating a genuine need for guidance in this area. To be honest, when I embarked on this journey, there were limited resources available for setting up Grafana and most of the features was still in Beta. Although relying on source code, GitHub discussions, and documentation sufficed, having tutorials and insights from others would have been immensely helpful during troubleshooting. Therefore I decided to provide you with an extensive post myself. If you missed it, you can find it on the link below.levelup.gitconnected.comThis post is also a great introduction into this post, because we will build upon it. Although we’ll specifically talk only about certain parts of the code, but in order to keep up with the post, you should have a decent understanding about the stack.To keep things concise and manageable, I have divided this topic into multiple posts. This article will focus solely on integrating Loki and Tempo, addressing the following question:How can I find the trace to which a particular log belongs?Grafana also enables to nicely integrate Mimir with Tempo or the other way around, but we’ll get to that sometime else. So let’s dive in, leaving aside any further preamble. Before we begin, ensure you have an application or component, such as a cronjob on a Kubernetes cluster, that emits both logs and traces to the Grafana Stack. If you need guidance on achieving this, refer to the post referenced earlier.To begin, we need to make slight modifications to the application responsible for emitting logs and traces. The key step is to include the trace ID somewhere within the log line. Here’s a simple script I created to illustrate this. Pay close attention to the print statement, as this is a cronjob running on my Kubernetes cluster, with the Grafana Agent scraping pod logs and forwarding them to the Loki instance. The same applies to traces, which are forwarded through the Grafana Agent to the Tempo instance.You might be wondering why it’s necessary to include the trace ID in the log lines. This brings us to the second part, where we utilize a feature called derivedFields in the Grafana datasource configuration. With derivedFields, we instruct Grafana to scan all log lines, search for the trace_id pattern, and construct a query to reference the corresponding trace in Tempo.Note that this is only the part of the configuration and is not deployable. For more information, check the documentation.After you correctly deploy it, you should be able to see a button under the log line, which you can click and the relevant trace should open up for you.That’s essentially all there is to it. While the process is relatively straightforward, it can be tedious without proper guidance. By integrating Loki and Tempo in your Grafana Monitoring stack, you can achieve seamless traceability and enhance your debugging capabilities. Remember, there’s much more to explore in this area, as you can not only reference traces but also metrics and vice versa. In a future post, I’ll delve deeper into these integrations and their benefits.I encourage you to experiment with your cluster and share your experiences of what worked and what didn’t. By actively engaging in the process, you can fine-tune your monitoring setup to meet the specific needs of your application and infrastructure.Thanks for being a part of our community! Before you go:🚀👉 Join the Level Up talent collective and find an amazing job----Level Up CodingA combination of Human and a Part-time Geek 🤓. I write about all perks of software and other boring stuff 👻Teodor J. Podobnik, @dorkamotorkainLevel Up Coding--Arslan AhmadinLevel Up Coding--12Gabe Araujo, M.Sc.inLevel Up Coding--13Teodor J. Podobnik, @dorkamotorkainLevel Up Coding--Love SharmainDev Genius--23Ahmed Elfakharany--3Love SharmainDev Genius--4JuinDevOps.dev--headinthecloudsinAWS Tip--1Prajwal T--HelpStatusWritersBlogCareersPrivacyTermsAboutText to speechTeams