Posted on Oct 21 Hello 👋, in the previous post, we have seen about Grafana Cloud with an AWS subscription. I this post, we are going to see about Amazon Grafana(which is a fully managed service in AWS), with some demo on couple of metrics collected via Prometheus from EKS. Let's get into action!!!I have created an EKS cluster for the purpose of this exercise. For more info on creating clusters you can see this video.We would be fetching metrics with prometheus, hence we can add the helm repo for prometheus on the local system.Scope to the cluster and create a separate namespace on which we can install prometheus.The prometheus chart comes with a stack of components, by default. We don't need all of those. For this exercise, we'd just need the prometheus server and kube-state-metrics. We also don't need persistent volume for this lab. So we can define a values file like below.We can now install a prometheus helm release with the prometheus helm chart.The status of helm release installation can be checked.And the pods should be running.We can access prometheus via port-forwarding.Note: you can press ctrl c to stop port fowarding...You should be able to access the prometheus expression browser on http://localhost:8080.Let's try a sample query kube_apiserver_clusterip_allocator_allocated_ips with promql.So you are seeing a couple of hits, it seems good.We would now setup Amazon Prometheus. The idea is to forward metrics from the local prometheus we installed previously, to Amazon managed Prometheus.I have created a workspace in Amazon Prometheus, with the name grafana-demo-k8s. Once created, it would show you all useful details that are required to rewrite metrics from our local prometheus. Make a note of the remote write URL on this page.Follow this link to set up service roles for the ingestion of metrics from Amazon EKS clusters.Setup variables for the AWS account ID and AMP rewrite URL.Update values file with the details of AMP.Substitue the variables in the values file and make a new file.Upgrade the helm release with the new values file.Now let's set up Amazon Grafana.I am creating a workspace here, with name grafana-demo-eks.I have selected AWS IAM Identity Center as the authentication method, and created a user with email, first and last name.. A verification link will be sent to this email using which the password could be set.I am choosing the same VPC as that of the kubernetes cluster. For the AZs I have choosen the private subnet in both. And, default for the security group.Select the data source as AMP.I have kept rest of the settings to their default and created the workspace. One thing I have noted is there were only two availabe versions of Grafana which are 9.4 and 8.4 for now in Amazon Grafana, where as Grafana cloud had the latest version 10.2.0. Once the workspace is created, you should see a workspace URL that can be used to access the Grafana portal.In th worskspace page, under the authentication tab click Assign new user or group and select the correct user that was created earlier. Also make the user admin, as by default it will be given a viewer role.You can login to the workspace URL with the IAM identity email and the password set during email verification.You should see a page like this:Add datasource in GrafanaGo to Home > Apps > AWS Data Sources > Data sources and add the AMP datasource like below.Let's try adding a panel📊. Go to Home > Dashboards > New Dashboard and Add a new panel here. Let's try to see the number of pods in each namespace. The query we use for this purpose is sum(kube_pod_info) by(namespace). Note that the legend at the bottom would show that Last null value as configured in the right side of the screenshot below.So it says the default namespace has pod, the prometheus namespace has 2 pods, and the kube-system has 10 pods. Let's try validating this with kubectl.Awesome, it's matching. I am saving this panel with name Pod count per namespace, and given the dashboard the name grafana-demo-eks.Now let's try with one other panel, this time we will see the no. of services by namespace. We can use this query sum(kube_service_info) by(namespace). And instead of the default timeseries visualization, let's try with pie chart. We'll show both value and percent in the pie chart as shown in the screenshot below.We shall validate this too, with kubectl.Cool, the service count per namespace is matching.Alright, we had some practice with tools like prometheus and grafana in the AWS world with kubernetes. And, we have tried visualizing metrics that are collected by kube-state-metrics. We used prometheus for scraping the metrics from kube-state-metrics, which is ingested to the AMP workspace. We added AMP as a datasource in Grafana to build the panels. So, here is the final look of our 2 panel dashboard.So the AWS resources we have created in this exercise are: An EKS cluster, a grafana workspace, and a prometheus workspace. So make sure you are deleting those when you are done, to avoid billing💰... For EKS cluster jus do it from the cli: eksctl delete cluster --name grafana-demo-eks.Thanks for reading !!!Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well Confirm For further actions, you may consider blocking this person and/or reporting abuse Dmitriy A. - Sep 19 Rahul Ladumor - Sep 19 urgensherpa - Oct 2 Reza Alipour - Oct 1 Would you like to become an AWS Community Builder? Learn more about the program and apply to join when applications are open next. Once suspended, aws-builders will not be able to comment or publish posts until their suspension is removed. Once unsuspended, aws-builders will be able to comment and publish posts again. Once unpublished, all posts by aws-builders will become hidden and only accessible to themselves. If aws-builders is not suspended, they can still re-publish their posts from their dashboard. Note: Once unpublished, this post will become invisible to the public and only accessible to Shakir. They can still re-publish the post if they are not suspended. Thanks for keeping DEV Community safe. Here is what you can do to flag aws-builders: aws-builders consistently posts content that violates DEV Community's code of conduct because it is harassing, offensive or spammy. Unflagging aws-builders will restore default visibility to their posts. DEV Community — A constructive and inclusive social network for software developers. With you every step of your journey. Built on Forem — the open source software that powers DEV and other inclusive communities.Made with love and Ruby on Rails. DEV Community © 2016 - 2023. We're a place where coders share, stay up-to-date and grow their careers.