Posted on Aug 22 Recently, I've been spending a lot of time writing a Kubernetes operator using the go operator-sdk, which is built on top of the Kubebuilder framework. This is a list of a few tips and tricks that I've compiled over the past few months working with these frameworks.Kubebuilder, like much of the k8s ecosystem, utilizes zap for logging. Out of the box, the Kubebuilder zap configuration outputs a timestamp for each log, which gets formatted using scientific notation. This makes it difficult for me to read the time of an event just by glancing at it. Personally, I prefer ISO 8601, so let's change it!In your scaffolding's main.go, you can configure your current logger format by modifying the zap.Options struct and calling ctrl.SetLogger.In this case, I added the zapcore.ISO8601TimeEncoder, which encodes timestamps to human-readable ISO 8601-formatted strings. It took a bit of digging, along with a bit of help from the Kubernetes Slack org, to figure this one out. But it's been a huge quality-of-life improvement when debugging complex reconcile loops, especially in a multithreaded environment.Speaking of multithreaded environments, by default, an operator will only run a single reconcile loop per-Controller. However, in practice, especially when running a globally-scoped controller, it's useful to run multiple concurrent reconcile loops to simultaneously handle many Resource changes at once. Luckily, the Operator SDK makes this incredibly easy with the MaxConcurrentReconciles setting. We can set this up in a new controller's SetupWithManager func:I've created a command line arg in my main.go file that allows the user to set this value to any integer value, since this will likely be a tweaked over time depending on how the controller performs in a production cluster.One of the basic functions of a controller is to act as a parent to Kubernetes resources. This allows the controller to "own"these objects such that when it is deleted, all child objects are automatically garbage collected by the Kubernetes runtime.I like this small function that can be called for any client.Object to add a parent reference to the controllerthat you're writing.You can then add Owns watches for these resources in your SetupWithManager func. These will instruct your controller to listen for changes in child resources of the specified types, triggering a reconcile loop on each change.Your controller can also watch resources that it doesn't own. This is useful for when you need to watch for changes in globally-scoped resources like PersistentVolumes or Nodes. Here's an example of how you would register this watch in your SetupWithManager func.In this case, you need to implement myNodeFilterFunc to acceptan obj client.Object and return []reconcile.Request. Using theResourceVersionChangedPredicate triggers the filter function for every change on that resource type, so it's important towrite your filter function to be as efficient as possible, since there is a chance that it could be called quite a bit, especiallyif your controller is globally-scoped.One gotcha that I encountered happened when trying to query for a list of Pods that are running on a particular Node. This query uses a FieldSelector filter, asseen here:This codepath led to the following error: Index with name field:spec.nodeName does not exist. After some googling around, Ifound this GitHub issue that referenceda Kubebuilder docs page which contained the answer.Controllers created using operator-sdk and Kubebuilder use a built-in caching mechanism to store results of API requests. This is to preventspamming the K8s API, as well as improve reconciliation performance.When performing resource lookups using FieldSelectors, you first need to add your desired search field to an indexthat the cache can use for lookups. Here's an example that will build this index for a Pod's nodeName:Now, we can run the List function from above with the FieldSelector with no issues.If you've ever written controllers, you're probably very familiar with the error Operation cannot be fulfilled on ...: the object has been modified; please apply your changes to the latest version and try againThis occurs when the version of the resource that you're currently reconciling in your controller is out-of-date with what's in latest version of the K8s cluster state. If you're retrying your reconciliation loop on any errors, your controller will eventually reconcile the resource, but this can really pollute your logs and make it difficult to spot more important errors.After reading through the k8s source, I found the solution to this: RetryOnConflict. It's a utility function in the client-go package that runs a function and automatically retries on conflict, up to a certain point.Now, you can just wrap your logic inside this function argument, and never have to worry about this issue again! And the added benefit is that you just get to return err instead of return ctrl.Result{}, err, which makes your code that much easier to read.Here are some useful code markers that I've found while developing my operator.To add custom columns to your custom resource's description (when running kubectl get), you can add annotations to your API object like these:To add a shortname to your custom resource (like pvc for PersistentVolumeClaim for example), you can add this annotation:More docs on kubebuilder markers can be found here:https://book.kubebuilder.io/reference/markers/crd.htmlOriginally published on my blog: https://sklar.rocks/kubebuilder-tips/Templates let you quickly answer FAQs or store snippets for re-use.Thanks for the solid write up, especially outlining edge case issues. Also hats of for going so far as to check source code to figure out what's happening! Great write up and keep up the good work! Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well Confirm For further actions, you may consider blocking this person and/or reporting abuse Gerasimos (Makis) Maropoulos - Jul 2 Oshi Gupta - Jul 3 Maxime Guilbert - Jul 3 Kaike Castro - Jul 3 Once suspended, sklarsa will not be able to comment or publish posts until their suspension is removed. Once unsuspended, sklarsa will be able to comment and publish posts again. Once unpublished, all posts by sklarsa will become hidden and only accessible to themselves. If sklarsa is not suspended, they can still re-publish their posts from their dashboard. Note: Once unpublished, this post will become invisible to the public and only accessible to Steven Sklar. They can still re-publish the post if they are not suspended. Thanks for keeping DEV Community safe. Here is what you can do to flag sklarsa: sklarsa consistently posts content that violates DEV Community's code of conduct because it is harassing, offensive or spammy. Unflagging sklarsa will restore default visibility to their posts. DEV Community — A constructive and inclusive social network for software developers. With you every step of your journey. Built on Forem — the open source software that powers DEV and other inclusive communities.Made with love and Ruby on Rails. DEV Community © 2016 - 2023. We're a place where coders share, stay up-to-date and grow their careers.