Posted on Jul 21 We were running into an issue recently where Launchdarkly wasn't evaluating on celery servers. Opening a python shell on the boxes showed the keys were setup correctly, and that if we called the methods directly, the flags evaluated as expected. However, when called as a delayed task, the flags weren't evaluating.LaunchDarkly makes use of fork in python, and requires that only one LaunchDarkly client instance exists. This post is a good primer on forking in python. It appeared that this was the issue.My coworker Doug explained it thusly:The LaunchDarkly library makes use of threading, Celery starts the main "control" process that uses fork() to start n workers, based on your concurrency setting.Forking copies the current process into a new one, but in Python it kills off all but the thread doing the forking. All others are stopped. So the threads that LaunchDarkly starts up during initialization (e.g., EventDispatcher or StreamingUpdateProcessor) end up "defunct" or unpredictable.The Locks used within LaunchDarkly are thread independent, but because threads are killed off in the child, you end up with an invalid state and can’t trust things will work.Further, LaunchDarkly recommends a post fork hook to initialize the client.However, our Django application uses asgi, which doesn't currently have this hook. This is our current LaunchDarkly configuration launch_darkly.py:The LDClient class allows us to ignore new instantiations of the ldclient library if it's already been loaded.And the general use is:After a lot of bashing through walls, aka iterative development, we discovered two things:There was a module-level instantiation of the LaunchDarkly client that was causing the library to initialize before the fork. Basically, the above code, but instead:So that code was removed / refactored.In our celery.py code, we added a worker_process_init hook to initialize the library properly. This ensures that when the celery workers fork, there is definitely a ldclient ready to go for any code that requires it.To aid in future discovery and debugging, we also created a celery task that we can call on the fly to make sure things are working:Lastly, we will likely iterate on the LDClient class to deal with issues regarding the fork on the fly.Let me know if this helps you in your code, or sparks any ideas for you!Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well Confirm For further actions, you may consider blocking this person and/or reporting abuse Foteini Savvidou - Jul 5 Dhruv Joshi - Jun 30 Alireza Razinejad - Jun 5 STEVE - Jul 4 Once suspended, pjhoberman will not be able to comment or publish posts until their suspension is removed. Once unsuspended, pjhoberman will be able to comment and publish posts again. Once unpublished, all posts by pjhoberman will become hidden and only accessible to themselves. If pjhoberman is not suspended, they can still re-publish their posts from their dashboard. Note: Once unpublished, this post will become invisible to the public and only accessible to PJ Hoberman. They can still re-publish the post if they are not suspended. Thanks for keeping DEV Community safe. Here is what you can do to flag pjhoberman: pjhoberman consistently posts content that violates DEV Community's code of conduct because it is harassing, offensive or spammy. Unflagging pjhoberman will restore default visibility to their posts. DEV Community — A constructive and inclusive social network for software developers. With you every step of your journey. Built on Forem — the open source software that powers DEV and other inclusive communities.Made with love and Ruby on Rails. DEV Community © 2016 - 2023. We're a place where coders share, stay up-to-date and grow their careers.