Posted on Aug 14 Originally posted on: scapegoat.devI am notoriously distracted and both my long and short term memories are less reliable than a hobbyist's kubernetes cluster. After 20 years of python, I still can't remember what the arguments to split are. Thankfully, these days are now over, as I can ask a just as clueless LLM to answer the myriad of questions I would otherwise have to google everyday (please don't use google, use kagi, you'll thank me later). Being part of a secret society of robo-scientists, I have been building some visionary software over the last couple of months, drawing inspiration from many sources, because writing software is about being in touch with the universe.More prosaically, I have been writing some LLM-powered Command line tools, as many people have before me and many people will after my bones are dust and my ideas forgotten. All the tools I present in this blog post have been written in a single day, because that's the world we now live in.Here is a little taste of the things that are possible with these tools. All of these examples were built in a couple of hours altogether. By the end of the article, you will be able to build them too, with no code involved. You might see a a mix of pinocchio commands and llmX commands. The llmX commands are just aliases to pinocchio prompts code X, the llmX-3.5 are aliases to use the gpt-3.5-turbo model (pinocchio prompts code X --openai-engine gpt-3.5-turbo).Let's start with a simple interview question: reverse a string in go.Too lazy for manpages? Extremely reliable LLM to the rescue!Some more reliable, trusted help.Recording these little screencast videos is tedious, let's get some help.What the heck did I do 2 months ago?Too lazy to read API docs? Just paste a curl request in there! Documentation is for chumps!I'm sure this won't miss a single thing. It sure is more annoying than my colleague Janine.But don't be so blunt.Tufte said slides shouldn't have bullet points. We disagree.In order for you, dear reader, to follow along on this adventure, you first need to install pinocchio. Nothing could be easier as we provide binaries for linux and mac (x86 and arm), as well as RPM and deb packages. All the GO GO GADGETS are also published on homebrew and as docker images, if that's your jam.After configuring your OpenAI API key, you should be all set to go. For more precise instructions, check out the installation guide.Over the last couple of months, like every other developer out there, I have been working on a LLM framework that will make it trivial to build AMAZING CRAZY STUFF.The core architecture of this very advanced framework boils down to:(This is why they call me senior engineer).The framework's claim to fame, besides having a cool name (geppetto is a client for GPT and pinocchio, the puppet borne of geppetto's woodworking, is the CLI tool used to run applications built with the framework. Of course, LLMs being what they are, pinocchio often lies).Jokes aside, general consensus amongst the GO GO GOLEMS has it that geppetto and pinocchio are pretty cool. They are both based on the glazed framework, more precisely based on the Command abstraction. This groundbreaking abstraction of a "general purpose command" consists of:Most of the actual engineering in glazed focuses on:After handling \r\n and \n, converting between strings and integers is the second hardest engineering problem known to man. This is why this framework is senior principal staff engineer grade.For example, the flags for the go command shown in the first video look like this:glazed provides a variety of standard types: strings, list of strings, objects, strings from files, booleans, integers, you name it.These declarative flags as well as layers added by the application all get wired up as CLI flags. Because this is user-first development, the output is colored.If we were to use something like parka, these flags would be exposed as REST APIs, HTML forms and file download URLs. For example, sqleton can be run as a webserver for easy querying of a database.But that is a story for another time.Now that we know how to define a command's input parameters, let's get to what a geppetto command is actually made of. A geppetto command:Geppetto supports specifying functions too, but I've never used them because I try to use boring technology, not the latest fancy shiny new thing.None of this requires massive amounts of clever computation, which is why we can also fit all of this into the same YAML file. For example, the go command from before looks like this:You can see here how we we first specify the OpenAI API parameters: Of course, all these settings can be overridden from the command line.After the (omitted) flags definition from before, we get to the special sauce:While what pinocchio does borders on magic (as all things having to do with LLMs), it is not too complicated to follow:The nice part (the really nice part) about being able to create "rich" command line applications using glazed and geppetto is that you can now experiment with prompt engineering by using command line flags instead of having to write custom test runners and data tables. This simple example-driven template is usually enough to reproduce most papers from 2021 and 2022, as shown by various datasets used in the Chain Of Thought paper.And of course, it is entirely possible to generate GeppettoCommands using GeppettoCommands, as shown by this example which can be used to generate SqletonCommands (the same concept, but for SQL).Using a GeppettoCommand that generates GeppettoCommands to generate itself risks immanentizing the eschaton and bringing singularity into being, so use these new-found powers with caution.Let's imagine that we want to generate SqletonCommands using the schema from a table in the database, but we are so lazy that we don't want to copy that schema. Wouldn't it be nice if we had a SqletonCommand to introspect a schema?To which the LLM responds:Here is an example of a command that outputs the schema of a table in MySQL using the provided YAML template:In this example, the tables flag accepts a list of table names, the tables_like flag accepts a list of table name patterns to match using the LIKE operator, the columns flag accepts a list of column names, the columns_like flag accepts a list of column name patterns to match using the LIKE operator, and the type flag accepts a column data type.The query uses the INFORMATION_SCHEMA.COLUMNS table to retrieve the schema information. The WHERE clause is dynamically generated based on the provided flags. The ORDER BY clause orders the results by the table name.Note that the sqlStringIn function is used to generate the IN clause for string values, and the last function is used to determine if it is the last element in a list during iteration.Storing that command as mysql schema, we can now easily introspect various tables, leveraging all the affordances provided by glazed.Or slightly smaller:We can easily use our new mysql schema command to create new SqletonCommands, say one to query our giftcards table.To which the LLM responds:Here is the updated YAML template for querying the giftcards table:In this template, I have added flags for the different columns in the giftcards table and modified the WHERE clause to include the new flags. I have also added support for the codes_like flag, which takes a list of giftcard codes and generates a filter statement with multiple LIKE queries joined by OR. The order_by flag is used instead of sort_by to specify the order of the results.We can now of course run this command and benefit from all the cool glazed bells and whistles.Once you get ahold of such powerful AI programs^W^W^Wcrude word templates, you can stash them in what is called a "repository". A repository is a directory containing YAML files (plans exist to have repositories backed by a database, but that technology is not within our reach yet). The directory structure is mirrored as verb structure in the CLI app (or URL path when deploying as an API), and the individual YAML represent actual commands. These repositories can be configured in the config file as well, as described in the README.You can also create aliases using the --create-alias NAME flag. The resulting YAML has to be stored under the same verb path as the original command. So, an alias for prompts code php will have to be stored under prompts/code/php/ in one of your repositories.Let's say that we want to get a rundown of possible unit tests:By storing the result as suggest-unit-test.yaml in prompts/code/professional, we can now easily suggest unit tests:Of course, all these commands can themselves be aliased to shorter commands using standard shell aliases.An earth-shattering consequence of this heavenly design is that you can add repositories for various GO GO GADGETS such as sqleton, escuse-me, geppetto or oak to your projects' codebases, point to them in your config file, and BAM, you now have a rich set of CLI tools that is automatically shared across your team and kept in source control right along the rest of your code.This is especially useful in order to encode custom refactoring or other rote operations (say, scaffolding the nth API glue to import data into your system). Whereas you would have to spend the effort to build a proper AST, AST transformer, output template, you can now write refactoring tools with basically the same effort as writing "well, actually, wouldn't it be nice if…" on slack.Remember the words that once echoed through these desolate halls:WHEN LIFE GIVES YOU STYLE GUIDES, DON'T NITPICK IN CODE REVIEWS. MAKE EXECUTABLE STYLE GUIDES! DEMAND TO SEE TECHNOLOGY'S MANAGER! MAKE IT RUE THE DAY WHERE IT GAVE US EXECUTABLE SPEECH! DO YOU KNOW WHO WE ARE? WE ARE THE GO GO GOLEMS THAT ARE GOING TO BURN YOUR CODEBASE DOWN.This can look as follows, to convert existing classes to property promotion constructors in PHP8.This I think is the most exciting aspect of LLMs. They make it possible to build ad-hoc tooling that is able (even if stochastically hitting or missing) to do rote but complex, ill-defined, time-consuming work that is very specific to the problem and situation at hand. I have been using tools like the above to cut through unknown codebases, add unit tests, clean up READMEs, write CLI tools, generate reports, and much more.The ones I built today are quite useful and make for cool demos, but the real value comes lies in being able to do very fuzzy, ad-hoc work that needs to be repeated at scale.I hope to make it easier to share and package these custom prompts and slowly start advertising GO GO GOLEMS and its ecosystem more, in order to get community contributions.A second project that is underway is building agent tooling. Single prompt applications are useful but often suffer from limited context sizes and lack of iteration. I have been working on a "step" abstraction that should make it possible to run complex agent workflows supporting iteration, error handling, user interaction, caching.A third project underway is a "prompt context manager" to make it easy to augment the LLM applications with additional information, coming either from external documents, live queries against a codebase, external APIs, saved snippets and summaries from past conversations, etc…Finally, I have a currently closed-source framework that makes it possible to deploy any glazed command (i.e., a single YAML file) as a web API, a lambda function or a WASM plugin. I hope to be able to port this part to opensource, as it makes the tools exponentially more useful.Templates let you quickly answer FAQs or store snippets for re-use.I strongly believe that ad-hoc, custom, personal tools powered by large language models is going to be an important paradigm for future use. What do you think? Do you build your own tools for LLMs instead of just chatting with GPT directly? Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well Confirm For further actions, you may consider blocking this person and/or reporting abuse Olabisi Olaoye - Jul 30 Velda Kiara - Jul 31 Sajeeb Das Shuvo - Aug 1 Ayyash - Aug 1 Once suspended, wesen will not be able to comment or publish posts until their suspension is removed. Once unsuspended, wesen will be able to comment and publish posts again. Once unpublished, all posts by wesen will become hidden and only accessible to themselves. If wesen is not suspended, they can still re-publish their posts from their dashboard. Note: Once unpublished, this post will become invisible to the public and only accessible to Manuel Odendahl. They can still re-publish the post if they are not suspended. Thanks for keeping DEV Community safe. Here is what you can do to flag wesen: wesen consistently posts content that violates DEV Community's code of conduct because it is harassing, offensive or spammy. Unflagging wesen will restore default visibility to their posts. DEV Community — A constructive and inclusive social network for software developers. With you every step of your journey. Built on Forem — the open source software that powers DEV and other inclusive communities.Made with love and Ruby on Rails. DEV Community © 2016 - 2023. We're a place where coders share, stay up-to-date and grow their careers.