The Creature Agent is what makes my creatures feel like they’re aware of the world around them. It’s a Swift command-line tool that listens to MQTT topics from Home Assistant and uses an LLM to generate spoken responses. When something happens in the house — a door opens, motion is detected, the temperature changes — the agent picks up the event, decides if the creature should say something about it, and makes it happen.
This is part of the shared Swift package in the creature-console repo on GitHub. It’s built on the same Common library that the Creature Console uses.
The magic here is that the responses are contextual. The agent doesn’t just announce “the front door opened.” It sends the event to an LLM with a system prompt tuned to the creature’s personality, and the creature reacts in character. Beaky might say something silly about you leaving the door open. Mango might compare the door being open to his favorite database, MongoDB. Same event, different personality. 😍
How It Works
The agent connects to an MQTT broker and subscribes to a set of topics configured in a YAML file. Each topic can be associated with an “area” that has its own system prompt and cooldown period. When a message arrives on a subscribed topic, the agent:
- Checks if the cooldown for that area has elapsed (so the creature doesn’t comment on every motion event)
- Deduplicates repeated events
- Sends the event context to an LLM to generate a response
- Sends the response to the Creature Server to create an ad-hoc animation with speech
If the LLM is unavailable for any reason, there’s a fallback so the creature still says something rather than just sitting there silently.
LLM Backends
The agent supports both OpenAI’s API and local LLMs (via llama.cpp’s server). I mostly use local models since everything else in the system runs locally too, and it keeps the latency low. It maintains a conversation history so responses can build on what’s been said recently.
Configuration
Everything is driven by a YAML config file — the MQTT broker, which topics to watch, cooldown periods per area, which LLM to use, and the creature’s personality prompt. This makes it easy to run a separate agent instance for each creature, each with its own personality and set of topics it cares about.
Having creatures that react to things happening in the house is one of those features that really makes people do a double-take. It’s one thing to have an animatronic parrot that plays canned animations. It’s another thing entirely when it comments on you walking into the room. 💜