AI Vision for Home Assistant
Leverage the power of multimodal LLMs to automate your smart home
LLM Vision
Endless Possibilities for Automations
Notifications
Stay in the loop
Analyzes camera feeds and Frigate events in real time to keep you updated.
Insight
Just ask
LLM Vision builds a detailed timeline of events around your home, so you know what happened and when. You can even ask it about a specific event.
Optimized
Image Pipeline
Advanced computer vision algorithms pick the most relevant images to analyze and optimize them for inference. The image pipeline is designed to take advantage of hardware acceleration and reduce latency.
Providers
Powerful Integrations
Integrate any AI provider you already use, or host your own LLM for maximum privacy.
Usage
Easy to use
Get started with LLM Vision with just a few simple lines of code.
Overview
Actions
LLM Vision is a Home Assistant integration that can analyze images, videos, and
camera streams using multimodal LLMs.
It exposes four actions that are simple
to use, yet enable powerful automations.
Analyzes one or multiple images from different camera or image entities or image files.
- videocam Use any camera. If you can view the camera stream in Home Assistant, it'll work with LLM Vision without any additional setup.
- video_file Handles files too. Use image or video files stored locally as reference for people or actions.
- bolt Low token usage. To improve latency and reduce the amount of tokens used, the built-in downscaler can reduce resolution while retaining details.
Extracts and analyzes frames from a video file or Frigate event.
- speed_camera Use any camera. If you can view the camera stream in Home Assistant, it'll work with LLM Vision without any additional setup.
- video_file Handles files too. Use image or video files stored locally as reference for people or actions.
- bolt Low token usage. To improve latency and reduce the amount of tokens used, the built-in downscaler can reduce resolution while retaining details.
Captures and analyzes frames from camera entities.
- speed_camera Use any camera. If you can view the camera stream in Home Assistant, it'll work with LLM Vision without any additional setup.
- video_file Handles files too. Use image or video files stored locally as reference for people or actions.
- bolt Low token usage. To improve latency and reduce the amount of tokens used, the built-in downscaler can reduce resolution while retaining details.
Analyzes and updates a Home Assistant entity with data extracted from an image based on your prompt.
- speed_camera Use any camera. If you can view the camera stream in Home Assistant, it'll work with LLM Vision without any additional setup.
- video_file Handles files too. Use image or video files stored locally as reference for people or actions.
- bolt Low token usage. To improve latency and reduce the amount of tokens used, the built-in downscaler can reduce resolution while retaining details.
Usage Examples
Discover What's Possible with LLM Vision
Explore community-built automations, scripts, and more—powered by LLM Vision.
Visit Gallery-
mic
Ask Assist - Integrate LLM Vision with Home Assistant Assist to ask anything about the surroundings of your home.
-
thunderstorm
Weather Alerts - Receive AI-powered storm analyses extracted from your local weather radar.
Events
Know what happened
LLM Vision can remember events so you can ask about them later. A conversation agent such as Extended OpenAI Conversation is required.
- lock You are in control. Events are only stored if 'remember' is set to true. If 'important' is enabled, events will be classified automatically and only remembered if deemed important.
- hard_disk Stored locally. Once analyzed, events never leave your Home Assistant instance. For the most private setup, a local provider such as Ollama or LocalAI is recommended.
- automation Integrates seamlessly. Events are exposed as calendar entities within Home Assistant so you can integrate LLM Vision Events into your dashboard or automations.
Person seen
08:52 AM
White SUV seen
08:55 AM
Courier seen
10:11 AM
Data Analyzer
Turn images into data
LLM Vision can extract data from images and update Home Assistant entities.
- code_off Easy to use. Simply prompt the model what data to extract, provide a source and an entity to update and LLM Vision will handle the rest.
- input Multiple Sources. Use a camera, image entity or image file as input. Video input is not yet supported.
-
devices Multiple entity types. Currently supports these entity types:
numberandinput_numbertextandinput_textselectandinput_selectinput_boolean
Main Gate
Open
Supported Providers
LLM Vision connects to these AI providers out of the box.
Providers that implement an OpenAI compatible API endpoint can also be used. Additional setup
may be required.
LLM Vision is not endorsed by
or
associated
with any of the listed providers. All trademarks are property of their respective
owners.
New
Preview Card
The new Preview Card lets you see the most recent event at a glance. Tap it to see even all the details. Includes all the same advanced filters from the Timeline Card.
Customizable
Timeline Card
A detailed timeline of events, with smart categories and filters. Tap an event for more details.
Translations
Speaks your language
Currently supports English, German, Dutch, French, Spanish, Portuguese, Italian, Polish, and Swedish. More languages are added frequently.
Frequently Asked Questions
What is LLM Vision?
LLM Vision is a Home Assistant integration that can analyze images, videos, live camera feeds and frigate events using the vision capabilities of multimodal LLMs.
How do I install LLM Vision?
You can install LLM Vision via HACS using the GitHub repository URL. For more details, visit the LLM Vision documentation.
Is LLM Vision free to use?
While the integration itself is free, depending on your provider you'll pay per query. For more information on pricing, visit Provider Comparison.
Upgrade your smart home today
Get Started