Endless possibilities for automations

Notifications

Stay in the loop

Analyzes camera feeds and Frigate events to keep you updated.

Has my package been delivered yet?

Yes, a courier was seen carrying a package towards your house at 1:14 PM. It looks like your package has been delivered!

Insight

Just ask

LLM Vision builds a detailed timeline of events around your home, so you know what happened and when. You can even ask it about a specific event.

Optimized

Image Pipeline

Advanced computer vision algorithms pick the most relevant images to analyze and optimize them for inference. The image pipeline is designed to take advantage of hardware acceleration and reduce latency.

Providers

Powerful Integrations

Integrate any AI provider you already use, or host your own LLM for maximum privacy.

example.yaml

action: llmvision.image_analyzer
data:
  message: Who's at the door?
  image_entity:
    - camera.front_door
  max_tokens: 50
  temperature: 0.2
  include_filename: false

Usage

Easy to use

Get started with LLM Vision with just a few simple lines of code.

How it works

Overview

LLM Vision is a Home Assistant integration that can analyze images, videos, live camera feeds and Frigate events using the vision capabilities of multimodal LLMs.

The integration exposes 4 services:

image_analyzer
Analyzes one or multiple images from different camera or image entities or image files.
video_analyzer
Extracts and analyzes frames from a video file or frigate event.
stream_analyzer
Captures and analyzes frames from camera entities.
data_analyzer
Analyzes and updates a Home Assistant entity with data extracted from an image based on your prompt.

Frames are fetched through Home Assistant's Camera API. When using video- or stream analyzer, frames are preprocessed locally and only frames with enough movement are sent to the provider. This ensures low latency as well as lower token usage.

speed_camera Use any camera. If you can view the camera stream in Home Assistant, it'll work with LLM Vision without any additional setup.
video_file Handles files too. Use image or video files stored locally as reference for people or actions.
bolt Low token usage. To improve latency and reduce the amount of tokens used, the built-in downscaler can reduce resolution while retaining details.

Events

Know what happened

LLM Vision can remember events so you can ask about them later. A conversation agent such as Extended OpenAI Conversation is required.

lock You are in control. Events are only stored if 'remember' is set to true. If 'important' is enabled, events will be classified automatically and only remembered if deemed important.
hard_disk Stored locally. Once analyzed, events never leave your Home Assistant instance. For the most private setup, a local provider such as Ollama or LocalAI is recommended.
automation Integrates seamlessly. Events are exposed as calendar entities within Home Assistant so you can integrate LLM Vision Events into your dashboard or automations.

Has my package been delivered yet?

Yes, a courier was seen carrying a package towards your house at 1:14 PM. It looks like your package has been delivered!

Data Analyzer

Turn images into data

LLM Vision can extract data from images and update Home Assistant entities.

code_off Easy to use. Simply prompt the model what data to extract, provide a source and an entity to update and LLM Vision will handle the rest.
input Multiple Sources. Use a camera, image entity or image file as input. Video input is not yet supported.
devices Multiple entity types. Currently supports these entity types:

arrow_downward_alt

directions_car

Cars in Driveway

Frequently Asked Questions

What is LLM Vision?

LLM Vision is a Home Assistant integration that can analyze images, videos, live camera feeds and frigate events using the vision capabilities of multimodal LLMs.

How do I install LLM Vision?

You can install LLM Vision via HACS using the GitHub repository URL. For more details, visit the LLM Vision documentation.

Is LLM Vision free to use?

While the integration itself is free, depending on your provider you'll pay per query. For more information on pricing, visit Provider Comparison.

AI Vision for Home Assistant