AI Vision Capabilities for Home Assistant

Leverage the power of multimodal LLMs to automate your smart home.

Get started View on GitHub

Endless possibilities for automations

Notifications

Stay in the loop

Analyzes camera feeds and Frigate events to keep you updated.

Has my package been delivered yet?
Yes, a courier was seen carrying a package towards your house at 1:14 PM. It looks like your package has been delivered!

Insight

Just ask

LLM Vision builds a detailed timeline of events around your home, so you know what happened and when. You can even ask it about a specific event.

Speed

Optimized

Lightweight local preprocessing automatically picks the right frames to analyze.

Providers

Supports most AI Providers

Integrate any AI provider you already use, or host your own LLM for maximum privacy.

Usage

Easy to use

Get started with LLM Vision with just a few simple lines of code.

How it works

Overview

LLM Vision is a Home Assistant integration that can analyze images, videos, live camera feeds and Frigate events using the vision capabilities of multimodal LLMs.

The integration exposes 4 services:

  • image_analyzer

    Analyzes one or multiple images from different camera or image entities or image files.

  • video_analyzer

    Extracts and analyzes frames from a video file or frigate event.

  • stream_analyzer

    Captures and analyzes frames from camera entities.

  • data_analyzer

    Analyzes and updates a Home Assistant entity with data extracted from an image based on your prompt.

Frames are fetched through Home Assistant's Camera API. When using video- or stream analyzer, frames are preprocessed locally and only frames with enough movement are sent to the provider. This ensures low latency as well as lower token usage.

  • speed_camera Use any camera. If you can view the camera stream in Home Assistant, it'll work with LLM Vision without any additional setup.
  • video_file Handles files too. Use image or video files stored locally as reference for people or actions.
  • bolt Low token usage. To improve latency and reduce the amount of tokens used, the built-in downscaler can reduce resolution while retaining details.

Events

Know what happened

LLM Vision can remember events so you can ask about them later. A conversation agent such as Extended OpenAI Conversation is required.

  • lock You are in control. Events are only stored if 'remember' is set to true. If 'important' is enabled, events will be classified automatically and only remembered if deemed important.
  • hard_disk Stored locally. Once analyzed, events never leave your Home Assistant instance. For the most private setup, a local provider such as Ollama or LocalAI is recommended.
  • automation Integrates seamlessly. Events are exposed as calendar entities within Home Assistant so you can integrate LLM Vision Events into your dashboard or automations.
Has my package been delivered yet?
Yes, a courier was seen carrying a package towards your house at 1:14 PM. It looks like your package has been delivered!

Data Analyzer

Turn images into data

LLM Vision can extract data from images and update Home Assistant entities.

  • code_off Easy to use. Simply prompt the model what data to extract, provide a source and an entity to update and LLM Vision will handle the rest.
  • input Multiple Sources. Use a camera, image entity or image file as input. Video input is not yet supported.
  • devices Multiple entity types. Currently supports these entity types:
      • number and input_number
      • text and input_text
      • select and input_select
      • input_boolean
Events
arrow_downward_alt
Events

Supported Providers

LLM Vision connects to these AI providers* through their API

grid_view Compare Providers

*LLM Vision is not endorsed by or associated with any of the listed providers. All trademarks are property of their respective owners.

Frequently Asked Questions

What is LLM Vision?

LLM Vision is a Home Assistant integration that can analyze images, videos, live camera feeds and frigate events using the vision capabilities of multimodal LLMs.

How do I install LLM Vision?

You can install LLM Vision via HACS using the GitHub repository URL. For more details, visit the LLM Vision documentation.

Is LLM Vision free to use?

While the integration itself is free, depending on your provider you'll pay per query. For more information on pricing, visit Provider Comparison.