Use LTS from websocket API to fetch history for training models

[RudolfRendier]

Home Assistant has a websocket API that allows fetching the history of the long term statistics.
It offers data at a 5min, hour, day, week or month resolution.

Here's yesterday*, with hourly resolution for brevity (and as markdown):
*) Yesterday is also available in the regular recorder, but the retention for LTS is endless.

Is this something EMHASS could use to train ML-models?

Statistics Response:

Statistic ID	Start Time (UTC)	End Time (UTC)	Mean	Min	Max
sensor.solar_panels_total_output_power	2025-04-13 14:00:00	2025-04-13 15:00:00	1219.97	648.7	1779.2
sensor.solar_panels_total_output_power	2025-04-13 15:00:00	2025-04-13 16:00:00	747.781	321	1222.3
sensor.solar_panels_total_output_power	2025-04-13 16:00:00	2025-04-13 17:00:00	322.089	178.6	761
sensor.solar_panels_total_output_power	2025-04-13 17:00:00	2025-04-13 18:00:00	134.608	78.4	280.1
sensor.solar_panels_total_output_power	2025-04-13 18:00:00	2025-04-13 19:00:00	16.4212	1.5	125.4
sensor.solar_panels_total_output_power	2025-04-13 19:00:00	2025-04-13 20:00:00	0.290901	0	9.3
sensor.solar_panels_total_output_power	2025-04-13 20:00:00	2025-04-13 21:00:00	0	0	0
sensor.solar_panels_total_output_power	2025-04-13 21:00:00	2025-04-13 22:00:00	0	0	0
sensor.solar_panels_total_output_power	2025-04-13 22:00:00	2025-04-13 23:00:00	0	0	0
sensor.solar_panels_total_output_power	2025-04-13 23:00:00	2025-04-14 00:00:00	0	0	0
sensor.solar_panels_total_output_power	2025-04-14 00:00:00	2025-04-14 01:00:00	0	0	0
sensor.solar_panels_total_output_power	2025-04-14 01:00:00	2025-04-14 02:00:00	0	0	0
sensor.solar_panels_total_output_power	2025-04-14 02:00:00	2025-04-14 03:00:00	0	0	0
sensor.solar_panels_total_output_power	2025-04-14 03:00:00	2025-04-14 04:00:00	0	0	0
sensor.solar_panels_total_output_power	2025-04-14 04:00:00	2025-04-14 05:00:00	1.65232	0	28.2
sensor.solar_panels_total_output_power	2025-04-14 05:00:00	2025-04-14 06:00:00	149.33	28.2	276
sensor.solar_panels_total_output_power	2025-04-14 06:00:00	2025-04-14 07:00:00	310.03	189.5	583.7
sensor.solar_panels_total_output_power	2025-04-14 07:00:00	2025-04-14 08:00:00	684.022	303.3	1776.8

Here's some example python code to fetch the data for a my solar panels.
There's not much documentation on the API, and it might be subject to change.
I based it on the example found here:
home-assistant/core#88757

Here's the core-code:
https://github.com/home-assistant/core/blob/6d5c000e1fce94bc4ab5dcf453bde9a646a52737/homeassistant/components/recorder/websocket_api.py#L164

wss_uri = "wss://REDACTED:8123/api/websocket"
my_token = "REDACTED"

import asyncio
import websockets
import json
from datetime import datetime
from tabulate import tabulate


async def get_statistics():
    uri = wss_uri
    async with websockets.connect(uri) as websocket:
        # Authenticate with Home Assistant
        auth_message = {
            "type": "auth",
            "access_token": my_token
        }
        await websocket.send(json.dumps(auth_message))
        auth_response = await websocket.recv()
        print(auth_response)

        # Prepare the statistics request
        request_id = 36
        statistics_request = {
            "id": request_id,
            "type": "recorder/statistics_during_period",
            "start_time": "2025-04-13T13:48:18.341Z",
            "statistic_ids": ["sensor.solar_panels_total_output_power"],
            "period": "hour", # 5minute, hour, day, week, month
            "units": {"power": "W"},
            "types": ["mean", "min", "max"]
        }


        # Send the statistics request
        await websocket.send(json.dumps(statistics_request))
        # Handle the response
        while True:
            response = await websocket.recv()
            response_data = json.loads(response)
            if response_data.get("id") == request_id:
                print("Statistics Response:")
                headers = ["Statistic ID", "Start Time", "End Time", "Mean", "Min", "Max"]
                table_data = []
                for stat in response_data.get("result", {}).get("sensor.solar_panels_total_output_power", []):
                    row = [
                        "sensor.solar_panels_total_output_power",
                        datetime.utcfromtimestamp(stat.get("start") / 1000).strftime('%Y-%m-%d %H:%M:%S'),
                        datetime.utcfromtimestamp(stat.get("end") / 1000).strftime('%Y-%m-%d %H:%M:%S'),
                        stat.get("mean"),
                        stat.get("min"),
                        stat.get("max")
                    ]
                    table_data.append(row)
                print(tabulate(table_data, headers=headers, tablefmt="grid"))
                break


# Run the async function
asyncio.get_event_loop().run_until_complete(get_statistics())

[purcell-lab]

Thanks, this would be welcome for access to long term history data without relying on a large recorder database.

[davidusb-geek]

Same reply as in the community forum, just to keep track of this ;-)

LTS is not available to retrieve using the official HA API from which EMHASS is based on to fetch data.

Here are the docs for the core REST API: REST API | Home Assistant Developer Docs

And the supervisor API: Endpoints | Home Assistant Developer Docs

If this evolve, I would gladly support LTS. We need to ask this to HA devs.
For now the solution is to configure (as you said) the recorder to store more data that the default 10 days.
Personally I store 1 year of my data with a custom list of sensors to be stored, is quite a long list but my DB is always steady at 6-7 Gb