Connecting P1 port to RPi and Loxone

Purpose of project

I recently upgraded our energy meter for electricity and gas at my home to a digital meter. The electrical meter has a P1 and S1 port. The S1 port (fast port) provides a raw data stream with in frequency of <1ms. The P1 port is slower and has an update frequency of about 1 second. The P1 port is therefore most suitable for home automation and energy management applications. The gasmeter sends its information back to the net operator via the electricity meter and can also be retrieved through the P1 port. 

The goal of this project will be to interface Loxone with the P1 port of the energymeter. It should be possible to visualize the electricity and gasmeter in the Loxone app. 

Considerations

• Minimize additional hardware needs

• I currently have already a Loxone installation and a Raspberry PI with Openhab for voicecontrol. Ideally this hardware can be used.

• I would like to be able to use the data also in other applications running in the cloud in a next project. 

Hardware selection

I checked with the Loxone helpdesk how they suggest the P1 meter can be best integrated. It is currently already possible to read the digital meters via the P1 port via various devices, for example, you have Youless, Negotica via modbus and also a KNX interface. There are probably more on the market. Previous devices work quite well. If you want to stick to basic information such as gas and consumption / delivery, then the Youless and works well is a simple solution that does not cost much. If you want that little bit more information, say all the info, then the Negotica P1 interface via modbus is the right device to use. This is also easy to integrate with us via Modbus TCP or RTU, templates are also available in the partner portal. This device may cost a bit more but offers the widest range of possibilities and can also be updated remotely so that new changes in the protocol remain supported.

Hardware option 1 - Youless

• https://www.youless.nl/hoewerkt.html

• Youless LS120 - https://www.youless.nl/winkel/product/ls120.html

• Costs about 89 EUR

• How to integrate Youless 120 to Loxone (Loxone training)

• Benefit - this solution allows to connect at relatively low cost and is able to directly connect to your network. Loxone is able to read it therefore out via TCP/IP. It is ideal if you don't already have a RPi or other hardware to connect to. 

Hardware option 2- Negotica via modus 

• I did not further explore these options. The reason is that the hardware cost would be high.

• Modbus Negotica would cost north of 300 EUR. Modbus TCP would be the preferred way to communicate.

• (Modbus RTU is also possible, but then a Modbus extension is required in Loxone -- setting you back another 300 EUR )

Hardware option 3 - KNX interface

Also KNX interfaces exist for the P1 port. Since the old Loxone miniservers have a KNX interface this could be interesting, but the drawback is that you still would need KNX programming software and a KNX lite license also will set you back 200 EUR. 

In this article by JensD it is explained how to interface with the P1 port directly. The things I would need to read this out would be  An RJ11 connector and an UTP cable

Hardware option 4 - A Raspberry PI with network card

Using a Raspberry PI to read out the P1 meter seems to be the most optimal solution. It will give most flexibility to use the signal in using different protocols and it will be most cost effective. 

One option would be to make the cable to read the serial signal directly from the P1 port. The other options is to buy a USB-to-Serial converter. 

If you would like to make your own cable here are some good references which contain the soldering scheme and code to read out the serial signal. 

• Circuit required to invert the serial signal and picture how to connect the P1 port to the UART RX port on RPi: https://developers.nl/blog/10/uitlezen-nederlandse-slimme-meter-p1-poort-over-netwerk

• Make your own Cable P1 to USB : https://www.domoticz.com/forum/viewtopic.php?t=4970

• https://nielsriekert.nl/posts/smart-meter-p1-port/

• Code to read from serial port on RPI : https://pimylifeup.com/raspberry-pi-serial/

Making my own cable would be interesting, but would certainly be more time consuming. Therefore, I concluded that buying a 10m P1-to-USB cable was my best option. The cost is only 24 EUR and this would allow me to use my existing Raspberry PI which I also use for my Openhab-GoogleHome voice control. 

• https://www.sossolutions.nl/slimme-meter-kabel-p1-kabel-10-meter

• https://webshop.cedel.nl/Slimme-meter-kabel-P1-naar-USB

Acitivating the port

Before it is possible to interface with the P1 port, it needs to be activated on the Fluvius website. It is possible to do this here --> https://mijn.fluvius.be/poortbeheer/

It is possible to phyiscally measure whether the port is active by measuring the voltage between the two outer pins of the P1 port. If there is 5V between these two pins, the port is activated. 

Programming the Rapberry PI

Writing the code 

To create the code I have used Visual Studio code and set up a virtual environment 

1. Set up virual enviroment

   • python -m venv .venv

   • add .venv to .gitignore

2. Install dependencies to the virtual environment 

   • Normally running of scripts is blocked in Visual Studio Code, here is how you can run the acitivation script

     • Run VSCode as administator and open a new terminal

     • >> Get-ExecutionPolicy --> will be ' restricted' 

     • >> Set-ExecutionPolicy RemoteSigned

     • >> .venv/scripts/activate

   • If all is well you should now see 

     • (.venv) PS <<Path/To/Your/VirtualEnvironment/

3. Next we are going to install the dependencies to the virtual enviroment

   • >> pip install serial

   • >> pip install flask

4. We will make a Requirements.txt file to have the versioning saved in the project.

   • >> pip3 freeze > requirements.txt 

5. Run code using CTRL+SHIFT+P and select the venv in Python: Select Interpreter

Reading the signal from the P1 serial

• A very good blog to start from is the script made by Jens Depuydt to parse the P1 meter. Detailed blog can be found here. This script is based on the 

• Needed to start : Installed Raspian on a SDCard and enabled SSH Access and enabled serial port hardware (using raspi-config) 

• Installed the dependencies and the github repository
  >>> git clone https://github.com/jensdepuydt/belgian_digitalmeter_p1
  >>> sudo apt update
  >>> sudo apt install python3-serial python3-crcmod python3-tabulate
  >>> sudo chmod o+rw /dev/ttyUSB0 #setting the rights to read from the USB port
  >>> python3 -m serial.tools.miniterm /dev/ttyUSB0 115200 --xonxoff #reading from the port

•  If the cable is not yet connected (or if there is a problem with the cable or the P1 port is not yet opened) you will get an error stating '' /dev/ttyUSB0 not found ''
  As soon as the cable is connected data will start flowing in. 

• => /dev/ttyUSB0 should appear once the P1 port is connected, this will be for when the new cable has arrived. 

Communicating between the Raspberry PI and Loxone using REST API

My first idea was to use MQTT as an interface to Loxone. In principle using an MQTT broker to publish the results and using MQTT to Loxone could be considered. However I did not find a way to interface directly to an MQTT broker using Loxone and the only workable solution would require Loxberry to be installed on a Rasberry PI to allow to interface. This is quite cumbersome. 

Loxone is able to best communicate with external data sources using HTTP REST API. I will therefore create a simple API using flask to publish the meter results to Loxone using a REST API. 

As a test to see what comes out I have used ChatGPT using the following command to generate the framework of the code:
" Please write python code to run on a raspberry pi, read data from a P1 energy meter from the usb serial port, parse the data using regex and publish the data using a simple REST API server via a json. " 

This is what came back:

import serial

import re

from flask import Flask, jsonify

# Open serial connection to P1 energy meter

ser = serial.Serial('/dev/ttyUSB0', 9600)

# Initialize Flask app for REST API

app = Flask(__name__)

@app.route('/energy', methods=['GET'])

def get_energy_data():

    # Read data from P1 energy meter

    data = ser.readline().decode('utf-8')

    # Use regular expressions to parse data

    match = re.search("(?P<power>[0-9.]+) kW", data)

    power = float(match.group("power"))

    # Return data as JSON

    return jsonify(power=power)

if __name__ == '__main__':

    app.run(host='0.0.0.0', port=5000)

This is quite a good starting point and I will further modify this to get the final code.

Code with REST API and Serial reader from P1 meter

• Github repository can be found here :  https://github.com/bramvg87/P1LoxoneInterface  

Install code to Raspberry PI and configure to run automatically 

• To clone the github repository to your raspberry PI (assuming it is a private repository) you first need to generate a personal access token 

  • Generate a personal access token on github 

  • Clone the code from the repository to the Raspberry PI :

    • git clone https://<pat>@github.com/<your account or organization>/<repo>.git

  • Once the repository has been made public you can run the following command on Rapsberry PI

    • git clone https://github.com/bramvg87/P1LoxoneInterface

  • After this updates can be installed using

    • git pull origin main

• Navigate to the folder with the installed repository and install the requirement packages

  • pip install -r requirements.txt

• To add running the code on the background when starting raspberry PI I have found 2 options :

  • Option 1 : Using Cron

    • Open : 

$ crontab -e

• • • Then add the line:

@reboot /home/vivek/bin/python_script.py

• • • Save and close, and then run

# update-rc.d cron defaults

• Option 2 : Using Systemd (source)

• Create a .service file for your service as below:

sudo nano /lib/systemd/system/P1Reader.service

• Add the below text:

[Unit]

Description=P1 Reader Service

After=multi-user.target

[Service]

Type=idle 

ExecStart=/usr/bin/python /home/pi/P1meter/P1LoxoneInterface/main.py 

User=pi 

[Install] 

WantedBy=multi-user.target

• Save and exit the nano editor (by pressing Ctrl+X).

• The permission on the unit file needs to be set to 644 :

sudo chmod 644 /lib/systemd/system/P1Reader.service

• Reload the system manager configuration by using the following command:

sudo systemctl daemon-reload

• Start the service using the following command:

sudo systemctl start P1Reader.service

• Stop the service using the following command:

sudo systemctl stop P1Reader.service

• You can enable the service to start at boot as below:

sudo systemctl enable P1Reader.service

• You can check the service status using:

sudo systemctl status P1Reader.service

The P1 meter is now connected to my raspberry PI, the code runs in the background and I'm able to retrieve the data from the P1 meter on my phone running Raspberry-IP:5000/P1

Interfacing with Loxone

First I upgrade my miniserver to version Loxone config 13.1.11.17, as this version contains the most recent energy flow monitoring block.

You can read values from a json file into the miniserver, you need a Virtual Http Input. In the Virtual HTTP Input, the URL needs to be set to the Raspberry-IP:5000/P1

Next, for each meter input a Virtual HTTP Input command needs to be created. The Command Recognition field needs to be populated with the identifier from the meter, for example for the 'All phases consumption' this will look as follows: 

\i"All phases consumption",\i\v

 the \iText\i encapsulates the value to look for in the file and the \v indicates that the miniserver should take the numeric value following the \v. The value ends up in your Virtual Http Input variable and you can use it however you like from there. Don't forget to add the UoM in the Display>Unit setting. 

I have added the following values. 

• Rate 1 (day) - total consumption

• Rate 2 (night) - total consumption

• Rate 1 (day) - total production

• Rate 2 (night) - total production

• All phases consumption

• All phases production

• L1 consumption

• L2 consumption

• L3 consumption

• L1 production

• L2 production

• L3 production

• Gas consumption

I used the Meter object to create a meter for the total electricity consumption & production (don't forget to sum the day and night totals). 

If you just want to see trend data this is sufficient.
You can now download the code to Loxone

Add it to production RPi running OpenHabian

Now that it works as a next step I move the code to my Raspberry PI running Openhab. The first thing I confirmed it that the port used for the Flask REST API is still unused. This is the case. As a recap following instructions were used to deploy the P1LoxoneInterface code using SSH.

mkdir P1meter

cd P1meter

git clone https://github.com/bramvg87/P1LoxoneInterface

cd P1LoxoneInterface

pip install -r requirements.txt

#autostart

sudo nano /lib/systemd/system/P1Reader.service

[Unit]

Description=P1 Reader Service

After=multi-user.target

[Service]

Type=idle 

ExecStart=/usr/bin/python /home/openhabian/P1meter/P1LoxoneInterface/main.py 

User=openhabian 

[Install] 

WantedBy=multi-user.target

sudo chmod 644 /lib/systemd/system/P1Reader.service

sudo systemctl start P1Reader.service

sudo systemctl status P1Reader.service

sudo systemctl enable P1Reader.service

192.168.178.28:5000/P1

Backup SDCard

Now the next thing I wanted to do is make a backup of my SDCard, so that in case anything happens to my Raspberry PI I can more easily redeploy Openhab and the code I installed. 

I will be using this manual from pimylifeup.com

1) Safely stop the Raspberry PI

sudo shutdown now

2) Move SD Card to PC with SD-card reader

3) Install Win32diskimager

4) I was going to write, now create the image file, but as it turns out nothing is that easy. Win32diskimager did not open on my Windows 11 machine --> Quitting google drive seemed to do the trick.

5) Now create the image file. This little manouver is going to cost us 27 minutes.

Next steps

• Test Loxone visuals

• Send P1 data to database (SQLite or PostgreSQL & test to make visualization PowerBI)

• Test NodeRed visual

  • http://domoticx.com/p1-poort-slimme-meter-data-naar-node-red-mqtt/

• Interface with Openhab

  • Connect to P1 port and communicate with MQTT broker + instructions to communicate with Openhab : https://github.com/tribp/DSMR-Fluvius-MQTT-Openhab