Building a RS485-Ethernet-Modbus-tunnel

 The plan sounds simple: install solar panels and an inverter in the garage, and enjoy the sun delivering power for the house. But how to do this if 1. the garage is 50m away from the house and the only connection is a 5 wire power cord and an ethernet connection and 2. no power should be fed back into the outside power network. The latter requires a measurement device at the main power and a connection between this device and the inverter. Typically, these devices are connected via an RS485 connection, and the SDM630 meter that I am trying to connected uses the Modbus protocol. So, one possible solution to the problem of connecting the inverter to the powermeter is to use 2 wires of the ethernet cable for a direct connection. But that means to drop gigabit support, because for a gigabit connection all wire pairs are actually used. Also, it is not very elegant to hack the cable to split a wire pair from the rest.

So, what would be a more elegant solution? There are commercial RS485-to-ethernet bridges that are advertised for exactly this scenario. But they are really expensive. And there are cheap RS485-/Modbus-to-Ethernet bridges, f.e. the USR-DR302. They support many modes and I was hoping that it is possible to use them for a simply RS485/Modbus tunnel. Interestingly, I could not find a single article about such a setup, and the manual is no big help either. So I tried, using an ESP32 with a RS485 shield simulating a measuring device, and a computer with an RS485-USB adapter simulating the inverter. First, I connected the devices directly, then I tried to put the DR302 devices as tunneling devices in between. To do this, the DR302 which is connected to the meter is running in "TCP server" mode, while the DR302 connected to the computer is running in "TCP client mode", connecting to the IP address and port of the TCP server. Also, I selected  the "Modbus TCP" to use modbus TCP as an intermediate protocol. First try, switch on, and: success! This setup works exactly as hoped!

NGINX as a tls-enabled socket tunnel

The use-case of a TCP-tunnel with TLS might be somewhat esoteric, but nonetheless useful in certain situations. Imagine having two servers that need to talk to each other via TCP sockets, but the application should not be concerned with the extra overhead and configuration hassle of transport layer security. This is where NGINX comes into play. We use NGINX as a reverse proxy on both servers, effectively tunneling the TCP socket from one machine to the other via a TLS secured connection. The server initiating the connection is "Server A".

NGINX configuration on Server A:

stream {
       server {
               listen 127.0.0.1:10000;
               proxy_pass serverB:443;
               proxy_ssl  on;
               proxy_ssl_certificate client.cert.pem;
               proxy_ssl_certificate_key client.key.pem;
       }
}

NGINX configuration on Server B:

stream {
       server {
               listen 443 ssl;
               proxy_pass localhost:10000;
               ssl_certificate server.cert.pem;
               ssl_certificate_key server.key.pem;
               ssl_session_cache  builtin:1000  shared:SSLS:10m;
               ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
               ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
               ssl_prefer_server_ciphers on;
               ssl_client_certificate ca-chain.cert.pem;
               ssl_verify_client on;
               ssl_verify_depth 2;
       }
}

The used certificates can be self-signed, Server A uses client certificates that are signed by the ca-chain used in the configuration of Server B. 

NWP tutorial @32c3

I am going to give a Numerical Weather Prediction (NWP) tutorial at the 32nd Chaos Communication Congress in Hamburg. It will be mainly about installing and running WRF, with some excursions to related topics here and there. And I am really excited about this!

Challenges

Is it just me, or is there a surge of "challenges" / "competitions" / "contests" of all kinds and colors lately? It appears to be a very cheap way to get people working for you (or for your objectives). So you need to boost your business and you don't have good ideas yourself and/or the capabilities to realize them? No problem, start a "hey-give-me-your-ideas"-challenge! Hopefully a number of good ideas will emerge and eventually you can take advantage of many of those ideas while "paying" only the winner, who probably spent a lot more time on it than you have to pay him for (if there is any price money involved at all).
Don't get me wrong, I do like the general idea of these challenges, and in particular I like participating - for example in the yearly Copernicus Masters Challenge. But the balance between how much effort the participants put into it and the actual rewards needs to be fair. Anyway, just my two cents on that topic (and maybe this year I'll make it to the finals :-) ).

EMS/ECAM meeting

Last week I attended the EMS (European Meteorological Society) / ECAM (European Conference on Applications of Meteorology) meeting in Sofia, Bulgaria. In summary, it was a very nice meeting, I learned a lot and now I am even more convinced that I am on the right track developing the web-based model verification system.
The first part of the conference was a bit off target for me, but still very interesting - there were sessions about observations, climate & a lot about severe weather warnings (including communication and impact research - very, very interesting stuff). The second part of the conference was more about verification and model evaluation, which is also what my poster was about. The verification sessions were all very interesting and sparked a lot of new ideas... moreover, there is a "find new verifiation measures"-competition by the WMO working group - I'll probably give that a shot! Finally, there was the MesoVICT meeting, which is more of a "meta"-comparison of verification schemes; maybe I'll also take part in this to use it as a testbed for the verification system and the new verification measures that I've come up with.
Besides the lack of a continuous coffee supply chain, the conference was well organized, and I really enjoyed it! There is just one more thing that stroke me rather odd: some of the sessions (at least 2) turned out to be more-or-less one-man-shows, and in my opinion that just is not what a conference like this should be about: in one of the sessions the session chair already had 2 scheduled talks (which is already questionable), but finally turned out to give 5 (out of 6) talks in that very session.

Custom SRIDs in PostGIS

One small, but occasionally very important feature of PostGIS is the ability to deal with custom spatial reference systems (SRS, in PostGIS SRS are defined using an SRID). Given data in some obscure projection, it is not required to transform the data to some well-known SRS, but it is possible to store the data directly, provided that a custom SRID entry is stored into the spatial_ref_sys relation in PostGIS. The columns in that table are basically the SRID (anything > 100000 should be safe to use for custom projections) and the projection definition in two formats, namely well-known text and as a proj4 string. Note that certain proj4 strings cannot be properly expressed as well-known text (f.e. rotated poles), in that case the well-known text definition can be extended by EXTENSION["PROJ4","<some proj4 string> +wktext"]. Also note that it seems that the "-m" option in the proj4 string is ignored, so it is better to use an equivalent "+to_meter" option (but I haven't tested this in every detail).

KNN with PostGIS

This article is basically a note to my future self, in case I need to look up how to do KNN searches with PostGIS again. The technique is taken from the talk "Writing better PostGIS queries" by Regina Obe (given at FOSS4G 2014 in Portland).

The problem statement

Given a set of geometries G1, find the k nearest neighbours for each of these geometries from a second set of geometries G2. For simplicity, I assume G1 and G2 to be sets of points which are stored as simple POINT geometries in PostGIS. Note that G1 and G2 could be the same set, I will use this in the example code.

CREATE TABLE knntst
(
  gid serial NOT NULL,
  geom geometry,
  CONSTRAINT knntst_pk PRIMARY KEY (gid)
);
CREATE INDEX knntst_geom_gist
  ON knntst
  USING gist
  (geom);

INSERT INTO knntst (geom) VALUES ('POINT (0 0)'::geometry), ('POINT (1 0)'::geometry), ('POINT (0 1)'::geometry), ('POINT (1 1)'::geometry), ('POINT (3 0)'::geometry), ('POINT (4 0)'::geometry), ('POINT (3 3)'::geometry), ('POINT (4 5)'::geometry);

Using the KNN operator

There are many ways for solving this problem in PostGIS, but the trouble is to find a way that actually uses the spatial index and that does not require to set a maximum distance beforehand (and that provides the complete output, i.e. all k nearest neighbours for each of the points). The solution is to use the KNN operator "<->", which may be included in the ORDER BY clause. However, one argument to that operator needs to be constant in order to actually take advantage of the spatial index. So, for the given problem, a LATERAL join is required in the following way:

SELECT g1.gid, g2.gid, ST_AsText(g1.geom), ST_AsText(g2.geom), ST_Distance(g1.geom, g2.geom) FROM knntst AS g1 CROSS JOIN LATERAL (SELECT g3.gid, g3.geom FROM knntst AS g3 WHERE g1.gid!=g3.gid ORDER BY g3.geom <-> g1.geom LIMIT 3) AS g2

The output table contains the 3 nearest neighbours to each of the points in the source table, excluding the point itself.