Conduit is in no way compact either. I tuned its caches because two gigs of ram seemed ridiculous for a single-user instance but I only got the mobile client sync lag as a result.

XMPP used to be so much nicer…

This is the best answer. Your router protects you from the outside, but a local firewall can protect you from someone prodding your lan from a hacked camera or some other IoT device. By having a firewall locally you just minimize the attack surface further.

Matrix is spectacularly cursed to the point of being unusable if you self-host it. The protocol is dumb enough to lock you out of rooms hosted on another server forever if anything goes wrong with the key rotation.