Few of the answers given were concrete. So here’s my take.
I am able to run singleuser Continuwuity on a 8GB RAM Pi machine with 4 cores, and join many large rooms (around >=1000 users, although the number of homeservers in the room is a more suitable metric). It would use around 2GB RAM, but you can tune it for less (basically reduce cache values, but ask in the room for more advice).
After a few months the database hovers at around 2GB, because the database uses zstd compression by default. It’s not anyhow a major problem like Synapse, just don’t use HDD for storage and you should be fine.
For best experience, I also selfhost a dedicated caching resolver (unbound) for continuwuity. That takes like a few hundred more MBs of memory.
Given the fact you’d like to play around with it, a mid-tier VM/VPS (2CPU, 2GB RAM, 20GB SSD) is a reasonable starting choice. For a non-federating server, it can take a lot less resource than this.
If you want a non-federating LAN-only Matrix server, then STUN/TURN can be behind the NAT. Since you have Tailscale, STUN/TURN can also expose itself on the Tailscale VPN too. Just configure proper DNS records per-interface and you should be fine.
Since calls are p2p, the purpose of STUN is to determine a client’s (usually public) IP address, and TURN is to relay the connection if they can’t connect directly (i.e. behind NAT). If your clients are on the same LAN/VPN with unrestrictive firewalls then you might not even need any STUN/TURN altogether.