On first look, there is no open source code support for GT-AC5300. Tell me, if I am wrong about. There are routers with community or manufacturer support, you can find a list here
for start. You want custom stuff, you better pick up something from that list.
If you just pick up a router, without any developer support, and would build a custom firmware, that is not simple, but hell of a fight.