I think there are a few issues with it, the biggest being that with both mechanical and optical components you get the failure modes of both. If you're using a gear type thing for the detents, I think you should probably make it out of something less abrasive than PCB material. Maybe something like phenolic sheet. That can be cut by anyone with a CNC router, not just a PCB house. I'd also try different tooth profiles, to see what feels best(probably want a pointy tip instead of a plateau). As for the encoder part, I don't know how hard it is electrically, but maybe you could make a capacitive encoder, similar to what they use in digital calipers. I think that would be the most reliable option provided you want to make the encoder out of PCB material. I wonder if those capacitive touch ICs would work.
For another high reliability option, maybe replace the mechanical encoder with a ring magnet (multi pole, diametrically magnetized). Position information obtained from a couple hall effect sensors. This would have to be done in large volumes to be cost competitive, and would probably still end up slightly more expensive than the mechanical encoder. I think it would be possible to get close (under $1/mouse) if you use the same stuff they make those flexible fridge magnets out of.
As for low cost, low weight, and simplicity in use, it's hard to beat an off the shelf mechanical encoder. I think if I were to spend the time to engineer something more reliable, It would take several times longer than I would otherwise spend replacing however many mechanical encoders I would go through for the rest of my life. As a feature/selling point don't think it would let you increase the price of a mouse more than say, the ability to open a mouse without peeling the feet off.
Last edited by TranquilTempest; 05-08-2019 at 06:34 AM.