I don't remember what TDP the Strix 1080ti has, but it's not over 350W. The rest of your system probably draws about 200W unless you have a crazy amount of pumps or fans. 650W should be more than enough. And from my experience if the PSU doesn't do a good enough job then the PSU will shut itself down, not crash the specific application that runs.
I have used FireStrike Ultra for a while and while it is the best stress test in the world, it has problems with power limited cards like Pascal and Turing. Still, it shouldn't crash on stock clocks. It's most likely a faulty card. You could maybe fix some things yourself, but I wouldn't bother with it.
Can you download MSI afterburner, let it monitor GPU clock, temperature, voltage and power limit, and run FireStrike Ultra again? I want to see what's going on.