If DxTory is unable to record due to MSI's AB being active and overlaying, then you should probably disable it.
In order to disable the OSD and Recording features of MSI AB, while keeping the OC/Monitoring/Fan control section of it, simply do the following.
Once you have a hotkey set for OSD Blocking, simply press it once and MSI AB will function without the Rivatuner OSD/Recording features, so MSI AB should work alongside DxTory.
As for your FPS Dropping when recording with AB.. I haven't experienced it myself.
I use the Full Frame/Uncompressed settings for 1080p30 to a RAID0 drive and regardless of what game I play the FPS almost never changes when I hit record. Without real-time encoding, the CPU load of saving uncompressed frames is so small.
The new version has four different quality options, maybe if you need to do real-time encoding to reduce the file sizes and HDD transfers you should change them around a bit, I know MJPG is a pretty light load and has a quality slider.
It's not the program that determines quality and performance, it's more what options are available and which you choose.
For me I switched to MSI AB from FRAPS since I didn't like FRAPS's default quality, and the Lossless option required a lot of CPU power. Depending on the game, sometimes I'd see almost no FPS Drop, and in others it would drop me down a lot. The lack of consistency sucked, so I went looking for a way to reduce the varying CPU usage when recording and that's where Uncompressed came in.
Uncompressed requires almost no CPU power, instead it just needs a capable dedicated recording drive.
I went with a (2x1TB) 2TB RAID0 setup that gets 280MB/s sequential write speeds, and since 1080p30 only needs around 180MB/s I'm way over the requirement and can always record stuff.Uncompressed RecordingsThe Bad
-Huge file sizes - Sometimes over 300GB/recordingThe Good
-Best quality possible
-Lowest FPS Drop possible, aside from getting a dedicated capture card
-Compatible with every
editor out there
-Results in the best quality after Encoding, due to 0 artifacts being present before final encode
-Encodes the fastest. (Due to not needing to be decoded as it re-encodes it)