Your first going to want to set everything back to stock and run a stability test that's supported on Mojave, I'm guessing Prime 95 may be the go to here (never dealt with this OS). Running a blend test for 4-8 hours while keeping an eye of temperatures with a HW monitoring software.
Your CPU is probably fine at stock settings, so you'll first want to check if your ram is stable at both stock (2133 jedec standard, or auto in the memory tab settings) and xmp settings. I'm not familiar with any memory stress test software for a Mojave so you may want to check this link/software out
So in terms of a general process:
1) Set everything to stock, run a memory testing software with your memory set to auto/jedec standard 2133
If it passes -> set your memory to XMP and repeat step 1
If it fails -> you more than likely have a bad memory kit
2) Once you know your memory is stable you can begin to overclock your CPU (set your memory back to the auto setting/jedec standard while looking for a CPU overclock). If you have a target of 5ghz, set your multiplier/cpu ratio to 50 and disable spread spectrum (Advanced CPU Core Settings Menu); you may also have to set an AVX offset depending on temperatures or based upon what program you end up using to check system stability. Then set a starting voltage, for example the 1.3v you chose.
Run Prime 95 on blend with the above settings for 4-8 hours; pay attention to CPU temperatures and make sure there is no throttling/excessive temps (85c+)
If it fails/you get errors -> you need more vcore, increase in 20mV steps and rerun until you find a stable setting; if you are still failing and you are throttling/temps are exceeding 90c+ you will have to reduce your multiplier down to 49x and start back at 1.3v
If it passes -> reduce your vcore by 20mV steps until you can no longer pass your stress test/P95 for 8hrs without error -> once you've completed this you will have found the minimum required voltage for your overclock.
3) Once you've found a stable cpu multi/ratio and vcore required to be stable, you can now set your memory to XMP and retest both your CPU and Memory.
If you fail either in P95 or memtest -> you may need more vcore, more vccio, more vccsa, or a combination of all 3.
a) Start by increasing VCCIO/VCCSA to 1.1v, and rerun P95 on blend,
If you pass reduce VCCIO/VCCSA in 10-20mV steps until you find the minimum required VCCIO/VCCSA.
b) increase VCCIO/VCCSA or Vcore in 10-20mV steps until you find a stable combination of voltages - DO NOT exceed 1.25v on either VCCIO or VCCSA (you shouldn't need more than stock (1.05vccsa and 0.950 vccio) or up to 1.1v for either voltage with a DDR3000 kit).
If you pass both CPU and memory tests -> Congratulations your system is stable.
Once you've found a stable overclock, if you still have issues that seem to directly relate to specific programs or actions on your PC you may need to further diagnose different parts with stock settings.
You'll find a lot of helpful information in the Z390 Gigabyte Thread
, it will also be a great resource for asking questions.
Again, I'm NOT familiar with stress testing software on Mac based OS's so someone may chime in with better software options. The general overclocking process will remain the same however, you'll just be using different software to stress test your system.