Enable Windows High Performance Power Plan, Set CPU Min/Max states to 1/100. At idle your vcore/clocks should drop. But as soon as you move the mouse or do anything it spikes back up. But this seems to be the only Power Plan that behaves this way at idle when Steam is open. They all behave different than one another even with the same CPU Min/Max states set on some. I posted this to Reddit and a few people said it worked, while other people said their power plans behaved different from what I have described.
I made a quick gif to demonstrate. At the start of the gif, Ryzen High Performance mode is set with CPU State Min/Max 1/100 - the vcore and clocks are fluctuating a bit, like normal. Then I open Steam and iCue. You can see the vcore and clocks max out. A few seconds later I change plans to Windows High Performance 1/100 and the vcore and clocks instantly drop to a normal idle. See my ramblings below for more info on each power plan behavior.
System Info: 3800X / Aorus I X570 Pro Wifi / F4h BIOS / 3600 C14 / Windows 10 1903 / 'Normal' voltage set in BIOS / Chipset + Power plans reinstalled / Windows Plans set to default.
[Ryzen High Performance] at Min/Max 100/100 (Default) has clocks and core voltages bouncing between 1.100v-1.488v and 3513-4441Mhz. At 1/1 and clocks and voltage lock to 0.948v/3515MHz.
[Ryzen Balanced] at State 0/100 (Default) and 1/1 behave identical - it doesn't drop clocks or voltage on either setting beyond normal fluctuations 3513-4441Mhz. Acts the same as High Performance 100/100 on any setting.
[Ryzen Power Saver] acts exactly like High Performance; CPU state 0/100 (Default) gives the same clock and core read outs and 100/100, and 1/1 will drop the clocks to 0.948v/2195MHz.
[Windows Balanced] at State 5/100 (Default) and 1/1 behave identical; the clocks/core don't drop. Acts the same as 100/100. (Note: this is the plan AMD Robert said to set at 85/100 to 'fix' vcore, but it behaves the same for me regardless of the settings; 1/1, 85/100, 100/100).
[Windows Power Saver] Here's where things get a bit weird. The Max setting doesn't seem to have an effect on anything. Default is 5/100, but it acts the same as the other plans on 1/1 and core/clocks lock to 0.931v/2196MHz. But if the Min state is set to anything higher and your vcore stays locked at 0.948v until about 80 and your clocks rise. State 99 @ 1.092v, and 100/1 sends the vcore back up to 1.488v/4441Mhz
[Windows High Performance] Acts like every other plan @ 100/100. But curious enough, you set it to 1/100 and it starts to behave like a normal power plan. Cores drop to 0.948v and speeds to 2196Mhz when actually at idle with Steam and iCue open.
So TLDR; I set every power plan to 1/100, sat idle, and Windows High Performance was the only plan to actually idle with Steam/iCue open. When those apps are closed the idle is pretty normal on all plans at any 1/100 setting. Also, don't change your power plans with the drop down menu. Go back to the menu with the radio buttons. Selecting from the drop down only changes settings, but doesn't apply the plan.