Been using mostly trash tier memory on my DDR4 platforms myself, due to memory prices rapidly rising shortly after I built my X99 systems and only coming back down recently.
Will probably grab a good set of Samsung stuff for the Zen2 setup I plan on building this summer, but in the mean time, my five year old kit of Ballistix Sport 2400 CL16 is still holding in there on my 5820K:
~30% faster than the XMP profile, mostly from timing tweaks.
Anyway, there aren't any LGA-2011v3 CPUs that will have significantly better light or moderately threaded performance than your 4.4GHz 5820K already has and the gain you'd get from faster memory will be modest. I'd recommend tuning what you've got to perfection and riding it out until your next platform upgrade.
...rightful liberty is unobstructed action according to our will within limits drawn around us by the equal rights of others. I do not add 'within the limits of the law,' because law is often but the tyrant's will, and always so when it violates the right of an individual. -- Thomas Jefferson