Used CPU Discussion

[crashtech]

I thought it might be useful to have a catch-all thread to discuss used CPUs, particularly which ones currently represent the best value for DC.

To begin, I have two dual CPU Xeon E5-2678v3 systems that I have been thinking of replacing. Due to space constraints, I need all my rigs to run at least one GPU, and I need them to be able to run a 1080ti or faster at minimum. But sometimes I have seen the Xeon rigs struggle with even a 1070ti, which is why they are running 1070's now, which I want to have retire in 2021. The most economical way to solve the problem might be to identify replacement CPUs that have the ability to drive a larger GPU. Does anyone have experience running, say an E5-26XXv4 Broadwell CPU with a large GPU? Or should I just give up on this idea and plan for entire platform replacements?

[Skillz]

I think the best bang for your buck Distributed Computing CPU will be AMD Ryzen based CPUs. Even first generation such as the 1950x or the first generation epyc CPUs. They're cheaper than Intel based on core count alone and usually have more cores than Intel does which distributed computing can utilize.

[Icecold]

Which projects are you running that the Xeon is falling short on? I'm running an RTX 2070 on a Xeon E5-2670V3 and I've never noticed it not keeping the GPU at 100% or projects running slower than my 2070 in a Ryzen machine.

I'm running a 3070 off of a used i5 6400 and it seems to run great.

[crashtech]

I think the best bang for your buck Distributed Computing CPU will be AMD Ryzen based CPUs. Even first generation such as the 1950x or the first generation epyc CPUs. They're cheaper than Intel based on core count alone and usually have more cores than Intel does which distributed computing can utilize.

I think you have a good point, but even though I am still running a couple of first-gen Zen CPUs, I have a couple of objections to using them as Xeon replacements. First, because their AVX2 performance lags far behind Haswell, and second, because the cost of replacing the whole platform is going to be pretty expensive. I think I would rather jump straight to Zen2, or at least to a platform that would support Zen2 if I was to go this route. I don't know anything about Epyc or Threadripper platforms, but for sure the motherboards and coolers will be more expensive than other alternatives.

[crashtech]

Which projects are you running that the Xeon is falling short on? I'm running an RTX 2070 on a Xeon E5-2670V3 and I've never noticed it not keeping the GPU at 100% or projects running slower than my 2070 in a Ryzen machine.

I'm running a 3070 off of a used i5 6400 and it seems to run great.

Well, I haven't kept good notes, but I think it has to do with which combination of projects is running. Some scenarios are definitely more challenging than others, and I am pretty certain I saw instances where the Ryzens were definitely outperforming the Xeons while running a CPU and GPU project concurrently.

[Icecold]

Which projects are you running that the Xeon is falling short on? I'm running an RTX 2070 on a Xeon E5-2670V3 and I've never noticed it not keeping the GPU at 100% or projects running slower than my 2070 in a Ryzen machine.

I'm running a 3070 off of a used i5 6400 and it seems to run great.

Well, I haven't kept good notes, but I think it has to do with which combination of projects is running. Some scenarios are definitely more challenging than others, and I am pretty certain I saw instances where the Ryzens were definitely outperforming the Xeons while running a CPU and GPU project concurrently.

Interesting. I'll have to try to keep a closer eye on the Xeon I have with the 2070 and make sure I don't see it bottlenecking.

[StefanR5R]

@crashtech,
GTX 1080Ti are running identically on i7-7700K/Linux and E5-2696 v4/Linux. Only the pre- or postprocessing portions of an application which run some calculations on the CPU only (e.g. the first part of the Amicable application) are respectively slower on the Xeon.

Note however that this E5 is a single-socket system. I have no experience with GPUs in dual-socket systems. I *suspect* that the tasks which access a GPU are not pinned to logical CPUs which belong to the socket to which the GPU is attached. But I am not sure. BTW, if there is a problem with that, then Xen/Xen+ based Threadrippers and Epycs are affected by it too, just that their inter-"socket" interface is on-package Infinity Fabric, not off-package QPI.

[crashtech]

I wish I could remember when the performance penalty was observed. It occurred while running a GPU and CPU contest simultaneously, something I don't think that you do often or at all, @StefanR5R. But I have to try to keep my compute power relatively compact, so every PC must have at least one powerful GPU. It became clear to me that under certain circumstances, the Zen 2 systems had a distinct advantage when running the same GPU and CPU project across all hosts. I'm assuming at this point that even going with the best Broadwell CPUs I can find and afford, they will only net me about 300 Mhz additional speed plus a small IPC gain. Probably not worth it!

[StefanR5R]

These i7s and the E5 have in common that some combined CPU and GPU projects workloads cause reduced GPU utilization unless the total thread count is restricted to the number of physical cores. IOW, in some cases it is required for top GPU utilization to leave Intel HyperThreading unused. But again, this applies to both desktop Skylake+ and server Broadwell alike.

I have no direct comparison to AMD SMT, because my AMD based computers don't have GPUs.

[Icecold]

I didn't mention this in my post @crashtech but I almost always leave 2 threads open per GPU even on my 3900x machines. My hope being that this allows it to feed it with a physical core, or at least 2 hyperthreaded/SMT ones. I have noticed some slowdown/bottlenecking on the Xeon when only leaving 1 thread open for the 2070.

[crashtech]

Yes, in fact on the dual socket Xeons I leave a minimum of 2 logical cores for the GPU, sometimes more depending on the project. At other times it does not seem to matter how few cores I try to use for a CPU project, the GPU is impacted unless CPU computing is suspended. It's not always like that, and I'm kicking myself for not noting which combinations exhibited this behavior. I'd consider this to be "normal," but since I'm shepherding Xeons and Zen 2 alike, the difference becomes noticeable in psensor.

[StefanR5R]

Regarding my suspicion of GPU applications running on the wrong socket/ allocating memory at the wrong socket:

Two boinc clients could be used in parallel, one started normally and used for CPU projects, and the other one started with an affinity mask which restricts it and its child processes to run on the socket which has got the GPU attached. (Or to be sure, the other client could be started with an affinity mask which is the complement of the GPU client's mask.)

[crashtech]

In my usual un-scientific and impulsive way, I found a pair of E5-2690 v4 CPUs for $450. I should be able to get about $150 for the pair of E5-2678 v3's, so this is a ~$300 experiment. I will get slightly better IPC, 300 MHz more speed at full load, 8 more threads, and possibly lower power consumption per thread. I'll soon be able to compare the two systems and see if there is any difference when they are running identical projects. I want to believe, lol.