ixgbe: stable UDP but volatile TCP performance

[dreamliner787-9]

For the x86 ixgbe driver in #682. On single core, I observed that for UDP it hits 10Gbps consistently. But for TCP, it varies wildly between runs. For example, at 5Gbps requested throughput, the result varied from 2 to 4 Gbps.

I discussed this with @Courtney3141 but we aren't sure what's going on. This issue isn't a blocker for getting the ixgbe driver merged so I'm opening an issue for tracking.

A theory of mine is that the tsc_hpet timer driver's set timeout latency is still too high. I measured it taking >60,000 CPU cycles to service a set timeout PPC.

[omeh-a]

This could be related to traffic getting clogged up in the sDDF queues and messing with TCP flow control ... we expect this at high speeds with modelling from my thesis. The timer latency would contribute if so.

[dreamliner787-9]

I just had a chance to test the driver @53c943e with a Linux VM as the client using virtIO net, so packets had to be copied between the sDDF data region and guest RAM by the VMM and all checksums are computed in software.

On single core I was able to get 2Gbps TCP on iperf3 with 3x 1Gbps load generators. Given that our native TCP performance is just slightly higher than the Linux VM number which incurred much more overhead, my theory is that this is something to do with lwIP itself, rather than something wrong with the sDDF net subsystem or the driver.

[omeh-a]

I'm not convinced personally. LWIP may be a part of it, as it is what implements flow control. But there is a solid mathematical basis to expect TCP to start behaving weirdly with our queueing design.

iperf3 with 3x 1Gbps load generators

How did you use multiple load generators with iperf3? Doesn't it only allow point-to-point testing? I'm personally not sure that doing many iperf3 tests at once will give results that are reflective of the true performance given how the tool works ... we should ideally have a 10gbe machine to be the other end of the test. We started working on setting this up on Horde yesterday though!

[dreamliner787-9]

How did you use multiple load generators with iperf3?

I just ran 3 instances of iperf3 on the VM.

we should ideally have a 10gbe machine to be the other end of the test.

But yea that would be ideal.