https://knightoftheinter.net/blog/{20181206,tinkerboard-iperf3-ipv4-ipv6-pfsense}{.xhtml,.odt}

 pfsense is pretty neat. If I visit any bandwidth performance site, I get the full speed of my internet connection. Or so those sites claim anyway. But there’s definitely some quirks that I’m not sure how to figure out to explain.

 Let me give an example. I have an Asus Tinkerboard connected on my LAN. For a while, I had a workstation virtual machine with bridged networking to the LAN so that it could connect directly to the Tinkerboard. I could `ssh` into the machine and run `iperf3` and get the advertised 350Mbit performance. I could do that in both directions.

 When I added pfsense to the mix: I put the workstation virtual machine behind that. Now, I get 250Mbit performance going out from the pfsense (LAN → WAN) but going in is much slower and wildly variable; as slow as a few tens of Mbits/sec typically. See below.

        In the above screenshot, the left column is the client and the right column is the server. In the top row, I am sending data from the Tinkerboard to the workstation (WAN → LAN). In the bottom row, I am sending data from the workstation to the Tinkerboard (LAN → WAN). To demonstrate that it’s not some weird quirk with the connection, I can run `iperf3` in reverse on the same `ssh` connections and get the same logical result:

 Now the top row is LAN → WAN and the bottom is WAN → LAN, reusing the same `ssh` connections.

        Okay, so I don’t really know how to explain that performance difference. So I went over a bunch of settings in pfSense and nothing seemed to make a difference. I sat on this for a few weeks (almost six, actually, since October 27th) thinking over the different settings and configurations that might affect it.

 I thought perhaps there’s something up with Snort, so I tried turning that on and off; that didn’t change the performance characteristics.

        I thought perhaps there’s something up with the way SSH was encrypting data. There’s software and then there’s software. Not all softwares are made equal and not all encryptions have the same performance. I know my CPU has AES-NI and should be able to handle the performance just fine. But what about the Tinkerboard?

        Even so, that didn’t make so much sense. If the Tinkerboard could keep up with a single connection on the physical-to-physical host (not shown) then why wouldn’t it keep up over pfSense through a virtual NIC? Back to the drawing board. The first difference I thought of was the virtual NIC which, as I understand it, forces packets through the CPU and that could be a bottleneck. But if that’s the case then why is the bottleneck apparently on a single direction? The next difference is IPv4 vs IPv6: I’d been tinkering (hah) with IPv6 link-local addresses (which you can’t see). What if I connect to the Tinkerboard over IPv4?

        Take a look:

 Okay, so the left column is WAN → LAN while the right column is LAN → WAN. And look at that, connecting over IPv4 to the Tinkerboard makes the bandwidth symmetric. That’s very interesting.

 I’m still not sure what’s going on, but it’s clearly something to do with IPv4 vs IPv6.

Cheers,
inetknght

blogspam@knightoftheinter.net