When I sent this before it had a different title - you probably missed it because of that. Date was 4/12. -Dave -------- Forwarded Message -------- Subject: Fwd: CS Network Throughput Date: Thu, 12 Apr 2018 16:48:03 -0400 From: Dave Hall To: Craig M. Houck Craig, Here's the note I typed up for one of the CS faculty. Near the bottom I referenced some disk subsystem tricks I've learned that can further boost performance by getting XFS to spread it's activity across multiple cores, and also to spread the actual IO activity across multiple paths/controllers. These involve LVM and were taken from a Dell packaged NFS offering that subdivided that physical disks into 8 - RAID6 (8 + 2) partitions. The partitions were spread across the disk controllers and linked back together in the server with striped LVM such that the OS tends to queue I/O to each partition in turn. I know some of the fancier Dell disk systems claim to take care of all that kind of stuff under the covers, and Nimble even more so, but the trick is to leverage a large server to maximize throughput. My NFS server is 256G RAM and dual 14 core CPUs, and 4 disk partitions on dual 12Gb controllers. I've gotten above 10gb/s (network) on a sustained file transfer. -Dave Dave Hall Binghamton University kdhall@binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office) -------- Forwarded Message -------- Subject: CS Network Throughput Date: Tue, 20 Feb 2018 18:45:25 -0500 From: Dave Hall To: Aravind Prakash Aravind, I mentioned yesterday that there are internal kernel factors that can effect network throughput. I started a document at one point, but when I went back to it I found it to mostly be large grabs from a few websites I found. So for now, here are some settings that I derived for our NFS servers. I'd suggest that your students who are seeing poor network throughput might try some informed variations of these settings while considering that going too extreme will impact the interactive performance. These are stored in files in /etc/sysctl.d so they get injected on every boot: # cat /etc/sysctl.d/vm.conf vm.dirty_expire_centisecs=100 vm.dirty_writeback_centisecs=100 vm.dirty_background_ratio = 5 vm.dirty_ratio = 80 vm.vfs_cache_pressure = 50 vm.swappiness=10 # min_free_kbytes is tuned to the amount of RAM installed. vm.min_free_kbytes = 262144 cat /etc/sysctl.d/tcp.conf # increase TCP max buffer size settable using setsockopt() net.core.rmem_max = 67108864 net.core.wmem_max = 67108864 # increase Linux autotuning TCP buffer limit net.ipv4.tcp_rmem = 4096 87380 67108864 net.ipv4.tcp_wmem = 4096 65536 67108864 # increase the length of the processor input queue net.core.netdev_max_backlog = 250000 # recommended default congestion control is htcp net.ipv4.tcp_congestion_control=htcp # recommended for hosts with jumbo frames enabled net.ipv4.tcp_mtu_probing=1 It is also worthwhile to change the IO scheduler (/sys/block/*/queue/scheduler) to 'noop' or 'deadline', but this is a bit less obvious to do since systemd no longer executes /etc/rc.local. I've learned a bit more about systemd lately so now I think I know how I'd do this. The most important point is that the default kernel vm settings can really choke off data transfers unless they are changed to aggressively write and release dirty buffers (received data). This would also speed up any large compiles. The TCP settings are important as well - to keep the NIC from getting clogged up with incoming data and to allow the sending system to keep the queue full. Another important issue arises when transferring many small files: cp and rsync both build maps of the entire file tree in memory to keep track of progress. This seriously increases memory clog and hammers throughput. A better approach is to use xargs or parallel to break the transfer into many small file trees. Not only can you engage more cores, but the aggregate of mapped file trees at any given moment is much smaller. (I have a good war story to back this up.) Lastly, for larger RAID systems that are natively connected to a Linux server, there are some subtle and slightly complex tricks that can be used to assure that the server avoids file system bottlenecks by spreading file system transactions across a higher number of cores. We learned about this from a packaged Dell solution that used active/passive servers and 5 shelves of disks (16 x 8TB per shelf). However, the principle yields performance improvements with as few as 16 disks.