Original article was published by Emily Potyraj (Watkins) on Deep Learning on Medium
Use nconnect to effortlessly increase NFS performance
Having only 1 network connection per NFS mount is so 2019.
Linux has a new feature called “nconnect” that enables multiple TCP connections for a single NFS mount. Setting nconnect as a mount option enables the NFS client to open multiple “transport connections” for the same host.
nconnect is included in linux kernel versions >= 5.3. (So it’s available in Ubuntu 20.04, which uses Linux kernel 5.4).
It’s easy to utilize. Just add an
nconnect=_ value to the options list (
mount -t nfs -o ro,nconnect=16 198.18.0.100:/datasets /mnt/datasets
Note: The current limit of client-server connections opened by nconnect is 16.
Now, completely under the hood, your NFS client will open multiple TCP connections. Applications don’t need to change since a single stream of data is still returned to them.
Let’s look at a performance comparison for a single host with an example NFS server (Pure Storage FlashBlade).
In the “Before” test, the host is bottlenecked by the throughput limits that a single TCP connection can provide, i.e. the max bandwidth from a single NIC on the host to a single blade on the storage server. In this example, the limit is 10 Gbps per TCP connection.
Since the storage system includes more than one blade, we should take advantage of its full potential.
In the “After” test, the host can read ~70 Gbps, which is the limit for my example setup that has 7 blades.
Previously, single-client bandwidth-intensive applications were throttled because they had a hard time accessing the full power of their infrastructure. A performant application running on performant compute hardware connected to performant storage was limited by how much traffic it could access over a single TCP connection.
Even some of the most compute-intensive workloads, like deep learning training, are using applications that read data from storage as a single client.
Now, with the nconnect mount option, applications can easily take full advantage of the infrastructure’s capabilities and deliver faster results.
Note: This post is meant as an intro to the nconnect feature — not as a complete guide. Before using nconnect with my example NFS device, FlashBlade, please reach out to firstname.lastname@example.org to learn more about their current best practices.