Putting the below in /etc/sysctl.conf both client and server machine, helped me a whole lot.  I was seeing a lot of hesitations and heavy memory and swap usage on the server, especially after time idle, as well as overall desktop latency and audio breakup; these knocked it all out.   2G RAM on 1GHz single-CPU old laptop client, 4G on quad-2.6GHz AMD X2 server.  

net.ipv4.tcp_rfc1337=1
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_workaround_signed_windows=1
net.ipv4.tcp_sack=1
net.ipv4.tcp_fack=1
net.ipv4.tcp_low_latency=1
net.ipv4.ip_no_pmtu_disc=0
net.ipv4.tcp_mtu_probing=1
net.ipv4.tcp_frto=2
net.ipv4.tcp_frto_response=2
net.ipv4.tcp_congestion_control=illinois
vm.swappiness=10