The idea is that you have a super-high bandwidth, super-low latency connection. Your first instinct is to use a "transparent" bridge and just connect together the memory spaces of the two systems. Then you think some more and you realize that this has no security and no robustness at all. With some more thinking you can solve these issues and lash the two systems together with hardware that is minimal and fast, but the bridge is arbitrating things so that the systems can still exchange data at mind-blowing rates and latencies.
This is precisely the mission statement of Infiniband.
I've worked with Infiniband RDMA which is pretty similar. One system allocates a shared memory buffer and hands a reference to the other. The other system writes to the memory block and then signals that it's done. It looks a lot like the classic inter-process "shared memory". This stuff is all happening in nanosecond timeframes, so running a network protocol on top of this is just a sad loss of bandwidth and latency. If you really want to see the pure speed then you have to program at the RDMA layer.
I've set up NFS over Infiniband RDMA using a fast RAID array. This is not your grandfather's NFS. It's not really possible to distinguish it from local disk, even a monster RAID array. The bonnie results on the client and the server are pretty similar.
If you want to mess around with Infiniband, you can get used gear on ebay for super cheap. It's probably going to lose out in the long run as an interconnect, but it's fun to play with and you can't argue with the prices.
Interesting. I can see that using LAN as means for connecting things can be slower, but by what amount? Also, what connectors would NTB use? I can see Thunderbolt as a possibility, but I'd imagine there should be something else as well.