I recently completed UT Austin's Distributed Computing class, taught by Lorenzo Alvisi. My project partner Pato and I decided to give it a go in Haskell, and ended up using Haskell for all project assignments (Chandy-Lamport, Paxos, and Bayou).
This turned out to be an excellent idea. We put in a fraction of the time most implementations (in Java, Python, or C++) required.
This article represents a summary of what I learned over the course of the class, as well as an example program applying these principles. Some of this is Haskell-specific, and some is more general.
Every time we used multiple blocking reads from the network, it came back to haunt us. For example, we would send a Ping and wait for a Pong before continuing. This leads to all sorts of bad behavior. What happens if both servers Ping each other at the same time? Deadlock. Even blocking reads that seemed innocuous at first usually led to confusion and race conditions later on.
Instead, use an asynchronous architecture. Your program should block in exactly one place. Each node in the system should have a "superloop" that performs blocking reads from the network using an efficient epoll-like mechanism and then dispatches messages appropriately.