The last few days I've been heads-down working most of the time on a distributed system that involves going deeply into the arcana of TCP/IP. It's an eye-opening experience. First, because Java proves once more to be a rock, obtaining pretty high data transfer rates (currently 37 KB/sec on a 50 KB/sec connection) even with all the twists and turns the code is taking. Second, but most important, it has reminded me of how little we know of distributed systems, how to build them properly, and how to test them.

By distributed systems, I mean truly distributed. I mean that you can't count on a server to be happily taking over stuff for you with a bunch of TCP ports open and a four-way processor core ready to handle incoming tasks. We might be tempted to call this peer-to-peer (as opposed to client-server) but not really, since I could easily see this being used on a "traditional" client-server server environment. The difference is subtle, in terms of what you assume on the server side, and how you get around the constraints imposed by today's Internet.

That aside, being a test-first-code-later kind of person, I tend to put the burden on testing, or the testing framework rather. So I thought I'd write down my wish-list for a distributed testing framework (as food for thought more than anything else). This framework would work as follows: you'd have a "test listener" that can run on any machine, and a "test controller" app that can run on your desktop. Once the listener is running on the other machines (and maybe even on your desktop too) you can easily choose a JAR to deploy to all the target machines, then run it. The system automatically routes the output (System.err and System.out) to your "test controller" in multiple windows. You can control any of the clients through simple play/pause/stop/restart buttons. Clear the consoles, etc. You would be able to script it, so that this whole process can be run in loops, or automatically every day or every week, or whatever. You would be able to define output values to check for that can alert you of results that don't match expectations.

Looking around, I found the DTF at SourceForge, but it seems to be dead (no binaries, and no updates since February this year). I found papers (if you look hard enough, you can find papers on every conceivable topic I guess, so this doesn't mean much), like this one. But not much, really. Or is there some vast download area somewhere that I'm overlooking?

In any case, I know for a fact that CS curricula still don't pay enough attention to testing, much less to distributed testing. For one, distributed testing is difficult to generalize. But there should be more in this area happening, shouldn't it? Or does anyone doubt that half the future lies with large scale distributed applications? (The other half is web services :-)).