Now blogging at diego's weblog. See you over there!

an update on spaces


So I missed my self-imposed end-of-february deadline for the release of the spaces beta. I'm not terribly happy about it, and I want to apologize to those users that have been waiting for it. But there wasn't much I could do. Let me explain.

On one hand, there have been several things going on besides spaces. For example, the term at Trinity (where I am a teaching assistant in the CS department) just ended (last friday, yay!), and it was slightly crazier than usual; because of the way the course is organized this term is the one where the students have to hand in most of their work, and that means a lot of bureaucratic crap to deal with, not to mention having to correct dozens of 20-page assignments (not fun).

On the other hand, my thesis work was on the forefront at the end of january (as I was releasing alpha 1.8) and at the beginning of february. Usually I manage my time reasonably well and balance between the different things I do, but this time the thesis got to the point where I needed one last big push to get the code running. And it did! The first complete version of the code ran for the first time in the second week of february.

Now, those weeks where the thesis took overriding priority over everything else (namely, sleep, eating, and so on, with many 16 or 18-hour days) are the main factor that accounts for the beta not being out by now. There's another however.

The other factor is that I've decided that the beta would have to include the memory/storage optimizations that have been waiting on the wings since the alpha release. The goal would be for spaces to run comfortably under the default settings of the JVM (usually 64 megs of maximum heap, although on I've seen some installations where it defaults to 128 megs). This applies to all operations, including import. Simply put, I wanted the "java.lang.OutOfMemoryError" messages to be a thing of the past. Also, reliability had to be rock solid. Many people are already using spaces for their everyday mail, and although they know it's alpha, that's no excuse. It should be fixed. And the time is now.

The original version of the storage system that underlies spaces alpha 1.x was optimized to minimize disk usage and maximize performance, at the expense of memory. To update it, I am rewriting several components that will change the equation to minimize memory usage and maximize performance. The new version will take up maybe 10-20% more on disk (with a higher peak usage as well), but will have upper bounds on the RAM used. The goal is, again, never to breach the default maximum JVM heap of 64 megabytes when the number of items stored (email, RSS, calendar entries, etc) is 100,000 (yes, one hundred thousand items). It should use less memory than 64 megs, probably in the neighborhood of 32, but 64 is the upper bound. This would mean that spaces will run in about the same memory other programs currently demand under similarly heavy use. In the process, I am adding more features for stability, reliability, and storage management, and preparing the core to support versioning of items, which will be necessary in the near future.

Now, the storage system change is well underway. I've already been running preliminary tests on it with 100,000 thousand items inserted into the database, and memory usage has decreased by a factor of ten in some cases. It's amazing how changing the focus on optimization can affect different measures of performance: the original design of the storage had in mind a few tens of thousands of items, rather than hundreds of thousands. And even though not that many people will deal with hundreds of thousands of items, RSS Feeds create large numbers of items, so tens of thousands of items for a user is not a ridiculous idea.

After the storage change is complete (probably in the next couple of days), I will finish work on the IMAP implementation and make the UI changes that were planned for beta 1, which include making a space more "malleable" by allowing various types of sorting and filtering to be used. I will post more information on these changes for comments as as soon as there is something that others can actually see, or try out.

So that's about it for the moment, more to come in the next few days!

Categories: clevercactus
Posted by diego on March 10 2003 at 5:46 PM

Copyright © Diego Doval 2002-2011.
Powered by
Movable Type 4.37