diego's weblog: clevercactus archives
clevercactus: so long, and thanks for all the fish!
Today we removed most of our stuff from the clevercactus office, in preparation for a full shutdown early next week. I've already switched over the server to a placeholder page. Email will still work, at least for a while, and we'll do our best to ease the transition for our current users. We apologize to them in particular, but there was no way to keep the service running without significant investment in time and money, and we just couldn't handle it. That's how things go sometimes.
The last two and a half years have been an incredible experience, and I'm thankful for everything. To my business partner, Paul, and to everyone that was with us along the way, our families, friends, the design team at huskit, and of course the users: thanks. Really.
And now on to the next big thing...
Through the last week the clevercactus site has been sporadically unavailable, and it's down right now. This means no web, no service, no emails getting through.
If you're trying to get through to clevercactus and can't please let me know through a comment or email to my personal address.
What happened is that we were attacked (I'm not sure when) and someone left a number of scripts there that are flooding the system (they do other things too, but at least one of them is clearly written simply to flood the network and disable it). This is something obviously intended to bring down clevercactus, not just a simple hacking. Why? What do they gain by bringing down the service of a small company that is going through hard times?
This kind of thing makes me sad, and is really discouraging.
I had this whole thing planned for today, getting the manifold site up and so on but now I'm going to spend time trying to see how to route around the problem for now until we can determine the extent of the hack. I don't even know how they got in yet--we constantly update our software with the latest patches. Needless to say, I'm seriously reconsidering the whole of the software I use and how to set it up so that this doesn't happen again.
Anyway. We'll see how it goes.
location, location, location
First, the support I've received in the last 24 hours is regarding my post on clevercactus amazing and heartening. When I collect myself I'll be more specific, for for now I just wanted to mention this.
One of the comments I've heard most since yesterday has to do with the funding thing. As Dave said, "Consumer focused companies are always difficult to get funding for in Europe". I can definitely attest to that. Lots of enterprise and "vertical" focus, and generally low tolerance for the risk/opportunity equation presented by those opportunities. To their credit, VCs are very candid about this, so it's not as if it's a secret or anything.
One thing we tried to get across is that we'd be totally open to moving to the US if that's what it took to get funded. We'd have no problem with that. But then again, not being there, it's hard to get in the door.
But the "virtuous cycle" created by Silicon Valley is hard to beat. One of the conclusions I will take away from this experience is that all the talk about the "Indian Silicon Valley" or "European Silicon Valley" or "[Insert geolocation here] Silicon Valley" is just plain ridiculous. There isn't a place like it in terms of investors, press, talent, etc., all deeply interconnected (well, maybe a couple of places in the US come close, namely the Seattle/Redmond area, Route 128, and NYC). I remember going to Il Fornaio in Palo Alto and just feeling it. It's where things happen. (Plus it ain't a bad place to eat).
So: there's only one Valley, the rest are close, but no dice. :)
looking for the next big thing
So. A week has gone by with no posting. Lots has happened, but more than anything it's been a time of consolidation of what had been happening in the previous weeks. First, the short version (if you have a couple of minutes, I recommend you read the extended version below): tomorrow is my last day working for clevercactus. And that means I'm looking for the next thing to do. So if you know of anything you think I could be interested in, please let me know.
Now for the extended version.
For the last couple of months (and according to our plan) we have been looking for funding. Sadly, we haven't been able to get it. This hasn't just been a matter of what we were doing or how (although that must be partly a problem) but also a combination of factors: the funding "market" in Europe and more specifically in Ireland (what people put money into, etc), our target market (consumer) and other things. Suffice it to say that we really tried, and, well, clearly it was a possibility that we wouldn't be able to find it.
On top of this, I haven't been quite myself in the last few weeks, maybe even going back to September (and my erratic blogging probably is a measure of that). By then I was quite burned out. Last year was crazy in terms of work, and this one was no different: between January and the end of July I only took two days off work (yes, literally, a couple of Sundays) and the stress plus that obviously got to be too much. I see signs of recovery, but clearly this affected how much I could do in terms of moving the technology forward in recent weeks. Since there's only two of us, and it's only me coding (my partner deals with the business side of things), this wasn't the most appropriate time to have a burnout like that. I screwed up in not pacing myself better. Definitely a lesson learned there.
At this point, the company is running out of its seed funding and we don't have many options left. Even though it's possible that something would happen (e.g., acquisition), what we'll be doing now is to stop full time work on the company, which after all won't be able to pay for our salaries much longer, and look for alternatives since of course we need to, you know, buy food and such things. The service will remain up for the time being, and I'll try to gather my strength to make one last upgrade (long-planned) to the site and the app, if only just for the symmetry of the thing. Plus, you can't just make a service with thousands of users disappear overnight. Or rather, you can, but it wouldn't be a nice thing to do.
Now I have a few weeks before things get tight, and I'll use that time to get in the groove again and hopefully find something new to do that not only will help pay for the bills but is cool as well. Who knows? I might even end up in a different country! As I said at the beginning, if you know of something that I might find interesting, please send it my way. Both email and comments are fine (my email address can be found in my about page).
In the meantime, I'm going to start blogging more. No, really. I have some ideas I want to talk about, and maybe I can get back into shape by coding (or thinking about) something fun and harmless.
Or, as the amended H2G2 reads: Mostly harmless. :)
new version of share beta
clevercactus pro beta is back!
There's still much to do of course, there are some known problems with view updates under some circumstances, and new challenges that emerge from updating Atom support up to 0.3 (a few Atom 0.3 feeds confuse the parser by including XHTML un-escaped within the content element--they are labeled as xhtml+xml yes, so it's parser refinement that has to be done).
Now my focus goes back for the most part to share. There's some bugs to fix, improvements to make, and some important features to add (among those expose support for multiple locations for a single identity).
when task manager makes you smile...
... you probably need to get a life. But that's a topic for another day.
I was just final dogfood testing the upcoming new release of clevercactus pro on my own data and it made me happy enough that I decided to post about it.
Btw, by "dogfood testing" I mean passing the "eat your own dogfood" test, which stands for completely switching over to (and relying on) the shiny new widget that you're about to unleash on the world.
One of the major changes on the new release of pro is on performance+memory usage (which are closely related when dealing with a memory-managed environment). Previous releases of pro, while efficient, started to be a burden when getting to a certain DB size. In my case, with about 15,000 items on my mail store, pro regularly used up all of the available memory in the VM. It did work when constraining the available memory, but the point is not to work past the limit but never to reach the limit at all. So the last few days I've concentrated on further improving that aspect of it.
While I have been using the app myself for a while now --months actually--, I've relied on the previous version as well, just in case. Finally, today, I flipped the switch for good and ceremoniously deleted the previous version from my system. All that remained was double-checking the effect of the upgrades, including the most recent changes.
Lo and behold, when I loaded the database and checked memory usage the app was holding at around 15 MB for regular usage and peaking at around 50 MB when poking it hard enough. Over time, usage dropped down back to 15 MB. And minimizing it, that is, eliminating the memory needs of display, etc, usage dropped down to literally nothing: 2 MB.
Much work remains, and while you might argue (rightly so) that this only brings the app to par with others, it's still a small rush when it's your code that is doing it.
Yeah, I'm happy about the new RSS/Atom parser, or the other many improvements in it, but this was a long-standing goal that I'm glad to finally cross off the list.
Anyway, I'm itching to get this rev out and concentrate back on share. Now for a bit of rest--more tomorrow!
burnout? no, just busy
So yesterday as I'm pondering why I haven't posted anything in a few days I read this wired article on blogger's burnout. Although I've experienced lack of blogflow before, this time it was something different: just being too absorbed into what I was doing to do anything else.
So what was I doing? Simple: working on a new release of clevercactus pro. Not that share is taking a back-seat or anything, mind you, this is something that we had planned for a while and finally there was time to do it. (The release will be out sometime next week).
Anyway--I have had a couple of posts swirling in my head for a couple of days now, so I'll get to that now. :)
share v.1.40b released
In the end the new version of share took a couple of days more than I anticipated, but it was worth it. Here's the announcement in the forums. Probably the most visible change is the improved queue manager, which maintains local and remote queues synchronized transparently (but allows asynchronous queue management, that is, it is irrelevant whether the contact that is sharing the files is online or not).
This has kept me quite busy the last few days--now for a short break (and maybe some more blogging :)).
One thing I keep forgetting to do (or rather, not finding time for, since I clearly keep remembering I should do it!) is to thank some of the brave souls :) that helped in the initial beta testing phase of share (that is, before mid-June) and posted about their experience, in most cases with ideas and recommendations. Yes, I've said this to them privately but linkback never hurts :). Probably one reason I hesitate to do this is that I'm sure that I'm going to miss some of the links--so this list is not meant to be exhaustive. Anyway, here it goes (more or less in the order in which they appeared).
Erik, here (and more recently here) who also has given us great ideas and comments. Russ, noting some of the initial performance and L&F issues among other things (and who hung in there even when strange things seemed to be happening with his installation). Cristian, Frank , and Jim, all of them with nice comments. Don, who quickly came up with a wonderful idea for wishlists (which, sadly, I haven't added yet, but it's never been off my mind), and then followed up later with more detail (and some excellent questions and comments as usual). Dylan, who posted a long review, with many ideas and good criticism. Anne, with a great analysis of some of the sociable ideas in share. Also, Justin, and James.
Once again, thanks to them and to the many others that I can't link to (either because they don't have a weblog, or because they made comments privately through email) -- share is better today because of their help and feedback.
share: how and why
I've just posted a (slightly lengthy) description of how share works over at the cactus log. It probably doesn't answer all questions (does anything ever?) but it's a start.
More later. :)
best laid plans
While the idea of the bloomsday release was to come out sometime in the afternoon (as my twisted reference to 5 pm in that post shows), the plan became meaningless after Danny found the not-yet-public login page to the new version and outed it on a MeFi thread. Luckily enough, this happened not long before the day started (not that it had ended that long ago) and so most of what we had planned to do during the day had to be done incrementally making sure the system stayed up.
So, without further ado, here's entry on the cactus log that discusses the release.
Now, most of the changes are in place and we have been fixing certain problems, such as linux install issues (noted in the newly-released clevercactus forums). But there's still a lot to do: documentation (particularly in what relates to firewalls, connectivity, etc), config options, and improving portions of the app and features.
Anyway. Back to work.
ps: Happy Bloomsday!
coming up: clevercactus share, the public release
A bit ago we switched over the DNS records for the clevercactus site to its new location. Most people should have the updated version by Wednesday, when we'll come out with the so-called (by me :)) Bloomsday release.
ps: the link should go to the cactus log, but it might not work if the new DNS information hasn't reached you yet. Shouldn't take long though. :)
and clevercactus share...
... will see its next release becoming public within the next few days. All the work we've done since we launched it is coming to fruition.
Tired but happy is an accurate description of how I feel. Stay tuned!
clevercactus: year 2
Strictly speaking, the anniversary was that of the the "public release" of the company and product. Things don't happen on a clear-cut timelines, but it's nice to mark milestones.
Now, I had thought about it on and off for the last few weeks, but of course I had to actually get to the day to forget about it -- and about mentioning it! But since I have an irrepresible need to come up with something clever (no pun intended :)) to say, I'll just use my faulty memory as an excuse and make this an entry about the year that begins, as well as a celebration of the year that's gone by.
One year! Wow. Thanks to everyone who has helped us get here!
The best is yet to come. :)
Okay, so blogging is down again as more details of share come into place. For example, a couple of days ago we began to switch over the infrastructure to its final form (app-wise) by using HTTPS on the connections to provide a fully secure front-end to the API (which, yes, we will publish once the infrastructure is ready to handle it, along with a local API for the connection layer). There have been many revs over the last couple of weeks, and now we're nearing the next tipping point. More information on the product and the clevercactus weblog are also forthcoming.
Anyway, if I'm not blogging as much as I'd like to, that's the reason. It will ease up. Eventually. :)
...after my mini-hiatus. The last few days have seen important changes in the connectivity layer of share, improving online status detection and distributing the tasks completely into the client. We haven't finished verifying this yet but the current version should provide connectivity within environments with proxying firewalls (although not between an environment with a proxying firewall --as opposed to a transparent firewall-- and the outside world) when HTTP connectivity is available (used for basic contact sync). This is a common situation within many offices or corporate sites. Several bugs have been fixed as well. We're getting closer to opening up the floodgates so that anyone who wants to join, can.
Lots of other things to talk about. More later!
me = offline
Okay, a new version of clevercactus share is ready, with an improved connectivity layer and other changes. Phew! We've been catching up with emails over yesterday and today, so if you're waiting for a reply and/or you're wondering whether we got the message or not (one of our email addresses was non-operational for a couple of days) please let us know.
I need to take a break, so I will be going offline for a couple of days as of right now. Have a good rest of the weekend! :)
a couple of things about clevercactus, and share
A few things:
share on the mac
We had tested share on various OSes before release, including Mac OS X, but the initial integration with Mac wasn't that great. The next version, however (due out tonight) will integrate properly within the Mac environment. Check it out:
random simpsons quote
Mr. Burns, here are your messages:
From Homer the Smithers
clevercactus share, take 2
So, you're racing ahead to finish a release and all you see is code and product and small details and the million things that you think are crucial (that maybe are, and maybe aren't) and you miss other stuff.
Stuff, like explaining why there isn't a way to register.
Or explaining what does this have to do with what you were doing before, particularly for the generous and dedicated people that have been giving you feedback on your (other) product over months and months, to whom I'm very grateful for their support, people like John Rubier, who left a comment in the previous entry with some puzzlement (and rightfully so).
Or explaining why there isn't anything else on the website at this very minute moment aside from the application, which (again) you can't log in to unless you have been invited.
You know, small things like that.
So, first things first, an apology for the lack of context for share. As I'm writing this we are preparing a new release of the site with content that we've been working on over the last several days. But I wanted to start off here by explaining some of those things. Hopefully I'll be able to get enough in here to make it easier to understand, but if not please bear with me. If there are things that aren't clear, I'll be happy to answer them.
So, moving on to some of the, um, finer points.
What is this clevercactus share thing? Where is clevercactus pro? Why the sudden shift?
Let's take those questions in reverse order. First, the last: this is no sudden shift. We have been working on the concept and the implementation for share for several months, since late last year. Yes, we haven't been obvious about it. Part of the reason was that sometimes when you're trying to put together something which is new in some ways you can see it in your mind's eye, but you can't explain it clearly and concisely. Avoiding unnecessary hype is also important. Also, we're a small company, and we can only do so many things at once. :)
But that's as far as communication is concerned. As far as concepts, share still seems to be a bit off the target from pro. Which brings me to the second question of those three.
It's easy to see clevercactus pro and see only an email client, or a PIM, or whatever. From day one there was the plan to include P2P behavior (as was explained even on the original webpage for spaces). From very early on there was code in place to do this. We had internal versions of pro with P2P collaboration built-in since early 2003 (longtime clevercactus pro beta users might remember the "enable collaboration" switch on the configuration page, which was always turned off for public releases). But as work proceeded it became obvious that the question was less whether "to P2P or not to P2P" but finding effective solutions to do what we really want to do --easily-- and avoid constant, painful workarounds that increasingly don't work at all. Yes, this sounds like marketing-speak, but it doesn't make it any less true.
Email is broken. We all know it. But we keep using it. Why? Lack of choice is one of the main reasons. We use it to send files. To arrange meetings. To do all sorts of things it wasn't designed to do. And spam is a disaster for electronic life. But the important thing to see here is that email is not broken in itself, it is broken for the use we give it. This is an important distinction. It points to a "pincer movement" (you can tell I'm reading too many books about military history, can't you? :)) that has to be made to fix email, which really is just shorthand for "making internet collaboration & communication work better": improve email and other information management (clevercactus pro), but provide simple, easy to use tools to do what email was never meant to do in the first place.
Like, say, file sharing.
Put another way, what I'm saying is that the way to solve many of today's problems with email has nothing to do with email at all. And this happens in other areas too. Email is just one example that easy to see.
And since we're fixing broken stuff, why not other things too? Like, create an environment that is both secure and private. An environment where you know that the information is being transmitted encrypted and directly, in true P2P fashion, to the people you are sending it to. An environment where you can express properly your relationships with people you know and be able to use that (based on privacy settings that default to "maximum privacy" but can be gradually and selectively changed) to do things with your personal network, as you'd do in real life.
So clevercactus now is closing in on the second part of this "pincer movement". The release of share is not the end of clevercactus pro. It's a new beginning that finally brings to light the other part of the puzzle (as we understand it of course :-)).
As we bring back information into the site, clevercactus pro will also resurface over time within the full context that includes these ideas. Stay tuned.
And where's the registration button? I want in!
There is no registration button, since share is for now invitation-only. See the next point.
Invitation-only? I smell elitism!
Come on, admit it. That's what you were thinking. :)
At the moment, share is invitation-only for a simple reason: we care about creating a good product, and we want to improve the product before a general release that would allow anyone to create an account. We want to make sure that we have the main problems ironed out, and scalability issues solved. We will provide discussion forums, etc, and until we do it would be a disservice to users to put something out that would be confusing and where we would be unable to respond to feedback properly.
Additionally, we believe in the concept of eating our own dog food :). So we are using share ourselves as we would use it, with people we know, not sending out hundreds of invitations. That said, if you're really curious send me an email and I'll try to arrange something (no promises though). In fact I haven't yet invited all the people that I want to use the product with. So it's an ongoing process.
As you can imagine, we want people to use the product :-) but a big part of that is making the both product and the experience good, and the service reliable, and as much as we can do testing the only way to do that for us is through an incremental, limited release. We will do a general release when the kinks have been ironed out of course.
Where's the product information? Even if I did get an invitation, I want to check things out a bit before I go into the site.
As I mentioned above, we will be bringing more content online (including stuff similar to what I'm talking about in this post) over the next couple of days.
Tell me more about this P2P stuff. What's this like? Kazaa? Gnutella? Where does the data go? Will you see any of it?
This is P2P, yes, but it's not Kazaa, or Gnutella, or anything else that's out there for that matter, although the ideas have been surfacing recently. Don, for example, a few days ago talked about the pros and cons of P2P NG/Darknets, concepts with which share has some similarities.
P2P connections in share are secure and fully peer to peer. This is P2P like the Internet was at the beginning. There are no proxies involved, not even proxies within the P2P network. We only provide authentication and, if necessary, handshaking between clients. That is nice, but it creates a problem as far as connecting through some types of firewalls, and we'll be working on improving it over the next days and weeks.
This is why I was recently talking about sociable software that is useful for something more than browsing contact lists, and that allows us to use our network as a complement, rather than a replacement, of the real world, while maintaining privacy and expressing relationships properly.
So, please have some patience with us as we get our bearings. We will continue to bring new content online and I'll talk more about it here.
Phew! I might not be getting much sleep these days, but this sure is fun.
announcing: clevercactus share
There are million things I want to talk about, and I'm sure there will be many questions. But that will have to wait for a little bit (just a little bit) of sleep and rest. More tomorrow!
reversing the privacy flow
In a post talking about Python Jon mentioned, at the end, this (in reference to an email quote he made with permission):
Emails from Ross Mayfield, CEO of Socialtext, include a .sig that ends with:It is a great idea (I'll follow the meme as well :)). This led me to thinking further about what's underlying including a signature like that. It's not only a matter of privacy, but also of trust, ie., if you don't trust whoever you're sending the email to, that signature is worthless.this email is: [ ] bloggable [ x ] ask first [ ] privateGreat idea! I've added this to my own .sig.
So this is a kind of purely social control on the openness of a digital medium (in the case, email), and it underscores is that we need to start reversing the privacy flow of applications. What I mean by that is that many applications today (including most current social networking apps) don't have a poor "understanding" of privacy as much as they have a bent towards assuming that everything's public unless you say so. The notion of privacy "flows" from the user to the application, requesting adjustments to the user, instead of the user giving up elements of his privacy to the application (reverse flow). I'm sure many people are ok with that, but I prefer the idea that everything is private and you can open up elements as you prefer.
This makes it more difficult to do things like searching and browsing, since not all the information is public. So there has to be a fundamental design decision made to enable this behavior.
Plus: I find it interesting that the first option is "bloggable" which is basically as public as anything gets. Bloggable replaces public, and it's reasonable in the context of blogs as "information that can spread out of control." Could we not have a category called "public for coworkers" or "public for your friends"? Yes. But in this case the levers we can pull are useless, because your trust relationship (which determines whether that message is useful or not) is with the first person; once the information goes beyond that trust relationship (eg., to your trusted contact's coworkers) then they are not bound by it. Sure, your trusted contact might point out that it shouldn't go beyond them, but when they decide not to mention it they will have to act not based on their trust relationship of your contact, but on the aggregate of your contact's trust relationship to them and the perceived value (for them) of your contact's trust relationship to your (whom they presumably don't know). That simple statement, "bloggable" is an acknowledgement that once you go your immediate trust circle you have for all purposes lost whatever control you had of the information spread. All or nothing.
Makes sense? I think a diagram might be in order ...
social networks: the glue for next-generation internet applications
Bill Burnham (who, among other things, is a managing partner at Softbank Venture Capital) has an interesting piece today on social networking applications. Quote:
Without some kind of application to force the regular use and maintenance of such networks, pure play online social networks are destined to become as stale and appealing as two week old bread.Maybe. Interestingly I think that his mention of "pure play" has another dimension beyond what he means. I think that social networks as we know them today are, in part, in some weird way, piggybacking on the reality TV craze (or both are riding on an undercurrent of "reality fetish" that has emerged in our society). The idea of "play" (as in playfulness) is more powerful than it would seem, and it's definitely a factor. Partially, it's a game.
Or in other words, entertainment.
Now, this is not my cup of tea, so I naturally drift towards the first part of that quote, that without an application on which the network can sustain itself, it would eventually whither.
Social networks might be overhyped today, but no more so than the web browser was in 1995. Both, in my view, are critical pieces of infrastructure. Further, "social networks" have existed in limited forms online for some time now, so part of what's happening is the realization that the network of a person is crucial and can be nurtured to enhance the online experience in general and certain applications in particular.
What's powerful about this idea, to me, is not what they are today, but what this laser-like focus on their nature enables, just like one of the web browser's more important contributions was not the browser itself (ie., the component) but the notion that an application didn't necessarily belong either on the client or on the server, but could (and should when necessary) be split in two and thus improve the networked experience (ie., applications that embed browser or browser-like functionality). This has been the driving force behind my work in the last few months in fact: to allow people to leverage the qualities of personal networks and make them useful.
What are these qualities? In my opinion, above all, they can carry over a level of trust from the real world. This can then be applied to re-create real-world activities online, even if they still have a ways to go before they enable more than they restrict. Identity is enhanced. Interactions have less "friction" (friction is increased by anonymity, since we have to double-check the information we receive, etc).
There's another factor to take into consideration: that social networks, while infrastructure, are "soft", in the sense that they have no hard boundaries that define them (which lock out overlapping functionality). The web browser, for example, has hard boundaries, you don't generally use two of them, and in the end you don't care as long as it works. Social networks, however, are not only dynamic, as Bill notes, but also multiple and overlapping. Many people don't think twice about running two IM clients side by side, just as they don't think twice about having a group of friends from work and an entirely different group of friends from, say, college. Sometimes these groups intersect and/or interact, true, but they can be, and generally are, maintained and evolve in parallel.
Social networks are the glue for next-generation apps, and done properly they can bring together cyberspace and meatspace much closer than they are today. Explicit social network functionality might not become widespread by itself but for a few exceptions, but their principles and underlying notions will become part of the fabric of interconnected software, just as surely as the web browsing has become part of the fabric of applications today.
In other words, "pure play" social networks (in both senses) by themselves might not be sustainable beyond a few players , but their underlying principles will enable enable new types of applications and frameworks that can---and most certainly will be.
 If you're skeptical of even that, consider that Opera, with a product in an ultra-commoditized market, has grown to the point of filing for an IPO recently.
new clevercactus website... for a little bit
A new (temporary) clevercactus website is now online. Very mysterious :)
To those that have been reading this weblog for some time now, clevercactus share was code-named kitten. It's been a long road. Almost there!
Meet Tim McAuley (or his blog rather), the newest member of the clevercactus team. :-) (He actually starts working with us next week). When I met him a few weeks ago he was (as he mentions in his post) a bit of a skeptic regarding weblogs. Okay, maybe more than a bit. You can imagine, however, that I babbled on about the benefits of weblogs, decentralized communities, and so on, for quite a while, enough for him to consider giving it a try. Very cool.
Tim, welcome to the cat squad! :-)
first day at the new office
Here are some pictures I took of the offices. There aren't that many of our space (actually I appear to have a certain fixation with my own desk!). I did take a couple but they were out of focus so I didn't include them. We spent the afternoon rearranging the desks and so on. When I was getting back home I realized we should have taken pictures of the place before we got it ready. Oh well. I suppose we can always spend a day messing it up, taking pictures and then putting it back together. :)
Technically the first day was yesterday, but not all the legalities were finished and we just used one of the meeting rooms and couldn't set up the offices.
The phones we got installed are Cisco IP phones: they connect through Ethernet and have an Ethernet-100 output that gives us Internet access on each desk (from the phone!). I babble about the phone because this is the first desk phone that I've used that has its own IP address, network config, DNS setup, :)), ringtones like mobiles, LDAP directory integration (!), updatable firmware, and the first one for which I've found an online tutorial (!). Very cool. (And maybe I'm wrong, but I think those are the phones they use at the fictional CTU in the series 24, which I've always liked). Anyway, not that the cables are that important, once we finished arranging the space it was about 5 minutes before I set up an 802.11g router.
Anyway. It's been one of those days were you feel the change. We were talking with Paul about yesterday's meeting and he said "It feels like it was a week ago, doesn't it?", and it was true. I didn't want to leave :) but since we haven't received the machines yet there wasn't much I could do there (the laptop is good for some things, but can't handle all I need for full development). If I'm not very responsive to email (or IM, or whatever), this is why; just keep badgering me :). We've got so many things planned for the next few of weeks that it's a bit overwhelming, but that's how it is.
In case you missed the link above, one more time: here are the pictures.
:-) :-) :-)
and before I forget...
... another thing. On Friday I mentioned, among other clevercactus news, that we are hiring. I've gotten some comments on that, specifically that a) there is no info on what positions we're hiring for, and b) the website is not very up-to-date. Both true. I said in that entry that I would add more in the coming days but I wanted to clear this up now: we'll be making changes over the next week to the information on the site, etc. That is, we haven't posted any info yet on that. In the meantime, if you'd like to know more just send us an email.
There are lots of news related to clevercactus, and a number of other things I wanted to comment on. This last few weeks have been really busy (being away in December was also a factor) and as a result I haven't updated as often as I wanted. (Plus, I've had a bad cold that I haven't shaken up completely--what with the subzero temperatures and all).
Regarding clevercactus, a few things to start (I'll elaborate more on these points in future postings):
Things are just about to get a lot more interesting.
ps: the category for this entry is still "spaces"--it should be changed to "clevercactus". Will probably do that tomorrow. :)
Part of the work on the new beta of cactus has to do with making the calendar work properly and adding main missing features to it (such as recurrent events, or all day events). One interesting area in this sense is the calendar "properties" or configuration if you will. Here are the main elements that I'm working with at the moment:
Questions: what are other important calendar settings? Calendar functions in general would be interesting too, but general properties are the main focus at the moment. However, if there's any particular element of your current calendar software that you find annoying or some feature that you'd really, really like to have, let me know too. Drop a comment, or send me an email. Thanks!
a small news item
From NewsForge: a short (but cool!) mention of clevercactus on this article: "The Java-based clevercactus beta software looks quite promising." Thanks! :)
clevercactus bug/feature tracker online
Bugzilla was a complete nightmare. I didn't even get past installing all the required Perl packages: the CPAN automatic module installer kept finding dependencies that it wanted to download, apparently ad infinitum, and it kept asking questions to which I had no good answer ("Use package Test::whatever to do X Y and Z NOW? [yes]").
Both Mantis and Flyspray are quite nice, but Mantis is more complete, and it includes options (that require additional packages) that I'll look at in the next few days, such as adding a forum for discussion and anonymous reporting.
The current setup allows anyone to register and report bugs, I'll leave it like that for now and change it only if necessary.
Oh, right, the link: The database is here.
...for a feature-freeze in clevercactus and the subsequent bugfixing only period prior to final release. Part of that is installing a bug tracking system, and I'm now looking at bugzilla, scarab, FogBuz and JIRA. I've used scarab internally up until now, but I want to deploy this in the open and I'm not too happy with Scarab's complexity. FogBuz looks nice and its price is reasonable, although if I'm not mistaken it is fully hosted (an ASP model) for the trial and requires Windows if deployed, which wouldn't work for me. JIRA looks nice, but I wonder if I could use the trial as I want to. Bugzilla might be the way to go then; only problem is that the setup is a complete nightmare. We'll see.
clevercactus beta 3 in the oven -- and some comments
There are lots of things planned for clevercactus beta 3. I am eyeing the release of an internal version for tomorrow to solve some issues that surfaced in the last couple of weeks. Then there's the issue of not just completing/exposing features that are already there, but also of properly exposing/explaning a lot of the concepts and features that clevercactus has. Because it's (relatively) simple, and it has a (relatively) straightforward UI, users seem to see the features they're looking for, and the rest of features don't get noticed so much (until they are needed). This is great, but it's much better when you're aware of what something can do, and then use it when it's needed-- and improved documentation plays a big part in that, as well as adding more UI hints for other functionality without adding cognitive load to the UI. That's the problem with any platform I guess. A bunch of ideas have come together in the last few days, maybe not to much in terms of "features" but particularly on how to articulate how the cc functionality will solve some of the problems that we're facing today in terms of collaboration, information management, spam, viruses (yes, even that), and so on.
Something else I haven't done in a while is point to comments about cc (Some of the comments are a bit old, but they apply to beta 2 as far as I can tell). Here are a few I found after digging through my referrers (yes, yes, I miss a lot with that system, gotta get a better one...if you come across some please let me know): Gary, who is tracking the "information client" space, has added cc among a list of other distinguished entrants. Justin has a bunch of nice comments on it and would like to see more "correlation" tools added--as I do. Every information item we own is essentially the root of a tree of related elements, and it makes perfect sense that cc will eventually allow you to navigate those relation trees with ease. Kristina really liked the ideas behind it, but has been too busy to give it a full try yet :-). Josh also found it interesting and was looking for what it provides, but was slightly dissapointed to see that it was still in beta (the final 1.0 is approaching, worry not!). Then the folks at B.Mann consulting (did I get the name right?) had previously looked at spaces and have now found that clevercactus is the new name for it--and they still like it (and the Mac-UI problem is something that's pretty important too). And, from the I-can't-read-the-comment dept (okay, I can read it, but I can't get the nuances!) here are some comments of clevercactus in the context of advanced information management tools.
And, finally, this just in: clevercactus has been featured in the latest Swing Sightings!. Yeah! :-) Isn't it cool of the Swing team to maintain that service. And, yes, beta 3 is coming.
Thanks everyone for the comments!
Okay, enough babbling. Back to work. :)
news in one page -- but not a webpage
Another "clevercactus tip of the day". :-)
The other day, Dave was saying that using the three-pane paradigm to present news in an aggregator was not the way to go. Today, he linked to a piece by Adam Curry in which, among other things, Adam puts forward the same idea. One important difference (aside from the UI) between typical three-pane readers and web-page readers is what Dave refers to as "the queue". Adam mentions that another advantage is that part of the "freshness" of RSS feeds is how you can get a view of a number of different POVs on the same idea.
Both of these uses (and more) are not just legitimate, they are also clearly useful in many cases. But sometimes I want to see what just one person has said recently, and for that the three-pane view rules. Additionally, at times an item might contain more than a short description, it might contain the full text of the entry, for which three-pane views rule as well (since the full text could be really long).
So, is it possible to have both?
When reading feeds in clevercactus you can get the combined effect of a "one-page" view with the benefits of a three-pane view. As an example, consider the following two screenshots. Number one shows the "aggregated" view, where all the feeds are shown together. Number two shows only the space that is aggregating Scripting News. Just one click takes me from a bird's eye view to a more specific per-blog view.
How does this happen? In clevercactus, a space is recursive, that is, it automatically shows not only its contents but the contents of its subspaces (which are also recursive, and so on) with the proper ordering. So this effect happens transparently and automatically.
Now this is close, but no dice yet, since (as opposed to the way in which webpage aggregators do it) the news items remain there in the aggregator, which can potentially (and very quickly) create clutter (This is another one of Dave's good points). In an upcoming version of clevercactus, you'll be able to set the "decay rate" of items (automatically set by default to a sensible value). This means that after, say, seven days, feed items will be deleted automatically unless explicitly marked for permanent storage through a one-click procedure on the item itself (An icon saying "Store this") -- And bingo, we've got the functional equivalent of the aggregated webpage view, but in a three-pane UI, with "entry decay" for the first time.
Some users will still prefer a webpage view (which you can also get in clevercactus using webaccess, btw!).
And that's what it's all about isn't it? Giving the users choice.
Update: Ted was commenting on this entry and comparing clevercactus to FeedDemon. He said:
it seems to me that Clevercactus looks a lot like FeedDemon, except that FeedDemon can do a newspaper, and Clevercactus can't, and Clevercactus has PIM info, which FeedDemon doesn'tI didn't know about FeedDemon's Newspaper View. Very nice! However, I must say that clevercactus has a lot more than "PIM info"-- some examples are the webaccess feature, weblog posting, email, and Real Soon Now a bunch of other cool things, such as real-time collaboration and synchronization. Reading feeds (Both RSS and Atom) is just one more of the information channels that clevercactus can handle.
Update 2: Sam linked to this entry and Dare commented that SharpReader and RSS Bandit support this kind of functionality. Dare was also wondering whether I thought that cc was first on this, and I replied:
Clevercactus was first. CC was originally called spaces, and it was released publicly on November 11, 2002 (the name changed in April this year), supporting exactly that functionality (on both feeds and email), which predates most if not all the 3-pane aggregators that I know of, and certainly those that do recursive aggregation (of course, corrections are welcome--you never know :)).As usual, corrections or more comments are welcome. :-)
Update 3: In another comment on Sam's blog Dare noted that he implemented "entry decay" in RSS Bandit about a month ago. It's not deployed (then again clevercactus isn't either), but this predates my first public mention of this feature by at least a week (obviously more, since Dare noted the release of the code on that entry, rather than the idea as I did). Cool.
PS: Are we all in sync or what? :-)))
clevercactus beta 2 tip of the day
I mentioned this on the cc-devlist but didn't make a big fuss about it later--it's very useful though, so here it goes:
If you have installed the latest beta of clevercactus using any of the two Windows installers (from the download section) you can now go to Control Panel > Internet Options > Programs and select "clevercactus" from the list, to set cc as your default email client.
For other operating systems, you can set the default by using the options to load with a mailto: address, run the executable (or the JAR file) with the -mailto parameter, for example:
If clevercactus is loaded, the running copy will launch a new mail window, otherwise it will load, then launch the window. Additionally, if cc is already running, and you run it again, the effect will be to bring the currently running copy to the front.
Nice eh? :-)
identity != me
Joi on how identities are different than people. What can I say except: I agree with those concepts completely--and I think that the concept of identities should actually be one of the pivots of all our digital interactions, rather than some kind of useful add-on (not that Joi implied that it was an add-on, but sometimes they are discussed in ways that don't fully reflect their impact). In clevercactus, for example, you can define identities for different activities, which are "expressed" through synchronization devices that are currently email and weblog subscriptions. But this is just the beginning. We are extending this idea to other realms--real time sharing to begin with, and others down the road. Stay tuned.
clevercactus beta 2
Extra! Extra! Read all about it!
clevercactus beta 2 has just been released. A lot of improvements and fixes in this release, as well as a few hints of what's to come in the area of collaboration. Here's some of the most important changes:
...And more! :-)
You can get it from the download page (yes, I know I linked to it above... but can't I enjoy it a little? :))
Anyway, comments/questions/etc are welcome!
cc beta2 r2 & other things
Just released to the devlist a new rev of beta2, this one fixing two important problems that appeared since the first rev yesterday (those are: layout changes, and the vanishing of the "new weblog entry" button).
In a comment, Jamie was asking me to stop stonewalling and start talking already about my experiences with Symbian/J2ME (of course, he was a lot more diplomatic :-)). That's definitely coming up, I've been reading up on the subject, and I'm getting a better understanding not only of some of the strategies (e.g., the whole idea of Profiles in J2ME, or why PersonalJava was canned) as well as the development tools, etc. Cool stuff. Comments will follow. :-)
clevercactus beta 2 r1 internal release
"Good news, everyone!"
clevercactus beta2 r1 has just been released to the cactus dev-list. Many changes and fixes for this version, and a few tantalizing clues about upcoming features (Screenshot in this earlier entry). The new dynamic tabs are in, as are the "Query Spaces" (persistent query-based spaces) and the much discussed horizontal email layout (the default is the "standard" layout though). Also: preliminary (n)echo support, OPML import/export...
Anyway, the plan is to do about a week more of testing on the release, and then go for public beta2.
Double phew! :-)
rss performance improvements
New cactus beta2 feature of the day: RSS-download performance improvements! Aside from tweaking the parsing a bit, I was reading Mark today, and he was reminding everyone about some bandwidth- and performance-tips for RSS readers, including use of conditional GET (which Dylan first brought to my attention last year) for HTTP and GZIP encoded streams. So now cactus beta2 supports both. Huge difference in check times for feed changes, and for download times when there are changes.
And, btw, I've also added GZIP streams support to this site (after a bit of fiddling, as usual, the docs for the configuration are all over the place). Two in one: Bandwidth saver -- and a good way to test cactus!
clevercactus beta 2
Lots of improvements, fixes and new features coming up for clevercactus beta2. Because an image is worth more than a thousand words (and because I won't be able to write a thousand words), here's a screenshot. In it, among other things: the new tabs (the tabs are back! :)), the new horizontal layout, and display of a necho feed (actually, Mark's prototype necho feed). And, before anyone starts on how the horizontal three-pane view is bad, here is a screenshot taken a few seconds later but with the "standard" view; the layouts can be switched on the fly (through the View menu). New things in different areas, too (and the screenshots contain a hint of another upcoming feature...).
So, more updates soon! In the meantime, back to work. :)
necho & OPML in cactus
Today I added (among other things) preliminary necho support for the aggregator in clevercactus. It didn't take long, and detection of the feed type is transparent like before with no problems, which is basically what I expected.
Another important addition for the aggregator side of cactus is the new OPML import/export feature, or as I call it, "the Russ feature" since Russ has been its most vocal advocate :). Just like email import is essential to try out a new email program, an easy way to import (and export) of a user's feed subscriptions is also crucial to try out the aggregator. (OPML is actually a generic outliner format, more flexible than to be used only for subscription lists, but it's widely used for that anyway, and many news aggregators allow import/export to it).
Looking good so far. I'll probably post a screenshot later today, or tomorrow.
the three-pane question
Software that deals with information in general, and with email (or RSS) in particular, has, over the past few years used a relatively standard three-pane interface, as shown in the following sketch:
The "all horizontal" mode is better IMO, since it gives you more space to view information (which is typically "long" rather than "wide"). It also makes it easier to follow the progression of "contained" information: as you move along the x axis to the left, you increase specificity--this consistency can then help in other areas of the UI. Apple for example has used the all-horizontal paradigm for one of the settings of the file browser in the Finder, and it's much, much better than the alternative. Simple and easy to use.
The all-horizontal mode has one drawback: You need a bare minimum of 800x600 to be able to use it well. (or maybe 1024x768?). This limits it mass appeal a bit, but it's probably safe to say that people that use 640x480 are probably late adopters, and not likely to be trying out new software anyway.
So I'm considering making the horizontal view the default for the next beta of cactus, and it might even be useful to allow in-place editing of some elements (say, contact or task information). Comments?
invisible features[via Erik, Charles]: Joel gets it right in his short comment about invisible software features:
Unnecessary UIs [...] that pop up to brag about a cool feature the developer implemented are a little bit obnoxious. Too many software developers just can't bring themselves to implement completely invisible features. They need to show off about what a great feature they just implemented, even at the cost of confusing people. Really great UI design disappears. It's a matter of taking away, not adding.I agree 100%.
This is really hard to do, and it definitely takes discipline. For example, a lot of work in clevercactus has gone into creating deep integration without forcing abstractions, a bit like playing around with puzzle-pieces until they fit. The result is that, if you look at a screenshot, cactus looks very much like an email reader. This is both good and bad.
It's good because there is less cognitive overload (in UI-speak), and because features "appear" when you need them, and can be ignored when you don't. It's bad because users might tend to focus on one feature alone, and not see the other features at all. And, this not by any fault on the user's part, but because the program doesn't expose them properly.
For example, a couple of weeks ago Francois posted a comment about cactus on his weblog that is a good example of what I mean. He discovered things incrementally, things that were "invisible". Francois knows software, and he can poke around the program and understand what he's seeing. Most users, though, won't. They probably won't get to the point where they have to ask themselves "What is this weblog posting thing he's talking about?". They won't see it at all. And, arguably, no one should have to spend time discovering features.
So cactus needs more work in creating "soft exposure" of features that are available. A short tutorial would be useful, but the UI needs soft clues to show the user what the program can do. Clues that are soft enough so they can be safely ignored, but also visible enough so that if the user is curious, and has some time to spare, the feature can be checked, help on it obtained, etc.
Which brings me back to Joel's example.
Apparently, this is not necessarily the case Joel was mentioning. His example is a feature where there should only be one behavior, and so asking the user is overkill (and/or showing off). The feature should be invisible, yes. But somehow the user should be made aware of it. Why? Well, for starters, it's good to know what your software is doing to your data (if the data was created by for program for internal use it's a different matter), and then we have the more prosaic case in which a user wonders "I know this site changed its URL, I wonder how did the program deal with it". Giving information in this case seems eminently useful. So how to do it? In this particular case, in my opinion the URL should update itself in the bookmarks with no message, but the next time the user opens the menu the bookmark will appear with a different color and maybe with a temporary submenu or other UI widget to "learn more" about what the color means. The temporary UI widget would explain what changed and why. User satisfied, feature exposed. But only if necessary.
clevercactus versioning information
There were a couple of comments by Greg and Dan to last week's entry on cactus' html editing (thanks!). Dan in particular raised a good point that I want to clarify. Dan said:
I keep checking the clevercactus website for an update. But, the beta page doesn't mention anything about versioning. Is there a good place to go to check for future versions, or can I just assume you'll blog about any updates?For the moment, yes, I'll post information about cactus releases on this weblog (under the spaces category, which has its own feed to which you can subscribe, links on the right navbar of the page as well--I know I've got to change the name of the category but I didn't want to deal with the URL redirection config just yet). Another option is to subscribe to one (or both) of the clevercactus mailing lists, clevercactus-info in particular is for announcements only.
That said, with the next beta we'll start a clevercactus-only weblog that will be used for announcements and other things. As always, if there are comments or other ideas for improving this, I'm all "ears". :-)
clevercactus WYSIWYG HTML editing
I've been using the new clevercactus build (internal build, that is) to post weblog entries and edit them using the new WYSIWYG editor. The difference is amazing. Editing raw HTML can be a real pain, and it's easy to make mistakes. It also becomes much easier to comment on RSS feed entries. Still some problems to work out (Java's EditorKit package is really powerful, but incredibly confusing and hard to use, and with few examples for HTML editing) but it's already reasonably functional.
a cactus update
A short status update on clevercactus: lots of work in progress to improve current features and add some other important missing elements. On the works:
cc mailing lists
The clevercactus mailing lists are back! (In reality they never "left" :) but access to the subscription page was disabled after the transition of the site from spaces to cactus. Still more things to come on the site...)
more cc comments
Dan has been trying clevercactus as well as columba. He has some comments about the filtering capabilities (or lack thereof) in cactus--and he's right, there's little there except basic rule functionality and some of the automatic behaviors (for example, if you have a contact that matches an email the email will be automatically routed into that contact's space if no other rule matches). More filtering capabilities will follow in the next beta.
Ricardo has also posted some comments, particularly on the weblog sync feature. He notes that it doesn't yet sync back the current posts on the weblog (you have to subscribe for an RSS feed for that) and he's right. It's just that it takes a bit longer to clarify how to define what gets synched (it wouldn't be good to sync the entire blog if that's not what you want...) as well as coming up with a UI that lets the user deal uniformly with different weblog tools, which have different APIs and therefore will support different capabilities.
Thanks for the comments Dan and Ricardo!
Erik now has a column in the Java Developer's Journal Newsletter. Congrats Erik! (A link to just the column can be found here). And for his first column he included a link to the my entry announcing the clevercactus release--and some of the comments that followed. Thanks!.
some comments on the cactus release
The clevercactus public beta has been released, three weeks after the original internal beta release--along with the website and the formal announcement of the company and Paul Kenny's role as a co-founder. I've been working with Paul for the past several months; he's a great guy, and with tons of experience with technology startups.
It's definitely been a ride so far, fulfilling in more than one sense, not least because of all the great people I've met, that have helped along the way, and all I've learned. It was six months ago, almost exactly to the day, that I released the first public alpha of spaces. It feels like six days! Back then I said "this is just the beginning". Boy, was I right on that one. :-)
I'll be posting more information over the coming days--things are about to get even more interesting. And it's still just the beginning!
I'm a machead
I just got running an old G3 powermac (from the time when powermacs came in purple, like the original iMacs), with OS X to do some clever cactus testing and debugging, and, more importantly, to improve its L&F in the mac environment. I've got a flat panel with both digital and analog inputs; the digital input is connected to the PC and the analog to the Mac. A press of a button changes the selected input source.
Within a few minutes of playing around with the Mac, I was just amazed at how much I didn't want to go back to windows. When I did switch back, Windows just felt ... clunky.
I had a mac a few years ago (a powerbook G3) and it was nice, but it had OS 9. OS X is just amazing. The UI, both in L&F and behavior, is incredible. Maybe this is an effect that will wear off, but I hope not. :) I could live all day with a UNIX like this. Too bad the machine is not powerful enough to do development in it, but I'll certainly give it a try when I've got more time. In the meantime, I am just enjoying the experience.
OS X rocks.
(simple) roundtrip weblog/RSS/email integrationI've just completed (including testing) the integration of the weblog posting code and UI into clevercactus. Now I can not only post to my weblog from within cactus, but I can also repost an email to a weblog entry (after editing of course) and, even better, I can use an RSS feed item as the source. So commenting to other posts on my own weblog is now really, really easy. A clevercactus first! In simplicity of setup and use at least :-). No more copy/paste madness. It's compatible with Blogger, Radio and MovableType, which in the end meant that I had to implement different calls for each system (or slightly modified calls, in the case of Radio and MT--but enough to force me to test everything twice--last week's work on Weblog APIs, and all the comments in the entries, really helped me understand better the current state of things, what to expect, and not to expect).
The new functionality also means that if, say, two people are subscribed via RSS to a weblog and they can both post to the weblog as well (with different users for example) then presto!: you've got a weblog-based discussion space, mirrored on the client, with no need for administration aside from what you were already doing to maintain the weblog. You can publish new entries, or republish from any other data source with or without comments, either from cactus or from the weblog software. It's only scratching the surface of what cactus will be able to do, but it's a good first step. I'm happy. :-)
Now to fix some remaining bugs so I can release all of this in a new beta (r6) to the list tonight...
ps: of course, I'm posting this from within clevercactus. :-)
syncml and palm
Still in a rueful mood regarding the problems with Palm and its Java conduit SDK, suddenly I thought: wait a minute, what about SyncML? If that worked, there would only be need for a SyncML implementation, which would then work across Symbian, Palm, and maybe even (gasp!) PocketPC.
A quick search for SyncML and Palm turns up this old press release which identified Palm as being one of the founders of the SyncML initiative. While it doesn't seem that Palm has an SDK to support SyncML (searching for SyncML-related stuff at Palm's developers site turns up nothing of use except press releases), there are some indications that this should work. For example, this article from IBM DeveloperWorks that talks about SyncML device management talks about a reference implementation SDK that runs on Windows, Palm, and Linux. And maybe using Palm's Bluetooth SDK would help. Maybe, just maybe...
swamped by google
I was totally confused by all the extra traffic I started seeing yesterday especially since most of it was to a trackback entry.It is weird that this happened, but considering that the release and all the links leading to it happened close to the last Google Dance (when Google rearranges its rankings massively) it's likely that not enough link information existed and he ended up first for whatever reason. It will probably be corrected when the clevercactus website is up, and as Google updates its rankings fully again at the end of the current "dance" (which started yesterday), or maybe the next one, since current results for that search on all eight datacenters shows the results changing but not by much. In the meantime, I can only hope Koz's site will hold up! :-) Thanks Koz for the pointer and the comments.
first post from clevercactus
This is the first post to my weblog from within clevercactus. Several UI elements to figure out completely yet. But getting there! :-)
clevercactus palm synchronization
I had tested Palm's CDK a while ago, and it seemed to work properly. So, to integrate the feature into clevercactus, yesterday I dusted off those files, reinstalled the CDK in the new machine (after a painful 4 hour download), and went from there. Everything appears to be fine. Then I start to integrate the code into the cactus codebase. Compile. Sync.
This was working just a minute ago...
So I start to poke around and I find that JDK is complaining that the classfiles have the wrong minor/major version number. I check and sure enough JSync, the Java part of the conduit, is using a JRE installed in the Palm directory.
Version of the JRE: 1.3.
Okay, so I rename the directory that JSync is looking at, hoping that jsync13.dll will find the JDK 1.4 on its own.
Reload Hotsync Manager. Sync.
"jvm.dll not found".
At that point I suddenly ask myself: can they be so stupid as to completely hardwire their implementation to a single JDK release?
Yes, they can.
The registry is littered with references to that particular JDK. Furthermore, I changed the directories and copied, by hand, the JRE directories from 1.4.1_02 into the directory where JSync was expecting to find the JRE. Still no dice: JSync is even looking for jvm.dll in the bin\hotspot directory, and since 1.4 Hotspot is installed either in bin\client or bin\server. No problem, I say, rename "client" to "hotspot".
"Too many files." says Hotsync Manager. And dies. WTF?
Request a verbose output of the error from Hotsync, and what do I get? Sure enough "Invalid version."
So, the exact same code that runs on 1.3 doesn't run on JDK 1.4 because some genius hardwired the DLL to look exactly for version 1.3, or fail otherwise.
Google provides no help on the matter, nor do Palm's own forums (the only message in their developer forum for conduits that has to do with this is from February this year, and has received no replies). Apparently no one noticed that JDK 1.4 has been out for about a year (since the beta versions). Too bad, because the JSync suite is pretty well done, including installers that you can deploy with your application and so on. Back to a COM-based solution it seems.
a new beta (rev5) of clevercactus was released yesterday to the list, and so far it seems that the IMAP problems have been solved, with lots of help in in getting debug traces from the people in the mailing list. Also, I now have 5 different IMAP servers installed in my machine for testing. There are a couple of new problems, but they are not easy to reproduce; I'm working on them. However, since the core code for communications now is approaching stability, I have begun finalizing code that has been around for a while, but never properly integrated: code for synchronization. First in line is XML-RPC Weblog sync. I've been looking at what has changed on blogging software APIs since I last took a look at them in December/January (summary: not much). I've been writing up a review of how they compare, and will probably post it tomorrow. That aside, I'm also advancing on Palm and Symbian (SyncML) synchronization-- freshening my knowledge on both, looking for new development tools that might have appeared and so on. I had downloaded and installed the Palm Conduit Development Kit in my other machine but now I can't find the install binary--so I have to download it again. I was kind of hoping that I'd have DSL by now, but Eircom has outdone itself again--3 weeks and counting. We'll see if it comes in next week, in the meantime I've just started to download it, and at 56 Kbps those 45 MB will take about 3 hours. Oh well. I'll do something else in the meantime. The plan is to release a public beta by the end of next week. Lots of things to do!
And, another thing: Matthew Walker sent a shell script to run cactus on Mac OS X using the extensions defined by Apple, so that it can integrate better with the OS (e.g., the menu appears on the top OS menubar, rather than on the frame). He also sent links to a couple of screenshots showing the effect of the change: before and after. Very cool. Thanks Matthew!
clevercactus on google
That was fast: clevercactus now holds the first two pages of hits on google for the name with the two words together ("clevercactus") and it appears fifth (referencing one of my blog entries) when looking for the words separated by a space ("clever cactus"). Wow. Now that was fast. Considering the name was "released" only five days ago...
configuring courier imap: bring on the valium
You think I'm exaggerating eh?
Yeah, well, maybe. :-)
Then again, what to expect from software that, in the source distribution, comes with a file called "00.README.NOW.OR.SUFFER"???
I ended up getting an RPM package of it, for Red Hat Linux 7.2 (Mandrake). I created a new VM copy (for VMWare) and uninstalled UW IMAP from it, then installed courier.
It took me about an hour to discover that I had to create a ~user/Maildir directory using "maildirmake". Anyway, now it's configured and it appears that I can reproduce the IMAP problem that's been showing up with clevercactus in John Rubier's, and possibly also James Murphy's installation (James' problem could be different though, he uses an e-smith box, and as far as I can see that comes built it with UW IMAP--which I've tested against successfully. We'll see).
Anyway, now I can actually begin testing and finally fix this problem.
And I'll say again: VMWare rocks. :)
clevercactus: the UI changes
I've been meaning to post this since friday. So, finally, here it goes!
Some comments by email (particularly on the dev-list) had to do with the new UI of clevercactus. I want to talk a bit about that. But first, a screenshot:
(click on the image to see a larger version)The difference that can be seen right away with respect to the alpha is that the tabs are not there anymore. This is temporary--let me explain. I have already talked about it on the dev-list, but I thought it was useful to do it here as well.
Apart from the "navigation bar" and more filters (to, say, "see only the messages received in the last week") The new navigation interface includes the back and forward buttons. The buttons get activated when switching any of the parameters (say, going from Messages to Contacts, or changing the sorting, or, again, viewing only messages from last week), letting you navigate quickly between views. The buttons don't keep state between runs (although if it makes sense they could do that eventually), and they maintain state when switching between spaces. Soon, each space will have its own navigation history. Additionally, the tabs will be back, but they will be dynamic. The final result will look very much like a modern browser interface: the back/forward and the view filters define the current view, and you will be able to create multiple tabs dynamically, making it look like "the old spaces" if needed, or maintaining multiple views that you normally use, for example, view all Unread messages, and view all contacts.
The tabs are, IMO, crucial, and as I said they will come back, but improved. The default setting of the tabs will make it look like the spaces alpha, and you'll be able to ignore the navigation features if you want, or use them. The idea would be to move further into the concept of information as something dynamic, "browse-able", in a sense your own personal web of data, rather than the more static view that Folders provide today in other programs.
clevercactus imap update
Regarding the IMAP fix in yesterday's release of clevercactus beta 2, it turns out not all the problems were solved. Gary reported that beta 2 fixed problems for him, but John and James are still seeing issues. I just saw as well that Koz gave it a try and found login problems, thanks for the report (and the nice comments)! Koz also commented that it would be nice if RSS messages used the more of the RSS information: I agree: there are many cool things that can be done with RSS and weblogs as well. For example I've been looking at OPML, which I've noticed some people use as a way of holding information on the feeds they read (although that's not its primary purpose. I admit I'm a bit fuzzy on that, if anyone knows more, or thinks I'm wrong, please let me know)--it would be great if clevercactus could just create a set of RSS feed configurations from reading a single OPML file off somewhere. Ditto for RSD.
So the next step is to start testing with several different servers. So far I've been using a local server on WinXP and the one on my linux box. I've now downloaded courier 1.5.1 and 1.7.1 as well, which is what John Rubier uses, I'll try to get information on the server used from others for whom it's failing. I think it must be a small difference in how the command is set for those servers, or a slightly different interpretation of the standard.
Good progress so far though.
and, by the way...
...I just released clevercactus beta rev2 to the dev-list, with a few important fixes. Among them: one for IMAP folder parsing. Let's see if it behaves better now.
the day after
Okay, so in retrospect saying that I was going to post screenshots and other things I've been meaning to blog about later yesterday was overly optimistic. At that point I had been awake for 50 hours and it's probably to be expected that I was hallucinating :-).
I haven't yet caught up with all the emails/feedback on the beta, which flooded in yesterday after the release. I'll comment on some of those.
First, I am pleased with how the release went. There had been so many changes that upgrading a user base at this stage was difficult, but it went well. I want to thank those on the spaces-dev list for their feedback.
Curtis, who had tried unsuccessfully to import a large Eudora store into the spaces alpha, reported importing (after some hoops) 69,000 messages (!) into cactus, albeit after several tries (and with the problem that cactus was "splitting" mailing list digests into individual emails. Through some logs he sent, I have determined that the bug is in the parsing of multipart/digest MIME messages--so that will be fixed in the next rev. Memory usage also seems to be jumping over what should be expected, which is strange, since the VM memory settings are the default. Regardless, as I mentioned in a message to the list, the mere fact that we're discussing problems in dealing with 69,000 messages is, to me, a success. A lot of the work I've done over the past few months has been precisely aimed at dealing with large message stores, and it's getting there.
All considered, the migration process worked relatively well. Gerard, who has always given great feedback on spaces, was the only report of problems with the spaces-to-cactus migration (of the database) hopefully I'll get the error logs from him and so will be able to fix it.
There seems to be a problem with reading some messages--James Clarke has just reported something to that effect to the list, and I had seen it myself this morning: the messages appear empty after a point. I am looking at that now.
Another important problem that I'll work on over the next few days, before the public release, is some unexpected issues with IMAP: it seems that some servers are complying with RFC 3348, which specifies extended behavior for the IMAP server, and apparently the LIST commands are failing when looking at the contents of a mailbox. I would have expected that a server implementing extensions would implement standard RFC 2060 IMAP behavior as well, but that doesn't seem to be the case (admittedly, I might be wrong, but that's the best diagnostic of the problem I have at the moment). Both James Murphy, a.k.a. Murph, and Gary Coady have been very helpful with their reports, hopefully in a couple of days I'll fix the problems and they'll be able to use the new version. Murph in particular has been really patient in waiting for better IMAP support, and I don't want to dissapoint him :)
There were some comments on weblogs as well: Russ posted an entry with nice comments, and giving some of "Russ's rules for starting a company". Very cool. nf0 posted a comment and some screenshots of cactus running on OS X. Great! (Once the first glitches of the beta are fixed, better OS X integration will be a more important priority--I noticed that there are some layout problems still, which I've known to exist depending on font size settings--and they will be fixed). Matt also posted some comments, as did Anders. Thanks to everyone!
On the announcement-entry there were some comments from Bruno, Anthony and Russ regarding the name "clevercactus". Bruno noted that there is a product with the name "clever cactus" that is a ... "100% all natural prickly pear cactus perfect for grassland/desert tortoise and iguana species". LOL. This is the first hit for "clevercactus" on google (actually, the only hit for both words together, and the first for the words separated by a space. I had indeed seen it, and I don't think there should be problems with that. Trademarks apply in specific domains, and I think that software is far enough from cactus seeds for iguanas to avoid causing confusion. For example, if you look on google for "apache" you get, of course, the Apache Software Foundation and related links. However, if you look for "apache products" you'll notice that a lot of the links are a) companies that provide services for apache and, more importantly, b) companies called Apache or with products called Apache that sell Apache Indian stuff. That said, the naming of the product is not final at this point.
Anyway, I have some more specific comments on the new UI, but that's for the next post, along with the screenshot.
it's a CAT!
I've just released the beta of spaces, under its new name: clevercactus.
Lots of progress on the company side of things (more news soon). Clevercactus will be the company name, but the product name is still tentative, as is the logo--they're both in "beta" as well.
cactus is the cat's name, and the company's mascot. :-)
I released the beta internally to the spaces-dev list only, to allow for a few more days of testing, as well as polishing some details while starting to get feedback on it. The version will be released publicly next week. New features/Improvements include a streamlined UI with better navigation, more powerful filtering, optimized memory usage, speed, import speed and new implementation of the network protocols, among many others, and even a surprise new feature: Hotmail synchronization!.
If you want to download the beta to check it out, you can join the dev list, or and I will reply with download instructions. Offline for a while now! Over the past couple of days I've accumulated a few things I wanted to comment on, but I simply didn't have time. All of that (and some clevercactus beta screenshots as well!) later today.
full speed ahead...
...on the beta of spaces, and so zero-time for the blog, along with other minor things like sleeping. However, I do think that if I had DSL I would blog even when working--I do take short breaks, but I don't immediately think of connecting. When you are already connected, though, the situation changes...
Anyway. It's probably a couple of hours away from being release-ready: I'm doing final testing on several new features to make sure they work properly. More news soon.
a brush of slashdot
I have slept about 2 hours in the last 36, but the pre-beta release of spaces is almost done. In the meantime, I've been so consumed by it that I mostly stopped checking news, stats, technorati and everything else. And lo and behold, what happened? Yesterday there was, predictably, a thread on the Chandler 0.1 release, and two people made comments pointing to spaces (here and here) with the (also predictable) jump in downloads. It took a while to find (I first knew about it because of an email) because the first comment, which appears close to the beginning and so would generate the most hits, has the URL for dynamicobjects but not a link, so it wasn't showing up on the referrers list. To the people who posted the comments, thanks! :-)
Btw, the first comment on Chandler on the slashdot thread is funny. It reads:
already emulating Outlook wellThe perils of native code... not that Java will never ever crash, it just won't be a common occurrence and would normally result from a bug in the JVM.
I'll take that with some reverse engineering on the side
So I'm coding away, to the tune of U2, INXS, and Jerry Lee Lewis, as the pre-beta release of spaces is approaching fast.
I've spent the past couple of days redoing a lot of configuration infrastructure, trying to make it easier to configure mail accounts and so on, which are now Synchronization devices. To take breaks from the grueling UI work (adjusting locations or spacing by a few pixels to make things look good is never a lot of fun), I've been reverse engineering the HTTP access protocol of a certain web-based mail application <wink>, not just to include support for it in spaces, but also because it is useful to better understand different usage models for the internal synchronization abstractions that I'm using. WebDAV is used for some of the query mechanisms, so I've been looking at that. And for the reverse-engineering, I've found useful to fire up Network Probe. I've had to go through some hoops to do it, since I don't have DSL, and so my internet connection is modem-based... and Network Probe latches onto Ethernet or Firewire devices only. So I'm routing internet access through a wireless hub, dialing on the notebook, and using it as NAT router. Twisted, but it works.
In the process I discovered that this web-mail app is violating some parts of RFC 2109 (for cookies) among other things. Good thing I'm ignoring the cookies, or I'd have to find ways around that too.
Guess whose app it is. :-)
And, btw, yes, I can download messages now from this "certain web-mail app", although the ability to do so will remain limited until there is more time, after the pre-beta is out. Right now it can get messages from the Inbox, and the performance is not optimal. But it works!
Goodness, Gracious, Great Balls of Fire!
decentralization, take 2
The past few days I've been thinking a lot about synchronization, decentralization, sharing, and so on, as I move to connect the pieces that will make all of those things possible in spaces without making it difficult to use. Perchance, I found this article by Kevin Werbach on decentralization. I had linked to exactly the same article before, except that then I had found the CNET version of the article. The CNET and ZDNET articles where published in tandem, in a form of "decentralized synchronization." Heh
the beta is approaching
A short update on spaces: the beta is coming along well. One (further) reason for a delay is (aside from work unrelated to it that's going on as the term in Trinity finishes, and other things) is that I've been rewriting large portions of the JavaMail API. This is slightly nuts, I wish it didn't have to be done, but to support things like S/MIME mail there's little choice, since Sun doesn't have any support for that (not that S/MIME will be supported in the beta, but it should be supported eventually). Also, sharing through email (transparent store-and-forward sharing) requires a good grasp over the underlying message parsing and code/decode. Finally, performance can also be tuned better.
So that's going really well. More news soon.
I have a bad cold... so I'm spending time learning and trying out different code options for things that will have to change in spaces to make it more robust. One thing I have to grok is the IMAP spec (RFC 1730) and the other encodings, such as Base64 and Quoted-Printable, which are described in RFC 2045. I'd much rather not get too much into this, but the javax.mail.* parsers tend to fail in extreme cases and it gets difficult if not impossible to fix a problem. Oh well...
JDK 1.4.2 beta
The JDK 1.4.2 beta was released a couple of days ago. It includes an improved look and feel for Windows XP! Here are some screenshots. Too bad Sun has to chase a moving target in terms of UI conformity, but they are doing a reasonably good job. If JDK 1.4.2 is out within a month or two, then they'd be about 6 months behind the OS release for UI changes (not to mention good side effect of all the bugs they fix). Maybe I won't need to find another look and feel for spaces to have a more modern UI.
spaces on OS/2
Four words: "Write once, Run anywhere".
I know that this became the joke du jour in tech circles, quickly replaced by "Write once, debug everywhere." But now the joke no longer true. Spaces has so far run on every system that it's been tried on, including Windows, Linux and Mac OS/X, with no modifications or work required on my part. (Okay, I did have to fix one bug that was appearing only on Mac at the beginning, but that was it). In fact, I developed the program entirely on Windows machines and except for a friend that ocassionally did Linux testing I never ran it on anything else before the release, or after, until recently, when I got VMWare installed a few days ago.
And now, OS/2.
Over the past couple of weeks I've received several emails from happy users of spaces on OS/2. Very cool. Julio Cezar Salgueiro da Silva was kind enough to attach a screenshot:
It kinda looks like the Metal L&F doesn't it? What gives it away is the window controls at the top-right. Very OS/2. I guess that as more Java apps are available they will breathe new life into older systems.
more on version control
Still checking to look at the recent advances in source control management (SCM) systems. I began looking about two months ago and at that time I saw BitKeeper and subversion. Then Chris Bailey from CodeIntensity and Dylan both recommended Perforce. So now I've downloaded both Perforce and Bitkeeper. Installed Perforce, created a repository, etc. Still not installed BitKeeper. Will post impressions of both systems once I've given them a workout, which is not so hard since I already have a source structure that is reasonably complex.
synchronizing to multiple data sources
One of the most important questions remaining for spaces is the definition of how multiple sync sources are managed. In the current version (alpha 1.8) it is possible to synchronize a single space to an RSS feed. The mechanism is extensible: the context menu for a space includes an option "Space sync settings" that lets the user "attach" an RSS feed to the space. For a while I've intended IMAP to use a similar mechanism. Aside from the IMAP Folder Selection dialog, available when configuring an IMAP account (which creates a corresponding local space for each folder), there would be the possibility to sync a space with by using the context menu, attached to a particular IMAP account. So far so good.
The problem is that IMAP constantly synchronizes content to the server. Therefore, when moving content in and out of a space that is synchronizing with IMAP, it is useful that the user is aware of what is being done, preferably in a non-intrusive way.
A similar problem applies to synchronization with mobile devices. An eminently useful feature would be to synchronize a particular space with a particular mobile device, so if you have a personal mobile phone, you can synchronize your "Personal" space to it (and thus the contacts, calendar items, etc, that belong to it) and then synchronize the "Work" space to your PDA, which you use for the office. However, again, there must be some visual feedback that this will happen. So how to do it? The solution I am currently working on is shown on the screenshot below.
The idea is that sync sources can be displayed next to a space with an icon: mobile device, IMAP server, LDAP, RSS, whatever. To reduce load, each sync setting can be hidden, with appropriate defaults. For example, IMAP synchronization would appear, because it implies sending data to a server (potentially bad, since, for example, we might be sending content to a server that is not secure, or that belongs to the company), while RSS would not, since it is read-only (a space with RSS sync settings that allow weblog postings are another matter). A mobile device sync icon would also appear by default, because, again, the content is being synchronized "out" of the program, and so it makes sense for the user to be aware of that. Certainly power-users might prefer not to see all those icons, or be aware only of a few. Users that don't need IMAP would never have to deal with those options.
Also, note that the icon has the added advantage that it provides quick access to the configuration: when the mouse is over it a pull down is shown (see the screenshot). Access to the configuration would still be available through the context menu for a space.
As far as IMAP is concerned, there is another nice side-effect: transparent IMAP configuration. A big headache in IMAP systems is how to manage the server. With this interface if a space is created locally, and then attached to an IMAP account, a corresponding IMAP folder would be created in the next IMAP sync, carrying along its contents.
I hope this description was clear enough. I'd like to hear your comments, or questions.
Positive and encouraging comments overall, even those raising words of caution. I'd say there were three common themes, some comments talking about all of them, most about one or two:
Russ posted a cool comment on his weblog. There is one particular sentence he wrote that stuck:
[...] my thoughts are simply this: If you're thinking about starting a business, you must want to. If you want to, then you should.I can't add much to that, except to say: Onward!
look and feel, part 3
In another comment Roberto was wondering what are my thoughts on SWT (I've made some comments/references before on the state of Swing here and here). I have several reasons, not the least of which is that the idea of using an SDK that is more recent (and therefore more buggy) sounds a bit risky, but the main one is this: objects allocated in SWT have to be released "by hand". To me, this is unnacceptable. I don't want to go back into the having to find memory leaks. If they fix that, I might reconsider :-).
to be or not to be
Whether 'tis nobler in the mind to suffer The slings and arrows of outrageous fortune, Or to take arms against a sea of troubles, And, by opposing, end them?
Hamlet, Act 3, Scene 1
Okay, so maybe Shakespeare didn't quite intend it for what I mean.
Poetic license aside: I'm thinking of starting a company based on spaces and its related technologies. I have been talking to someone here who has experience with other companies, we've discussed possible names, how it would be done and so on.
Well, for one, to let spaces prosper and grow faster, and make it a more visible contender in what is becoming a highly competitive field.
To make sure that a free version can be supported by a structure that companies can be comfortable with, and would be willing to pay for.
To be able to get some sleep.
And so on. :-)
Regardless of my decision, I'd like to get other opinions.
Comments? Ideas? Questions? I'd like to hear what you think. Leave a comment on this post, or
looking for a new look and feel, cont.
A couple of comments on my previous entry about my search for a new look and feel. Matt commented that the Alloy L&F is quite good (also a comment here. I agree (I had mentioned it in passing in my entry), but as I said I find it slightly outdated in some sense I can't quite describe. I should try it a bit more though.
Aside from noting my search, Cristian said:
Decent font rendering by the Java VM is what I hope Sun will provide, sooner or later.I couldn't agree more. Although font rendering improved vastly with the addition of Java2D, there are still holes. In particular, an easy way to turn on anti-aliasing would be a godsend (right now the only way to do it is to redefine the paint() method of a component by subclassing and change the parameters on the Graphics2D object that is received). Well-designed, platform-independent font management would also be an important step forward.
new machine, cont.
The installation of the new environment is mostly done, and copying of files from the notebook is also largely complete. I've spent most of the day working on a new Look and Feel for spaces, adding a couple of new context commands for Contacts, and setting up some VMWare virtual machines. I just run the spaces beta in Linux/Gnome (Red Hat 7.2) and it looks great (More on my impressions on running linux again later). Screenshots are forthcoming shortly :-).
using all those GHz and MB
Now that I have a machine that can handle more than two applications at once, I installed VMWare Workstation 4 (Beta). Such an amazing application. So well done, so simple and powerful. I've already installed W2K Server in one VM, now I'm going for Red Hat 7.2 in another. Testing spaces is about to get a lot easier.
Something else that I've been looking at, in the interest of making life easier for users, is InstallShield multiplatform, which builds native installers for multiple target platforms from a single build configuration. Still haven't tamed it (somewhat non-intuitive at times) but getting there. In the end, for Windows installs the Multiplatform product might not be the best idea since the installation doesn't look too good. But for targeting multiple UNIX environments, plus MacOS and Windows, it's great. Then again, many people that use UNIX don't have much use for installers. ;-)
looking for a new look and feel
As part of changes for the beta I've been exploring the area of Pluggable Look and Feels for Java. There aren't that many really, and most are not very complete. Javootoo is a good listing of the best-known (there are several more out there, like the Alloy L&F). One that I found that looks really good is the Skin Look and Feel. Alloy also looks nice, although a bit dated. Oyoaha is also interesting. And finally, the Simple L&F. I've been modifying one of the L&Fs from Skin, and it looks good.
It's possible that a different look and feel might be the default for spaces, always having the option for reverting to "native" L&F. I think that wouldn't be too much of an impact for users, who are by now more used to seeing different UIs (Winamp, Quicktime, Media Player, games...)
reference handling in JNI
As I was working on memory optimizations for spaces I was also trying to fix a memory leak that seemed to be coming from the DLL used to import data from Outlook. Of course, this would only affect the Outlook import process, but it was nevertheless an important part. So I started looking at the JNI code.
The Outlook DLL import reads Outlook information using MAPI (actually wrapping the MAPI objects with ATL) and creates string pairs (field/value) that are then passed to Java. The C++ call looks like this:
jniEnv->NewStringUTF(fieldValue)Where fieldValue is a char pointer. Now, the JNI documentation doesn't say anything at all of having to release strings or data created with this particular method, but they have to be. The way to do it is by calling
const char* reschr2 = jniEnv->GetStringUTFChars(fieldName, JNI_FALSE);Now, once we've seen this code it sounds straightforward, but the JNI documentation does not specify that calling the ReleaseStringUTFChars and DeleteLocalRef methods is necessary, or, in fact, required to release the object allocated. Online discussions in various places are also silent on this. The JNI tutorial doesn't mention it when talking about how to create strings from the C/C++ side. Which raises the unsettling thought that there are thousands of JNI apps out there that almost certainly have memory leaks of this type in them, because unless you're creating tens of thousands of objects, it becomes difficult to see that the memory leak is actually there.
Anyway, the bug is now fixed and I just did an import from Outlook into spaces of 10,000 items (emails, calendar items, contacts, etc.) with the default memory settings for the JVM (64 MB max heap), and it worked fine, finishing in 20 minutes. According to different tests I've done that speed is linear, so about 500 items imported per minute. At some point I should create a bigger Outlook database to see how far it'll go.
More or less...
Good progress on the integration of the new storage code. Still some problems to fix but I could do an outlook import of 10,000 items within 128 MB of RAM on the VM (there's a leak in the DLL which is what causes the memory usage to be more than it should be). Also, some more optimizations remain to be done for reading data from the indexes. More testing to follow. New UI elements will be added on the next few days, and then I'll post some screenshots.
In any case, I'm a bit more relaxed now, once all the pieces are connected it's easier to find problems and improve things... doing those tasks in a vacuum is really difficult, since perception of what a particular optimization might do is usually out of sync with reality. It's really easy to get fooled into thinking that a certain optimization or improvement will be really important when actually it won't be--because the code "above it" uses it in a particular way. Good progress being made though.
more on the spaces storage changes
"Good news, everyone!"
Okay, so the basis of the new spaces storage system changes seems to be working, basically within the bounds that I wanted. The next couple of days will be spent integrating the new code back into the program, and in the process trying to optimize (memory-wise) things in the interfaces between the storage system and the UI. After that, UI changes and bugfixes and an initial beta release will be ready (no, I'm not forgetting about IMAP!).
Note to self: remember IMAP. :-)
I'll post another update as soon as the integration is done.
ps: Futurama rocks.
an update on spaces
So I missed my self-imposed end-of-february deadline for the release of the spaces beta. I'm not terribly happy about it, and I want to apologize to those users that have been waiting for it. But there wasn't much I could do. Let me explain.
On one hand, there have been several things going on besides spaces. For example, the term at Trinity (where I am a teaching assistant in the CS department) just ended (last friday, yay!), and it was slightly crazier than usual; because of the way the course is organized this term is the one where the students have to hand in most of their work, and that means a lot of bureaucratic crap to deal with, not to mention having to correct dozens of 20-page assignments (not fun).
On the other hand, my thesis work was on the forefront at the end of january (as I was releasing alpha 1.8) and at the beginning of february. Usually I manage my time reasonably well and balance between the different things I do, but this time the thesis got to the point where I needed one last big push to get the code running. And it did! The first complete version of the code ran for the first time in the second week of february.
Now, those weeks where the thesis took overriding priority over everything else (namely, sleep, eating, and so on, with many 16 or 18-hour days) are the main factor that accounts for the beta not being out by now. There's another however.
The other factor is that I've decided that the beta would have to include the memory/storage optimizations that have been waiting on the wings since the alpha release. The goal would be for spaces to run comfortably under the default settings of the JVM (usually 64 megs of maximum heap, although on I've seen some installations where it defaults to 128 megs). This applies to all operations, including import. Simply put, I wanted the "java.lang.OutOfMemoryError" messages to be a thing of the past. Also, reliability had to be rock solid. Many people are already using spaces for their everyday mail, and although they know it's alpha, that's no excuse. It should be fixed. And the time is now.
The original version of the storage system that underlies spaces alpha 1.x was optimized to minimize disk usage and maximize performance, at the expense of memory. To update it, I am rewriting several components that will change the equation to minimize memory usage and maximize performance. The new version will take up maybe 10-20% more on disk (with a higher peak usage as well), but will have upper bounds on the RAM used. The goal is, again, never to breach the default maximum JVM heap of 64 megabytes when the number of items stored (email, RSS, calendar entries, etc) is 100,000 (yes, one hundred thousand items). It should use less memory than 64 megs, probably in the neighborhood of 32, but 64 is the upper bound. This would mean that spaces will run in about the same memory other programs currently demand under similarly heavy use. In the process, I am adding more features for stability, reliability, and storage management, and preparing the core to support versioning of items, which will be necessary in the near future.
Now, the storage system change is well underway. I've already been running preliminary tests on it with 100,000 thousand items inserted into the database, and memory usage has decreased by a factor of ten in some cases. It's amazing how changing the focus on optimization can affect different measures of performance: the original design of the storage had in mind a few tens of thousands of items, rather than hundreds of thousands. And even though not that many people will deal with hundreds of thousands of items, RSS Feeds create large numbers of items, so tens of thousands of items for a user is not a ridiculous idea.
After the storage change is complete (probably in the next couple of days), I will finish work on the IMAP implementation and make the UI changes that were planned for beta 1, which include making a space more "malleable" by allowing various types of sorting and filtering to be used. I will post more information on these changes for comments as as soon as there is something that others can actually see, or try out.
So that's about it for the moment, more to come in the next few days!
a short article on spaces
Spaces was reviewed in a short article on the online edition The Hindu. (Well, maybe it's also on the printed version, but I have no way to check :-)). Cool.
You own your data
On an entry from a few months back, Jonas recounts his recent outlook migration experience. A sobering read, and one I can relate to, and not just from a user's perspective. The outlook import in spaces is by far the most complex of the imports, since it requires a native library to work.
Jonas ends the entry with:
ps. What are they (Microsoft) thinking? Why do they lock their users? Why aren't they embracing open standards? When will I be trusted with my own data?He's right. At the moment spaces does a full XML export of your data, but that's not enough. In the future, standards such as iCal or MBOX will also be supported for exports.
We own our data, and our software should acknowledge that.
spaces web access as default windows client
Spaces eventually will use native libraries to insert hooks into Windows and other OSes so that it can function as the default email client on a system. For the moment, though there's an alternative.
John Rubier posted to the spaces dev list this tip for making spaces web access the default mailto: handler. Here are his instructions:
To use Spaces as the default mailto: handler, we can use the Spaces Web Access feature. (I've modified his instructions to account for the VBS script being downloaded).Also, It should be possible to restore the old mailto: handler by going to Control Panel/Internet Properties/Programs tab and selecting the email program you want to use. This is a temporary solution, but useful if you want to connect IE to spaces alpha 1.8. Of course, be really careful and be sure to know what you're doing, since editing the Windows Registry might create problems. So use at your own risk. As John said, this code and instructions are provided "as is", with no warranty expressed or implied!.
Anyway, thanks John!
on data and views
A couple of days ago Greg Greg posted an entry on 'Data Views' and some things he would like to see in software that deals with information. I agree with most of what he says, although I think that sometimes it's not so easy to make views purely "virtual" while making it clear for the user what's going on. Many people assume that information is "physical" somehow, and they might be confused by seeing that an item deleted in a certain place disappears from another. So UI elements are really important to making this work properly and to avoid confusing the user. That said, spaces will definitely include a way to create "dynamic spaces", essentially views generated against a query. A space today is already that, although you're not allowed to really modify the query that creates them. That mechanism just needs a bit of an extension for dynamic spaces to exist.
Besides the idea of dynamic spaces there is something else I'm calling "cross cutting filters" (the 'cross-cutting' liberally borrowed from 'cross-cutting concerns' such as those that AspectJ deals with) that basically define an orthogonal category of filtering on a space. This is not new, other programs do it as well, although I want it to be easier and more frequently used in spaces. An example of a cross-cutting filter would be "see unread only" or "see sent msgs only" or "see RSS msgs only" and so on. Additionally, cross cutting filters will be accessible for user-defined tags for the items. (kind of like categories). This will allow to make a space (which is commonly a reflection of a real-world activity or task) to be filtered by information-dependent parameters, and so make it much easier to navigate. I will talk about this in more detail over the next few days, and post some screenshots.
Copyright © Diego Doval 2002-2007.
Movable Type 4.37