a google-rss bridge in java

I've been experimenting with the Google API for an upcoming Google-query feature in clevercactus and I thought that it would be cool to show how simple is to generate RSS 2.0 feeds (which validate) from within Java based on the Google query results, using only JAXP and SAX, which come built-in with JDK 1.4. (Incidentally, if you haven't checked the RSS spec recently I recommend another look, there have been a few --but key-- clarifications and improvements made to it in the past few weeks).

The result is Google RSS Feeds which is a simple (emphasis on simple :-)) web service using pure Servlets that generates an RSS feed based on a query. (The feed generation is encapsulated in a separate class though, so it can be reused in other contexts). The whole thing took about an hour to write, package and deploy. Not bad. :)

The page linked above is an example of the service working; and since the idea of all of this is to provide a sort of "code-tutorial" for Java, the sources are also available, provided under a simple (again!) open source license that allows modification and redistribution, both in source and binary forms, as long as the original copyright notice is maintained.

As always, comments (and in this case, improvements!) are most welcome.

Enjoy! :-)

clevercactus beta 3 in the oven -- and some comments

There are lots of things planned for clevercactus beta 3. I am eyeing the release of an internal version for tomorrow to solve some issues that surfaced in the last couple of weeks. Then there's the issue of not just completing/exposing features that are already there, but also of properly exposing/explaning a lot of the concepts and features that clevercactus has. Because it's (relatively) simple, and it has a (relatively) straightforward UI, users seem to see the features they're looking for, and the rest of features don't get noticed so much (until they are needed). This is great, but it's much better when you're aware of what something can do, and then use it when it's needed-- and improved documentation plays a big part in that, as well as adding more UI hints for other functionality without adding cognitive load to the UI. That's the problem with any platform I guess. A bunch of ideas have come together in the last few days, maybe not to much in terms of "features" but particularly on how to articulate how the cc functionality will solve some of the problems that we're facing today in terms of collaboration, information management, spam, viruses (yes, even that), and so on.

Something else I haven't done in a while is point to comments about cc (Some of the comments are a bit old, but they apply to beta 2 as far as I can tell). Here are a few I found after digging through my referrers (yes, yes, I miss a lot with that system, gotta get a better one...if you come across some please let me know): Gary, who is tracking the "information client" space, has added cc among a list of other distinguished entrants. Justin has a bunch of nice comments on it and would like to see more "correlation" tools added--as I do. Every information item we own is essentially the root of a tree of related elements, and it makes perfect sense that cc will eventually allow you to navigate those relation trees with ease. Kristina really liked the ideas behind it, but has been too busy to give it a full try yet :-). Josh also found it interesting and was looking for what it provides, but was slightly dissapointed to see that it was still in beta (the final 1.0 is approaching, worry not!). Then the folks at B.Mann consulting (did I get the name right?) had previously looked at spaces and have now found that clevercactus is the new name for it--and they still like it (and the Mac-UI problem is something that's pretty important too). And, from the I-can't-read-the-comment dept (okay, I can read it, but I can't get the nuances!) here are some comments of clevercactus in the context of advanced information management tools.

And, finally, this just in: clevercactus has been featured in the latest Swing Sightings!. Yeah! :-) Isn't it cool of the Swing team to maintain that service. And, yes, beta 3 is coming.

Thanks everyone for the comments!

Okay, enough babbling. Back to work. :)

viruses, OSes, and videotape

viruses

I was reading today this article from Salon about Microsoft's approach (or non-approach) to security. Some good comments in there, which reminded me of a piece by Cringely from last year on Palladium and what he called TCP/MS. Re-reading it now in the face of what happened in the last couple of weeks is quite enlightening, particularly when you think that MS is considering making automatic updates mandatory. How difficult would it be for MS to do what Cringely was talking about then, in the name of security (hey, not even that, they can say it's a bug. It seems to have worked fine so far...). Through Jon Udell I got to this entry by Chris Brumme, who works at Microsoft (unrelated topic: look at the permalink in Chris's entry. Damn.). Near the end he talks about security, and I can't help but thinking that if Microsoft as a company had his attitude, things would be better. I am sympathetic to him as a person caught in all of this, as I guess I'd be to other MS employees. But the company as a whole needs a change in attitude, and in priorities. Forget about conspiracy theories (after all, greed is always a more plausible reason), this is a problem that needs to be fixed, and now. Some of what I mentioned last week should work, properly applied, and it's not even new, as Cringely's article this week demonstrates in the section where he discusses the worms. It's not a technology problem. It's a problem of priorities, and company culture.

If you don't think it's company culture, consider this article from Fortune magazine. I'd call that article "the return of clipit". Of special note is the attitude of the Microsoft person that the author talked to. It was always "no, you don't get it at all".

My view is the opposite: if a person that is obviously interested, educated and motivated to look at your software doesn't "get it at all" for something so simple as auto-case-modification then, something's wrong with the software, not with the person!

And so the worms and viruses spread. In the past 48 hours my postfix filters have rejected more than ten thousand infected emails. So I guess it's stabilizing at about 5000 a day (!). Update: Others are reporting similar continuing problems, though not in the scale I'm seeing--I wonder why. It seems that mileage varies widely. Matthew who created and runs AlienCamel mentioned through email that they've seen quite a lot of traffic from it, and the CS servers of TCD they've stopped about 10,000 copies over the weekend, but for the entire department. Dave had 600 messages accumulate overnight, rendering his email useless. Wilson has also seen some traffic bumps, but not much. Grant hasn't been hit at all. I wonder if it has to do with who is running your server, whether you're (unknowingly) protected by other SMTP relays with checks along the way, instead of, for example, my case, where I run my own SMTP server. Hmpf.

In my idle moments I been thinking often about the issue of liability, which has been raised more in the last few days, as in the Salon article mentioned above, or on this News.com article specifically on the topic. Liability in some form might sound like a solution, but a closer look reveals many thorny questions. For example:

• If, like in my case, I am being seriously affected by a virus but I am not using the Microsoft product that has failed (Outlook, Outlook Express, etc), is Microsoft liable, or is the sender? In the real-world analogy, if a car breaks down and a pedestrian gets killed, then the driver is sued. However if the accident is due to a sistemic failure a class-action lawsuit would make the company pay, rather than the individual drivers
• And if such where the case, how do we determine the balance of bug/feature/error, plus the inevitable claims of misuse?
• Even more: in the case of the virus, if Microsoft was found guilty, couldn't Microsoft accuse all other OS makers of not being prepared to handle the situation created by the bug (on the grounds that "it could be reasonable that no system is bug-proof)? In a distributed system, is the sender less, as, or more responsible than the receiver?

Okay, these are only some of the issues, but I think it's clear what I mean. Digital communications and distributed systems create problems beyond what we've known in the past. It's going to take a while to figure out how to apply our laws (in some form) to them, and it would take even longer if something like Law 2.0 is needed.

OSes

Yesterday I spent most of the day working on the Linux machine. A real treat. Been using IDEA, Dia to create diagrams (some things, like font settings, are primitive in it, but it works) and then OpenOffice. I saw the Gnome desktop crash a couple of times (particularly when accessing network resources through SMB), but it recovered on its own with no problems. Tried out KDE, as Nex6 recommended, and installed the Windows true type fonts, as Juan Cruz recommended, both of them in comments to yesterday's linux entry. KDE does seem to be more solid, but it's also a bit less polished. Will play more with both though.

Configuring a remote printer (an HP deskjet shared on a Windows XP machine) on the Linux system was a breeze. Simple, fast, and it just worked. I loaded a PDF and pressed print, and nothing seemed to happen. I pressed again. Then I hear the printer in the other room. Oops. I hadn't expected it to work silently and transparently like that. Very cool.

What's weird is that Windows is actually getting more complex than Linux. Why? Because Windows is, at heart, a system designed for a disconnected world, while Linux (even though not fully a "networked OS" like the Spring Research OS was in the 90s) is much more aware and ready to deal with those things. Transparent firewall, IP routing, NAT, DHCP, etc, etc. All of that has been making the rounds on Linux for ages while Microsoft is just adding it now to Windows. The conclusion is that the playing field is leveled in that sense. Red Hat 9 already provides an easier (and a LOT more clear) way to configure the system's firewall than Windows XP does.

Also, I've been using GIMP a bit; I can't help it, I am graphics-dependent, and even for simple cases I end up doing image manipulation that is always better done with a good program (scaling algorithms are not all the same, you know :)). Found this really useful site on it and Gnome in general (including for example this nice GIMP tutorial).

And I still haven't commented on how cool (and useful!) the miriad of useful applets are in the Linux desktop. For later then...

Unrelated (more or less). I found Contiki. Useless for serious work, sure, but probably useful in many situations... and in any case, isn't it cute?

videotape

Right. Let's forget about the videotape for now. :-)

a linux tale

I started using linux back in early 1994, when I got a distribution on eight high-density floppy disks, with one of the first versions of LILO (the kernel was the "classic" 0.91pl14 if I remember correctly). I had been "converted" to UNIX a little bit earlier when I got my first job and I learned how the UNIX kernel worked, its design principles, and its inherent beauty. :-) Suddenly, I could see. I became a UNIXhead (TM), but of course (as mostly everyone else) I had to deal with Windows.

I used Linux heavily at work and at home for the next few years, mostly in dual-boot environments (before Red Hat, my favorite was the Slackware distribution-of-distributions, and the popular-for-a-while Plug and Play Linux from Yggdrasil Computing, remember that one?), until the versions of Java got completely out of sync (particularly on the graphics side of things, with the arrival of Java2D) and I couldn't use it as a decent Java platform anymore. In my last company in the US I installed Linux (RedHat 6 methinks) for use in server-side applications and as a NAT/transparent firewall until the company hired a sysadmin and they decided to simplify (right...) and went with an all-Windows infrastructure.

By the time I came left the US the Linux port of Java was respectable (certainly for server-side work) but a lot (most?) of what I was doing was client-side, so, again, no luck. I stayed with Windows (besides, I needed to make sure that then-spaces ran properly on Windows first). I switched to WinXP at the beginning of 2002 after a friend convinced me that, yes, it was slightly better than Windows 2000, and cleartype made it a good option on LCD displays. Then over the last two years or so I used Linux for testing, basically using an old version since getting a new version was too much hassle, and I (thought I) didn't really have the time, and I guess that inertia got the better of me as well.

But then...

Fast-forward to last week and the attack of the Windows worms that wrecked my mail (Since I activated the filters on my server 24 hours ago, 5000 messages---yes, that's five thousand--- have been rejected), and me getting pissed off enough at Microsoft that I thought that maybe change was in the cards. So I downloaded Red Hat 9 (took about 5-6 hours I guess?) burned the install ISOs, backed up my notebook, and started the install last night after I got the email situation sorted out.

I started selecting the packages to install, and in the end I decided that the "Install Everything" (4.7 GB) option was probably best, since I wanted to check out KDE, Gnome, etc, and I always end up using the development tools in Linux anyway. The full install took maybe 3 hours. After it was done, a tool called "kuzdu" detected the hardware that hadn't already been configured (including a 3COM modem, and the sound card, among other things) and after that X launched with no problems. I had to do some tweaking later to optimize the use of the LCD (which is the only thing that wasn't properly detected, and Thinkpads have pretty good LCD dpi resolution), but aside from that everything was fine. The next step was configuring my 802.11 (aka WiFi) card.

Coincidentally, I had gone through the config process of a WiFi adapter against my WAP only last week (with another machine), but that machine was running Windows XP Pro. It was a nightmare. The configuration options were impossible to decipher, including typing the WEP key. In all, it took more than half an hour, and in the end the connection was flaky: it kept dropping and had to be manually reconnected, which was, as you might imagine, a royal pain.

So I was a bit apprehensive about the Linux config, as it turns out, needlessly so. I just had to go to Main > System Settings > Network and the network device control panel showed up. I selected the wireless card, which had already been detected (a Linksys WPC-11 Type-II PC Card, which Linux identified by its chip, Orinoco), type in the HEX WEP key, choose DHCP to use my NAT gateway, and bingo. Select Save, and restart network interfaces. Done.

It just worked. After that, even though sometimes the connection dropped (rare, but possible) it reconnected on its own (obviously, who came up with the idea of manual reconnect in WinXP??).

I was slightly shocked. Could it really be that easy? It was 2 am, my eyes burned slightly, but I got more ambitious. What else? I thought. Now I had Internet access...

I started with Firebird, to replace the bloated sluggishness of Mozilla. Loaded Evolution just for fun <wink> and closed it again, while I got Firebird set up, which magically replaced my links to mozilla itself (I'm not even sure how that happened... something to do with the Nautilus desktop--but I don't care). After Firebird I downloaded JDK 1.4.2 for Linux. Strangely enough, all the Linux distributions I've seen set as default Java install one of the crappy open-source implementations (don't get me wrong, I find the work done on them amazing, but the truth is that they're not up to par in terms of Swing, Java2D, etc, and they're always one or two major versions behind). I suppose that might have to do with Sun-licensing stuff. Anyway, so I got Sun's distribution (RPM), installed it, all fine. The default java still pointed somewhere else but I didn't want to spend time then to look at how to replace it.

Now that I had proper Java 1.4.2 I got the latest binary of clevercactus (internal release :-)) and ran it. Set up a breeze, and it might have been a subjective hallucination but I thought that it looked excellent, and performed flawlessly. I was impressed. Heh.

Okay, enough with the self-promotion! After I got cc running I configured it to get my email, etc (the cactus-to-cactus sync is still not complete) and then moved on to development environment. I got the latest version of IDEA (3.0.5--plus updated website!) which I still prefer to Eclipse for reasons that are probably irrational while making perfect sense only to me. That took a bit longer (longer download, a couple of more things to set up).

Then some cvs configuration (against the CVS server for sources) and I was basically done. I still have to look for a few other things (among them, an IRC client, which I'm sure is around here somewhere), and it should be enough.

So now I've just finished updating the packages in my system with up2date (take that, Windows update! :-)) and am now enjoying a number of luxuries that I had forgotten, for example:

• A decent shell, with proper scripting and regular expressions (bash2)
• A decent built-in text editor (vi--I'm not an emacs kind of person)
• A decent multi-screen window system (X)
• A decent thread/task manager (ie., an OS Kernel that doesn't kill the entire machine even when one process is running at 100%)
• Good screensavers :-)
• gcc at my fingertips!

Among other things, like the cool desktop gizmos that Gnome/Red Hat has (I'll comment on that in another entry). And all of that, with excellent, zero-config Windows compatibility. Example: I go to Main > Network Servers open the Windows server. Get files. Then copy over a PPT file from the Win machine, double clicked, it opened. I can run the presentation, edit it, whatever. Everything just works. (Okay, and when it doesn't, there's tons of docs on how to do things, much more accessible than their Windows equivalents). It's definitely very, very close to being something that absolutely anyone could set up (the only problem that remains is that, when something goes wrong, you end up having to go to consoles, which I actually enjoy at times, but most people would be completely baffled by it. In Windows you just get a catastrophic error, and that's that. The console is harder to use, but it actually lets you fix the problem. :-))

Probably the only thing that I truly miss is cleartype within Firebird (Mozilla seems to have better font display for some reason). But so what.

I still have WinXP on the other machine, but I am beginning to wonder if that's necessary too. If I get VMWare for Linux, and then run WinXP in there... not now though, this has taken about 12 hours total--not bad, but there's a ton of things to do aside from this... we'll see.

One more thing: It's good to be back :-)

AOL goes blogging

Finally, it happened. Now let's see how long it takes Microsoft to jump in the fray... Yahoo! has already started dipping its toes....

a few minutes ago...

...I ran Gnome on Red Hat 9 for the first time. The last version I had used, for testing clevercactus, was 7.2.

I. Am. Speechless.

Or, in the spirit of what Comic Book Guy said once: "Vision ....blurring .... balance... failing... can't... go on.... describing.... symptoms...!"

(bonk)

config-day

At around noon, I thought the filters were more or less working. I was right. They worked so well, in fact, that even I wasn't allowed to send any email. Hmpf. More configuration. Verifying whether it was a problem with clevercactus in particular (it wasn't). Removed a couple of hostname checks that, while useful, prevent clients behind firewalls from sending email (because it can't do a reverse lookup on their name). Then spent another bunch of time until I realized that the pop-before-smtp process had died at some point, and that meant my client wasn't being properly authorized.

Anyway. On to Linux now. Verified that my 802.11 card works ok. Ready. Funny that a year ago I had shed the first layer of monopoly-skin (and I was using spaces back then, even though I hadn't mentioned it yet :-)).

mail's back

Okay, so, I admit, I wasn't ready to ditch email yet. This morning, the reasons remained: if I loaded my mail server I suddenly received a flood of messages, including the virus, spam, and "rejected" messages from addresses that had received the virus with my own address spoofed.

But even as the problems remained, I needed email, not least to reply to the clevercactus-dev list, and even do some work. Last week's crisis led me to think in new directions, and maybe we'll be able to come up with a good solution for this problem (or part of it). In the meantime, I had to get my mail back. I had no choice.

So I breathed deeply and started looking for configuration options for postfix, the mail server that I use. I found good information here, here and especially here. I started adding options and it took me some time to get them running, in particular the regular expressions that parse both the headers and the body of the message were a bit of a pain to get right, as usual. (I am now, for example, rejecting EXE, PIF, BAT and other MS-virus-related attachments, knowledge of MIME and how it is usually done has its uses :)).

I still have to tweak things a bit, but in principle it should be back to normal. Interestingly enough, most of the messages are being rejected with the error "Helo command rejected: need fully-qualified hostname". I wonder if this will affect legitimate email (I am not entirely sure which of the postfix settings is requiring this, maybe it's "reject_non_fqdn_sender" or "reject_non_fqdn_recipient" or "reject_unknown_client"...).

If you're trying to contact me via email and you can't, leave a comment here--also, if I haven't replied in the last few days just give me a few hours as I go through my queue.

Update: in the roughly three hours since I completed the filtering configuration, postfix has rejected 1128 emails, most of them infected with the Sobig virus. One thousand one hundred twenty-eight! Jeez. Anyway, it feels weird now. Like the quiet right after the storm has passed.

(In the time it took to write the previous paragraph, another nine invalid emails were bounced!)

hyperreality TV

Steven Soderbergh (and George Clooney) come out with a new HBO series, K Street:

In "K Street," a half-hour show that makes its debut on Sept. 14, HBO is aiming for something that Steven Soderbergh, a co-executive producer, calls "real-time fiction." The show will depict a make-believe firm of lobbyists and consultants, but will blend in real politicians, lawmakers and issues to give an insiderish flavor of how Washington wheels, deals, logrolls, backscratches and backstabs.

From Shakespeare's plays to "Ragtime" to Mr. Soderbergh's Oscar-winning film "Traffic," the technique of mixing the real and unreal in entertainment has a long tradition. But the characters in those works fall into familiar categories: they're either pure creations, fictional versions of real figures, or cameos. Here Mr. Soderbergh and his creative team, including the actor George Clooney, the writer Henry Bean and the producer Mark Sennet, are heightening the trompe l'oeil effect by having real lobbyists and consultants play alternate versions of themselves, while grappling with real issues about real people in a fake firm. Like "Law and Order," "K Street" will rip its plots from the headlines. But it will do so only days after those headlines appear, while the issues in question are still live ones, and do only as much fictionalizing as necessary to keep the plots interesting.

The idea is to be so topical that viewers are left "asking whether it's a documentary or fiction," as Mr. Soderbergh puts it. To remain on top of the news, the episodes will be outlined and shot on an extremely tight schedule. Each week's installment will be hammered out and finished in three days, beginning on the Monday before each Sunday's air date. Editing will be done on Thursday and Friday. "Everyone will come in on Monday having read the papers and seen the Sunday shows," Mr. Soderbergh explained during a recent interview in New York, wearing his trademark black T-shirt and black eyeglasses and sipping on a drink called an Arnold Palmer (a combination of iced tea and lemonade). He and his partners are hiring researchers to make sure the show gets its facts straight, and equipping two mobile vans to rush the cast and crew to real hearings and other events in the capital.

Wow. Hopefully they'll show it here in Ireland at some point. It takes a while sometimes, for example, I still haven't seen The Wire, which is apparently excellent.

on google and the markets

From News.com:

Despite frenzied speculation of an imminent Google public offering, company co-founder Sergey Brin said he's still casually debating the pros and cons with board members and has not yet set a date.

Given recent irrational behavior on part of the markets (you know, P/E ratios that don't make any sense... stocks of money-losing companies that cuadruple in value in a matter of weeks... things of that nature), a Google IPO could easily set off a chain reaction. It wouldn't last though--there's not a lot of money left to lose--and a few people would make a lot of money.

That aside, I was thinking today: how much more possible is it that Google will be acquired? .... No, not AOL--not enough cash, too many problems. But Microsoft? I admit, it's far fetched, highly unlikely, etcetera. But they could make "an offer they couldn't refuse". It's left as an excercise for the reader to figure out, in this hypothetical scenario, who would be Don Corleone, and who would be Luca Brasi. :-))

new passports for US visits--when?

According to the New York Times, it's on October 2004:

Technologies that scan faces and fingerprints will become a standard part of travel for foreign visitors next year, and for all travelers in the near future.

The technology, known as biometrics, has been developing for years, but largely because of security concerns after the attacks on Sept. 11, its arrival has been greatly accelerated.

One deadline looms large — Oct. 26, 2004. In a little more than a year, the State Department and immigration bureau must begin issuing visas and other documents with the body-identifying technologies to foreign visitors. The change is mandated by border security legislation passed by Congress last May. The federal government has started issuing border-crossing cards for Mexican citizens and green cards that display fingerprints and photos.

By the same deadline, the 27 countries whose citizens can travel to the United States without visas must begin issuing passports with computer chips containing facial recognition data or lose their special status. People from those countries with passports issued before the deadline may still travel to the United States without visas as long as their governments have begun biometric identification programs.

But I thought it was October this year. I'm confused.

And it sounds like quite a lot of information on foreign citizens doesn't it? Biometrics and so on.... I think it will be interesting to see what happens when (or if) they actually try to impose this same system for their own passports/documents. And if it doesn't happen, one would have to ask why should the rest of the world be treated with so much suspicion, and if that's something healthy for an open society.

novell's strategy

It's been a couple of weeks since Novell's announcement that it was acquiring Ximian. Since then, it has posted its results for the most recent quarter, including a net loss and a number of layoffs. Some of the comments to my previous entry were interesting in that they mentioned Novell's "Linux moves" and I must admit that I was sort of blindsided by this. But as this interview with Novell's vice-chairman Chris Stone makes clear, the Linux play is something that Novell has been working on for quite a while.

Putting two and two together now, it seems that Novell wants to mount a strong challenge to a) generic Linux, by leveraging their NetWare brand and technologies, including Ximian's Red Carpet software and b) complement that with a competitive move against Microsoft at a higher level (UI, messaging) through Ximian products. In the interview it becomes clear that there is a nice interaction between what Ximian and what Novell has, with no overlap. Even the mono project (to create an open-source implementation of a .Net-alike environment) fits: Novel understands development tools, toolkits and environments quite well, since Netware and its Netware Loadable Modules (NLMs) where once an important development platform for networked services.

I have to say: it actually seems to make sense. And all of it, without hype or pretense. Amazing.

openoffice for OS X... in 2006

Speaking of, er, alternatives. Got this on the #mobitopia channel yesterday: OpenOffice for Mac OS X delayed until 2006.

Plus the new version of OO by 2005? What?!? The Linux kernel itself has had delays, but this is too much.

Ridiculous.

And then we make fun of Microsoft for running over 2 or 3-year deadlines...

Maybe it's time for a renewed Corel to come up with some real competition for MS Office?

now downloading...

...all 1.4 GB of Red Hat 9. The plan is to attempt a full, clean install of it on my Thinkpad laptop. No Windows partition whatsoever. No FAT or NTFS. Between IDEA, OpenOffice, Firebird, and clevercactus, I should have all I need. We'll see how it goes.

what went wrong with columbia

An in-depth article from the Washington Post on the causes of the Columbia disaster back in January. Lots of lessons in it, in particular concerning communication between groups. And, isn't it amazing that the Shuttle lasted as long as it did? The Shuttle is clearly a great piece of engineering, its many problems notwhistanding.

diego's excellent symbian adventure, part two

in which diego discovers that J2ME is a lot less, and a lot more, than previously thought

[Part one, notes].

I concluded in part one that native development in Symbian is a difficult proposition at the moment if your main concern is, like mine, minimizing development time by targeting as many devices as possible with a single code base. Creating 80-85% portable "native" Symbian apps is possible, but complex, and difficult for new developers. Because of this, for many applications, J2ME will be the way to go.

micro-Java, not micro-J2SE

J2ME is truly a "micro" version of Java, rather than a reduced J2SE (Here's a good take on the topic from Russ). When J2ME was first launched there was another "contender" called Personal Java (based on JDK 1.1.8) which we'll ignore since it's currently being phased out by Sun. To get some context, let's look again at the varieties of Java:

• J2ME: a Java-based platform for small/memory limited devices.
• J2SE: the Standard Java platform for desktop/mobile computing and above.
• J2EE: Enterprise-oriented additions to J2SE, mostly in the forms of new libraries and APIs that plug into enterprise runtime environments.

J2ME targets areas that, unlike the (slightly) more uniform target of J2SE/J2EE, require support of widly different devices and capabilities, in terms of I/O, processing power, memory, and everything in between. As a consequence, J2ME is actually a set of specifications. Each specification targets a configuration, which then can be further defined by using it with different profiles.

And herein begins the acronym-fest. There are, currently, two main types of configurations. The CLDC and the CDC (not a bad name, in our virus-ridden times). The CLDC ("Connected Limited Device Configuration") is a low-end configuration target: cellphones, low-end PDAs, etc. CDC (Connected Device Configuration) on the other hand targets everything between the high-end of CLDC and the low end of devices that begin to support J2SE. (CDC + the Foundation Profile, the Personal Basis Profile and the Personal Profile define the equivalent of Personal Java but with more flexibility, which is why Personal Java has been discontinued, btw).

Now, both the CLDC and CDC reference VMs differ from the J2SE VM, and from each other. I'll be most interested in the CLDC VM (The KVM, or Kylobyte Virtual Machine), since it's the one used in most Symbian phones. The KVM is limited compared to a J2SE VM, not just in features (e.g., no advanced JIT techniques) but also in capabilities (e.g., floating point arithmetic is not required by the CLDC spec).

Talking about VMs is all well and good, but in the end what makes Java a platform is its libraries as much as it is its VM. So what about libraries in J2ME? Well, that's where the profiles come in.

profiles

Profiles add packages and classes to configurations, and each configuration has one or more associated profiles. And for CLDC, the most popular profile is one called the Mobile Information Device Profile or MIDP.

The MIDP adds basic networking, UI elements and minimal storage capabilities to the CLDC, and it is normally used for wireless devices (phones, PDAs, etc). Recently Sun announced the release of a new an improved MIDP, MIDP 2.0. But most Symbian phones implement CLDC with MIDP 1.0, and only one so far (the Nokia 6600) supports MIDP 2.0 (that I know of). SonyEricsson's P800 supports Personal Java, but that's a dead end since it has been discontinued.

In general, profiles try to use subsets of classes from J2SE. Whenever a class from J2SE is used, only methods already existing in the J2SE version can appear in the J2ME version (i.e., no new methods can be added). That's why the classes/packages some of what we already know from J2SE (say, java.lang.*), or are completely different (the storage classes in MIDP 1.0).

MIDP allows various optional packages such as the Mobile Media API and Wireless Messaging API which are included on some phones such as the Nokia 3650 and newer APIs such as the Bluetooth API on the 6600, as well as APIs in development such as 3D Graphics API for future models.

enough with the theory

After a while it's quite clear that the best way to do multi-device deployment with J2ME is to use MIDP 1.0. Using this profile, you can create Midlets, similar to Applets. Like applets, midlets are limited in their access to the local device, something that is even more visible because of the limitations of the MIDP 1.0 API (PDF, 171 KB).

Now for getting together a J2ME development environment for Symbian requires. To start with, I'll need Sun's J2ME WTK (Wireless Toolkit) either version 1.0.4 (for MIDP 1.0) or version 2.0 (for targeting MIDP 2.0). So I get WTK 1.0.4. Plus, let's say that I'd like to test with a Nokia simulator. I need to get the Nokia Developer's Suite for J2ME, plus an emulator such as the Series 60 Emulator/SDK (which for this phone includes cool things like the Bluetooth Java APIs--but careful! Since other MIDP 1.0 phones might not support that API yet).

The final element in all of this is the IDE integration. The steps necessary to go from cource to binary, and then to deploy it into the phone emulator (or into the phone) are Definitely Not Fun. So getting an IDE that will do them automatically is essential to keeping my sanity. The first option I looked at was CodeWarrior, but CW came integrated with WTK 1.0.3 (instead of 1.0.4) and didn't allow upgrading it. JBuilder 9 for Mobile Dev didn't have a trial download. Eclipse... well. Eclipse was an option :). But I thought that maybe IDEA, which I use for other Java development, had some Plug-In... and sure enough, there it was. Excellent. A bit of fiddling, and finally I was able to see my simple Hello World! application running in one of the default "phone covers".

Note: a big Thanks to Russ, who guided me in my search for tools and toolkits. Without his help, it would all have taken a lot longer than it did.

Now, this is all a lot more complicated than it needs to be, and Sun isn't necessarily helping matters. Hopefully the situation will improve in the near future, to make it easier for new developers to approach J2ME on Symbian platforms (and others, too).

native v. java

What is probably one of the most ridiculous limitations of MIDP 1.0 is its inability to access the device's store (e.g., contacts, calendar entries, notes, etc). This hugely limits the kinds of useful applications that can be written in Java (which in many cases will want to interact with phone data and services). And there's still no standard to access the bluetooth functionality (although it's coming, too).

At a minimum, creating a prototype with J2ME should be easy enough (once the development environment is set up) so that then certain Symbian devices can be specifically targeted with native development. J2ME apps can be targeted to devices beyond symbian, such as Palm. However, they have some serious limitations. If an application needs to access low-level functions or device data, native development is a must. Otherwise, J2ME might be a good alternative to simplify portability.

is it symbian-bashing?

My previous two Symbian articles (here and here), while, I think, balanced, elicited some comments that I was being too hard on SymbianOS and its efforts for the developer community. Before I continued writing about Symbian I wanted to clarify what I think, to avoid any further confusion.

First of all, something that probably didn't quite come across (but that I added in one of the comments I made to my first post in mobitopia) was that I consider Symbian's work so far excellent, and that this whole issue is something that in my opinion will be more of an issue in the near future.

I consider these problems a natural outgrowth of the nature of what Symbian does in terms of its multiple target platforms. No one has done anything like this before, and Symbian has done remarkably well so far, and the fact that this is coming out now only means that it's time to move to the next level.

For example, consider my comments on development environments (on the two entries linked above). Then read Russ's entry on the .Net development environment for smartphones, including the comments. I hate to say "I told you so," but there it is.

Information and toolkits exist, but they are not easy to find, or to put together. People with experience (even limited experience) on Symbian had said that I was wrong on this count, but I truly believe that this is a bit of "inbredness" that afflicts the platform at the moment. Once you are familiar with the technology and the documentation, things make sense, but that is true of anything, even Win32, which has to be the most confusing API ever devised by man.

Recently I've been doing a sort of "mini-evangelizing" of Symbian. And people don't get it. They just don't see what the OS can do, the tools that are available. Some of them don't even consider it a development platform. When I start to describe the capabilities of the new Symbian-enabled phones that are coming out, they open their eyes wide "Really?". These are technical people, and they are not hiding in some closet. They have been doing mobile development for some time now, in some cases for years, in many cases starting in research environments at the university and they have been using PocketPC and so on because a) it's easy to "see" what to do with it (an HP iPaq is more "visible" as a development platform than, say, a Nokia 3650) and b) because Microsoft has marketed the hell out of the platform to those early stage developers "See? Just Win32" MS says, and of course that's not true because the Win32 implementation of PocketPC is severely constrained compared to the PC-side of things. But they get started quickly, and that's all that matters. After a while, they are locked in. In my experience, pointing people to DevNet just baffles them completely. They see all these toolkits, pointers to each manufacturer's site, all these IDEs, and they assume (wrongly) that they have to develop for each platform separately.

Some developers take pride in the fact that sometimes information is obscure or difficult to get at, that you need hours of sweat to set up a development environment or to understand the advantages and limitations of a platform ("If you're not willing to suffer for a while, then we don't want you here"). It reminds me strongly of the Linux atmosphere in the mid-90s: "Documentation? Of course you've got gorgeous man pages, and maybe even comments in the source code! What? Graphical environments? Sure, just spend a few hours configuring the vertical scan rate in your XF86Config... And if you can't handle that, well, then you're just not ready for us..."

Linux got over that mostly, but there's one big difference: Symbian is the incumbent in the next-gen mobile phone market. And as such, it can't behave like an upstart. Much less when Microsoft is moving in.

The capabilities offered by Symbian OS and its target devices open up possibilities to create new applications, AND with a massive installed target base. I know people that are not expert programmers, and yet they have good ideas and would like to put them into practice. They are "formal" tinkerers, but there are many like them that don't necessarily have time to set up an environment from different sources, and understand that to use J2ME you have to get Sun's WTK as well as stuff from Nokia, for example. Those tinkerers will come up with the cool applications that will then seep back into the platform to make it vibrant, and within corporations, if someone can easily set up an environment and show a particular idea working easily, they are more likely to get it approved for further development.

Sure, this entails also "widespread evangelism", but simple, easy to use tools out and a single entry point for the information related to those tools (going to the manufacturer's site for device-specific information only) gets you half-way there.

Java.sun.com (and Javasoft's Java Developer Connection) is a good example of what I'm talking about. I think that when I'm looking for Java information I spend maybe 80% of my time there. While if I'm looking for information on Symbian, I spend 80% outside of Symbian's site. (Yes, I know that if I'm in Nokia's dev site I might be reading relevant information for any Symbian OS phone, but that's not so easy to see, is it, when everything is labeled "Nokia this" and "Nokia that"?). Things like this would go into that site. Even if it doesn't seem like it, The JavaSoft site, and JavaDoc, played a big role in getting Java adopted quickly.

The information is all there, and the only thing that's needed is better organization and centralization, particularly of forums for discussion. Symbian as a development environment has some problems but at this moment I give it the benefit of the doubt. The situation is improving. And Symbian has a great advantage, both in terms of development experience and deployment of its platform. But it has to continue evolving and becoming simpler, and moving into new areas.

So, is any of this Symbian-bashing? No. It's trying pointing out problems that I see, and doing what I can by continuing to write how-tos and such (and sometimes just a bit of frustration seeping out, yes).

I'd like to add: It's one thing for us to just talk about it, it's another thing entirely to actually do all of this while juggling new product releases, partners, company growth, etc., with the big Redmond Machine breathing down your neck.

Credit where it's due: If not for Symbian, MS would have already taken over the mobile phone space as well (or, nothing would have happened in the area at all). Google seems to get a lot of credit these days for keeping MS on its toes, but Symbian deserves a good amount of it for doing the same in the Mobile arena. (Google obviously has a better PR department :-)).

So, hats off to Symbian and its people!

Now, coming up on my end: a short review of J2ME on Symbian OS.

it's not the users, part 2

I was typing this as a comment but it got to be just too big. So here it goes. References for this entry are comments in my entry on software, developers, and users, in particular those by Bo and Roger.

First, thanks everyone for the comments. Now, my replies:

Bo said:

Are you willing to guarantee that your program behaves exactly the way it's supposed to on the infinite configurations of sofware and hardware out there? Are you willing to guarantee your program won't one day do something stupid leading to great monetary loss? What will be the EULA on clever cactus?

While liabilities and guarantees are legal matters, what I am talking about is simply a question of taking responsibility. Instead of taking responsibility and saying "We'll fix this for you, it's our fault", Microsoft says, "It's YOUR fault. But we'll see what we can do."

And it can be fixed. For example, any content downloaded from the Internet would be placed under quarantine, and as Christian said, scanned. Even then, it could be run within a sandbox, for example, NOT allowing access to your entire web browsing history, cookies, and all the network drives to infect. I don't know, there are a thousand things that could be done, but Microsoft hasn't done ANY of them since they started "trustworthy computing" almost two years ago.

In fact, can anyone name a single clearly defined advance that "trustworthy computing" has brought to Windows and/or Office? This would have to be something added to Win XP SP 1 or one of the updates to Office XP. And how is it that well after this "initiative" begun, they still keep finding buffer overflows all over the place, sometimes across ALL VERSIONS OF WINDOWS (!) including the "ultra secure", recently released Windows 2003? Yes, they've got millions of lines of code. But also have thousands of developers. Surely asking each person to run purify and a properly defined set of tests on each of their modules is not too much to ask for.

Even their new developments are not secure, they come up with C#, which is "secure like Java" and they break the security of the environment by letting the developer mess with memory directly. Result: I guarantee that there will be C# buffer overflow worms. Why is that? Because they don't care about security. Don't tell me that it's a compatibility problem, please. A company that is willing to do this doesn't care about compatibility too much.

Security is low priority for them, trustworthy computing notwhistanding. They think it's a "feature", and optional as such. Here's the proof.

So, as far as clevercactus is concerned, I can say what I will NOT do. I will not, facing a widespread security problem, start a "user education campaign", like Microsoft is doing this week. That is an insult. I will use the money on development. I will try to come up with innovative solutions for the problem, and I will try to understand how it can be solved, not just for the next version, but for current versions. I will ask users for input. And, you know, if I had 50 billion dollars in cash (heh) I would think about how to use it properly. Assumming that this is some intractable problem (it's not) you could fund a good number of crash research projects to find good fixes, no?

I don't know, I guess that I could not provide a guarantee (in terms of legal liability, no small company could, probably, and an EULA would reflect that), but at least I would be honest and humble in the face of a mistake, and I'd do my best to fix it, and communicate that to users (and get their input), instead of subjecting them to an "education campaign" which essentially arrogantly says that "we're fixing it up, you just get educated while we come up with something for morons like you", and "oh, here it is. BlahBlah XP is more secure. Pay up." and then have it blow up all over again. This is not a one-time problem. This is a pattern of problems that keeps showing up, over and over, and it's been happening for quite a while now. If it happens once, maybe even twice, it's an honest mistake. If the exact same thing happens three times, except that it gets worse every time, well then...

And, of course, as I've said before, Microsoft has even more responsibility because of its dominant position in the market, and its immense resources.

Roger said, essentially as the core of his argument:

People who open unknown attachments are jumping the curb. People who don't run antivirus software aren't wearing seatbelts. Safety is in the steps you take to protect yourself, not the responsibilities you shift to others.

Now, I can add to what Christian said, by taking each sentence in turn.

"People who open unknown attachments are jumping the curb."

No. Opening an attachment is trivial, it's two clicks and one confusing warning message, and you can get an amazing amount of damage from a simple action. "Jumping the curb" implies a lot, not least of which is the screams of the people you're running over, not to mention crashing trees, etc. I'm not making a literal comparison, I'm just saying that something that causes *so much damage* should be hard to do and to keep doing. That is not the case with attachments.

"People who don't run antivirus software aren't wearing seatbelts."

Conversely from my previous point, "safety" should be easy to obtain. Putting on the seatbelt is easy. Installing, maintaining, and updating AV software is hard. And expensive. The seatbelt comes built in. Easy to use. And it just works. AV software is a long way from that.

On PCs, particularly on Windows PCs, doing a lot of damage is easy, while avoiding damage is hard and expensive. It should be the other way around.

This ties in to Bo's first point of how hard it is to certify different configs, platforms, etc. I agree with Bo that it's difficult. What I am saying is that: a) Microsoft's gut reaction of "people are wrong in opening attachments" should change. If people keep doing it, then they are right, and Microsoft is wrong in assuming they won't. b) Doing damage should be difficult, increasing safety should be easy. Consider how difficult it is today to configure Internet Security options in IE, or how braindead is Outlook XP's "security" with attachments (Executables are not shown. That's it.)

I am not asking that Microsoft be perfect. I know the problems they face, they have a huge market, etc. I am just asking for something simple: I would just like to a) be treated with respect, as a user, that is, when there's a problem like this, please don't say "Oh, users are idiots, they don't do what we tell them to do," and b) that they, rather, try to find a way of solving the problem while maintaining functionality, that is, I would like to see progress in these areas, where solutions are truly solutions, and the way to stop an engine from blowing up is fixing the engine rather than preventing people from ever turning it on.

I don't think it's too much to ask for, no?

an evolvable military platform

From Wired The best defense is a good upgrade on the USS R. Reagan--the aircraft carrier, not the president :-).

it's not the users

Okay, Microsoft news of the day: first, they warn of three new "critical" IE flaws. Then they say that, by the way, Windows patches might become automatic. Have you ever read the licensing agreement of Microsoft software? It gives them rights to do almost whatever they want. Now, automated. Does anyone think that they would use it for something useful? Why would they want to deliver patches automatically, since patches don't seem to work anyway?

On the topic, Scott says:

So tell me when is someone going to sue Microsoft in a class action lawsuit about shoddy security practices? Couldn't this be a tobacco lawsuit kind of thing?

I am incredibly surprised that no one in the US wants to take Microsoft up to task with this. They've sued McDonald's for making people fat, for crying out loud.

In part, I think, this comes from a certain misplaced perception that is quite widespread. It goes like this: "Oh sure, Microsoft is bad, but users are part of the problem too. You know, if they just stopped opening attachments... and it's not like they've no warnings...".

Users are not the problem. Until we, in the software community, take responsibility for what we produce, this isn't gonna get better. Let me put it another way, an example that I came up on an IM conversation today.

Say that a car company, for example, Fanstastic Motors, creates a particular type of car that, when driven beyond 70 MPH, becomes so unstable that it rolls over and explodes. Because of this, everyone has warnings. You get a course, that says that you should never drive beyond 70 MPH. Whenever speed increases, you get warnings on the console. Mechanics that you meet on the street explain to you how you should never, ever drive fast. Now, everyone knows that it's just gonna happen that people will, intentionally, or by mistake, drive over the limit. People will die.

Now, in that case, would you blame Fantastic Motors, or say "Oh it's the drivers that never learn".

If you think that my example was ridiculous, think again. Maybe you remember that Ford was in seriously hot water a couple of years ago because of tire problems with their SUVs. The reason? The tires were being used "beyond spec". They were disintegrating mid-trip, causing catastrophic accidents. Of course, Ford told users to check tire pressure. Of course Ford told people not to do X and Y. Of course people were told to check their car regularly. And, of course, drivers, users, sometimes forgot, with terrible consequences.

Back then, was anyone saying, "oh, these drivers, they never learn."?

No.

Software is NOT different.

Update: The Register has an example that is, well... exactly like mine. Heh.

I always try to tell people, when they say, "I don't know what I did. The computer stopped working." I always ask them: 'If your fridge stops working, does it ever occur to you to say "I don't know what I did, the fridge stopped working". They say 'No. I'd say, "The fridge isn't working.''

There's a big difference.

I know of no other industry in which the customer willingly takes the blame for the stupidity of the provider of the good or service. Customer support people reinforce this tendency ('Are you sure the computer is connected to the power outlet?'). The way software is designed reinforces this tendency ('Please read the following carefully and select the appropriate option' --to which you could almost add 'you moron!'). Users get blamed all the time. Well. Sometimes the user might be at fault, but until software actually works properly, that's definitely not were we should start.

Users are not at fault, We, the developers, are. And the biggest of all is Microsoft. They should be ashamed. They should get their act together. It's called corporate responsibility. Which in their case is even bigger, because they own not one, but several monopolies.

I just turned on the email server for a moment to see what was happening. And I'm still getting one email per minute.

And you know what? It's not that "email is broken".

It's not the "users' fault".

It's Microsoft's.

symbian's numbers

Boom in Symbian handset shipments:

The cell-phone operating system maker announced on Thursday that 2.68 million Symbian-based handsets based on its software were shipped in the first six months of 2003. This is more than ten times as many as were sold in the same period a year ago.Cool! Which reminds me, pretty soon I'll be posting my part two of my symbian dev intro guide.

Plus: Russ's opinion on the news, from mobitopia.

email-less

I have just shut down my email server.

At a minimum, I plan to keep the server down for one day. I will soon set up a CGI form to send me emails. It's a waste of time, but it's going to be nothing compared to what I'm wasting with these virus-ridden-emails. Matthew, who develops and runs AlienCamel, offered to help. Thanks! I will seriously consider whether to switch to something like AlienCamel, or simply ditch email, or what. I will update here on the solutions I find, if any. For the moment, just don't send me emails, as they will bounce. I will post again when the CGI form is ready.

Oh, and BTW, I've received very little spam these last three days. What is up with that?

upgrade or die

Wow. From a News.com article:

As of Oct. 15, users of Microsoft's free Web-based MSN Messenger and its Windows XP-based Windows Messenger will need to upgrade their software to a newer version or be shut out of the service, the software giant said Wednesday. MSN Messenger users will need to upgrade to version 5.0 of higher; Windows Messengers customers will need to upgrade to version 4.7.2009 or higher; and consumers with MSN Messenger for Mac OS X will have to use version 3.5 or higher. The last MSN Messenger to be released was version 6.

According to Microsoft spokesman Sean Sundwall, "Security issues that could be posed (on older versions) require us to force an upgrade." He declined to detail the security issue, saying disclosure would "put customers at undue risk."

Meanwhile, Oct. 15 also will mark the deadline for Trillian support for MSN Messenger. Trillian is software that integrates multiple IM clients into a common interface. While it doesn't enable IM services to communicate directly with one another, it lets people view all of their buddy lists from various services under one window.

Now, ain't that a nice thing to do. It doesn't matter that people don't want to "upgrade" every two seconds. It doesn't matter that people are sick and tired of "security patches". Except now it's not just "upgrades" for "security reasons". It's upgrade now or it stops working. The new version will be more secure, right? Just like the new versions of Outlook and Windows were supposed to be more secure? Good, good, I see. Let me just get this abacus here for my computational needs, just in case, you know...

They have to pull this kind of stunt right in the middle of two of their most widespread security crises ever? Oh, right, of course, "security". I'm sure that wiping out Trillian connectivity in the process had nothing to do with that. Everyone will understand "security" these days. Sure. Sounds a lot like a "PATRIOT upgrade", if you know what I mean.

Oh, right, and I'm sure that this "upgrade" has nothing whatsoever to do with this, right? Of course not.

I am, quite simply, astonished that they are not more sensitive to their customers, to increasing interoperability, and to letting users choose which product they like best, dammit!

Yes, I'm still receiving one email a minute. Yes, I'm still incredibly pissed off at Microsoft.

The deadline approaches.

location and services

(Now to change the topic from all that email-related whining...) Jamie has an excellent entry comparing the differences between JXTA and Jabber. Choice quote:

The key thing JXTA can do which Jabber can't, is Discovery (Jabber browsing and any other Jabber discovery specs aren't really up to the task as far as I can see). But then Jabber has presence information, permanent user addresses and the facilities to handle 1 user, multiple devices, all of which would have to be built on top of JXTA.

Okay, since this is what my research is on ;-) I wanted to add my 2c about this topic. What Jamie refers to as Discovery is more generally considered to be part of Resource Location and Discovery which is a huge area that covers everything from DNS to LDAP to other seemingly unrelated stuff like Mobile IP (consider that a MobileIP node has a home address, and essentially systems that want to connect to the mobile node have to use that home address as a sort of mini-dynamically updated location server to find the node). Discovery implies finding resources that match certain characteristics, which might then require an additional step to locate them. Typical case of discovery is finding a printer in your immediate vicinity. One you've found the name, a location service of some sort will be used to connect to it.

Now, in that sense, Jabber does not have discovery, unless you count some kind of centralized directory of Jabber users as a source for discovering Jabber IDs, but it does provide location. JXTA, on the other hand, due to its nature allows discovery of services.

But, I think that was Jamie was referring to as "JXTA has discovery" (since he was talking about it in the context of ad hoc networks) was actually the potential of JXTA to perform self-organizing location (it can do discovery too, but that's another matter--and, Jamie, corrections welcome if I got it wrong :)). Consider that an ad hoc network might not be connected to the Internet. Then there's no fixed infrastructure available (to, say, talk to a Jabber server) and Jabber clients in an ad hoc environment won't be able to find each other. JXTA on the other hand works perfectly well. JXTA is "blowing up" the location service that Jabber provides on servers, making it work in self-organizing fashion.

Then there's the problem of routing. Once you've located the client you want to talk to, you want to connect to it to communicate, request tasks, etc. With Jabber, this happens over TCP/IP, which is simple and well understood. JXTA adds another layer on top of TCP/IP. This additional layer may or may not be written using a direct TCP/IP channel: the routing could be happening across the JXTA network itself. And that is a problem in some situations.

It's only a terminology mismatch between what's in my head and what Jamie was saying though, as his conclusions hit the mark: JXTA is good for location/discovery, particularly in ad hoc environments. But if you're on the Internet, or if you have access to fixed infrastructure, a centralized system like Jabber is probably the way to go. And, in all cases, it's uncommon that anyone will want to do routing over anything but the logical transport, such as TCP/IP or, in the case of wireless ad hoc, maybe DSR or AODV.

Plus: unrelated-- Jamie mentions a Slashdot discussion on the relevance of PhDs, and how they might affect job opportunities. Surprinsingly enough, the discussion is actually relatively civil (for /. that is). The idea that a PhD hurts your employment opportunities sounds just plain silly to me. Ethernet created a multibillion dollar industry, and quite literally changed our lives. For more recent developments, consider that REST was also the product of a PhD thesis.

There are people who fall on a PhD since they have no idea of what else to do, and that's lame. Any kind of multi-year commitment to something like that should imply that it's something that you really want to do, rather than doing it because it gives you status or because society saysut areas that interested me, and, if I could, create something new that could at so, or whatever (I feel the same way about any third-level education, btw). As for me, I got into it because I really wanted to do it, I wanted to create something new and learn about the "really bleeding edge" in the process. As far as I'm concerned, the same thing can be achieved in other ways. It's just a matter of which way seems right at a particular moment. (Very new-wavey kind of idea, I know. Heh).

I'm mad as hell... and I'll take it for one more day

Kevin Werbach is right. This last day could very well be the day email died. I am still getting one email per minute. I am even more pissed off at Microsoft. And even though I am seriously looking at how to do whitelisting/challenge-response in cc, I know that's only half of the solution since emails still clog my server inbox, and at 110 MB per day it's nothing to be sneered at.

If this keeps up I will ditch email. I'll give it one day, starting now. In the meantime I am looking at the latest version of Red Hat Linux. I still use Win XP as my OS, and I have to use it for development and such, but this is just abuse, even if I haven't been harrassed as others have been. The great Microsoft exodus should begin momentarily.

Update: From the comments, I guess that what I said wasn't clear (reading it over I admit it was a bit muddled :-)). The email I'm receiving has nothing whatsoever to do with me running Windows. I am being spammed with viruses from people that are infected. I am not contributing to the spread in any way. My email is received in a Linux box that I use for hosting. Essentially what Nex6 was saying in the comments. The only way I would have to stop these things from showing up in my mailbox is to put a virus filter at the SMTP level that would bounce to the sender when attempting to send the virus, but I don't have the software, I don't have the time to look for it, and I don't have the time to install it. Furthermore, I don't want to have to go through all this crap because of ActiveX and the shoddy security model of Windows. So.

Point number one is that if this keeps up by tomorrow I will disable my email account. Maybe switch to a new name. That will get the email bounced with no work on my part. I have specifically been thinking how I would simply stop using email at all. I am sure that it can be done. I just need to think of the cases that I want to cover, and how...

And then, point number two is to start ditching MS stuff whenever I kind, my own tiny bit of protesting. Like Scott says:

Outlook is a joke. No sane computer user today should use it. If your company makes you use it, go to your CEO and explain how much time and money his company is losing by using it. I use Eudora; there are several other good non-Microsoft products depending on what platform you're on.

Yes. Yes. Definitely. And clevercactus is one. But any client will do. The madness has to stop. Mass exodus now!

yep, that's the one

A few hours ago I started getting about one message per minute (!) containing the good ol' Sobig Worm for Outlook. Good thing clevercactus doesn't get hit by it :-). Anyway, I was curious as to what had happened, and to the rescue comes a News.com article:

The Sobig e-mail virus that caused havoc two months ago has reappeared in a virulent new form, according to e-mail service provider MessageLabs.

Yep, that's the one. MessageLabs is right, it's baaack, but worse. Much worse. So far the subject lines I've seen include: "Re: Approved" (the classic), "Thank you!", "Re: Thank you!", "Re: Details", "Re: that movie" and "Re: wicked screensaver".

Jeez. Such a waste of bandwidth, processing power, and time. Hopefully with this and other high-profile, recently noted occurrences Microsoft will finally take note and get their act together. Whatever they've done until now is clearly not enough.

Update: in a comment, Juan Cruz was pointing out a slashdot thread on this new Sobig variant, and mentioning that some people are saying this is the one that is supposed to erase the effects of the previous worm that was making the rounds last week. However, they're not the same. See here. This new Sobig worm is just incredibly annoying, and it has no redemptory qualities. :-)

Update 2: Btw, I'm still being hit by about one message per minute. Simple calculation: the worm is about 80 KB. That means in a day I'll get 24 x 60 x 80 KB = 110 MB give or take a few KB. One hundred and ten megabytes of traffic!!. I am really pissed off at MS. The addresses are spoofed, and at times my address is being spoofed, so I get rejected viruses from people that were receiving it. And I don't even want to multiply... say, by, 1% or email users that might be infected... say... five million? Ugh.

I can only wish MS will get hit by this worm as well, just as me and countless others are, and that will make them to realize what a mess this is, and force them to fix it.

one/Walking across the park along one of the less-traveled roads, I see a bird flying. The bird flies perpendicular to the road I'm on, on a trajectory that will cross mine, approaching, losing altitude. The bird touches down right on the curb opposite mine, walks across the street, tiny steps, and when it gets to the other side, only a few meters away from me, it takes off again, continuing along its original path. I stand there, looking at it until it becomes a point in the sky, then dissapears.

two/At a convenience store, getting some milk and bananas, a magazine in the newsstand reads, in bright, big type: "My husband's lover hired hitman to murder me." I keep thinking I need to get a blender.

three/Deep in work, I find an interesting paper titled "A Euclidean Ramsey Problem" by Geoffrey Exoo. I read it. It's more than useful. And yet all I can think of is how Exoo is a really cool name if you want to be, say, a topologist.

four/A song keeps playing in my head. It's Bad, not the studio version, but the one that is in Wide Awake in America. Then A sort of homecoming, which follows it. I listen to both. Repeatedly.

five/There is a spoon. But no one really knows who, or what, if anyone, or anything, put it there.

the last of futurama

Just saw the last episode of Futurama. I had read recently that Fox had disbanded the team that was working on it, and that they had one full season in the can, but I wasn't sure whether I was actually watching those episodes or not. It seems I was. I wonder if another show will ever follow in their "footsteps". The writing was as good (better, at times) than The Simpsons. And the characters... the ideas... oh, man....

Anyway.

Futurama is no more. Long live Futurama.

chaos, complexity and power cuts

Last Thursday, as I heard about the power cuts in the Eastern US, it got me thinking about Complexity and Chaos. Not that I think it was original, of course, but I was still a bit surprised to find an article written along those lines, published the very next day. Cool.

the one-minute-guide to JXTA

After my IEEE article on P2P network topologies I got some comments (including one in the entry) about JXTA. Recently I was looking in more detail at the JXTA site and I realized that there's a lot of documentation, but a lot (and I mean a lot) of it is in PDF form, and so not "browser ready". Heretics that suggest that opening a PDF inside a browser is still web browsing--so comparing HTML with what is essentially a Postscript derivative and the experience and simplicity of dealing with one as with the other--should continue along their path and don't bother trying to convince me that PDF is like the web but better, or something like that. It's a nice format for eBooks and documents meant to be printed or archived at high resolution, or where formatting is key. Period.

But I digress. Lack of HTML documentation in the JXTA site (at least docs that are easily accessible for intro material, the "tutorials page" is quite useful as a hands-on programming guide however) seems to be a problem to those looking for a short introduction. So here goes a one-minute introductory guide of JXTA concepts (ie., sans programming examples). For more detail check out this longer article from a couple of years ago at O'Reilly's website. Also useful.

intro

JXTA is a generic, protocol-independent P2P system. JXTA is not based, or dependent on, Java. JXTA depends on XML as a message passing format, and nothing else. Although the first implementation of JXTA was written in Java, there are JXTA implementations under way in other languages, such as C, Perl, Python and Ruby.

JXTA is open source, and its code is licensed with the Apache Software license. There always seems to be a fair amount of activity in the projects section (including the core), quite a number of projects going on, even if some of them are not evolving too fast.

abstractions and protocols

JXTA is a set of specifications to handle the core functions of P2P communication. Through these protocols and abstractions, JXTA establishes a P2P network on top of the Internet and non-IP networks, allowing peers to directly interact and organize independently of their network location. All nodes connected using the JXTA system form the JXTA network. JXTA employs five abstractions to abstract existing computing networks:

• Addressing. A uniform peer addressing scheme that spans the entire JXTA network. Every node is uniquely identified by a Peer ID, connected to a peer endpoint. Peer endpoints encapsulate all the network interfaces available to a node.
• Peergroups. Nodes can organize autonomously into protected domains called peergroups. A node can belong to as many peergroups as it wishes. Users, service providers, and network administrators can create peergroups to control peer interactions.
• Advertisements. Advertisements are XML documents that allow a node to to publish and discover network resources through a uniform interface.
• Binding. JXTA defines a universal binding mechanism, called the resolver, to perform all resolution operations required (for example, resolving a name to an IP address, binding an IP socket to a port, or locating a service).
• Pipes. Pipes are dynamically defined communication channels that enable services and applications to advertise communication access points through advertisements. Pipes allow nodes to dynamically connect to each other. Pipes can be of multiple types: point-to-point, point-to-multipoint, encrypted, and others.

These abstractions are handled through a set of protocols (although the mapping between abstractions and protocols is not
one-to-one), as follows:

• Peer Discovery Protocol. Enables nodes to discover services on the network.
• Peer Resolver Protocol. Allows nodes to send and process generic requests. This search/retrieval mechanism is what enables nodes to locate each other, to find peergroups, find services, and retrieve content, also performing authentication and verification of credentials.
• Rendezvous Protocol. Defines the details of message-propagation between nodes.
• Peer Information Protocol. Allows nodes to obtain information about other nodes in the network.
• Pipe Binding Protocol. Provides a mechanism to bind a virtual communication channel to a peer endpoint.
• Endpoint Routing Protocol. Provides a set of messages used to enable message routing from a source peer to a destination peer.

Each JXTA protocol defines a set of XML messages to coordinate one of the aspects of JXTA interactions. In JXTA, all resolution operations are unified under the simple discovery of one or more advertisements. All binding operations are implemented as the discovery or search of one or more XML documents. However, JXTA does not specify how the search of advertisements is
performed. Each peergroup can tailor its resolver implementation to use a decentralized, centralized, or hybrid search approach to match the peergroup's requirements.

back to the real world

JXTA has made a lot of progress recently, but using it can still be difficult at first, particularly if you'd like to tinker with the source (check out the instructions in the core build download page and see for yourself), and in some cases its performance is terrible. However, it's very useful as a base to whip up a quick P2P prototype (quick in terms of development times, not in performance :-)).

Hm. If I keep typing, this is going to take longer than one minute to read. Okay, I'll stop. Done.

flame warriors

[via Scripting News]: Flame Warriors, a taxonomy of people's roles on online discussions. Heh.

don't push!

Scott on Wired's "Push" story and a few other things:

In its heyday, Wired magazine gave the entire technology and Internet press a steady stream of wacky, outrageous material to react to. On the blog he has created to accompany his new history of Wired, "Wired: A Romance" (Andrew Leonard's Salon review is here), Gary Wolf is posting some reminiscences and other Wired miscellany.

I have to agree with his judgment that Wired's worst story ever was the "Push" cover story he was credited as co-author of. Wolf's recollections of how that absurd piece of puffery came into existence is illuminating and worth reading; Wired, it seems, was even more seat-of-the-pants in its editorial process than those of us on the outside could tell. I'll stand by my assessment of February, 1997, that the story wounded the publication's credibility. But reading Wolf's account, you can't help feeling a little more charitable toward the people responsible for the open-ended, improvisatory provocation that was the Wired game. Viewed as a moment rather than a movement, it all seems a little funnier and less heinous. After all, the next three years would see far vaster corporate scams unfold -- and ones with far less style.

I remember this story really well--and I have to admit that it really had me going for some time after. Of course, reality settled in pretty soon (not that Wired ever published a revision of some kind though--and they should have! A story as big, with the covered changed to say " don'tpush" or something). Misinformation was not spread "on purpose" but it was all part of the big machinery of hype that took over the tech world during those times; Wired just became the amplifier for all sorts of madness that was going on in The Valley. Strange how these feedback cycles can occur--and interesting how the process behind them eventually comes to light, to the everlasting wonder and amazement of all. :-)

Manhattan offline

Just saw this on CNN.com as a news alert (now there's a short story posted now), and went to check it on the TV: there's a massive power-outage on the east coast of the US, affecting, apparently, Manhattan, Boston, Cleveland, Detroit, and other cities, as well as some Canadian cities such as Ottawa and Toronto. At the moment, in Manhattan there are no public transportation services: no buses, subways or trains. Most of Downtown Manhattan, including Wall street, have shut down. Many airports closed. Massive traffic jams.

What a mess.

I was just looking at the TV images of people walking across the FDR bridge (obviously, vehicle traffic has slowed to a standstill) and it's kind of a surreal scene. No cars, just a sea of people, in, out, and on the bridge. Apparently the power cut started as a hardware problem at transformer at a ConEd plant in NYC, and then it started to spread (no indication whatsoever that sabotage was involved, even though the news people keep bringing it up). At the moment it appears to have affected the balance of the whole of the US power grid. Amazing how this kind of thing can happen. Just one component in one power plant, in one city, and you get this result. Apparently the problem is still spreading through the network as the "domino effect" takes hold (Reminds me of the effect that brought down the long-distance AT&T network on January 15, 1990, as documented by Bruce Sterling in his excellent book The Hacker Crackdown--although in this case it's a different type of overload that is spreading, but the effect is exactly the same: a station goes down, which breeds more overload on the stations that are still up, which then shut down, which...). Hopefully it's being contained though.

We tend to think of network effects and related ideas in terms of concepts, in particularly concepts related to computer science (these ideas are big in the social sciences, group psychology and Economics too, though). In case anyone had any doubt, this shows that, more and more, everything will be affected by it. The science of Complexity (and the related topic of Chaos) will only grow in importance in the years to come.

Update: this article from the New York Time dismisses the "transformer theory" along with others. So even two days later it's still unclear what caused the problem--they seem to agree it originated in the mid-west though. The Economist has more information, as well as a historical perspective.

And Bruno said, in a comment, that reports that said that Wall Street had shut down were wrong. There were lots of similar comments to this effect in the newspapers yesterday and today as well. Another one of the many innacuracies in initial reporting, along with the transformer story, Boston being affected (it wasn't--not that much at least), and so on.

"weblog" is one word

An article on weblogs in this week's Economist. Some notes of interest, but aside from all that, can I say something? (Diego asks, then Diego replies: Yes, of course you can! Heh). So here goes: Will some publications please stop writing "weblog" as "web log" (note the space). News.com does this often. The Economist has apparently followed suit. "Web log" reads... broken. Just write "weblog" or "blog" and be done with it. It's one word, not two. Or will they now start writing "cyber space" too?

news in one page -- but not a webpage

Another "clevercactus tip of the day". :-)

The other day, Dave was saying that using the three-pane paradigm to present news in an aggregator was not the way to go. Today, he linked to a piece by Adam Curry in which, among other things, Adam puts forward the same idea. One important difference (aside from the UI) between typical three-pane readers and web-page readers is what Dave refers to as "the queue". Adam mentions that another advantage is that part of the "freshness" of RSS feeds is how you can get a view of a number of different POVs on the same idea.

Both of these uses (and more) are not just legitimate, they are also clearly useful in many cases. But sometimes I want to see what just one person has said recently, and for that the three-pane view rules. Additionally, at times an item might contain more than a short description, it might contain the full text of the entry, for which three-pane views rule as well (since the full text could be really long).

So, is it possible to have both?

Absolutely :)

When reading feeds in clevercactus you can get the combined effect of a "one-page" view with the benefits of a three-pane view. As an example, consider the following two screenshots. Number one shows the "aggregated" view, where all the feeds are shown together. Number two shows only the space that is aggregating Scripting News. Just one click takes me from a bird's eye view to a more specific per-blog view.

How does this happen? In clevercactus, a space is recursive, that is, it automatically shows not only its contents but the contents of its subspaces (which are also recursive, and so on) with the proper ordering. So this effect happens transparently and automatically.

Now this is close, but no dice yet, since (as opposed to the way in which webpage aggregators do it) the news items remain there in the aggregator, which can potentially (and very quickly) create clutter (This is another one of Dave's good points). In an upcoming version of clevercactus, you'll be able to set the "decay rate" of items (automatically set by default to a sensible value). This means that after, say, seven days, feed items will be deleted automatically unless explicitly marked for permanent storage through a one-click procedure on the item itself (An icon saying "Store this") -- And bingo, we've got the functional equivalent of the aggregated webpage view, but in a three-pane UI, with "entry decay" for the first time.

Some users will still prefer a webpage view (which you can also get in clevercactus using webaccess, btw!).

And that's what it's all about isn't it? Giving the users choice.

Update: Ted was commenting on this entry and comparing clevercactus to FeedDemon. He said:

it seems to me that Clevercactus looks a lot like FeedDemon, except that FeedDemon can do a newspaper, and Clevercactus can't, and Clevercactus has PIM info, which FeedDemon doesn't

I didn't know about FeedDemon's Newspaper View. Very nice! However, I must say that clevercactus has a lot more than "PIM info"-- some examples are the webaccess feature, weblog posting, email, and Real Soon Now a bunch of other cool things, such as real-time collaboration and synchronization. Reading feeds (Both RSS and Atom) is just one more of the information channels that clevercactus can handle.

Update 2: Sam linked to this entry and Dare commented that SharpReader and RSS Bandit support this kind of functionality. Dare was also wondering whether I thought that cc was first on this, and I replied:

Clevercactus was first. CC was originally called spaces, and it was released publicly on November 11, 2002 (the name changed in April this year), supporting exactly that functionality (on both feeds and email), which predates most if not all the 3-pane aggregators that I know of, and certainly those that do recursive aggregation (of course, corrections are welcome--you never know :)).

That said, I was just pointing out that it didn't have to be a choice of either/or, and of course I used my own software as an example since I can fully explain what it does and how.

Additionally, when I said "for the first time" I was referring to the concept of "entry decay" plus the rest of the stuff (Of course, if anyone's aware of other aggregators that are currently doing that--i.e., allow you to set the time after which entries dissapear on a per-feed basis-- please let me know and I'll correct it). I was of course aware of others (although not of its FeedDemon's Newspaper view, which I noted in the post after Ted's comment).

As usual, corrections or more comments are welcome. :-)

Update 3: In another comment on Sam's blog Dare noted that he implemented "entry decay" in RSS Bandit about a month ago. It's not deployed (then again clevercactus isn't either), but this predates my first public mention of this feature by at least a week (obviously more, since Dare noted the release of the code on that entry, rather than the idea as I did). Cool.

PS: Are we all in sync or what? :-)))

two cool books

Today (finally!) I got two books that I needed for my thesis (or is it "wanted" instead of "needed"?). One of them is Packet Communication Robert Metcalfe's 1973 PhD thesis in which he set the basis for Ethernet (which was developed by him shortly thereafter while at Xerox PARC). The other one is Ruling the root, internet governance and the taming of cyberspace by Milton L. Mueller, which talks about the development of DNS and all the issues surrounding it. I've flipped through them, and I'm already drooling. I know most if not all the topics discussed in these books, some of them very well, but sometimes it' easy to get hooked on a single element ("Hey, how cool is this proof that algorithm Z can be relied on to behave at O(log N) complexity?") and losing sight of the larger picture. "Connecting the dots" it's called these days. More comments after I've read them.

in praise of RMI

I meant to write this sooner; now I'm running a long test on an algorithm and I've got some time; so here it goes.

Ever since Java RMI got released, it was derided by critics at all levels. Too slow, they said. Hides too many of the details, they said. Does not hide enough, said others. And everyone hated those checked RemoteExceptions. In my experience, however, all of those complaints are baseless, or simplymiss the point.

Not to boast >grin< but I was one of the first people to actually write a useful application in RMI (See this Sun list of applications--the one I wrote was the real time collaboration server for dynamicobjects' perspectives, and btw, dynamicobjects was not a corporation, it was a development group of me and three friends, but we were labeled as a "corporation" for some reason). When I wrote the application RMI had just been released, and, even then, it just worked. And it worked well. Of course, the context in which it worked was a LAN. Forget about the internet--too much latency. Problems with firewalls. Etcetera.

Since then, RMI has become a lot better: IIOP interoperability, proxies, and a very cool automatic fallback mechanism that defaults to using RMI-over-HTTP when a firewall is preventing TCP communication.

The "Internet problem" still remains, though. Latencies over TCP are an issue, the marshalling/unmarshalling of the objects is partially an issue (since it increases data transfer requirements through serialization), and the fact that RMI hides so many of the connection details from the developer means that it's more difficult to create a "controlled environment" that deals properly with firewall situations, etc.

Even so, RMI is still the best choice for two tasks:

• Prototyping of any Java networked application. Creating a network application with RMI is by far the fastest and more reliable method for getting it up and running quickly. Often, a first shot at an idea is enough to udnerstand the issues involved. Then, if the design was correct in the first place, you can just rewrite the transport mechanisms maybe even using Serialization directly over TCP sockets. This is exactly what I did recently when working on clevercactus collaboration (though it hasn't been released yet), and the process worked really well for me.
• LAN-only applications. If the application to be deployed is LAN-only, then there is no question that RMI is the way to go. RMI is easier to debug and maintain, and it performs really well over fast, low-latency networks without firewalls.

So, in conclusion: RMI is not for everything (and no one claimed it would be, but I think a lot of the criticism comes from the misplaced expectation that it should be good for everything), but, in many cases, it's exactly what is required.

reverse engineering: a case study

From CMU's Software Engineering Institute: Into the Black Box: A Case Study in Obtaining Visibility into Commercial Software:

We were recently involved with a project that faced an interesting and not uncommon dilemma. The project needed to programmatically extract private keys and digital certificates from the Netscape Communicator v4.5 database. Netscape documentation was inadequate for us to figure out how to do this. As it turns out, this inadequacy was intentional-Netscape was concerned that releasing this information might possibly violate export control laws concerning encryption technology. Since our interest was in building a system and not exporting cryptographic technology, we decided to further investigate how to achieve our objectives even without support from Netscape. We restricted ourselves to the use of Netscape-provided code and documentation, and to information available on the Web. Our objective was to build our system, and to provide feedback to Netscape on how to engineer their product to provide the capability that we (and others) need, while not making the product vulnerable or expose the vendor to violations of export control laws. This paper describes our experiences peering "into the black box."

Great analysis, very detailed and extremely interesting.

the java almanac

Another one: Russ mentioned the Java Almanac yesterday. Tons of code samples, including organization by packages that makes it even more useful. Great resource.

the danger for the US

[via Karlin]: Ian Clarke, creator of Freenet has said that he will leave the US. His reasons? In his own words:

As an Irish citizen living in the US - I have decided that it is time to leave this country - it is starting to look, smell, and act as Germany did during the 1930s. I wish you Americans luck in regaining civilized justice in your broken country, if not, I hope that the EU wi