Rasmus' Toys Page

SearchMonkey

rasmus@lerdorf.com (Rasmus) — 2008-05-08T21:50:28Z

One of the things I have been playing with lately is Yahoo!'s SearchMonkey project. It appeals to me on many different levels. The geeky name is a play on GreaseMonkey. But instead of writing plugins that run locally in the browser, SearchMonkey is a way to write plugins for the Yahoo! Search results page that change the appearance of the results themselves. Best explained with an example. Assume I am looking for a Japanese restaurant, and on my search results page I see:

That's ok, I guess. It tells me it is somewhere in Redwood City and that it is a neighborhood restaurant, whatever that means. Compare that to:

This gets me a real address and phone number plus a number of other useful bits of information. That is the first level SearchMonkey appeals to me on. The usefulness is obvious. My usefulness test is to see if I can explain it to my mother. Having her search for recipes and get pictures of dishes, ingredients and preparation times right on the search results page makes this an easy sell.

The second level this appeals to me on is the way it is implemented. Writing these SearchMonkey plugins becomes much simpler if the site you are writing the plugin for uses microformats of some sort. hCard, hCalendar, hReview, hAtom, xfn or generic structured eRDF or RDFa tags. The data can also be collected via a separate XML feed that can then be converted via XSLT in the SearchMonkey developer tool. The microformat data is collected and indexed and when you go to write a plugin and specify the url pattern you are writing the plugin for, it will find whatever indexed metadata it has for that url. If it doesn't have what you are looking for, you can still write a custom data scraper to get it, but that gets a bit more involved. I really like that the easy path is to add some sort of semantic markup to the pages. Yes, as Micah points out, this is not the (uppercase) Semantic Web, but it is still a push towards semantic markup. Having such a tangible and visible result of adding semantic tags is going to encourage people other than microformat geeks to do so. The more semantic markup we get, the better off the Web is.

The third part that appeals to me is the way the plugins are written. You write a little snippet of PHP. It is actually a method in a class you can't see, but its job is to return an associative array of data such as the title to display, the summary, extra links to show and whatever other key/value pairs you might want in the output. Because you have a full-featured scripting language available, you can write quite complicated logic in one of these plugins and pull whatever data you want from the site the plugin is written for. You can also write an add-on to your plugin which is called an Infobar. It is a little bar that is shown below the plugin and from an Infobar you can access arbitrary external services. This example shows it well:

This one shows an OpenTable reservation link and a Yelp review, but almost anything can go there as long as you can squeeze it into the limited space you have.

The SearchMonkey is still in its infancy. It needs developer support. If you are in Silicon Valley, please come to the Developer Launch Party next week on Thursday May 15. See the link for details. If you aren't in the area, or even if you are, sign up for a developer account at http://developer.yahoo.com/searchmonkey/preview.html and help encourage the Web to become more semantic.

Looking for a new 1U box

rasmus@lerdorf.com (Rasmus) — 2008-01-13T01:23:18Z

This server is well over 5 years old and really starting to show its age. It's a 2.66GHz P4 with a Gig of ram and mirrored 80G drives. I am constantly running out of disk space and spamd chews up all available cpu on a regular basis. Moving Bayes token expiration to a cron job helped spamd cpu usage a little bit, but it still isn't a happy server.
I need way more disk and way more ram. I'm looking for a dual-cpu, mirrored (software raid this time) 500G drives, and 4G of ram 1U box to stick in the colo. This http://www.siliconmechanics.com/i14740/Quad-Core-Xeon.php fits the bill, although I find the quad-core cpus a bit silly. I'd be fine with quick dual-core chips, but the bang-for-the-buck on the configs there seem to favour the 4-core chips.

This Supermicro http://www.xpcgear.com/sc813t50c1.html looks ok as well.

A Sun X2200 is a possibility, but they seem a bit pricy for what you get and I like the new 45nm Intel chips.

Has anybody run across any interesting new 1U servers?

New toys: iMac and Sonos

rasmus@lerdorf.com (Rasmus) — 2007-12-16T17:08:10Z

A couple of early Christmas presents for the new house. A new iMac for the kitchen nook and a Sonos system. I guess I haven't bought ram in a while because it was under $100 to upgrade the iMac to 4G with ram from OWC. The iMac is a perfect fit for the kitchen. The black and aluminum matches the counters and appliances in the kitchen nicely.
I had been eyeing a Mac Mini for ages, but Apple doesn't seem very interested in the Mini and for the price the iMac seemed like a much better deal. Bought it from Amazon and it didn't come with Leopard but Apple has an update program so the upgrade is mostly free. They want $10 for shipping me the CD. I already have another copy, so it would be nice if they would just give me a serial number.

As nice as the iMac is, the Sonos system is more interesting. I have had various mechanisms for playing music from computers to decent speakers over the years, but they were all inconvenient hacks. I don't want to have to use a computer to control the music, and I definitely don't want to do it via a clunky TV-based interface either. The Sonos with its controller does a great job of taking the hack out of the system. It is basically a wireless meshing modular music system. You put either an amplified box with speakers connected, or an unamplified box connected to an existing stereo in each room and the single controller can then control each zone individually or you can link them all up so all the zones play the same music. Each box also has a Line-In that can be used as a source and played in any other zone

I wanted the iMac to be able to go to sleep without killing the music, so I used a 250G Simpleshare drive I had sitting around. One of these days I need to figure out a real NAS system for the house, but for now 250G is plenty as a Sonos media source. I pointed iTunes at the Simpleshare and copied all the music to the drive, then I pointed the Sonos system at the drive as well and it worked nicely.

Internet radio streaming, Rhapsody and Pandora are all working very nicely. I realize this turned out to be a bit of a boring post since there were no technical hurdles and thus no interesting hacks involved in getting any of this working. But that is afterall why people buy things like iMacs and Sonos systems. If they didn't just work without days of fiddling there are plenty of cheaper options that will let you hack and fiddle for months and in the end you get something that almost sorta mostly works.

Pipes

rasmus@lerdorf.com (Rasmus) — 2007-02-08T10:03:22Z

http://pipes.yahoo.com is a cool toy, and by toy I mean it in the useful and cant-stop-playing-with-it sense. My first impression when I saw an early version a couple of months ago was, "How the heck did they do that?" I was reading the Javascript source code for quite a while. Once you get beyond the fact that this is a browser-based app doing this without Flash, or Java or any similar cheats, you get down to what the app actually does.

Years ago I wrote this silly little Mashup example:

http://buzz.progphp.com/?q=4

It grabs an RSS feed, in this case the top daily search term % movers from http://buzz.yahoo.com/feeds/buzzoverm.xml which gives you an indication of what is on the minds of web searchers right now. I took these searches and did a Yahoo Image search and a News search and combined them in that oval interface you see. I had to do a bit of RSS and XML parsing to take these different data sources and combine them. This is what Pipes is all about. It provides a visual environment for manipulating data sources and then provides a number of different ways to get the results and integrate them into other things. Directly in your RSS reader is probably the simplest, but you could also feed it to PHP and do further data manipulation.

A simplified Pipes version of the above takes the same Buzz.yahoo.com RSS feed and does a Flickr search on each search term. The result looks like this:

http://pipes.yahoo.com/pipes/DnudMlO32xGDcIu7pRr_og

The point here is not the visual output. It is meant to be fed to something else. Hover over the "Subscribe" link on the right there. Then click on the "How this pipe was made" image on the left to see how it works.

This is a particularly lame and simple pipe. Some much cooler ones include:

Blog Buzz for Pipes combines a couple of different blog watching feeds, filters out duplicates and gives you a combined feed in reverse chronological order. When you look at how it was made it becomes immediately obvious what it does. You can save a copy and make your own version that watches for whatever terms you want.

Another interesting one takes the New York Times front page, runs a content analysis on it to get a set of representative keywords and then does a Flickr search on each of those. http://pipes.yahoo.com/pipes/vvW1cD212xGMiR9aqu5lkA/.

Here is a much more complex pipe that takes some user input. It finds apartments near things. In this case it looks for apartments within 2 miles of a Park in Palo Alto, California by searching Craigslist, then doing a location extraction and then doing a Yahoo! Local Search for that location.

Even if you have no use for processing data sources this way, open up one of these Pipes and drag the boxes around and watch the pipes react. Web apps don't get any cooler than this right now.

Want a PHP job?

rasmus@lerdorf.com (Rasmus) — 2007-01-31T07:36:00Z

Want to work on some of the busiest and coolest web apps in the world?

Do you like Flickr, and want to work downtown San Francisco?

Or perhaps you are into music, movies or TV and want to work out of Santa Monica? Jumpcut? Or have you seen answers.yahoo.com? Address Book, Personals, Search, Premium Services, Hot Jobs? Want to do interesting things combining PHP and Flash?

Yes, I get a referral bonus, but I need more toys. You get a cool job though, so I think we are even.

Send me your resume and let me know what sort of stuff you are interested in or poke around on http://careers.yahoo.com/ and let me know which job interests you and I will forward your resume to the appropriate hiring manager.

[edited to remove RSS ad test I had forgotten about]

100 Runs

rasmus@lerdorf.com (Rasmus) — 2007-01-28T01:21:57Z

My previous entry way back on Aug.18 talked about the Nike+ipod widget I had picked up along with a first gen Nano. I figure it is time for an update. I still use it 5 or 6 times a week and I just logged my 100th run. There have been a few more actually, but I have had the odd Nano crash/corruption on me before I had a chance to upload a run.
Despite those occasional technical glitches, I still really like using it. I have been eyeing the Garmin Forerunner 305, but I actually like listening to music or a podcast as I run and it seems a bit much to strap a GPS to my wrist and also carry an mp3 player. Plus the thing costs $300+ vs. the $27 the Nike+ipod gadget costs, assuming you already have a Nano.

My 100th run looked like this:

620 miles later and some 80,000 calories burned I have gone from being a 205lb fat slow geek to a 175lb slow geek. Not entirely true, I lost about 10lb before I started running more seriously, so the Aug.18 to Jan.27 timeframe was about a 20lb drop. 80,000 calories and I read that one pound is about 3500 calories which seems to fit my numbers.

I ran a couple of hundred miles in a pair of Nike+ shoes. It is very convenient to have the sensor right in the sole of the shoe, but that is the only thing I liked about those shoes. Now I have a pair of New Balance 767's and have sewn the little sensor seated in a Switcheasy RunAway thing to the front of the shoe with the back looped through the bottom of the laces. That gets it as horizontal as possible and I find it is more accurate that way. Of course the RunAway plastic thing needs a bit of tape to make sure it doesn't swing or pop open. Here is a picture of the hack job:

I have also been playing with the run data a bit. Each run is stored in an XML file which is uploaded to the Nike site by itunes when you synch. You can easily read the files yourself by mounting the ipod and navigating to it. However, the red Nike interface you see above is a flash thing and since flash is obviously client-side the site needs some sort of API. With a little digging with Firebug it was trivial to figure out how it worked and I whipped up a little Serendipity plugin to show my stats. You can see it in the right column of this page. And I wrote a simple PHP 5 class that makes it rather easy to integrate this stuff into other apps. You can see the code here:

http://lerdorf.com/php/nikeplus.phps

Each run in the run list has an id and you can call the run() method in that class to get the raw data for that run. Graphing that gives us something like this for that same run I showed at the top:

The grey line is the raw data and the red line is a Lowess curve based on that. I find it interesting how noisy the raw data actually is. Especially for the shorter runs, but by applying a bit of local regression it cleans up nicely and data that seems accurate emerges. I can pick out the uphill stretches of my run from the graph above. If you click on the runs in the right sidebar section there you can see the graphs of my other runs.

If I could somehow find some free time I'd love to build a better tracking site than that Nike thing and then also have it support the Forerunner and whatever other devices are out there. But for now most of my free time is spent running. Probably better for me in the "long run" anyway.

Nike+iPod Goodness

rasmus@lerdorf.com (Rasmus) — 2006-08-18T16:25:28Z

On a whim I picked up the Nike+iPod doodad the other day and this morning did my first decent run with it and I am quite impressed. I'm not much of an Apple nor Nike fan. I do have a Powerbook, but doubt I would buy another one (see previous Toys entry), but this little device is definitely cool. It is going to make me run further and faster and at a more consistent pace. If you already have a Nano you really should pick one up. It's only $29, $27 if you work somewhere where you get the Employee discount. Even if you aren't a runner, this works perfectly well for walking as well. Even if you don't have a Nano, you can pick one up for $108 in the refurbished section of the Apple store right now. Refurbished Apple items seem to be indistinguishable from new items in my experience.

It comes as two tiny devices. Doodad1 and Doodad2. Doodad1 you attach to your Nano as pictured on the right. It only comes in white so it doesn't match my black Nano all that well. Not that I care. Doodad2 you attach securely to your shoe. It should be on tightly such that it moves with your foot and doesn't dangle in a pouch or something. You can of course buy the $100 Nike+ shoes which I am sure is the goal of this, but I find it works perfectly well stuffed under my sock on top of my foot with the laces tight below and above it. I don't feel it and it seems to be very accurate measuring distances. It uses some sort of piezoelectric accelerometer, like in a speaker, to measure the amount of time your foot spends on the ground combined with the time between foot strikes to figure out how fast and how far you are running. After I calibrated mine over a 1 mile distance it seems to be very accurate. I tried running 400 meters with long strides and then the same 400m with short strides and it didn't get confused. Walking the same 400m it managed to measure accurately as well. EEtimes had a good article on it.

The guts of the Doodads look like the image below.

While you are running the screen shows your progress. Distance, pace and time. If you hit the center button it will use either a male or a female voice to tell you the same information so you don't need to look at your screen. If you choose a set distance for your workout it will tell you that you are "halfway", "400 meters to go", "300 meters to go"... I thought it was interesting that it used meters even though I have mine set to give me everything in miles. I go both ways on the miles/meters thing so I don't care, but I could see someone being confused by that. When you finish your workout and you have beaten your previous best time or distance record you get a little congratulatory voice by Lance Armstrong or a couple of others, I think. I have only gotten Lance so far.

You then plug your iPod back into your computer and it uploads (if you let it) your workout to the Nike+ site. This site has a very nice Flash app (for some definition of nice tempered by the fact that it is Flash) which keeps track of your runs. It's a very shiny app that shows each individual run with time, distance, pace and calories burned (you enter your weight during setup).

The yellow line shows your speed during a run. The dots along the run there appear to be the times I pressed the center button to hear my progress. When you mouse over them in the app you see the distance and pace at that point. The speed seems to match my mental state and the hills along the run pretty well. And I find it really nice to be able to quickly check if I am falling behind the pace I know I can run. It is too easy to trick yourself into believing you are running at your optimal pace.

There is an overview screen where you see your runs. I have only done 2 so far. A short 1 mile run to check the calibration and then this morning's 10km run. When you mouse over the bars it gives you the details on each run.

And there is a summary screen showing your farthest run, best 1mi, 5km and 10km runs.

There are also ways to set goals and to create groups where I assume you can see other peoples' runs and motivate each other to run more that way. I don't know anybody else with one of these yet, so I haven't played with that feature. Perhaps we need to set up a running group for fat grumpy open source developers. It might help turn us into just grumpy open source developers. My wife showed an immediate interest in it as well, but it doesn't support multiple profiles on the same sensor. You can however have multiple sensors and share the same iPod, so I already ordered another $27 sensor for her.

Time to buy a Thinkpad again?

rasmus@lerdorf.com (Rasmus) — 2006-08-13T18:18:43Z

About 18 months ago I had a very nice perfectly configured Debian T42p Thinkpad stolen at a conference. I had less than two weeks before the next conference and I didn't have time to fiddle with my OS to get a new Thinkpad up and running with Debian. It also annoyed me that there was no way to buy a new Thinkpad without paying Microsoft for yet another Windows XP license I would never use. So I swallowed some Open Source pride and bought a Powerbook G4. It took a bit of getting used to, but overall it was a pleasant experience for a while. It was a Unix laptop which just worked and I was up and running quickly.

The Powerbook hasn't done so well over time though. It is not nearly robust enough for my hectic travel schedule which include plenty of mad dashes through airports in strange countries. The clasp to hold it closed broke long ago. The power plug thing is so bent out of shape it won't hold the power connector solidly. These two factors probably contributed to my battery completely dying after about a year and I had to replace it. Also, whenever I put it down the CD Rom tries to eject a non-existant disk, and the screen has a long dent in it which causes a dark area on the display. The cheap silver paint has rubbed off the case where my wrist rests, the key labels are disappearing and my cursor-left key isn't there anymore. Half the time it won't come out of sleep mode requiring a reboot and the other half it simply turns itself off. It also gets amazingly hot. I do have full Applecare on it, but it doesn't appear like they will fix any of this stuff because the 4 or 5 big dents in it is obvious signs of it being dropped and they state they won't fix damage due to a drop.

I have of course been eyeing the new MacBookPro, but I see no signs that it would be any more robust. And even worse, now with their Intel move they are using Intel's Trusted Computing Platform for who knows what. I haven't seen much on what they are doing with the TPM beyond tieing the OS to the hardware, but since they are not a member of the Trusted Computing Group there is no oversight. I also find myself using very little Apple software. It is shiny and pretty for the most part, but not really something an old UNIX hack like me would use on a day to day basis. Compiling things on a G4 is painful. gcc and gdb do work, but Valgrind and Callgrind don't which means I can't do any serious development on it.

So, back to the Thinkpad. With Lenovo's announcement that they will sell the T60p with Suse pre-installed I am assuming this means I can finally buy a Thinkpad without paying the Microsoft tax. Combine that with the amazing improvements of Linux on the desktop spearheaded by Ubuntu and it all means that my Apple experiment is coming to an end after about 18 months.

Of course now I need to figure out how to navigate the Lenovo ordering process. It doesn't appear that they have this non-Windows T60p available yet. Probably need to wait until after Linuxworld next week. Will also need to figure out whether EPP (employee) or SPP (stockholder) pricing is cheaper, or if there is some other mechanism to get a bit of a discount. How about an AOSD (Annoying Open Source Developer) discount?

Get your geo plugin here

rasmus@lerdorf.com (Rasmus) — 2006-03-05T23:50:00Z

PHP 5, JSON from pecl cvs, Map Tile API and the Yui Connection Manager. Toss them in a bag and shake and you get something like this. Enter a city name, or even a pseudo-name like Philly or SFO and hit return. Each time you enter a new name you will get a map tile for that city.

It's not all that fancy, but it sure is easy to do:

33 lines total which includes about 6 lines of PHP which consists mostly of building the URL to the map tile service. Then the Javascript which has a callback function to read the form values and make the backend GET request and a simple function to take the JSON response and add the map tile to the document. And finally the actual HTML form.

With a slight tweak you can change it to make a nice geocode lookup field.

So, fire up your text editors and start writing some plugins for blog and forum packages and perhaps image gallery applications as well.

The no-framework PHP MVC framework

rasmus@lerdorf.com (Rasmus) — 2006-02-27T22:39:00Z

March 1 - Disclaimer: Since a lot of people seem to me misunderstanding this article. It isn't about OOP vs. Procedural programming styles. I happen to lean more towards procedural, but could easily have gone more OOP. I simplified the code a bit for brevity, but have added a light OO layer back in the model now. Not that it makes a difference. What I was hoping to get across here is a simple example of how you can use PHP as-is, without additional complex external layers, to apply an MVC approach with clean and simple views and still have all the goodness of fancy Web 2.0 features. If you think I am out to personally offend you and your favourite framework, then you have the wrong idea. I just happen find most of them too complex for my needs and this is a proposed alternative. If you have found a framework that works for you, great.

So you want to build the next fancy Web 2.0 site? You'll need some gear. Most likely in the form of a big complex MVC framework with plenty of layers that abstracts away your database, your HTML, your Javascript and in the end your application itself. If it is a really good framework it will provide a dozen things you'll never need.
I am obviously not a fan of such frameworks. I like stuff I can understand in an instant. Both because it lets me be productive right away and because 6 months from now when I come back to fix something, again I will only need an instant to figure out what is going on. So, here is my current approach to building rich web applications. The main pieces are:

MVC?

I don't have much of a problem with MVC itself. It's the framework baggage that usually comes along with it that I avoid. Parts of frameworks can be useful as long as you can separate the parts out that you need. As for MVC, if you use it carefully, it can be useful in a web application. Just make sure you avoid the temptation of creating a single monolithic controller. A web application by its very nature is a series of small discrete requests. If you send all of your requests through a single controller on a single machine you have just defeated this very important architecture. Discreteness gives you scalability and modularity. You can break large problems up into a series of very small and modular solutions and you can deploy these across as many servers as you like. You need to tie them together to some extent most likely through some backend datastore, but keep them as separate as possible. This means you want your views and controllers very close to each other and you want to keep your controllers as small as possible.

Goals for this approach

Clean and simple design

HTML should look like HTML
Keep the PHP code in the views extremely simple: function calls, simple loops and variable substitutions should be all you need

Secure

Input validation using pecl/filter as a data firewall
When possible, avoid layers and other complexities to make code easier to audit

Fast

Avoid include_once and require_once
Use APC and apc_store/apc_fetch for caching data that rarely changes
Stay with procedural style unless something is truly an object
Avoid locks at all costs

Continue reading "The no-framework PHP MVC framework"