detriot.org

Benjamin Chodoroff's blog

A bit about my recent work:

As the web dev for the <a title=”Allied Media Conference” href=”http://alliedmedia.org”>Allied Media Conference</a>, I’ve been tasked with creating a system for participants to liveblog about all the sessions at this year’s conference.  I decided on setting up a three-part system: a tweet searcher+archiver, a JSON server, and the existing <a title=”Discuss the 2010 Allied Media Conference and US Social Forum” href=”http://talk.alliedmedia.org”>2010 AMC and USSF discussion message board</a> software.

• Tweet searcher+archiver: a node.js app that reads a filtered twitter feed from the Streaming Twitter API (using a version of twitter-node, separates out relevant hashtags (each relating to a conference session) and stores them in a MySQL db. (80% complete)
• Tweet server: another node.js app, this time a HTTP server using djangode, that responds to queries for certain hashtags and returns a JSON stream of tweets with that tag, or alternatively returns a "last tweet" timestamp for comparison on the frontend. (10% complete)
• Frontend: a modified Drupal forum setup with a special field on topics that offers the relevant hashtag as an element on the page.  Javascript by PaulH will form a HTTP request using that hashtag to retrieve tweets from the tweet server and, based on the twitter timestamp, intertwine them with the topic replies. (80% complete)

This may seem like a complicated system.  Why not just draw tweets using clientside javascript during pageload?  Why use bespoke node.js servers instead of doing a Drupal implementation or buying cloud services?  Respectively, temporariness and efficiency.  With our setup, the twitter conversation isn’t temporary anymore:  we can archive the conversation and guarantee that it lasts for future viewers (no need to rely on twitter to keep on serving tweets).  It’s also more efficient: for one thing, it’s only one stream request from twitter (maybe two if i want to do redundancy) instead of a bunch of in-browser requests, and the streaming API seems to be more timely and reliable.  Node.js lends itself very well to this type of system.

While the current implementation is definitely off-the-cuff hackish, I’m going to work on generalizing the code and will release it on github afterwards.  If anyone is interested in collaboration, please comment!