AMC 2010 liveblogging: gleaning tweets with node.js and mysql

June 11, 2010

A bit about my recent work:

As the web dev for the Allied Media Conference, I’ve been tasked with creating a system for participants to liveblog about all the sessions at this year’s conference.  I decided on setting up a three-part system: a tweet searcher+archiver, a JSON server, and the existing 2010 AMC and USSF discussion message board software.

  • Tweet searcher+archiver: a node.js app that reads a filtered twitter feed from the Streaming Twitter API (using a version of twitter-node, separates out relevant hashtags (each relating to a conference session) and stores them in a MySQL db. (80% complete)
  • Tweet server: another node.js app, this time a HTTP server using djangode, that responds to queries for certain hashtags and returns a JSON stream of tweets with that tag, or alternatively returns a "last tweet" timestamp for comparison on the frontend. (10% complete)
  • Frontend: a modified Drupal forum setup with a special field on topics that offers the relevant hashtag as an element on the page.  Javascript by PaulH will form a HTTP request using that hashtag to retrieve tweets from the tweet server and, based on the twitter timestamp, intertwine them with the topic replies. (80% complete)
This may seem like a complicated system.  Why not just draw tweets using clientside javascript during pageload?  Why use bespoke node.js servers instead of doing a Drupal implementation or buying cloud services?  Respectively, temporariness and efficiency.  With our setup, the twitter conversation isn't temporary anymore:  we can archive the conversation and guarantee that it lasts for future viewers (no need to rely on twitter to keep on serving tweets).  It's also more efficient: for one thing, it's only one stream request from twitter (maybe two if i want to do redundancy) instead of a bunch of in-browser requests, and the streaming API seems to be more timely and reliable.  Node.js lends itself very well to this type of system.

While the current implementation is definitely off-the-cuff hackish, I’m going to work on generalizing the code and will release it on github afterwards.  If anyone is interested in collaboration, please comment!

Benjamin Chodoroff

This is the blog of Benjamin Chodoroff. See my twitter or github or mastodon.