stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

Foundation and Earth

Sat, 22 Nov 2008 12:05:00 GMT

ISBN: 0586071105
LibraryThing

I really like how Asimov wraps up the extended Foundation series. Specifically, I'd previously complained while reading Pebble in the Sky that it was hard to believe that everyone simply forgets that they originated on Earth -- this book and Foundation's Edge go a long way to resolving that annoyance for me. Its also good to find out what happened to Aurora and Solaria finally -- especially given the Solaria mystery has been bothering me since Robots and Empire.

Speaking just about this book so a moment, I do find the use of sex as a plot development method quite odd. There are three examples that bother me -- when Bliss is slipped through interstellar customs with the explanation that she's just a whore and therefore not important enough to make an issue of; the second is when Trevize basically shags his way out of an awkward situation, despite the other protagonist being quite hostile initially; and finally where he bonks someone on a rural world. I find all three of those incidents a little out of place with the rest of the book, and in fact the rest of the series. Other authors use those kinds of plot elements, but they seem out of place in Asimov's work.

Overall, I loved this book and it was a good conclusion to the series.

Tags for this post: book() Isaac_Asimov()

Comment

Foundation's Edge

Mon, 17 Nov 2008 18:40:00 GMT

ISBN: 0586058397
LibraryThing

I'm back to reading Foundation Series books actually written by Isaac Asimov. This one is the fourth in the Foundation Series if you count them in the order they were written, but is the second last in chronological terms. Its set 500 years after the failure of the first galactic empire, and follows the first Foundation's attempt to discover if the second Foundation still exists. Well, its a bit more complicated than that, but I don't want to ruin it for you.

As an aside, the user interface described for the ship's computer is really cool. Its a bit like augmented reality, mixed with gesture control, mixed with a direct interface into the brain. I'm not saying I want one in my house, but its cool that a book written in 1983 still has a user interface description which isn't dated, and still seems plausible.

This book has minor inconsistencies with the story presented in the second foundation trilogy (Foundation's Fear, Foundation and Chaos and Foundation's Triumph), but I see that more as a failure in those followup authors than in this book. In fact, I've already complained about how untrue to Asimov's vision some of those books are elsewhere.

This is a good read, and I enjoyed it greatly.

Tags for this post: book() Isaac_Asimov()

Comment

Automatically creating folders for mailing lists

Mon, 17 Nov 2008 14:59:00 GMT

I've been using some simple procmail rules to automatically create folders for mailing lists for ages. Tony asked me for those rules today, so I figured I'd just put them online.

##########################################################################
# Mailman

:0:
* List-Id:.*<\/[^>]*
$MATCH

:0:
* List-Post: ]*
$MATCH

##########################################################################
# Majordomo lists (sometimes don't have <>'s around the address

:0:
* X-Mailing-List:.*<\/[^>]*
$MATCH

:0:
* X-Mailing-List:.*\/.*
$MATCH

##########################################################################
# Ezmlm

:0:
* Mailing-List: .* \/[^ ;]*
$MATCH

##########################################################################
# I'm not sure what creates this one...

:0:
* X-Loop: \/.*
$MATCH

Tags for this post: procmail()

Comment

Wanted: someone to edit / review some MythTV stuff for me

Sun, 16 Nov 2008 19:00:00 GMT

I'm looking for someone with solid MythTV experience and a good grasp of the English language to help me out with a project. All I can promise in return is glory, and that will be proportional to the eventual success of the project. If you're interested in spending some time (probably around 40 hours or so, spread over a couple of months) on such a project drop me a line.

Tags for this post: mythtv()

Comment

Andrew and Matthew turn 2 and 3

Sat, 15 Nov 2008 21:34:00 GMT

Wow, these pictures are really old. I'm in the process of going through my photo collection and regenerating all the associated HTML. In the process I found these pictures of Andrew and Matthew's shared birthday party from when they turned 3 and 2 respectively. Heck, its only three years late, so I guess I should put them online.

Looking back, I think I forgot to put these online because about that time I was run off my feet with LCA 2005. That's my excuse and I'm sticking to it.

See more thumbnails

Tags for this post: events() pictures() 20050314()

Comment

The Riftwar Series

Sat, 15 Nov 2008 14:39:00 GMT

This series follows to young kids who grow up in a rural castle on the edge of an empire. Pug ends up being the greatest Magician to ever live, and Thomas ends up being merged with an ancient being of massive power. Its a good series, even if people accuse it of being steeped in cliche.

Year	Title
1982	Magician (later sold as Magician: Apprentice and Magician: Master)
1985	Silverthorn
1986	A Darkness at Sethanon

Tags for this post: book() Raymond_E_Feist()

Comment

A Darkness at Sethanon

Sat, 15 Nov 2008 14:33:00 GMT

ISBN: 0553263285
LibraryThing

This book took longer to read that I would have liked, because I have been busy with other things. Its a good book though, and a fine conclusion to the Riftwar Series. I liked this book a lot, although I do think that Magician (Apprentice and Master) was a better book.

Tags for this post: book() Raymond_E_Feist()

Comment

On a memeomatic

Fri, 14 Nov 2008 22:04:00 GMT

I'm on vacation today, and so I had a bit more time that usual to just think. So, when Jeff posited a meme detector for planets, I wrote one. Except its of course never just that simple. My initial implementation only took a few minutes to write, but sucked.

What I did was I wrote a script which scanned through the list of posts from the planet's RSS feed, and kept a tally of which sequences of words (let's call them sentences, even though they're not) appear in which posts. Then, if a sentence appears in more than four posts, and those posts are from at least two domains, we've found a meme.

That's actually a reasonable algorithm. Its big advantage is that it only has to take one pass through the posts, which means its order is linear -- O(n). Now, the problem with that algorithm is that there a small differences in some of the sentences (for example people mistype a sentence), and I ended up finding too many copies of the same meme.

Here's some sample output from that version:

"the cool book or the intellectual one pick the closest": Chris Neugebauer: Meme #42, Thomas Karpiniec: The Book Thing, Jeremy Visser: Book meme, Michael Davies: Meme #42, James Purser: The Meme of the Book, Jon Oxer: Phrase from nearest book, Josh Stewart: random book content, Stewart Smith: phrase from nearest book
"find the fifth sentence post the text of the sentence": Donna Benjamin: meme-ege, Chris Neugebauer: Meme #42, Thomas Karpiniec: The Book Thing, Jeremy Visser: Book meme, Michael Davies: Meme #42, James Purser: The Meme of the Book, Peter Lieverdink: blog meme #42, Jon Oxer: Phrase from nearest book, Josh Stewart: random book content, Stewart Smith: phrase from nearest book
"journal along with these instructions dont dig for your favorite": Jeremy Visser: Book meme, Michael Davies: Meme #42, James Purser: The Meme of the Book, Peter Lieverdink: blog meme #42, Jon Oxer: Phrase from nearest book, Josh Stewart: random book content, Stewart Smith: phrase from nearest book
"grab the nearest book open it to page 56 find": Chris Neugebauer: Meme #42, Thomas Karpiniec: The Book Thing, Jeremy Visser: Book meme, Michael Davies: Meme #42, James Purser: The Meme of the Book, Peter Lieverdink: blog meme #42, Jon Oxer: Phrase from nearest book, Josh Stewart: random book content, Stewart Smith: phrase from nearest book mi

If you look at those you'll see that they're all the same meme, but the code found it three different ways. I need an algorithm which accurately finds the meme only once.

I should stop here and mention that I think this problem would be an excellent interview question. If you were going to ask the question in an interview you'd probably phrase it more as:

Given a list of strings, find substrings repeated between the strings, and return a list of the substrings and the strings containing them.

When the problem is phrased like that, I am sure that some folk think of an algorithm which compares each string with each other, looks for some sort of largest substring between the two, and then builds a table of those. However, the problem with that is that the order would be O(N^2), which is ok for a planet RSS feed, but wouldn't be so great if the set of strings you wanted to compare was something like every page on the Internet.

Anyway, I think its possible to rescue my initial implementation by providing a final pass which checks if matches overlap and combines them if they do. For example, if the only difference between two detected memes is one post, then they're probably the same meme and can be combined.

That's a interesting problem in itself. Its easy to measure the difference in the list of matching posts for two memes, but that comparison has O(N^2), which I just said was a bad thing. However, this is a vacation day and I couldn't think of anything better, so that's what I ended up using. I guess I'll wait for a smart interview candidate to think of a better way for me.

You can see output from memeomatic in this blather post for today. The blather code I wrote a while ago makes it really to post messages to my site, which is why I've reused it here (you just call a method on a python module, and a pre-existing Rube Goldberg machine takes care of the rest).

My code:

import feedparser
import os
import re
import shelve
import sys
import unicodedata
import urllib


_SENTENCE_LENGTH = 5


def Normalize(value):
  normalized = unicodedata.normalize('NFKD', unicode(value))
  normalized = normalized.encode('ascii', 'ignore')
  return normalized


def ListDifference(l1, l2):
  delta = []
  
  for l in l2:
    if l not in l1:
      delta.append(l)

  return len(delta)


plugins_dir = '%s/plugins' % os.getcwd()
print 'Appending %s to module path' % plugins_dir
sys.path.append(plugins_dir)
import blather

data = shelve.open('memes.slf', writeback=True)
data.setdefault('sentences', {})
data.setdefault('titles', {})
data.setdefault('content', {})
data.setdefault('content_orig', {})
data.setdefault('memes', [])

ds = blather.DataStore()

changed = False

# Scan feeds, looking for new posts. This just populates the database.
for feed in data['feeds']:
  print
  print 'Fetching %s' % feed
  d = feedparser.parse(feed)

  # Newest entries are first
  entries = d.entries
  entries.reverse()
  
  for ent in entries:
    print '  Considering %s' % ent.title
    data['titles'][ent.link] = ent.title
    
    content = Normalize(ent.description)
    data['content_orig'][ent.link] = content
    content = ' '.join(content.split('\n'))
    content = re.sub('<[^>]*>', '', content)
    content = re.sub('[^\w]+', ' ', content)
    content = content.lower()
    data['content'][ent.link] = content

    words = content.split()
    for i in range(len(words) - _SENTENCE_LENGTH):
      key = ' '.join(words[i:i + _SENTENCE_LENGTH])
      data['sentences'].setdefault(key, [])

      if not ent.link in data['sentences'][key]:
        data['sentences'][key].append(ent.link)

# Now we have a database of sentences and the posts which share them. What we
# really want is a collection of shared sentences that form a meme, and the
# posts which contain those sentences.
for sentence in data['sentences']:
  found = False

  if len(data['sentences'][sentence]) > 4:
    domains = {}
    
    # Its possible that they're all from one domain...
    for url in data['sentences'][key]:
      domain = url.strip('http://').split('/')[0]
      domains[domain] = True

    # Its not a meme unless the sentence is shared by at least four posts.
    # Try to find an existing meme which contains these posts.
    for (sentences, urls, published) in data['memes']:
      if not found and ListDifference(urls, data['sentences'][sentence]) < 2:
        data['memes'].remove((sentences, urls, published))

        if sentence not in sentences:
          sentences.append(sentence)

        new_titles = []
        for u in data['sentences'][sentence]:
          if not u in urls:
            urls.append(u)
            new_titles.append('<a href="%s">%s</a>'
                              %(u, data['titles'][u]))

        data['memes'].append((sentences, urls, published))
        found = True

        if published and new_titles:
          print 'Added posts to an existing meme'
          ds.AddMessage('Memeomatic extended an existing meme: %s'
                        % ', '.join(new_titles))
          changed = True

    if not found and len(domains) > 1:
      print ('Created a new meme for "%s" with %s'
             %(sentence, data['sentences'][sentence]))
      data['memes'].append(([sentence], data['sentences'][sentence], False))

# Publish new memes
for meme in data['memes']:
  (sentences, urls, published) = meme
  if not published:
    titles = []
    for url in urls:
      titles.append('<a href="%s">%s</a>' %(url, data['titles'][url]))
    
    ds.AddMessage('Memeomatic found a new meme: %s' % ', '.join(titles))
    data['memes'].remove((sentences, urls, published))
    data['memes'].append((sentences, urls, True))
    print 'Published a new meme'
    changed = True

if changed:
  ds.Save()
data.close()

So there you go. I haven't set this as a cron job yet, as I want to baby sit it for a while to make sure its doing the right thing. I might one day get around to trusting it enough to just turn it on.

Tags for this post: meme()

Comment

It seems planet is a bit too trusting with dates?

Fri, 14 Nov 2008 21:55:00 GMT

It seems that planet is a bit too trusting with dates. For example, if you have a post with a date well into the future, then you can keep that post at the top of the planet output until that date comes around. Its interesting that no one has used that maliciously yet.

You can see an example of what I'm talking about at Planet Linux Australia, where some forward dated posts sit at the top of the page...

Tags for this post: blog()

Comment

Book meme de jour

Fri, 14 Nov 2008 21:29:00 GMT

I don't normally get involved in this whole meme thing, but I want to test memeomatic some more. So, here goes...

Instructions:

Grab the nearest book.
Open it to page 56.
Find the fifth sentence.
Post the text of the sentence in your journal along with these instructions.
Don't dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

So, I'm currently reading A Darkness at Sethanon, which means its close to hand. The sentence is "They are correct as written, Commander."

Tags for this post: meme()

Comment

Bypassing Australia's imminent internet filter

Thu, 13 Nov 2008 21:25:00 GMT

Paul has thoughts on how to avoid Rudd's internet filter. I am left wondering why he doesn't just suggest Tor though. Its designed for exactly this sort of censorship, requires no account in another country, and is cross platform. The only catch is that Tor does block some traffic (for example bittorrent), so you can't just use it for all your traffic.

Tags for this post: blog()

Comment

Another dynamic element to the site

Thu, 13 Nov 2008 19:55:00 GMT

I got adventurous tonight, and whipped up some javascript which updates the sentence at the end of each post which lists how many comments there are on a post. This means that the site is always up to date, even though all the HTML is static files on disk. It also means I can finally kill that silly hourly regenerate cron job.

Oh, and this is post 3,000 on this site.

Tags for this post: site()

Comment

irtrans remote control stops working

Tue, 11 Nov 2008 19:04:00 GMT

I've been using a Zalman HD160B media center case for quite some time, and I love it. It came with an irtrans based LCD and remote control built into the case, which has always worked nicely as well. Until yesterday when the remote stopped working.

When I run the irserver command in a terminal so I can see the output, I get this:

# ./irserver -codedump -debug_code -loglevel 4 /dev/ttyUSB0
Name   : 
Version: D5.03.08
FW SNo : 18780
Capab  : Power On; 
FW Cap : 3964953
USB SNo: 
Node   : /dev/ttyUSB0

IRServer Version 5.7.08
[ 0]:                      D5.03.08     SN: 18780
Remote zalman               compiled:     1 Timings -     45 Commands
Total:   1 Remotes  -   1 Timings -  45 Commands
         0 CCF Data -   0 CCF RAW -   0 CCF Error

No joy. Rebooting the machine, replacing all the batteries, and restarting the server all did nothing. The server still doesn't see events from the remote. I'm not sure how to determine if this is a receiver hardware problem or not -- I'm kinda out of other ideas. Suggestions welcome.

Update: of course, disassembling the PC and reseating all the cables fixed the problem. I wonder if it is temperature related?

Tags for this post: mythtv()

Comment

Finding locking deadlocks in python

Tue, 11 Nov 2008 15:46:00 GMT

I re-factored some code today, and in the process managed to create a lock deadlock for myself. In the end it turned out to be an exception was being thrown when a lock was held, and adding a try / finally resolved the real underlying problem. However, in the process I ended up writing this little helper that I am sure will be useful in the future.

import gflags
import thread
import threading
import traceback
import logging

...

FLAGS = gflags.FLAGS
gflags.DEFINE_boolean('dumplocks', False,
                      'If true, dumps information about lock activity')
...

class LockHelper(object):
  """A wrapper which makes it easier to see what locks are doing."""

  lock = thread.allocate_lock()

  def acquire(self):
    if FLAGS.dumplocks:
      logging.info('%s acquiring lock' % threading.currentThread().getName())
      for s in traceback.extract_stack():
        logging.info('  Trace %s:%s [%s] %s' % s)
    self.lock.acquire()

  def release(self):
    if FLAGS.dumplocks:
      logging.info('%s releasing lock' % threading.currentThread().getName())
      for s in traceback.extract_stack():
        logging.info('  Trace %s:%s [%s] %s' % s)
    self.lock.release()

Now I can just use this helper in the place of thread.allocate_lock() when I want to see what is happening with locking. It saved me a lot of staring at random code today.

Tags for this post: python()

Comment

Slight site hickup

Sun, 09 Nov 2008 18:52:00 GMT

I pushed a bunch of photo files to stillhq.com tonight, all of which I thought were previously published. Unfortunately a bunch of them had the wrong dates associated with them, which spammed the front page and RSS feed with old photo posts. The problem has been corrected now, and hopefully not too many people grabbed the bogus RSS feeds. Sorry for any inconvenience.

Tags for this post: site()

Comment

Bill the Galactic Hero on the Planet of Bottled Brains

Sun, 09 Nov 2008 13:39:00 GMT

ISBN: 0380756625
LibraryThing

(Edited by Harry Harrison.) This book started off better than Planet of the Robot Slaves, but I found the random Star Trek and Star Wars rip offs in the middle of the book intensely annoying. This isn't so much a satire it is a disjointed list of ideas. I'm disappointed to be honest.

Tags for this post: book() Robert_Sheckley()

Comment

James and Jeanette's wedding down the beach

Sun, 09 Nov 2008 11:45:00 GMT

I've been uber slack about putting pictures of James and Jeanette's wedding online. Given their wedding was ages ago, and the reception last weekend, it seems like I am running out of excuses.

See more thumbnails

Tags for this post: events() pictures() 20080531-sjrwedding()

Comment

Foundation's Triumph

Fri, 07 Nov 2008 21:04:00 GMT

ISBN: 0061056391
LibraryThing

This book is pretty good. I'd say its the best of the three Second Foundation Trilogy books in fact. Unfortunately, you need to read the other two in order for this one to make any sense, which is a shame because the first one sucked, and the second one was ok.

A lot of loose ends get cleaned up in this book. Why did Earth get abandoned? Why did everyone forget their history? Why is Trantor built much like the cities in the Naked Sun? Why are there all those habitable worlds for the galactic empire to reside on? It seems odd that there would be 25 million habitable worlds out there. There are other examples as well, but I wont bore you with them all.

Another good bit of this book is the time line of all Asimov Foundation stories at the back of the book. I am sure it would have been useful to know about that earlier.

Tags for this post: book() David_Brin()

Comment

Belgarath the Sorcerer

Tue, 04 Nov 2008 22:52:00 GMT

ISBN: 0345403959
LibraryThing

This book is a follow on from The Belgariad and The Malloreon written from the perspective of one of the protagonists in those two series -- Belgarath the Sorcerer. This book has quite a different style from the others. It is written in the first person as Belgarath's personal memoir. The book is really long at over 700 pages, and covers a 6,000 year (ish) period. That means that the book tends to skip around and over some things at a very high level. That's probably a good thing, as it stops you from getting bogged down in boring detail you don't care about.

One problem with this book is that to people who have read the previous series, this is all old territory. You totally know what is about to happen, and that makes it a lot less fun to read. I'm also not sure I'm a big fan of the first person style either. However, I did enjoy this book more than some of the others I have read recently, despite it not being Eddings' best work.

Tags for this post: book() David_Eddings()

Comment

Adium crashes on launch

Mon, 03 Nov 2008 17:16:00 GMT

Dear Mac Nerds,

Adium upgraded itself this morning, and now crashes on launch. This still happens after rebooting the machine. I'm a MacOS n00b, and have no idea how to debug what is happening. Please be with the helping?

Tags for this post: macintosh() osx()

Comment