stillhq.com : Mikal, a geek from Canberra living in Silicon Valley

Foundation and Earth

Sat, 22 Nov 2008 12:05:00 PST

ISBN: 0586071105
LibraryThing

I really like how Asimov wraps up the extended Foundation series. Specifically, I'd previously complained while reading Pebble in the Sky that it was hard to believe that everyone simply forgets that they originated on Earth -- this book and Foundation's Edge go a long way to resolving that annoyance for me. Its also good to find out what happened to Aurora and Solaria finally -- especially given the Solaria mystery has been bothering me since Robots and Empire.

Speaking just about this book so a moment, I do find the use of sex as a plot development method quite odd. There are three examples that bother me -- when Bliss is slipped through interstellar customs with the explanation that she's just a whore and therefore not important enough to make an issue of; the second is when Trevize basically shags his way out of an awkward situation, despite the other protagonist being quite hostile initially; and finally where he bonks someone on a rural world. I find all three of those incidents a little out of place with the rest of the book, and in fact the rest of the series. Other authors use those kinds of plot elements, but they seem out of place in Asimov's work.

Overall, I loved this book and it was a good conclusion to the series.

Tags for this post: book() Isaac_Asimov()

Comment

Blathering for Friday, 21 November 2008

Fri, 21 Nov 2008 15:00:00 GMT

09:45: Mikal shared: Buy One Dodge Ram, Get One Free [Deals]

You know the US auto industry is in trouble when they start offering buy one get one free deals on cars.

15:00: Mikal shared: Article about backyard chicken owners

I didn't realize that other people found chickens entertaining too. I figured it was just me. There is nothing more entertaining than throwing a mound of kitchen scraps into the coup and then watching the chickens argue over a banana peel. Its hard to explain... Perhaps when I move back to Australia I'll setup ChickenCam.

Tags for this post: blather() Comment RSS with no blather

Blathering for Wednesday, 19 November 2008

Wed, 19 Nov 2008 22:00:00 GMT

09:15: Mikal shared: Are You Spending $145 A Year Just To Leave Your PS3 On All The Time? [Frugality]

Wow. A PS3 left on permanently will cost $145 USD a year in electricity. I knew there was a reason I was turning that thing off.

22:00: Mikal shared: 15 Minute Lunch: Strap in, shut up and hold on. We're going back.

I want that jumpsuit. 1977 fashion was awesome.

Tags for this post: blather() Comment RSS with no blather

Foundation's Edge

Mon, 17 Nov 2008 18:40:00 PST

ISBN: 0586058397
LibraryThing

I'm back to reading Foundation Series books actually written by Isaac Asimov. This one is the fourth in the Foundation Series if you count them in the order they were written, but is the second last in chronological terms. Its set 500 years after the failure of the first galactic empire, and follows the first Foundation's attempt to discover if the second Foundation still exists. Well, its a bit more complicated than that, but I don't want to ruin it for you.

As an aside, the user interface described for the ship's computer is really cool. Its a bit like augmented reality, mixed with gesture control, mixed with a direct interface into the brain. I'm not saying I want one in my house, but its cool that a book written in 1983 still has a user interface description which isn't dated, and still seems plausible.

This book has minor inconsistencies with the story presented in the second foundation trilogy (Foundation's Fear, Foundation and Chaos and Foundation's Triumph), but I see that more as a failure in those followup authors than in this book. In fact, I've already complained about how untrue to Asimov's vision some of those books are elsewhere.

This is a good read, and I enjoyed it greatly.

Tags for this post: book() Isaac_Asimov()

Comment

Blathering for Monday, 17 November 2008

Mon, 17 Nov 2008 16:15:00 GMT

16:15: Mikal shared: 20% Of Citigroup Cardholders Can Expect Rate Increases For 2009 [Citibank]

Here comes the credit card crunch. Its interesting that Citi isn't increasing the interest rate on the riskiest 20% of card, just a semi-random 20%. That doesn't make sense to me at all.

Tags for this post: blather() Comment RSS with no blather

Automatically creating folders for mailing lists

Mon, 17 Nov 2008 14:59:00 PST

I've been using some simple procmail rules to automatically create folders for mailing lists for ages. Tony asked me for those rules today, so I figured I'd just put them online.

##########################################################################
# Mailman

:0:
* List-Id:.*<\/[^>]*
$MATCH

:0:
* List-Post: ]*
$MATCH

##########################################################################
# Majordomo lists (sometimes don't have <>'s around the address

:0:
* X-Mailing-List:.*<\/[^>]*
$MATCH

:0:
* X-Mailing-List:.*\/.*
$MATCH

##########################################################################
# Ezmlm

:0:
* Mailing-List: .* \/[^ ;]*
$MATCH

##########################################################################
# I'm not sure what creates this one...

:0:
* X-Loop: \/.*
$MATCH

Tags for this post: procmail() Comment

Wanted: someone to edit / review some MythTV stuff for me

Sun, 16 Nov 2008 19:00:00 PST

I'm looking for someone with solid MythTV experience and a good grasp of the English language to help me out with a project. All I can promise in return is glory, and that will be proportional to the eventual success of the project. If you're interested in spending some time (probably around 40 hours or so, spread over a couple of months) on such a project drop me a line.

Tags for this post: mythtv() Comment

Andrew and Matthew turn 2 and 3

Sat, 15 Nov 2008 21:34:00 PST

Wow, these pictures are really old. I'm in the process of going through my photo collection and regenerating all the associated HTML. In the process I found these pictures of Andrew and Matthew's shared birthday party from when they turned 3 and 2 respectively. Heck, its only three years late, so I guess I should put them online.

Looking back, I think I forgot to put these online because about that time I was run off my feet with LCA 2005. That's my excuse and I'm sticking to it.

See more thumbnails

Tags for this post: events() pictures() 20050314() Comment

Blathering for Saturday, 15 November 2008

Sat, 15 Nov 2008 19:30:00 GMT

19:30: Mikal shared: Territory and Municipal Services - Green Vehicles Stamp Duty Scheme

Interesting. Transferring registration of a vehicle in Australia incurs a tax called "stamp duty". In the ACT, the amount you pay is now dependent on how environmentally friendly the car is. That's a good idea.

19:30: Mikal shared: Say Goodbye to BlackBerry? If Obama Has to, Yes He Can - NYTimes.com

Huh. The Presidential Records Act might require Obama to give up standard business tools like smart phones and email. I guess we'll allow him a set of crayolas?

Tags for this post: blather() Comment RSS with no blather

The Riftwar Series

Sat, 15 Nov 2008 14:39:00 PST

This series follows to young kids who grow up in a rural castle on the edge of an empire. Pug ends up being the greatest Magician to ever live, and Thomas ends up being merged with an ancient being of massive power. Its a good series, even if people accuse it of being steeped in cliche.

Year	Title
1982	Magician (later sold as Magician: Apprentice and Magician: Master)
1985	Silverthorn
1986	A Darkness at Sethanon

Tags for this post: book() Raymond_E_Feist() Comment

A Darkness at Sethanon

Sat, 15 Nov 2008 14:33:00 PST

ISBN: 0553263285
LibraryThing

This book took longer to read that I would have liked, because I have been busy with other things. Its a good book though, and a fine conclusion to the Riftwar Series. I liked this book a lot, although I do think that Magician (Apprentice and Master) was a better book.

Tags for this post: book() Raymond_E_Feist()

Comment

Blathering for Friday, 14 November 2008

Fri, 14 Nov 2008 22:29:00 GMT

20:53: Memeomatic found a new meme: Stewart Smith: phrase from nearest book, Josh Stewart: random book content, Jon Oxer: Phrase from nearest book, Peter Lieverdink: blog meme #42, James Purser: The Meme of the Book, Michael Davies: Meme #42, Jeremy Visser: Book meme, Thomas Karpiniec: The Book Thing, Chris Neugebauer: Meme #42, Donna Benjamin: meme-ege, Simon Lyall: Economist plus meme
21:14: Memeomatic extended an existing meme: Chris Samuel: Book Meme
21:41: Memeomatic extended an existing meme: Book meme de jour
22:05: Memeomatic extended an existing meme: On a memeomatic
22:21: Memeomatic extended an existing meme: Kees Cook: phrase from nearest book meme
22:21: Memeomatic extended an existing meme: Mike Hommey: Book meme reloaded
22:29: Memeomatic extended an existing meme: Jacob Peddicord: Open Source Bridge

Tags for this post: blather() Comment RSS with no blather

On a memeomatic

Fri, 14 Nov 2008 22:04:00 PST

I'm on vacation today, and so I had a bit more time that usual to just think. So, when Jeff posited a meme detector for planets, I wrote one. Except its of course never just that simple. My initial implementation only took a few minutes to write, but sucked.

What I did was I wrote a script which scanned through the list of posts from the planet's RSS feed, and kept a tally of which sequences of words (let's call them sentences, even though they're not) appear in which posts. Then, if a sentence appears in more than four posts, and those posts are from at least two domains, we've found a meme.

That's actually a reasonable algorithm. Its big advantage is that it only has to take one pass through the posts, which means its order is linear -- O(n). Now, the problem with that algorithm is that there a small differences in some of the sentences (for example people mistype a sentence), and I ended up finding too many copies of the same meme.

Here's some sample output from that version:

"the cool book or the intellectual one pick the closest": Chris Neugebauer: Meme #42, Thomas Karpiniec: The Book Thing, Jeremy Visser: Book meme, Michael Davies: Meme #42, James Purser: The Meme of the Book, Jon Oxer: Phrase from nearest book, Josh Stewart: random book content, Stewart Smith: phrase from nearest book
"find the fifth sentence post the text of the sentence": Donna Benjamin: meme-ege, Chris Neugebauer: Meme #42, Thomas Karpiniec: The Book Thing, Jeremy Visser: Book meme, Michael Davies: Meme #42, James Purser: The Meme of the Book, Peter Lieverdink: blog meme #42, Jon Oxer: Phrase from nearest book, Josh Stewart: random book content, Stewart Smith: phrase from nearest book
"journal along with these instructions dont dig for your favorite": Jeremy Visser: Book meme, Michael Davies: Meme #42, James Purser: The Meme of the Book, Peter Lieverdink: blog meme #42, Jon Oxer: Phrase from nearest book, Josh Stewart: random book content, Stewart Smith: phrase from nearest book
"grab the nearest book open it to page 56 find": Chris Neugebauer: Meme #42, Thomas Karpiniec: The Book Thing, Jeremy Visser: Book meme, Michael Davies: Meme #42, James Purser: The Meme of the Book, Peter Lieverdink: blog meme #42, Jon Oxer: Phrase from nearest book, Josh Stewart: random book content, Stewart Smith: phrase from nearest book mi

If you look at those you'll see that they're all the same meme, but the code found it three different ways. I need an algorithm which accurately finds the meme only once.

I should stop here and mention that I think this problem would be an excellent interview question. If you were going to ask the question in an interview you'd probably phrase it more as:

Given a list of strings, find substrings repeated between the strings, and return a list of the substrings and the strings containing them.

When the problem is phrased like that, I am sure that some folk think of an algorithm which compares each string with each other, looks for some sort of largest substring between the two, and then builds a table of those. However, the problem with that is that the order would be O(N^2), which is ok for a planet RSS feed, but wouldn't be so great if the set of strings you wanted to compare was something like every page on the Internet.

Anyway, I think its possible to rescue my initial implementation by providing a final pass which checks if matches overlap and combines them if they do. For example, if the only difference between two detected memes is one post, then they're probably the same meme and can be combined.

That's a interesting problem in itself. Its easy to measure the difference in the list of matching posts for two memes, but that comparison has O(N^2), which I just said was a bad thing. However, this is a vacation day and I couldn't think of anything better, so that's what I ended up using. I guess I'll wait for a smart interview candidate to think of a better way for me.

You can see output from memeomatic in this blather post for today. The blather code I wrote a while ago makes it really to post messages to my site, which is why I've reused it here (you just call a method on a python module, and a pre-existing Rube Goldberg machine takes care of the rest).

My code:

import feedparser
import os
import re
import shelve
import sys
import unicodedata
import urllib


_SENTENCE_LENGTH = 5


def Normalize(value):
  normalized = unicodedata.normalize('NFKD', unicode(value))
  normalized = normalized.encode('ascii', 'ignore')
  return normalized


def ListDifference(l1, l2):
  delta = []
  
  for l in l2:
    if l not in l1:
      delta.append(l)

  return len(delta)


plugins_dir = '%s/plugins' % os.getcwd()
print 'Appending %s to module path' % plugins_dir
sys.path.append(plugins_dir)
import blather

data = shelve.open('memes.slf', writeback=True)
data.setdefault('sentences', {})
data.setdefault('titles', {})
data.setdefault('content', {})
data.setdefault('content_orig', {})
data.setdefault('memes', [])

ds = blather.DataStore()

changed = False

# Scan feeds, looking for new posts. This just populates the database.
for feed in data['feeds']:
  print
  print 'Fetching %s' % feed
  d = feedparser.parse(feed)

  # Newest entries are first
  entries = d.entries
  entries.reverse()
  
  for ent in entries:
    print '  Considering %s' % ent.title
    data['titles'][ent.link] = ent.title
    
    content = Normalize(ent.description)
    data['content_orig'][ent.link] = content
    content = ' '.join(content.split('\n'))
    content = re.sub('<[^>]*>', '', content)
    content = re.sub('[^\w]+', ' ', content)
    content = content.lower()
    data['content'][ent.link] = content

    words = content.split()
    for i in range(len(words) - _SENTENCE_LENGTH):
      key = ' '.join(words[i:i + _SENTENCE_LENGTH])
      data['sentences'].setdefault(key, [])

      if not ent.link in data['sentences'][key]:
        data['sentences'][key].append(ent.link)

# Now we have a database of sentences and the posts which share them. What we
# really want is a collection of shared sentences that form a meme, and the
# posts which contain those sentences.
for sentence in data['sentences']:
  found = False

  if len(data['sentences'][sentence]) > 4:
    domains = {}
    
    # Its possible that they're all from one domain...
    for url in data['sentences'][key]:
      domain = url.strip('http://').split('/')[0]
      domains[domain] = True

    # Its not a meme unless the sentence is shared by at least four posts.
    # Try to find an existing meme which contains these posts.
    for (sentences, urls, published) in data['memes']:
      if not found and ListDifference(urls, data['sentences'][sentence]) < 2:
        data['memes'].remove((sentences, urls, published))

        if sentence not in sentences:
          sentences.append(sentence)

        new_titles = []
        for u in data['sentences'][sentence]:
          if not u in urls:
            urls.append(u)
            new_titles.append('<a href="%s">%s</a>'
                              %(u, data['titles'][u]))

        data['memes'].append((sentences, urls, published))
        found = True

        if published and new_titles:
          print 'Added posts to an existing meme'
          ds.AddMessage('Memeomatic extended an existing meme: %s'
                        % ', '.join(new_titles))
          changed = True

    if not found and len(domains) > 1:
      print ('Created a new meme for "%s" with %s'
             %(sentence, data['sentences'][sentence]))
      data['memes'].append(([sentence], data['sentences'][sentence], False))

# Publish new memes
for meme in data['memes']:
  (sentences, urls, published) = meme
  if not published:
    titles = []
    for url in urls:
      titles.append('<a href="%s">%s</a>' %(url, data['titles'][url]))
    
    ds.AddMessage('Memeomatic found a new meme: %s' % ', '.join(titles))
    data['memes'].remove((sentences, urls, published))
    data['memes'].append((sentences, urls, True))
    print 'Published a new meme'
    changed = True

if changed:
  ds.Save()
data.close()

So there you go. I haven't set this as a cron job yet, as I want to baby sit it for a while to make sure its doing the right thing. I might one day get around to trusting it enough to just turn it on.

Tags for this post: meme() Comment

It seems planet is a bit too trusting with dates?

Fri, 14 Nov 2008 21:55:00 PST

It seems that planet is a bit too trusting with dates. For example, if you have a post with a date well into the future, then you can keep that post at the top of the planet output until that date comes around. Its interesting that no one has used that maliciously yet.

You can see an example of what I'm talking about at Planet Linux Australia, where some forward dated posts sit at the top of the page...

Tags for this post: blog() Comment

Book meme de jour

Fri, 14 Nov 2008 21:29:00 PST

I don't normally get involved in this whole meme thing, but I want to test memeomatic some more. So, here goes...

Instructions:

Grab the nearest book.
Open it to page 56.
Find the fifth sentence.
Post the text of the sentence in your journal along with these instructions.
Don't dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

So, I'm currently reading A Darkness at Sethanon, which means its close to hand. The sentence is "They are correct as written, Commander."

Tags for this post: meme() Comment

Bypassing Australia's imminent internet filter

Thu, 13 Nov 2008 21:25:00 PST

Paul has thoughts on how to avoid Rudd's internet filter. I am left wondering why he doesn't just suggest Tor though. Its designed for exactly this sort of censorship, requires no account in another country, and is cross platform. The only catch is that Tor does block some traffic (for example bittorrent), so you can't just use it for all your traffic.

Tags for this post: blog() Comment

Another dynamic element to the site

Thu, 13 Nov 2008 19:55:00 PST

I got adventurous tonight, and whipped up some javascript which updates the sentence at the end of each post which lists how many comments there are on a post. This means that the site is always up to date, even though all the HTML is static files on disk. It also means I can finally kill that silly hourly regenerate cron job.

Oh, and this is post 3,000 on this site.

Tags for this post: site() Comment

Blathering for Thursday, 13 November 2008

Thu, 13 Nov 2008 12:45:00 GMT

09:15: Mikal shared: Banks Want To Forgive Credit Card Debt -- But The Government Says No [Money

Hmmm, here comes the credit card crisis. I don't know how I feel about forgiving credit card debt. As someone who has used his cards responsibly (and therefore wont have any debt forgiven) it seems unfair. Then again, I guess I benefit from a smaller recession.

12:45: Mikal shared: The Doppler Effect: A Knee-Slappin' Good Time

So, Hodgman came to talk at Google. These pictures taken by a coworker are awesome.

Tags for this post: blather() Comment RSS with no blather

irtrans remote control stops working

Tue, 11 Nov 2008 19:04:00 PST

I've been using a Zalman HD160B media center case for quite some time, and I love it. It came with an irtrans based LCD and remote control built into the case, which has always worked nicely as well. Until yesterday when the remote stopped working.

When I run the irserver command in a terminal so I can see the output, I get this:

# ./irserver -codedump -debug_code -loglevel 4 /dev/ttyUSB0
Name   : 
Version: D5.03.08
FW SNo : 18780
Capab  : Power On; 
FW Cap : 3964953
USB SNo: 
Node   : /dev/ttyUSB0

IRServer Version 5.7.08
[ 0]:                      D5.03.08     SN: 18780
Remote zalman               compiled:     1 Timings -     45 Commands
Total:   1 Remotes  -   1 Timings -  45 Commands
         0 CCF Data -   0 CCF RAW -   0 CCF Error

No joy. Rebooting the machine, replacing all the batteries, and restarting the server all did nothing. The server still doesn't see events from the remote. I'm not sure how to determine if this is a receiver hardware problem or not -- I'm kinda out of other ideas. Suggestions welcome.

Update: of course, disassembling the PC and reseating all the cables fixed the problem. I wonder if it is temperature related?

Tags for this post: mythtv() Comment

Finding locking deadlocks in python

Tue, 11 Nov 2008 15:46:00 PST

I re-factored some code today, and in the process managed to create a lock deadlock for myself. In the end it turned out to be an exception was being thrown when a lock was held, and adding a try / finally resolved the real underlying problem. However, in the process I ended up writing this little helper that I am sure will be useful in the future.

import gflags
import thread
import threading
import traceback
import logging

...

FLAGS = gflags.FLAGS
gflags.DEFINE_boolean('dumplocks', False,
                      'If true, dumps information about lock activity')
...

class LockHelper(object):
  """A wrapper which makes it easier to see what locks are doing."""

  lock = thread.allocate_lock()

  def acquire(self):
    if FLAGS.dumplocks:
      logging.info('%s acquiring lock' % threading.currentThread().getName())
      for s in traceback.extract_stack():
        logging.info('  Trace %s:%s [%s] %s' % s)
    self.lock.acquire()

  def release(self):
    if FLAGS.dumplocks:
      logging.info('%s releasing lock' % threading.currentThread().getName())
      for s in traceback.extract_stack():
        logging.info('  Trace %s:%s [%s] %s' % s)
    self.lock.release()

Now I can just use this helper in the place of thread.allocate_lock() when I want to see what is happening with locking. It saved me a lot of staring at random code today.

Tags for this post: python() Comment