Rockband – with Computer Vision


Rockband Vision from njoubert on Vimeo.

http://www.vimeo.com/1045326

We built this computer vision system that can play Rockband as our final project for CS184 at UC Berkeley.

In the space of two weekends we designed and built a system that uses computer vision to monitor the xbox display through a camera and play the game. More details can be found at inst.eecs.berkeley.edu/~njoubert/cs184/Rockband.pdf

We were, well, sleep deprived for a good section of this work, which explains the craziness in some parts of the video.

Our system is similar to Slashbot and AutoGuitarHero, but we do not take a video feed from the console – no, we’re doing it through a panasonic handycam pointed at the screen. We’re interested in machine vision, and this was a fun project to get into the field!

Building OpenCV in Ubuntu 8.04

I’m using OpenCV for my current computer graphics project – hacking Harmonix’ Rock Band – so naturally I have to build it from source in Ubuntu. I downloaded the source from Sourceforge.

The procedure was fairly simple – the most important part was the packages needed to satisfy all the requirements. OpenCV depends on several other libraries to really get the full potential of our system (although simple installs are possible).

Since I wanted to do image input/output I apt-get’ted the following packages:

  • libpng-dev
  • libjpeg-dev

To do ffmpeg development – which is the library OpenCV uses for video campture:

  • libavcodec-dev – development files for libavcodec
  • libavformat-dev – development files for libavformat
  • libavutil-dev – development files for libavutil
  • libpostproc-dev – development files for libpostproc
  • libswscale-dev – development files for libswscale
  • libdlna-dev – development files for libdlna
  • libmpeg4ip-dev – end-to-end system to explore streaming multimedia

For all the funky GUI development:

  • libgtk2.0-0
  • libgtk2.0-dev

You can install all of this using audo apt-get install

Once this is done, I unpacked the TAR file, cd’d to the directory and ran the good old standard set of building commands:
<br />sudo ./configure<br />sudo make<br />sudo make install<br />

That’s it!

CHI 2008: Yay we made it in!

The CHI 2008 conference is winding down today, and I’m still excited that our Work in Progress paper got accepted to the conference!

CHI is arguably the biggest conference HCI/Design conference around – from their website: “CHI 2008 focuses on the balance between art and science, design and research, practical motivation and the process that leads the way to innovative excellence.”

Our paper was titled “Enhancing online personal connections through the synchronized sharing of online video” and came from the work that Ayman, myself, Marcello and Yiding did at Yahoo Research Berkeley during 2007. Some of our prototypes are making it into http://live.yahoo.com/ that allows for synchronous sharing of video. Also, our Yahoo messenger plugin Zync is not officially integrated into Yahoo Messenger – just click the “Watch with me” button when dropping in a video, and you get to watch video synchronously with the person you’re chatting with. Cool stuff!

DSLExtreme – my choice as ISP

“At last!” was the words I uttered after finally ending my Comcast contract. Not long after, my DSLExtreme modem (free with every contract) arrived by UPS, and I plugged it in with much anticipation. I called up their technical support since I was not getting an IP from their servers, and within 3 minutes from picking up the phone, I had an IP on my modem and I was surfing away. (Apparently you have to manually request dynamic IPs from them, so it was partially my fault that I was not getting an IP – although the process is a bit strange!)

Anyways, the average latency for web pages is half that of Comcast, and the download stats looks great too:

Unfortunately, you can’t compare this with any Comcast connection test, since their new “SpeedBurst” technology completely skews the result of any bandwidth test – they blast the first 10MB of data to you at high speeds, making these tests give you a very different result of what your long-running throughput will be.

Thus, my vote goes to DSLExtreme! They’re great. http://www.dslextreme.com/

The Tree Sitters of Berkeley

If you walked along Piedmont at the top of UC Berkeley’s campus sometime during the last months you would encounter an interesting sight – several people sitting high up in the oaks surrounding Memorial Stadium. These days they camp outside the fence erected to keep them out. What do these tree dwellers want? They are, according to their campaign boards, saving the trees. Since the University wants to upgrade memorial stadium by building a new complex where these trees are currently growing.

I’ve been fairly neutral in the situation – yes, i like trees, and berkeley happens to have lots of ‘em – but on the other hand i dont really see why those 38 oaks are the most important trees to fight for. I’ve been strangely pursued by a letter posted on craiglist, that made sme extremely valid points. Read it here:

http://www.craigslist.org/about/best/sfo/466281874.html

The main point that the person makes, which I have come to support wholeheartedly, is that those 38 oaks are not endangered and are completely negligible against the deforestation happening in the Amazon. The man-power of these protesters and the police that are forced to check up on them are much better spent in places where it would actually make a difference!

How to get your roommates to clean up the apartment, or, writing SMS Gateway Apps

Update: After contacting the guys from Clickatell they sorted out my billing issues and were friendly enough to throw in some free credits. Very nice of them, thanks guys!

I’m always a big fan of interfacing with as many possible data inputs and outputs when i’m working on projects. So much so, that sometimes I do a project purely to plan with novel ways to interact with people. After being introduces to the CEO of Clickatell, a international messaging company that runs a very nifty SMS (Text Message) API, I figured I’d better build something with it.

Two interesting problems came up almost simultaneously that I attacked with a little text messaging scripts in both cases – server monitoring and, well, roommate monitoring!

As one of the development members on 2draw.net, an online art community started by Marcello Bastae-Forte, I’ve been dealing with keeping the server alive and running. Since we’re a team of only 4 at the moment, there is the distinct possibility that the server can go down, and without adequate monitoring utilities, none of us will realize this for a significant amount of time. Thus, what could be better than a text message crying for help if something goes wrong! In the same gist, I would like to remind my roommates (and myself) to go take out the trash and run the dishwasher according to our very intricate schedule we posted on the fridge. Thus, I registered a Clickatell account to start playing with text gateways.

Clickatell gave me a great initial impression. Their website is clean and informative, and their API is great to use. Once you’ve registered, you request an application ID and corresponding secret ID. Using these numbers for authentication you can send batch message jobs easily and securely. Most attractive to my problem at hand was their, as I call it, “one request, one message” API call. With one HTTP POST I can send all the necessary information to get a text message sent to multiple phone numbers. This allowed me to write a total of 12 lines of Ruby to wrap their API in a simple Ruby object.

Unfortunately, I hit quite a snag in trying to procure some text message credits from them. For some strange reason, I kept getting a message saying “Required Field” next to the State form field on their billing page. I did, quite naturally, fill out the form multiple times and made sure to fill out every field I could find, but to no avail. Their customer service response was prompt, but unfortunately did not help to solve my problem. Being slightly confused about how to continue using their service, I shopped around to see what else is out there. (I did manage to fix this issue, so keep reading!) I gave Ez Texting, UpsideWireless and CellTrust a try to compare. Unfortunately, each of these services has caveats that, if I had some credit on Clickatell, I would not have bothered to register. Since I did not, I pressed on and gave them all a try.

Ez Texting is, as their name suggests, super easy. Register, and you get 10 messages to use. They have the cheapest rates going (over half the price of Clickatell – $12 for 500 credits versus $12 for 200 credits on Clickatell), and their site indicates this strongly with a design that doesn’t exactly spell professional. I was here for the API, though, and it was a big disappointment when I was notified that I needed special permission to use the API, and to please send an email to their customer support group. Seriously, I don’t want to waste the time waiting for them. I did send them an email though, but have yet to hear back. They do have the same feature as Clickatell to send a text message with a single HTTP call, but since they do not support HTTPS your password will be floating along the lines in cleartext. Need I say more about security. Their messages also contain “Subject: ” and “Message: ” parts that you cannot get rid of, which looks stupid and annoyed me to no end. As an easy to use, online text center, they work well though.

UpsideWireless and CellTrust both looked like strong competitors to Clickatell, and probably are in the business world. Neither catered to the hackers out there, though, and to buy credit or get access to the API you had to get quotes from their sales department and access rights to their APIs. (I’m generalizing now, and each has their own process, but lets get to the code shall we?)

By now, 2 days later, I hoped that whatever bug was in Clickatell’s billing script would be sorted out by now. Alas, it wasn’t, but in filling out all my details on both the Administrative and Billing profile sections, and filling out their billing page that forms part of the buying process, I made it to the checkout screen. I was taken aback that they required a phone call to verify my credit card (for a total of $13), but they called within 10 minutes after filling out the form, so I did not complain too loudly! With such a prompt response, I applaud them for taking the security measures.

Once I had my credits and my ruby code, I could finally start doing some cool stuff.

To wrap the Clickatell API, I put together a Notify object and, with some Monkey-patching to the URI class, I was all set:

<br />require 'net/http'<br />require 'uri'<br />require 'cgi'<br /><br />#Slight modification to the URI class:<br />class URI::Generic<br /> #appends more data to the current query.<br /> def append_query(param, arg)<br /> if self.query<br /> self.query += "&#{param.to_s}=#{arg.to_s}"<br /> else<br /> self.query = "#{param.to_s}=#{arg.to_s}"<br /> end<br /> return self<br /> end<br />end<br /> <br />class Notifier<br />public<br /> #Takes a YAML object that contains the <br /> #hashes "phones" and "emails" to notify people<br /> def initialize(yaml_conf)<br /> @yaml_conf = yaml_conf<br /> end<br /> <br /> #see https://www.clickatell.com/downloads/http/Clickatell_HTTP.pdf<br /> def clickatell_text(message, phonenrs)<br /> if (message == false)<br /> message = "_hmm, nothing's up. _is the script buggy?"<br /> end<br /> url = URI.parse(@yaml_conf["clickatell"]["send_uri"].to_s)<br /> url.append_query("api_id", CGI::escape(@yaml_conf["clickatell"]["api_id"].to_s))<br /> url.append_query("user", CGI::escape(@yaml_conf["clickatell"]["user"].to_s))<br /> url.append_query("password", CGI::escape(@yaml_conf["clickatell"]["password"].to_s))<br /> nrs = ""<br /> phonenrs.each do |value| <br /> nrs += value.to_s + "," <br /> end<br /> nrs[-1] = ''<br /> url.append_query("to", CGI::escape(nrs.to_s))<br /> url.append_query("text", CGI::escape(message))<br /> response = Net::HTTP.get_response(url)<br /> puts " code = #{response.code}"<br /> puts " message = #{response.message}"<br /> puts " response = #{response.body}"<br /> return (response.code.to_i == 200 && response.body.to_s.include?("id: "))<br /> end<br />end<br />

I tend to use YAML to store my config files, which is exactly what I did in this case. My config.yml file looked something like this:

<br />clickatell:<br />&nbsp&nbsp user: user<br />&nbsp&nbsp password: pass<br />&nbsp&nbsp api_id: appid<br />&nbsp&nbsp send_uri: http://api.clickatell.com/http/sendmsg?<br /> <br />phones:<br />&nbsp&nbsp person1: 19999999999<br />&nbsp&nbsp person2: 19999999999<br />

The script to remind my roommates and I of our daily duties was a simple couple of lines on top of this:

<br />require 'notifier.rb'<br />require 'yaml'<br /><br />$LOAD_PATH.unshift(File.dirname(__FILE__)) <br />CHECKERCONF_FILE = "dutiesconf.yml"<br /><br />yaml_conf = YAML.load_file("#{File.dirname(__FILE__)}/#{CHECKERCONF_FILE}")<br />notify = Notifier.new(yaml_conf)<br />message = "Yo " + yaml_conf["people"][Time.now.wday].to_s + "! Letting you know that House Duties is yours today. Aiight peace! -KitchenBot"<br /><br />notify.clickatell_text(message, [yaml_conf["phones"][Time.now.wday]]).to_s<br />

I added a “people” and “phones” field, and indexed both of these by the day of the week, thus sending personalized text messages to whoever is the lucky person that day. A simple cron job entry on my little home server, and I was all set for a blissfully clean kitchen. (I’ll have to let you guys know how that works out though…)

Overall, Clickatell is easy to use once you have it set up, their response time if fast, and their service works great. Once their initial process of buying credits is more streamlined, they will undoubtedly be the top choice for anyone wanting to do some text message hacking. I haven’t yet tried using their 2-way text messaging, but if this is as easy to use as their one-way API, I’m sold.

If Obama doesn’t win, I’m emigrating

I’m a 21 year old student. A foreign-born student, on top of that. And a immigrant with a great respect and love for the United States of America. I arrived in the US not too long after September 11, when Bush has one of the highest approval ratings of any president in the history of the US. Now, I live in a country that my international friends despise and fear, and where my American friends are hopeless and angry at the injustice done to the country.

And now we finally have candidates for a new election that could change all of this. But then I read the election news. And I follow the democratic campaigns. And what do I keep seeing? Hillary and her back-alley-fighting campaign smearing Obama’s name on every occasion they can.

Seriously. If Obama doesn’t win, I’m leaving for greener pastures.

Wanna reset your MySQL root password on Ubuntu?

Its easy!

sudo /etc/init.d/mysql reset-password

SVN: Converting from BerkeleyDB to FSFS

I’ve been working with Marcello on http://2draw.net/ and after posting up the Alt framework we’re working on (http://alt.cellosoft.net/) onto ajaxian.com our SVN suddenly stopped working. What a mess, eh? Indeed. BerkeleyDB was apparently at fault, since our DB got corrupted as we were being hit by the masses.

Since we have about 8 repositories, I wrote up a little shell script to do the housekeeping in making the transition. The usage case is pretty simple:

$ ./convert.sh repo_dir

be sure to run this with the pure repository directory, no trailing slash!

<br />#!/bin/sh<br />echo "Converting $repo from BerkeleyDB to FSFS"<br />repo=$1<br />echo "Dumping to $repo.svn.backup"<br />svnadmin dump $repo > $repo.svn.backup<br />mv $repo $repo.bdb<br />svnadmin create --fs-type fsfs $repo<br />svnadmin load $repo < $repo.svn.backup<br />chown -R svn:svn $repo && chmod -R g+r $repo<br />cp -R $repo.bdb/conf/ $repo/<br />cp -R $repo.bdb/hooks/ $repo/<br />rm -rf $repo.bdb<br />rm $repo.svn.backup<br />

Shaders!


I’m late and I have interviews to get to, but here is the fruits of my night’s labor in writing shading algorithms. My first real C++ program, so proud!

Wiring the apartment – setting up DHCP and Routing on Ubuntu

I have a confession to make. I live with 11 engineers. In two connected apartments. And when we moved it, there was no wiring in the house… Only a telephone jack in each room. And now, we all have ethernet drops and multiple wifi access points through the area. I do hope our landlord sees this as a significant improvement!

In the process I bumped up my networking skill quite significantly. The network layout looks something like this:

[Border Router]
| |
[Switches for Subnet 0]
| |
{eth0}[Server]{eth1}
| |
[Switches for Subnet 1]
| |
[Border Router]

Each switch is connected to about 8 desktops and two wireless access points, forming their own network. The idea is that each apartment (thus each subnet) has its own internet access point, but the fileserver is accessible from both apartments, and also routes between the two apartments so that it appears that they are both on the same network to things like SMB and the like.

I use DHCP on the server itself to set the correct information on all the hosts. Here is my dhcpd.conf file:

# Custom DHCP3 server for the EECS House
# Niels Joubert njoubert@gmail.com

### Add classless-static-routes option:
option classless-static-routes code 121 = array of { ip-address, ip-address };
option new-static-routes code 249 = string;
###

lease-file-name "/var/lib/dhcp3/dhcpd.leases";

default-lease-time 86400;
max-lease-time 604800;

authoritative;

subnet 192.168.0.0 netmask 255.255.255.0 {
option ip-forwarding off;

range 192.168.0.100 192.168.0.200;
option subnet-mask 255.255.255.0;
option broadcast-address 192.168.0.255;

#option domain-name "";
option domain-name-servers 192.168.1.1, 68.87.76.178, 68.87.78.130;

option netbios-name-servers 192.168.0.10;
option netbios-node-type 8;
option routers 192.168.0.10;

#This is to get the machines on the two subnets to talk:
option new-static-routes 18:c0:a8:01:c0:a8:00:0a;

}

subnet 192.168.1.0 netmask 255.255.255.0 {
option ip-forwarding off;

range 192.168.1.100 192.168.1.200;
option subnet-mask 255.255.255.0;
option broadcast-address 192.168.1.255;

#option domain-name "";
option domain-name-servers 192.168.1.1, 68.87.76.178, 68.87.78.130;

option netbios-name-servers 192.168.1.10;
option netbios-node-type 8;
option routers 192.168.1.1;

#This is to get the machines on the two subnets to talk:
option new-static-routes 18:c0:a8:00:c0:a8:01:0a;

}

There is a couple of hacks in here:
- to get machines from the one subnet to see the other subnet, i use the classless-static-routes DHCP option, that puts an entry into any host that gets a DHCP offer from this box. Unfortunately this only works for windows hosts! This is a huge caveat that I have not been able to solve.
For SMB use, I set the server itself as the netbios name server (more on this later). For each subnet, I set the default route as its border router. With this setup it is enough that each subnet has internet and should be able to talk to the other subnet through the server.

IPTABLES

To forward anything through the server itself, I use a custom iptables setup that resides in a script that gets automatically run, as follows:

# NIELS JOUBERT
# Custom iptables script

# Clean up iptables (flush it)
iptables -F
iptables -t nat -F
iptables -X

# Enable IP MASQUERADING/NAT
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# Set firewall policies (default behaviour)
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

The idea here is to set up a nat table to masquerade ips, and the forwarding table to accept all forwarding. Simple, straight-forward and general. Possibly not the best security, but we are self-contained within the border routers right now, so if one of my roommates wants to be an ass and attack someone else, its easy enough to go hit him physically, right?

SAMBA

Samba itself also needs to be configured. Now, Samba is HUGE, as are its config files. I recommend a good book on Samba if you are going to do anything more than the most basic sharing with it. I personally prefer “The Official Samba-3 HOWTO and Reference Guide”.

The gist of my smb.conf setup looks like this:

[global]
workgroup = workgroup
netbios name = ubuntu
server string = Niels Server
dns proxy = no
name resolve order = lmhosts wins host bcast
smb ports = 139 240
# This tells Samba to use a separate log file for each machine
log file = /var/log/samba/log.%m
max log size = 1000
syslog = 0

####### WINS Services ######
wins support = yes

# Forces us to be the local master browser for WINS
local master = yes
preferred master = yes
os level = 34
domain master = yes

# Cache TTL
max ttl = 86400
max wins ttl = 86400

The important thing to note is that samba works on all interfaces and is the local wins browser (netbios name server).

This is the setup! It works fairly well too.

Reliability through two interfaces!

The one apartment’s internet died today because we’re in the middle of switching from ComCast (ewww) to DSLExtreme (YAY!). I remedied 7 very angry internet-less engineers by a very simple change in routing.

1) Change the subnet without border router to use the server as border router
2) Change the server to use as default route the border router of the other subnet

This took about 5 minutes, and we were piping the internet through the other apartment into our without a hiccup. Hooray for networking!

Feel free to ask me about my setup or use my config files.

Why I took my Shuttle XPC Back – Linux on the Shuttle XPC SG33G5

I returned my Shuttle XPC to Fry’s Electronics over the weekend. Not because I didn’t like it, but because of my issues to run Linux on it.

I managed to get Ubuntu 7.10 working on it as I outlined in a previous blog post, but Windows was complaining about the BIOS settings needed to get Linux working. Frankly, although I’m fairly certain that it is possible to run Linux and windows XP dual-boot on the shuttle, it is far too much of a hassle to get this working. Thus, back it goes. It was a nice box, but alas I won’t be running it anymore! My hard drive was also running far too hot in it to be happy with. Thus, on to the next post for my new baby…

Ubuntu on Shuttle XPC SG33G5 (Intel G33 chipset)

I recently acquired one of the very very nice shuttle small form factor PCs – the Shuttle XPC SG33G5. This is one of their mid-range boxes, and I spec’d it out the following:

Chipset (Standard): Intel G33
CPU: Intel Core 2 Duo E6550 2.33GHz
RAM: 2x 1Gb OCZ Ram
HDD: Western Digital Raptop WD1500

I installed my (legal) copy of Windows XP on it, and with the driver CD that came with the box I was up and running in less than an hour (fast hard drive!). The little box is responsive, fast, and runs fairly quite. I was very happy with my $600 investment!

This was all about to change when I popped in the current Ubuntu CD… To make a long story short, every kernel, distro version and boot flag I tried failed miserably.

I finally came up with something of a solution. A combination of the latest kernel, the correct boot flags and the correct bios settings are needed. They are as follows:

Distribution: Ubuntu 8.04 Alpha 3
Kernel: 2.6.24

BIOS Settings (From http://gentoo-wiki.com/HARDWARE_Inte…d_Q33_Chipsets)
SATA = AHCI Native Mode
AHCI = On
processor = Native Mode

Boot Options:
acpi=off noacpi nolacpi pci=conf1

This got me into the LiveCD fine. I’m still having some issues once I try to run GNOME after my install completed, but at least its a lot further down the line than no kernel at all! I’ll post more as I hack it.

ASUS eeePC – First Looks

After many a hesitant glance, I finally convinced myself to buy one of the new Asus eeePCs. For those who haven’t been following the news, the eeePC is a small form factor laptop with a 7″ screen, weighing in just under 1kg (about 1.5 pounds). It runs a custom interface on top of Xandros linux, and has all the programs the road warrior expects. Firefox, Thunderbird, OpenOffice and the like come pre-installed and it is fairly easy to add and linux program to your distribution.

I’ve only had it for about 24 hours, but I’m impressed so far! The build quality is excellent and it feels very sturdy. The keyboard is a bit small, but I’m already getting used to it. Its a pretty amazing little machine!

I’ll post more as I explore this gadget, but for now I’ll recommend it to everyone!

"Topgun according to Tarantino"

I never really understood what Top Gun was about until I watched this:

I love Quentin Tarantino

NowTune.com now with hotlinking playlists

I have a couple of issues with Nowtune.com so far:

- Uploads only work if they’re all from the same directory. Bugfix I guess?
- You can’t delete tracks from a playlist!
- SLOW with lots of files. I have about 600 tracks in my main playlist and it takes about 5 to 10 seconds to load. And this is on a blazingly fast DS3 line.
- Crashes now and then, uploads fail…

BUT

I can do this: http://nowtune.com/njoubert/loungelatenight

and you can listen to the best late night lounge / study music Global Underground has to offer!

Copywrite in the Music Industry

I just happened across a great article on music copywrites, by jwz himself. Finals week, so no time for chitchat, but check it out:

http://www.dnalounge.com/backstage/webcasting.html

NowTune.com live

A buddy at Cal sent me a link – www.nowtune.com – to play around with. Concept: upload your music to their site and listen to it online anywhere from the internet. make friends and listen to their playlists in a radio station fashion.

I like the idea, although i don’t know how feasible the idea of uploading my entire music collection to their servers (which happen to be Amazon S3 in the back), the idea is great. I’ve been struggling with the same problem – making my music accessible from anywhere – and this is one solution. Winamp has tried the same thing with their Winamp Remote, which runs a local HTTP server on your computer, and gives you access to your library through their web interface. Nowtune.come gives you the social aspect, and you don’t need to have your local computer running. then again, you need to upload your music.

I did find a bit of a security hole (if you can call it that) that i’ve decided to keep private for now. Youtube, google video, all these sites have basically the same problem they’re looking at, but there are ways around it in this case, which i’ll gladly give them the time to fix.

Anyways, definitely something to look out for.

Music

OiNK, probably the best online music source, has been shut down a while ago, and record sales are going south like a duck in winter. The public is pretty outraged, but for many of us its still a blind rage at why your college friends are being sued and why you have to pay $18 for CDs if you really just want the third track on the bonus disc. I’m pretty much in the same boat, until I read this fantastic post by a former record company employee.

Husky Rescue

For enlightenment through music