View Full Version : BrainBot Work

10-10-2007, 08:04 PM
So, I've now got two BrainBots put together here, not including the third one that is down in New Hampshire at the Brain Engineering Lab.


That's the latest one, with a second tracked base. Its mostly together, but I need to do some more wiring and get the gumstix mounted in the chest.

Here's the two of them together:


Still lots of work to do, but things are coming along...

- Jon

10-15-2007, 01:02 PM
So last night I finally got wifi running with the gumstix, and now BrainBot can tool around completely wirelessly. Today at lunch I made a short video of BrainBot driving around my driveway (I'm controlling it with a joystick attached to my PC).


Things are starting to happen...

- Jon

10-15-2007, 02:27 PM
Great work Jon! It's only a matter of time until you have a swarm of "huvoloids" running around and doing your bidding, haha;)

Have you worked out the feeds coming in from the cameras? I'd love to see that vision software in action:D

EDIT: BTW, how does the whole communication with the gumstix thing work? I have an idea of it, but I haven't had the opportunity to mess around with it personally. Is it a master slave scenario, where the gumstix run embedded Linux and talks to a Windows PC via WiFi? I thought you briefly explained this once before in our forums, but I can't seem to find the link anymore...

10-15-2007, 02:43 PM
The cameras are wireless, so we connect the receiver to a frame grabber running on the PC.

The communications works like this:

At the bottom level, I have a PCB made which is a Bioloid bus device - see the second board mentioned in this blog post (http://www.huv.com/blog/2007/07/catching-up.html) for details. That becomes just another device on the Bioloid bus, which is a 1.0 Mbps serial multi-drop bus. The gumstix talks to the Bioloid bus using an FT-232 chip from FTDI, through one of its host USB ports. That gives me a virtual serial port from the gumstix to the Bioloid bus, also at 1.0 Mbps.

The gumstix has an application on it (written by myself and a friend) that sets up a socket, and waits for a connection.

The PC runs another custom application, which connects to the gumstix socket, and sends Bioloid bus commands to the gumstix, which forwards them to the devices on the bus. The socket communication takes place over wifi, which gives us a full 1.0 Mbps wireless connection from a PC to the Bioloid bus...

- Jon

10-15-2007, 04:03 PM
I was pretty close, haha:) Thanks for the info!

10-25-2007, 05:13 PM
Last night I got another piece working - I was able to stream audio from my PC to my gumstix, over wifi. I took an MP3 song and converted it to a WAV file (40 MB). I ran socat on the gumstix (which basically sets up a socket server, and streams incoming data to stdout. I piped the output of that to bplay, which plays WAV files over a speaker plugged into the audiostix.

I set up a client socket in Squeak on my PC, connected to the gumstix, and started sending chunks of the WAV file over the socket. The sound started playing on the gumstix almost instantly, which is pretty nice.

The song was 4 minutes long, and it streamed the audio flawlessly for the entire period.

One of the major goals of the BrainBot project is to stream audio in both directions (from the microphones on the robot to the PC, and to the speaker on the robot from the PC). Having this working so nicely is a major part of that, and streaming in the other direction works basically the same.

- Jon

10-29-2007, 03:32 PM
Nice:) So when are we going to have a dance room full of these BrainBots, haha.

Seriously though, so the purpose of streaming the audio is so that you could give the BrainBot voice commands on either the robot (gumstix) end, or the computer end? Or am I totally misunderstanding the point?

10-29-2007, 07:00 PM
The purpose of two way streaming audio is so the Brain Engineering guys can do language research with it. There are three main goals to the BrainBot robot: manipulation, vision, and language. We now have the capability to do all three, although all three require more polishing and setup. But we've basically got a proof of concept for all three.

- Jon

10-30-2007, 10:05 AM
We now have the capability to do all three, although all three require more polishing and setup. But we've basically got a proof of concept for all three.

Awesome Jon! So does this mean that we should expect to see some cool new things done in the vision, language and manipulation areas with your BrainBot in the future?

10-30-2007, 12:50 PM
Well, I sure hope so :-)

We're still waiting for a research grant to come through so I can work on this stuff full time. If/when that happens, things will definitely get interesting...

- Jon

10-30-2007, 12:53 PM

Good luck on that grant Jon. With all of the work that you guys have been doing, you definitely deserve it!

11-09-2007, 03:36 PM
Jon that looks like a great project.

I used a MS based solution on a similar project, using WIFI between a host PC and an iPaq. I had all communication go through the WIFI channel, including TTS, mic input, and lip sync data.

In hindsight, it might have worked better for me if I had used a separate channel for the sound. One idea that came later was a usb cellphone headset that communicated directly with the host PC. This could have helped with my ASR interactivity -- perhaps it could with your project as well.

11-12-2007, 12:27 PM

I'm curious - what problem did you have with sending audio over wifi? Was it strictly a bandwidth thing?

- Jon

11-13-2007, 10:51 AM
There are 2 parts actually. The first is the sometimes-connected-sometimes-not nature of wifi. What I had set up at first was a tcp socket by which I was uploading wav files to the server for ASR. Whenever the wireless hiccupped, I'd have a socket reset scenario. So I came up with another communication method using UDP. The UDP was largely to let the server know that a new wav file had been recorded. The server would then use http to download the wav file from a web server running on the ipaq. Likewise the ipaq would use http to download wav files from the server (tts files). This worked reasonably well in my setup, but with so many pieces was a little taxing to debug.

The 2nd part is less obvious at first: sending wavs back and forth is a turn-based protocol. It means that the ASR can't start processing the wav until you stop speaking, which really slows things down for a long utterance. It wasn't until I was beyond the testing digits phase that I realized what a problem this was. I know that some folks at Boulder have used voip with ASR (the Sonic project) and that might be a way to go if you wanted to use the internet. However, I suggest trying a bluetooth or RF microphone that connects directly to the machine doing the ASR. This makes sense to me for several reasons:
-you don't really need wifi to send sound data back and forth.
-many ASR acoustic models are 8khz and 16khz, so you can easily use a phone-based system and process the sound using existing acoustic models (my ipaq only had the hardware to do 8khz anyways)
-there are fewer points of failure and less messy code, since you've cut one computer out of the loop (the first part above)

Sorry this is a bit long. It's also very biased based on my experience with this one project. Your mileage may vary.

I've enjoyed reading about your BrainBot project and look forward to more posts


11-13-2007, 03:17 PM
I figured out how to stream wav files in both directions simultaneously, over sockets with wifi.

So I start sending a large (3-minute song) wav file from my PC, and it starts playing immediately on the gumstix. I then start recording using a mic on the gumstix, and that data gets streamed in real time to the PC. This is all done (ironically enough) using a built-in linux utility - socat.

- Jon

11-14-2007, 09:47 AM
Very cool -- this is the first time I've seen socat.

With all the possible options, I'm not sure exactly how you are using it. Are you basically doing voip with socat?


11-14-2007, 10:05 AM
To record from the gumstix:

brec -w -S -s 48000 -b 16 | socat -u - TCP:

This pipes the output of brec through a socket connection to a listener-socket on my PC.

To play from the gumstix:

socat -u tcp-listen:12346 - | bplay

This sets up a listener socket on the gumstix, which I can connect to from my PC, and start sending WAV file data...

- Jon