Sleep Waking

[brendanburns]

Hello,
Fernando Orellana and I (Brendan Burns) have created a piece of robotic art entitled "Sleep Waking" (http://www.fernandoorellana.com) its on display at the exit art gallery (http://exitart.org) in Manhattan until April 19th.

Using recorded brainwave activity and eye movements during REM sleep to determine robot behaviors and head positions, "Sleep Waking" acts as a way to "play-back" dreams. Through this piece we hope to investigate one of the possible human-robot relationships.

Video:
1RkM1Bt2b3k

Technical Details:
The robot is a modified KHR-2HV with 2 Gyros and an accelerometer (to detect falls) It is hard wired into a power source so that it can run for a extended period of time (it will be on display for 2 months!)

An overhead camera (not pictured in the video) is used to provide robot position/orientation data so that it can navigate around the pedestal without falling off.

The recorded eye muscle movements are used to drive head motion, so the head movement is a literal representation of Fernando's recorded eye movement. We also used machine learning to find patterns in the EEG eletrode data. These patterns were mapped to specific behaviors (shown in the video) which are remembered from our dreams.

Thanks!
--brendan ^ fernando

[darkback2]

I followed the link to the website for the bot. Really cool movement. How does it choose how to move around? As in, are there a set of preset movements which are triggered by the recorded eye movements? or is it based on data collected from the EEG?

In anycase...cool

[Zenta]

Very exciting and interesting project you got there!

Are the movements based upon a library (set of different sequences) that is triggered by your recorded brainwaves?? Or does it walk and moves based upon walking algorithms (realtime calculations) triggered by the brainwaves?? Or?..., how do you do that, tell us more! Please!

Awesome! :D

[brendanburns]

Thanks!

The eye movements are literally the recorded eye movements. The electrical potential used to stimulate muscle movement was recorded and then translated directly into the servo position of the head.

For the larger motions (e.g. flying, crouching, etc) we found patterns in a subset of the EEG voltage data. (actually in the frequencies of the voltage wave) and mapped those patterns onto pre-programmed behaviors which were inspired by dreams we remembered.

Then we process the rest of the EEG voltage data through a classifier which identifies the most likely pattern given the current window of data and causes the robot to perform the corresponding motion.

The walking is done using motion planning and movement primitives.
e.g. when it wants to go sit down, it uses a planning algorithm that looks basically like:

while (not near bench) {
pos = current position // from camera
if (pos.angle > 45)
turn right
else if (pos.angle < 45)
turn left
else if (pos.x,pos.y is to the left of the bench)
sidestep right
else if (pos.x,pos.y is to the right of the bench)
sidestep left
else if (pos.x,pos.y is far from the bench)
back up
}

each of the primitive actions (e.g. turn left, turn right) is a small pre-programmed motion.
--brendan

I hope that makes sense! (let me know if it doesn't...)
--brendan

[MinyFly92]

Thats really cool. It has great fluid movement.