Robotic Eye-Hand Coordination

**Pi Robot** · 03-02-2010

Hello,

I just finished up some work on using RoboRealm to guide my robot as it reaches toward a target object. The ultimate goal is for the robot to be able to pick up the object from a random location or take it from someone's hands. For now, I simply wanted to work out the coordinate transformations from visual space to arm space to get the two hands to point in the right direction as the target is moved about. The following video shows the results so far:

I don't have a full write-up yet on how I did this but it basically just uses 3-d coordinate transformations from the head angles and distance to the target (as measured by sonar and IR sensors mounted near the camera lens) to a frame of reference attached to each shoulder joint. The Dynamixel AX-12 servos are nice for this application since they can be queried for their current position info. The distance to the balloon as measured by the sonar and IR sensors is a little hit and miss and I think I'd get better performance using stereo vision instead.

--patrick

http://www.pirobot.org

**Orac** · 03-03-2010

Wow, very good, I have been playing with Roborealm 3d positioning, but no where near as good as that. Kind of spooky to watch, you think it might cry if you don't give it that balloon soon

**Pi Robot** · 03-03-2010

Thanks Orac! Right now I am using RoboRealm only to get the x-y coordinates of the balloon in the camera frame of reference. I then use a sonar and IR sensor mounted near the lens to get the z-coordinate or distance to the balloon. One could also use the width of the balloon's blob in RoboRealm if you assume it has a fixed size, but I wanted to allow for different sized objects. When STeven and company get their Stereo module completed, I think that will be a good way to go, but for now this seems to get the job done.

And yes, the next step is to actually give the balloon to the robot before it comes after me flailing those arms. The AX-12's can pack quite a punch.

--patrick

**bonmot** · 03-08-2010

Wow, I am looking forward to see you stereo vision logic working. I guess then two hands and the ball are the 3 visual objects in bot's brain to control the it grabbing the ball. Am I right about your plan?
This is what humand do, isn't it? We don't calculate the xyz of our hand to grab a cup :-D

**bonmot** · 03-08-2010

$89 for this software is really good. The steven guy you mentioned is from Roborealm?
If $89 can buy a stereo vision SDK, that will be a steal :-)
Thanks

**Pi Robot** · 03-09-2010

Actually, I imagine that the human brain (as well as other animal brains) does do some form of calculation without having to look at the hands. After all, you can reach for a cup with your eyes closed (though you might be slightly off target). Also, whenever you begin to reach for an object, your hands are not necessarily in view until they are part way to the goal. So you have to get the hands moving in the right direction before you see them.

Having said that, yes, at some point I was thinking of putting some small fiducials on each hand and these could be used to visually guide the hands to the target.

--patrick

**bonmot** · 03-09-2010

Agreed. rough calculation can be done to guide the hand to the area, then use vision system to do the "pick up".
The non linearity of the lenses, servo postion errors, mechanical errors, etc... all add up.
Calibration will solve just small part of it.

The "brain calculation" you mentioned is not actual "calculation", but "imagination". With eyes closed, brain still uses the "vision system" to guide your action by the image created by "imagination".
"Simulation" maybe another word.

**Pi Robot** · 03-09-2010

Yes, "simulation" seems like a better word than "calculation". Or perhaps simply "neural computation". As you probably know, much work has been done on this kind of thing in the fields of psychology and neurophysiology. I found this article (abstract) particularly interesting:

http://www.ncbi.nlm.nih.gov/pubmed/1...ubmed_RVDocSum