Trying to debug HROS1 with RPI talking to Arbotix Pro

[KurtEck]

I am sort of curious on how many people are using the RPI2 versus the Edison and I wondering about how well the Arbotix Pro is talking to the the Raspberry PI.

When I try to run some of the different project programs, such as the PS3 demo as well as the DXL manager, probably over 50 percent of the time when I try to run one of these programs, the system fails to initialize properly and errors out. Example:

[ID:200(SUB_BOARD)] ^[email protected] ~/HROS1-Framework-Kurte/Linux/project/dxl_monitor $ ./dxl_monitor

[Dynamixel Monitor for DARwIn v1.00]
Failed to connect CM-730!
Terminated DXL Manager.
[email protected] ~/HROS1-Framework-Kurte/

If I try to run it again it often will work. I am pretty sure it is a timeout issue.

Likewise most of the time I can not reliably switch to different IDS. For example:

[email protected] ~/HROS1-Framework-Kurte/Linux/project/dxl_monitor $ ./dxl_monitor

[Dynamixel Monitor for DARwIn v1.00]

 Check ID:1(R_SHOULDER_PITCH)     ... OK
 Check ID:2(L_SHOULDER_PITCH)     ... OK
 Check ID:3(R_SHOULDER_ROLL)      ... OK
 Check ID:4(L_SHOULDER_ROLL)      ... OK
 Check ID:5(R_ELBOW)              ... OK
 Check ID:6(L_ELBOW)              ... OK
 Check ID:7(R_HIP_YAW)            ... OK
 Check ID:8(L_HIP_YAW)            ... OK
 Check ID:9(R_HIP_ROLL)           ... OK
 Check ID:10(L_HIP_ROLL)          ... OK
 Check ID:11(R_HIP_PITCH)         ... OK
 Check ID:12(L_HIP_PITCH)         ... OK
 Check ID:13(R_KNEE)              ... OK
 Check ID:14(L_KNEE)              ... OK
 Check ID:15(R_ANKLE_PITCH)       ... OK
 Check ID:16(L_ANKLE_PITCH)       ... OK
 Check ID:17(R_ANKLE_ROLL)        ... OK
 Check ID:18(L_ANKLE_ROLL)        ... OK
 Check ID:19(HEAD_PAN)            ... OK
 Check ID:20(HEAD_TILT)           ... OK
 Check ID:200(SUB_BOARD)          ... OK

[ID:200(SUB_BOARD)] id 1
[ID:1(R_SHOULDER_PITCH)] wr 30 400
 Fail to write!
[ID:1(R_SHOULDER_PITCH)] id 2
 Invalid ID(2):4!
[ID:1(R_SHOULDER_PITCH)]

The first set of id works, I am also able to write to servo, but then if I try to switch to different ID it most often fails.
I updated the code to print out error code (4), which is timeout.

Looking at the underlying code, it tries to switch servos by doing a ping of it. The ping is failing. The underlying code that ping is calling, sets a timeout of 2* (length of ping packet)*time_to_send_a_byte + 5 ms.

I am not sure how this equation was decided on. It looks semi-reasonable to me if I was talking directly to the servos locally on the processor, as you need the amount of time to actually send the transmit the packet, plus the time the servo says to delay starting to send a response, plus the times it takes to send the bytes back over the UART. But this does not take into account the overhead for sending the packet over USB to the Arbotix, and then once the Arbotix gets the response back you have the delay to send the data back over USB... Also when I was earlier debugging some stuff using USB2AX, I also saw that at times there were some reasonable gaps in the servo response bytes...

So again I was wondering if others have already run into this and debugged it? Also wondering if those running using the Edison are also seeing similar issues?

Otherwise I will probably start debugging this more. Will probably use a logic analyzer to see how if valid packets are actually being returned and if so how long are they taking.

Other suggestions?

Kurt

[LloydF]

Hi there, just wondering have you verified you Arbotix-Pro to raspberry-pi connect is not lose? As I have not seen these issues, knock on wood, yet.

[KurtEck]

Thanks,
I double checked it earlier, but will do so again. Also may try a real USB cable to see if that makes any difference.

May also start reducing how many servos are connected to see if that changes the response or lack of.

Thanks again

[KurtEck]

Another sort-of update.

For the fun of it I swapped in the Edison for the RPI2. This took me longer than it should have as I like to do things the hard way and I wanted to build up my own image for it.

I first installed the latest official Edison image and it could not find the Arbotix Pro. That is when I remembered that the current official images do not include USB FTDI support. I tried to install it from AlexT's repo which installed but did not work, as the Linux images you build do not match the linux name of the official image, so the driver module did not load.

So I went through the process and built the module on my NUC under Ubuntu 14.04 and I used the menu config stuff to include the FDTI driver and then I reflashed. I now have /dev/ttyUSB0

I followed the stuff in my my forum posting: https://communities.intel.com/message/265911#265911

I then installed git, downloaded my fork of the HROS1-Framework, and built portions of it and starting to test.

I needed to install git as well as libjpeg-dev

So far I have done some testing with the dxl_monitor program and it appears to be much happier with the Arbotix Pro. I am not getting any of the failures yet.

I now have the ps3_demo program built, and now need to redo the sixpair as it was built with libusb-0.1.so.4 which is not on my machine yet, but that should be easy. And then next up try it out with my own PS3 driver.

Kurt

[r3n33]

Nice work though.. I went down a similar path not too long ago by installing the latest official image for Edison, finding no built in FTDI support, and unable to load the kernel module from AlexT's repo. Alex did get back to me about being unable to load the kernel module due to an infamous difference in kernel naming between the released binaries and the kernel produced by the build. In order to use his kernel modules he said you'd have to first install the kernel from his repo. By the time I was informed I had already moved on to something else.

I'm starting to get curious about the Arbotix communication troubles you are experiencing on the Pi. Also a little relieved you are getting the same good results on the Edison.

[LloydF]

Well dang. I went back and played with the dxl_monitor and got the same result you had. Must be when I check it, it ran so I just left it.
I am sure it is a timing issue and not you. (It is the code) This kernel has a ton of latency issues. There is a sanppy unbuntu kernel that can be interrupted it is used mostly for drones were telemeter can't handle latency issues. I use it with a NAVIO+ hat on one of my drones. I'll do some research.

[LloydF]

Holy cow, ok that fixed it and it runs everything way faster (I know were snappy came from LOL). I got the image from here https://drive.google.com/file/d/0Bxa...ew?usp=sharing. (P.S No desk top so everything is ssh.) This article gives details on how to build the Kernel. Look at the real time for Linux part. The Real-time patch lowers the worst case latency to tens of microseconds vs Hundreds of microseconds.(use raspi-config to set password and resize your sd card). http://docs.emlid.com/Navio-APM/installation-and-running/

[KurtEck]

Thanks r3n33 - Yep I have had a few postings over the last long while about some of the issues of official builds having one name and builds we do from the official sources have a different name. I thought the last semi-silent update was going to resolve it, but...

I am still trying to do the things the hard way With my build of the official sources, I can talk to the /dev/ttyUSB0 object which is the Arbotix Pro. As I mentioned in previous post it appears to not have (or at least show up) the same issues as I was having with the RPI2.

However I so far don't have a /dev/input/js0 object for the Sony PS3. I am part way there. The Bluez5 that is installed with the current sources did not have in the fixes for the Sony HID, however their github linux files (https://github.com/01org/edison-linu...edison-3.10.17) show they have installed a patch to fix it which has not propagated to a build yet.

So I made a patch file from that delta and then added that patch file to the build and I think that part is in now. However I then found that the joystick device drivers were not installed, such as the generic analog joystick. I think I may now have built a version with those installed, but still no /dev/input/js0 object (even when plugged in USB). Looking at my NUC with Ubuntu I see there are rules
(/lib/udev/rules.d) in a couple of files that I believe are responsible for generating these nodes. So I will try adding the stuff to the appropriate rules files on the Edison....

As I said the hard way. But maybe hopefully I can then get the Intel people to setup all of the needed stuff in an official build. Or at least minimally maybe have most of it in the build and some of it in stuff you can opkg install from AlexT's repos...

LloydF - Sounds good. Looks like you found a workaround, which I may want to look into. I still wonder if this is more an RPI2 issue or an issue with the Arbotix-Pro? That is I did not have issues with running AX servos on the older RPI with USB2AX or USB2DYNAMIXEL. I have not done much testing with these on the RPI2 yet, but I would think the RPI2 would be faster...

At times I wish I had a USB sniffer device to see the messages going back and forth. For example I believe the USB2AX is setup to send stuff packed with if I remember correctly 16 data bytes per packet (maybe 32?), I wonder how many data bytes are being sent per packet with the Arbotix-Pro.... But that is me just guessing.

Will back to doing stuff the hard way :lol:
Kurt

[tician]

The arbotix-pro uses the usual FT232RL USB2UART IC, so it will try to send back as much data as the STM32 has available in its buffer to dump over USB. It is quite likely an issue with the FTDI driver and/or its configuration. I know there have been many people afflicted by lots of lost packets on FT232RL-based devices (for both windows and linux) because the driver's packet timeout was set to be way too long compared to the timeout calculated in the robotis dynamixel library. Changing the driver's timeout to a smaller value, and/or increasing the calculated timeout in the framework might help. Might also be helped by following tigakub's suggestion of turning off Nagle's algorithm, but might work better only for outgoing packets to USB devices and not incoming to the host (since devices cannot actually signal they have data ready to send unless polled by the host).

Although it does not seem to want to work anymore on my netbook, I had used wireshark for snooping USB packets when working with the OpenCM (especially in figuring out the upload sequence).

[LloydF]

Yea, The NAVIO+ Guys have a nice cyclic test which show off the latencie issues with the rpi and rpi2 kernels, they must have gone through some sutff to get the ( Interipatable ) RT-snappy Unbuntu core up and running, but there in the Business of sensors and have a hat for the edison coming out soon (uav things).There version comes with a raspi-config program to expand the SD so the using of fdisk will kill it, just a warning.