reading entire Dynamixel table in one packet
[Edit: This thread has evolved beyond this initial problem into a discussion of general Dynamixel bus read performance and strategies.]
Hi everyone,
I'm seeing some strange behavior, wanted to see if this is something that others have encountered before as well. I'm seeing extra data bytes in the params when I read the whole table at once. I don't have any problems reading one or two bytes at a time.
Here is some output of reading an AX-12+ table one address at a time:
Code:
12 [address=0,name=Model Number(L),initialValue=12,area=EEPROM,access=R,size=Word,index=0]
0 [address=1,name=Model Number(H),initialValue=0,area=EEPROM,access=R,size=Word,index=1]
24 [address=2,name=Version of Firmware,initialValue=-1,area=EEPROM,access=R,size=Byte,index=2]
3 [address=3,name=ID,initialValue=1,area=EEPROM,access=RW,size=Byte,index=3]
1 [address=4,name=Baud Rate,initialValue=1,area=EEPROM,access=RW,size=Byte,index=4]
0 [address=5,name=Return Delay Time,initialValue=250,area=EEPROM,access=RW,size=Byte,index=5]
200 [address=6,name=CW Angle Limit(L),initialValue=0,area=EEPROM,access=RW,size=Word,index=6]
0 [address=7,name=CW Angle Limit(H),initialValue=0,area=EEPROM,access=RW,size=Word,index=7]
55 [address=8,name=CCW Angle Limit(L),initialValue=255,area=EEPROM,access=RW,size=Word,index=8]
3 [address=9,name=CCW Angle Limit(H),initialValue=3,area=EEPROM,access=RW,size=Word,index=9]
70 [address=11,name=Highest Limit Temperature,initialValue=70,area=EEPROM,access=RW,size=Byte,index=10]
60 [address=12,name=Lowest Limit Voltage,initialValue=60,area=EEPROM,access=RW,size=Byte,index=11]
140 [address=13,name=Highest Limit Voltage,initialValue=140,area=EEPROM,access=RW,size=Byte,index=12]
255 [address=14,name=Max Torque(L),initialValue=255,area=EEPROM,access=RW,size=Word,index=13]
3 [address=15,name=Max Torque(H),initialValue=3,area=EEPROM,access=RW,size=Word,index=14]
2 [address=16,name=Status Return Level,initialValue=2,area=EEPROM,access=RW,size=Byte,index=15]
36 [address=17,name=Alarm LED,initialValue=36,area=EEPROM,access=RW,size=Byte,index=16]
36 [address=18,name=Alarm Shutdown,initialValue=36,area=EEPROM,access=RW,size=Byte,index=17]
0 [address=24,name=Torque Enable,initialValue=0,area=RAM,access=RW,size=Byte,index=18]
0 [address=25,name=LED,initialValue=0,area=RAM,access=RW,size=Byte,index=19]
1 [address=26,name=CW Compliance Margin,initialValue=1,area=RAM,access=RW,size=Byte,index=20]
1 [address=27,name=CCW Compliance Margin,initialValue=1,area=RAM,access=RW,size=Byte,index=21]
32 [address=28,name=CW Compliance Slope,initialValue=32,area=RAM,access=RW,size=Byte,index=22]
32 [address=29,name=CCW Compliance Slope,initialValue=32,area=RAM,access=RW,size=Byte,index=23]
248 [address=30,name=Goal Position(L),initialValue=-1,area=RAM,access=RW,size=Word,index=24]
1 [address=31,name=Goal Position(H),initialValue=-1,area=RAM,access=RW,size=Word,index=25]
0 [address=32,name=Moving Speed(L),initialValue=-1,area=RAM,access=RW,size=Word,index=26]
0 [address=33,name=Moving Speed(H),initialValue=-1,area=RAM,access=RW,size=Word,index=27]
255 [address=34,name=Torque Limit(L),initialValue=255,area=RAM,access=RW,size=Word,index=28]
3 [address=35,name=Torque Limit(H),initialValue=3,area=RAM,access=RW,size=Word,index=29]
248 [address=36,name=Present Position(L),initialValue=-1,area=RAM,access=R,size=Word,index=30]
1 [address=37,name=Present Position(H),initialValue=-1,area=RAM,access=R,size=Word,index=31]
0 [address=38,name=Present Speed(L),initialValue=-1,area=RAM,access=R,size=Word,index=32]
0 [address=39,name=Present Speed(H),initialValue=-1,area=RAM,access=R,size=Word,index=33]
0 [address=40,name=Present Load(L),initialValue=-1,area=RAM,access=R,size=Word,index=34]
0 [address=41,name=Present Load(H),initialValue=-1,area=RAM,access=R,size=Word,index=35]
123 [address=42,name=Present Voltage,initialValue=-1,area=RAM,access=R,size=Byte,index=36]
28 [address=43,name=Present Temperature,initialValue=-1,area=RAM,access=R,size=Byte,index=37]
0 [address=44,name=Registered,initialValue=0,area=RAM,access=R,size=Byte,index=38]
0 [address=46,name=Moving,initialValue=0,area=RAM,access=R,size=Byte,index=39]
0 [address=47,name=Lock,initialValue=0,area=RAM,access=RW,size=Byte,index=40]
32 [address=48,name=Punch(L),initialValue=32,area=RAM,access=RW,size=Word,index=41]
0 [address=49,name=Punch(H),initialValue=0,area=RAM,access=RW,size=Word,index=42]
Here is some output of reading all 43 addresses of an AX-12+ table at one time (comments below):
Code:
12 [address=0,name=Model Number(L),initialValue=12,area=EEPROM,access=R,size=Word,index=0]
0 [address=1,name=Model Number(H),initialValue=0,area=EEPROM,access=R,size=Word,index=1]
24 [address=2,name=Version of Firmware,initialValue=-1,area=EEPROM,access=R,size=Byte,index=2]
3 [address=3,name=ID,initialValue=1,area=EEPROM,access=RW,size=Byte,index=3]
1 [address=4,name=Baud Rate,initialValue=1,area=EEPROM,access=RW,size=Byte,index=4]
0 [address=5,name=Return Delay Time,initialValue=250,area=EEPROM,access=RW,size=Byte,index=5]
200 [address=6,name=CW Angle Limit(L),initialValue=0,area=EEPROM,access=RW,size=Word,index=6]
0 [address=7,name=CW Angle Limit(H),initialValue=0,area=EEPROM,access=RW,size=Word,index=7]
55 [address=8,name=CCW Angle Limit(L),initialValue=255,area=EEPROM,access=RW,size=Word,index=8]
3 [address=9,name=CCW Angle Limit(H),initialValue=3,area=EEPROM,access=RW,size=Word,index=9]
131 [address=11,name=Highest Limit Temperature,initialValue=70,area=EEPROM,access=RW,size=Byte,index=10]
70 [address=12,name=Lowest Limit Voltage,initialValue=60,area=EEPROM,access=RW,size=Byte,index=11]
60 [address=13,name=Highest Limit Voltage,initialValue=140,area=EEPROM,access=RW,size=Byte,index=12]
140 [address=14,name=Max Torque(L),initialValue=255,area=EEPROM,access=RW,size=Word,index=13]
255 [address=15,name=Max Torque(H),initialValue=3,area=EEPROM,access=RW,size=Word,index=14]
3 [address=16,name=Status Return Level,initialValue=2,area=EEPROM,access=RW,size=Byte,index=15]
2 [address=17,name=Alarm LED,initialValue=36,area=EEPROM,access=RW,size=Byte,index=16]
36 [address=18,name=Alarm Shutdown,initialValue=36,area=EEPROM,access=RW,size=Byte,index=17]
36 [address=24,name=Torque Enable,initialValue=0,area=RAM,access=RW,size=Byte,index=18]
0 [address=25,name=LED,initialValue=0,area=RAM,access=RW,size=Byte,index=19]
44 [address=26,name=CW Compliance Margin,initialValue=1,area=RAM,access=RW,size=Byte,index=20]
0 [address=27,name=CCW Compliance Margin,initialValue=1,area=RAM,access=RW,size=Byte,index=21]
206 [address=28,name=CW Compliance Slope,initialValue=32,area=RAM,access=RW,size=Byte,index=22]
3 [address=29,name=CCW Compliance Slope,initialValue=32,area=RAM,access=RW,size=Byte,index=23]
0 [address=30,name=Goal Position(L),initialValue=-1,area=RAM,access=RW,size=Word,index=24]
0 [address=31,name=Goal Position(H),initialValue=-1,area=RAM,access=RW,size=Word,index=25]
1 [address=32,name=Moving Speed(L),initialValue=-1,area=RAM,access=RW,size=Word,index=26]
1 [address=33,name=Moving Speed(H),initialValue=-1,area=RAM,access=RW,size=Word,index=27]
32 [address=34,name=Torque Limit(L),initialValue=255,area=RAM,access=RW,size=Word,index=28]
32 [address=35,name=Torque Limit(H),initialValue=3,area=RAM,access=RW,size=Word,index=29]
248 [address=36,name=Present Position(L),initialValue=-1,area=RAM,access=R,size=Word,index=30]
1 [address=37,name=Present Position(H),initialValue=-1,area=RAM,access=R,size=Word,index=31]
0 [address=38,name=Present Speed(L),initialValue=-1,area=RAM,access=R,size=Word,index=32]
0 [address=39,name=Present Speed(H),initialValue=-1,area=RAM,access=R,size=Word,index=33]
255 [address=40,name=Present Load(L),initialValue=-1,area=RAM,access=R,size=Word,index=34]
3 [address=41,name=Present Load(H),initialValue=-1,area=RAM,access=R,size=Word,index=35]
248 [address=42,name=Present Voltage,initialValue=-1,area=RAM,access=R,size=Byte,index=36]
1 [address=43,name=Present Temperature,initialValue=-1,area=RAM,access=R,size=Byte,index=37]
0 [address=44,name=Registered,initialValue=0,area=RAM,access=R,size=Byte,index=38]
0 [address=46,name=Moving,initialValue=0,area=RAM,access=R,size=Byte,index=39]
0 [address=47,name=Lock,initialValue=0,area=RAM,access=RW,size=Byte,index=40]
0 [address=48,name=Punch(L),initialValue=32,area=RAM,access=RW,size=Word,index=41]
123 [address=49,name=Punch(H),initialValue=0,area=RAM,access=RW,size=Word,index=42]
You can see that the first nine values are the same, then there is a 131 thrown in there. Another nine values match (but shifted down), and then there is this sequence inserted:
Code:
44 [address=26,name=CW Compliance Margin,initialValue=1,area=RAM,access=RW,size=Byte,index=20]
0 [address=27,name=CCW Compliance Margin,initialValue=1,area=RAM,access=RW,size=Byte,index=21]
206 [address=28,name=CW Compliance Slope,initialValue=32,area=RAM,access=RW,size=Byte,index=22]
3 [address=29,name=CCW Compliance Slope,initialValue=32,area=RAM,access=RW,size=Byte,index=23]
0 [address=30,name=Goal Position(L),initialValue=-1,area=RAM,access=RW,size=Word,index=24]
Then it matches again, but of course everything is shifted, and while I get 43 bytes, the last byte I get is address 42, not 49.
It's not that the data is getting randomly garbled either, because I get this byte sequence every time, even after power/port cycling. If I try a different AX-12+, I get the bytes in the same positions but different values. It's interesting that it's so consistent. I don't see anything in the documentation about this.
FYI, here is some data where I get a single address if it's a byte and two addresses if it's a word (only low address is printed but with combined word value):
Code:
12 [address=0,name=Model Number(L),initialValue=12,area=EEPROM,access=R,size=Word,index=0]
24 [address=2,name=Version of Firmware,initialValue=-1,area=EEPROM,access=R,size=Byte,index=2]
3 [address=3,name=ID,initialValue=1,area=EEPROM,access=RW,size=Byte,index=3]
1 [address=4,name=Baud Rate,initialValue=1,area=EEPROM,access=RW,size=Byte,index=4]
0 [address=5,name=Return Delay Time,initialValue=250,area=EEPROM,access=RW,size=Byte,index=5]
200 [address=6,name=CW Angle Limit(L),initialValue=0,area=EEPROM,access=RW,size=Word,index=6]
823 [address=8,name=CCW Angle Limit(L),initialValue=255,area=EEPROM,access=RW,size=Word,index=8]
70 [address=11,name=Highest Limit Temperature,initialValue=70,area=EEPROM,access=RW,size=Byte,index=10]
60 [address=12,name=Lowest Limit Voltage,initialValue=60,area=EEPROM,access=RW,size=Byte,index=11]
140 [address=13,name=Highest Limit Voltage,initialValue=140,area=EEPROM,access=RW,size=Byte,index=12]
1023 [address=14,name=Max Torque(L),initialValue=255,area=EEPROM,access=RW,size=Word,index=13]
2 [address=16,name=Status Return Level,initialValue=2,area=EEPROM,access=RW,size=Byte,index=15]
36 [address=17,name=Alarm LED,initialValue=36,area=EEPROM,access=RW,size=Byte,index=16]
36 [address=18,name=Alarm Shutdown,initialValue=36,area=EEPROM,access=RW,size=Byte,index=17]
0 [address=24,name=Torque Enable,initialValue=0,area=RAM,access=RW,size=Byte,index=18]
0 [address=25,name=LED,initialValue=0,area=RAM,access=RW,size=Byte,index=19]
1 [address=26,name=CW Compliance Margin,initialValue=1,area=RAM,access=RW,size=Byte,index=20]
1 [address=27,name=CCW Compliance Margin,initialValue=1,area=RAM,access=RW,size=Byte,index=21]
32 [address=28,name=CW Compliance Slope,initialValue=32,area=RAM,access=RW,size=Byte,index=22]
32 [address=29,name=CCW Compliance Slope,initialValue=32,area=RAM,access=RW,size=Byte,index=23]
504 [address=30,name=Goal Position(L),initialValue=-1,area=RAM,access=RW,size=Word,index=24]
0 [address=32,name=Moving Speed(L),initialValue=-1,area=RAM,access=RW,size=Word,index=26]
1023 [address=34,name=Torque Limit(L),initialValue=255,area=RAM,access=RW,size=Word,index=28]
504 [address=36,name=Present Position(L),initialValue=-1,area=RAM,access=R,size=Word,index=30]
0 [address=38,name=Present Speed(L),initialValue=-1,area=RAM,access=R,size=Word,index=32]
0 [address=40,name=Present Load(L),initialValue=-1,area=RAM,access=R,size=Word,index=34]
123 [address=42,name=Present Voltage,initialValue=-1,area=RAM,access=R,size=Byte,index=36]
31 [address=43,name=Present Temperature,initialValue=-1,area=RAM,access=R,size=Byte,index=37]
0 [address=44,name=Registered,initialValue=0,area=RAM,access=R,size=Byte,index=38]
0 [address=46,name=Moving,initialValue=0,area=RAM,access=R,size=Byte,index=39]
0 [address=47,name=Lock,initialValue=0,area=RAM,access=RW,size=Byte,index=40]
32 [address=48,name=Punch(L),initialValue=32,area=RAM,access=RW,size=Word,index=41]
Any thoughts?
Re: reading entire Dynamixel table in one packet
I have found reading too many values at once can be problematic.
Did you check the status of your RX, to make sure that it was valid?
Kurt
Re: reading entire Dynamixel table in one packet
I agree with Kurt, reading the whole thing every time is never consistently clean. Never looked into it cause it wasn't something done in normal use cases. Let us know if you figure it out.
Re: reading entire Dynamixel table in one packet
Just checking: what do you use to read data (which interface, and how do you print the resulting data?)
And do you check the CRC on the packets?
Re: reading entire Dynamixel table in one packet
I was working on the read instruction this morning, then I posted this thread right before I had to leave for a trip. About an hour into the drive, I realized my mistake. The extra bytes are the gaps in the addresses that the control table uses (10, 19-23, 45). So I need to get 50 bytes starting from 0, not 43, and index by address. Silly me. Sorry if I wasted anyone's brain cells.
I will write a unit test when I get home Sunday night to confirm the behavior. I suspect that this will work fine.
Re: reading entire Dynamixel table in one packet
Right now I'm just prototyping and writing core classes and methods. Getting the whole table may not be that practical. But if I find that I'm getting several values from every joint in the bot, I think it would be useful. We'll see.
I will also stress test it on a bus of several dynamixels and make sure that there are no errors and the values are consistent.
Re: reading entire Dynamixel table in one packet
So I have this working reliably and also understand why an "unclean" result can be read.
For my test, I scan the bus, and for each ID found, I read the control table and put it in a map by ID. Then I loop over the IDs, get the control table, compare them, then sleep for half a second. Rinse and repeat. For the comparison, I test equality for non-variable fields, and for variable fields (voltage, etc) I test that the values are within a certain delta of each other.
The reliability is a function of how long you are waiting in between the serial port send and receive. The necessary wait time will likely vary from setup to setup.
Reading one or two bytes is faster than reading the whole table. I can wait 10ms and get a good result every time with 1-2 bytes. When reading the whole table, the shorter wait time is long enough more than 99% of the time, but it is not 100% reliable. I will occasionally get incomplete tables. I've found that the serial port receive will report bytes even if the results are not complete. You have to give it enough time for all the packet bytes to get into the buffer.
For my tests, using a fast Core i7 Windows 8 machine and USB2Dynamixel, I get 100% reliability with a 20ms wait time between send and receive. I'll test the USB2AX on Windows tomorrow, maybe it will be faster. Linux may give different results as well...
I think reading the whole table will be beneficial. It's not unreasonable to want all these values every cycle, maybe even more:
- Present Position(L)
- Present Position(H)
- Present Speed(L)
- Present Speed(H)
- Present Load(L)
- Present Load(H)
- Present Voltage
- Present Temperature
- Moving
If the robot has 20 Dynamixels, that's 9 values (or 6 if reading words as one packet). That's minimum 120 reads every cycle if reading each one separately. It starts to add up.
Actually, I think the fastest way to do this is just read the chunk of the table from about address 36 - 46 in one packet (for AX). Most of the table is static data anyway.
Re: reading entire Dynamixel table in one packet
Sounds interesting and you are having some fun.
Sorry in advance if I stray a little off topic here. Also note that these are only my thoughts and other people may differ.
For me everything associated with communicating with the AX servos is a trade off.
That is with an infinite speed buss I would be grabbing all of the data all of the time, but it is not, so I need to prioritize.
With AX servos I have not had luck with the servos doing their own interpolation, to move from one poses position to the next, so instead I have the host do it. Which is what the standard bioloid library does. Many options here, but assume fixed time frame per iteration, example suppose 50 iterations per second. Again assume SYNC_WRITE, so the more precise that we can time these packets such that the checksum of each of these packets is sent out at 1/50th (20ms) of a second from the previous, the smoother the operation. Note: often we go for more iterations per second. I often use 100 per second, others have tried much higher.
So assuming the SYNC_WRITE includes all servos and just Goal Position, we have 60+ bytes being sent.
So then the next question is how much time do I have between each of these packets, to be able to reliably do reads from the servos. If worst case is 20ms per read, that already can screw up the timing, as we need at about 1ms to send out the SYNC_WRITE...
But lets assume maybe 5ms per read, then maybe we can do 3 reads per cycle...
So again need to prioritize. What data do I really need/want. For me, I don't care about Present Position/Speed/Moving as if my servo iteration loop is fast enough typically I don't move more than 1 or 2 units per iteration. The others maybe, as you can use them to detect overheating hopefully before damage, and maybe use to detect hitting something.. Likewise detect voltage to hopefully shut down before my battery dies. Do I need this from every servo on every servo iteration? Maybe not, maybe either round robin detect, or maybe again prioritize, depending on what each servo is doing. If I am lifting a leg maybe not important, but if lowering or moving forward more important.
Next question when you do a query, where is the time being spent? Host USB to send, Adapter USB to receive, AX Buss to send, Servo time to respond, AX Buss, Adapter USB to send, Host USB to receive... Note: there may be some other delays as well, example Arbotix Pro uses an FTDI chip, which then has to send UART data to processor... But simplify here, how much of the time is used up by USB versus all the rest. With some processors I have tried to measure this, by for example using an IO pin on the HOST, set high when I start the request and set low when the request finishes, and then use Logic Analyzer connected to this IO pin as well as AX buss. Also on some servo adapters, I have also setup to use IO pins to see when they do things as well...
One obvious place time is spent is on AX-BUSS, which you can reasonably calculate, but then how much time is spent by the servos?
To minimize some of this, you should check/set each of your servos AX_RETURN_DELAY_TIME to zero. Also note, if you watch the AX buss, you will often see servos sputtering out their response. That is don't assume that once it starts a response it will complete the whole packet one byte after another. Sometimes can be longer delays between bytes (like an ms)
What I have found is that a lot of this time is spent in the USB stuff. Which again is why some of the servo controllers give you ways to try to minimize how many USB transactions are needed. Examples: USB2AX has the SYNC_READ, Arobotix Pro, has BULK_READ, which again can read in values for multiple servos per USB transaction. Also even using these you may want to experiment as reading in more bytes per servo and/or more servos may result in more USB packets, as each packet has a fixed size...
Also found with some adapters, especially some FTDI chips, I could reduce the USB overhead if I could force the driver to flush(tcdrain) the data out after the host writes out a packet, but with other servo drivers, the addition of calling tcdrain added a large delay.
Again sorry if some of this was slightly off your topic. Looking forward to seeing what all you find out.
Side note, I was running into not reading everything reliably with one read earlier and played around with my AXServoTest program with my read and display all of the values for a servo, which currently looks like:
Code:
void PrintServoValues(void) {
#define MAX_BYTES_TO_READ 10
// Version that does one read ...
word wID;
word wEndReg;
word w;
if (!FGetNextCmdNum(&wID))
return;
if (!FGetNextCmdNum(&wEndReg))
wEndReg = 50;
// Lets try breaking this up into reads of up to 10 bytes each time...
word first_byte_index = 0;
word bytes_to_read = 10;
int result;
while (wEndReg) {
bytes_to_read = (wEndReg > MAX_BYTES_TO_READ)? MAX_BYTES_TO_READ : wEndReg;
dxl_set_txpacket_id(wID);
dxl_set_txpacket_instruction(INST_READ);
dxl_set_txpacket_parameter(0, first_byte_index);
dxl_set_txpacket_parameter(1, bytes_to_read);
dxl_set_txpacket_length(4);
dxl_txrx_packet();
result = dxl_get_result();
if (result != COMM_RXSUCCESS)
break; // abort this loop...
for (int i = 0; i < bytes_to_read; i++) {
printf("%d:%x ", first_byte_index + i, dxl_get_rxpacket_parameter(i));
if ((i%10) == 9)
printf("\n");
}
first_byte_index += bytes_to_read;
wEndReg -= bytes_to_read;
}
if (result != COMM_RXSUCCESS) {
printf("dxl bulk Read error: %d on index: %d\n", result, first_byte_index);
wEndReg += first_byte_index; // lets restore the last item
for (int i = first_byte_index; i < wEndReg; i++) {
printf("%d:", i);
w = dxl_read_byte(wID, i);
result = dxl_get_result();
if (result != COMM_RXSUCCESS) {
w = dxl_read_byte(wID, i);
result = dxl_get_result();
}
if (result == COMM_RXSUCCESS)
printf("%x ", w);
else
printf("** ");
if ((i%10) == 9)
Serial.println("");
Serial.flush(); // try to avoid any interrupts while processing.
}
}
printf("\n");
}
Kurt
Re: reading entire Dynamixel table in one packet
It sounds like you do:
Code:
sendcommand
sleep
readfullcommand
This is not a great way of doing it, as you discover. Instead, a more robust structure is:
Code:
sendcommand
while (!receivedenough) {
receivemore
}
parsecommand
In real code, it might look something like:
Code:
unsigned char buf[260]; // max possible cmd size
unsigned int ptr;
unsigned long timeout = millis() + 30; // or however long you will want to wait before giving up
while (millis() < timeout && (ptr < 4 || (ptr < buf[3]+2))) {
while (SERIAL.available() && (ptr < 4 || (ptr < buf[3] + 2))) {
buf[ptr++] = SERIAL.read();
if (ptr <= 2 && buf[ptr-1] != 0xff) {
// junk data found when expecting packet start!
ptr = 0;
}
}
}
if (ptr < 4 || (ptr != buf[3] + 2)) {
// timed out -- report error
}
calculate checksum, decode packet, etc
Re: reading entire Dynamixel table in one packet
Completely on topic and thanks for detailed response. You make a lot of good points. It's a lot to digest so I will think about it some more.
I'll be looking at this again tonight, plan is to test USB2AX and see if it is faster than USB2Dynamixel. I hope so.
One thing I've noticed is that the time it takes to get a response is variable. I can talk more about how to test for this if anyone is interested. I'm not sure where the variability is being introduced but as you say there are a lot of hoops the bytes have to jump through.
I think this topic is something that I will continue to learn about over time as I try to do more interesting things. For now I'm trying to get all of the instructions to work reliably, will optimize for different use cases later.
Re: reading entire Dynamixel table in one packet
Jwatte, I was thinking about what you are saying last night, just didn't get around to experimenting with it yet. Basically, instead of a long wait, have a short wait and keep reading new bytes until the full packet is received. I'm new to doing this kind of byte-level stuff over serial ports, so I'm learning as I go. I will try to implement this tonight if I have time.
Re: reading entire Dynamixel table in one packet
There are indeed a lot of things to consider if you want fast reading of a lot of data.
The way to read data that Jwatte suggests is indeed much needed to avoid waiting for nothing.
On the host, there are delays that can be created by the OS interrupting your task: on a platform with limited ressources (embedded computer) it might play a big role.
Then the driver has buffers and timeout to flush them when they are not full. This is a big problem with FTDI chips, that can be mitigated by setting the buffer size low and the timeout to the lowest value (still 1ms...) (see http://knowledge.digi.com/articles/K...-port-settings for example).
Then the USB layer sends the data in Bulk transfer mode (for FTDI an USB2AX alike), which has a lower priority than some other transfer modes, meaning that if you have other things transmitting a lot on the USB controller (a controller usually serves multiple USB ports, and in laptops some of these can be used internally), the lowly virtual serial packets will not move as fast as possible. So the more stuff you have on the USB ports of the machine, and the more talking they do, the worst your timing repeatability and overall latency will be. When checking for performances of other parts of the system, remove all other USB devices to cut the noise down.
The adapter itself can have some latency, for example the USB2AX parses all the trafic coming from the Host to see if there are commands aimed at it (it behaves like a limited Dynamixel device for some things), but this is comparatively small (maybe a few µs tops).
Sending stuff on the bus takes a known time: take the baud rate, and divid by 10, that's the number of bytes you can send/receive in a second (because the bus uses 8N1 encoding, which transmits 8 bytes, plus a start bit and 1 stop bit, so 10 signs per character).
So if you do not use the fastest baud rate, you're losing microseconds or maybe even miliseconds...
At 1Mbps, you can transmit 1 char in 10us, so a 60 bytes sync_write will take 0.6ms just to get out on the bus (if there aren't any other delays)...
Then the servo takes some time to parse and validate the packet and start queuing up the response data. This will depend on the lenght of the packet, and the command itself (some take more time to be processed). This usually takes about 10 to 25us, maybe more on very complex packets (reading the full table might be in that category).
During the time it responds, the AX servo can stop for a moment between two char while the motor control task is done. This can add up to 130us delay to the response of the servo. This task is done once every miliseconds, so it can only affect a servo once every ms.
Then the data goes the other way, and can have the same problems. The USB2AX transmits data from the servo to the host directly, as fast as possible, with as little latency as possible. The FTDI chips might sit on data for some time, waiting for more to come...
Then data goes back up to the host, lingering in a buffer of the driver/OS, until your application finally reads it.
The way this buffer is accessed can make a lot of differences depending on the OS and libs used. Checking the number of bytes available might take as long as to get one (and manage the error if there were none), getting one might take as long as getting 20... then reading that data byte by byte might take some more time (I'm looking at you Python)... check your code for preformance, you might be able to gain some more here, mostly on slow embedded plateforms.
As mentionned, some precautions have to be taken on the servo side to make sure you're getting acceptable performances.
- Return delay time : set to zero. It's an artificial delay that the servo can wait before responding in case the thing that is talking to it is very slow. And it's set to 250 by default (1ms delay before responding, the support website http://support.robotis.com/en/ is still wrong about saying it's 2us per unit: tests show it's 4us per unit).
- baudrate : set to 1mbps. Technically you could push to 2mpbs, as jwatte does, but most adapter do not support such speed (USB2dynamixel and USB2AX included).
- Status return level : if you plan on ignoring the return packet when sending write_data commands, then just tell the servo not to send them, that's something les to think about. There are no status packet when sending broadcast commands (like sync_write) so it might not matter to you.
Check your cables as the timeout delays will kill any perf you gained elsewhere tenfold, plus you will get unreliable results.
Then in your code, you can optimize the way access to the servo is done.
For writing positions (writing values at the same spot on multiple servos), just sync_write. Unless you want to check that every transaction has not raised any error, in which case abandon the idea of doing things fast...
For reading stuff, the best thing right now is to use the sync_read of the USB2AX or the similar "one packet to read multiple servos" on the Arbotix, or maybe others. The time it takes to read something from a servo is very largely dominated by the USB transaction, which in itself does not vary noticably based on the quantity of data exchanged. Reading 1 byte or the whole table will have the same USB overhead, and it's still the same for one sync_read operation which can read many bytes from 25 servos.
A loop that reads and writes from every of the sevos of a robot can take very little time with sync_write + sync_read. All the more if you use sync_read asynchronously (ie send the packet, go do something else, then come back to read the result). If you do this: send in one big chunk the sync_write and sync_read packets (can save some more compared to writing the two packets one after the other, and even more compared to writing byte by byte), then go do something else (like computing IK from the data collected with the previous sync_read), and when you're finished start again, then you can effectively take the time lost with communications down to 2~3ms, with a loop time of maybe 6 to 10ms for a full robot.
Some tests I did a few years ago with the USB2AX and the slightly updated Dynamixel SDK : https://docs.google.com/document/d/1...it?usp=sharing
You can also use multiple lines in parallele. And mixed with the previous trick, you can read a HUGE number of servos in very little time... On the USB2AX reading 6 bytes and writing 4 each loop, it looked like 8~12 servos per line would be the sweet spot.
In my Xachikoma robot I used 2 lines (it was also easier to wire that way), and the Poppy Humanoid too ( poppy-project.org ).
If you are using the USB2Dynamixel however, you can not get to such fast loops. But you can still do some fun tricks to speed things up.
For reading things from many servos, the only thing that you actually have to avoid is the communication on the bus getting corrupted by having the servo and USB2dynamixel try to talk on the bus at the same time. Anything else if fair game. And since you know that the resonse of the servo will have passed on the bus in less than Host to device delay + device write to bus delay + servo parse delay + servo resonse delay + servo motor control delay...
By sending read commands one after the other with a small delay in between, and letting the return packets pile up in the driver's input buffer before reading them all at once at the end, you can interleave the USB overheads, effectively decreasing the time it took to get all the data out of the sevos.
The delay should be long enough to make sure the data is out, but not long enough for the return packet to be back all the way up to the host. I did it with about 0.4ms successfully on a Beagleboard xM running python code, so not the fastest thing around.
When reading all the packets, you just have to match each with the servo it concerns and know what register adress were queried. It might require to move away from the "one servo - one command sent - one response received" paradigm though.
There, that's about most of what I have gathered on the subject. I really should have written that low-latency communication guide I've been thinking about for years... :/
Re: reading entire Dynamixel table in one packet
Very interesting topic indeed. Xevel's post make this thread sticky-worthy. Looking forward to read more about your findings.
Re: reading entire Dynamixel table in one packet
Wow, Xevel my mind is blown by your post. So much there to digest and comment on, when I have more time. I have to say, this community may be small these days, but I am truly inspired by the depth of knowledge and willingness to share displayed here.
Even though I will likely abandon the "naive" approach of waiting N milliseconds before reading, I decided to still go ahead and do some performance testing with that approach using the USB2AX. Again, this is AX full control table read (50 bytes) at 1Mbps.
Here is some code just so that it's clear what I'm talking about:
Code:
public synchronized StatusPacket sendInstruction(InstructionPacket instruction) throws DynamixelException {
send(instruction.getPacketBytes());
sleep(receiveWaitTimeMillis);
return new StatusPacket(instruction, receive());
}
With the USB2Dynamixel, I was using a 20ms timeout, and I was getting greater than 99.99% reliability, but I was getting an occasional failure on a longer run that I tried.
It turns out that the USB2AX is ridiculously fast. Ludicrous speed. Maybe it's more accurate to say the USB2Dynamixel is ridiculously slow, but the speed difference is dramatic. With the USB2AX, I was able to run the same benchmark that was 99.99% reliable with the USB2Dynamixel at 20ms and get 100% reliability from the USB2AX at 15ms, then 10ms, then 5ms, then 2ms. I don't really feel its necessary to go any further with this approach. Point has been made.
I also wrote code to use the method jwatte described, and I did a little testing, but I don't have enough data to share yet. I'm cobbling together some new classes in my mini benchmarking suite to be able to run various types of reads over a period of time and calculate average total read time. Should have that testing wrapped up tomorrow night, and I'll have a good idea of exactly how fast these devices are in different scenarios.
I may need to go to nanoseconds to benchmark the USB2AX! Beautiful piece of hardware.
When I have a bigger assembly going, I will test the SYNC_READ as well. I'm curious to see how that performs vs reading each individually. Also, one final note since it's been mentioned a couple times, Return Delay Time is still default (250), will change that to 0 when I implement WRITE_DATA, probably tomorrow. Also curious to see if that ends up mattering much in terms of total read time.
Thanks again to all who have commented. You folks are brilliant.
Re: reading entire Dynamixel table in one packet
You're welcome :)
Quote:
Originally Posted by
_ADAM_
Also, one final note since it's been mentioned a couple times, Return Delay Time is still default (250), will change that to 0 when I implement WRITE_DATA, probably tomorrow. Also curious to see if that ends up mattering much in terms of total read time.
Yeah, I do think that removing the 1ms delay in the servo response might help xD
To be fair to the USB2Dynamixel, I think that you could have gotten better results with it by adjusting the driver settings. With an internal timeout and buffers set to the minimum, I used to get pretty reliable reads at <2ms. But then with the USB2AX I could still go lower than that...
Re: reading entire Dynamixel table in one packet
Didn't get too much time to spend on this tonight, but I did run some timing tests using the new sendInstruction implementation.
Test setup is three AX-12+ on a bus with 1Mbps baud rate and reading the entire 50 byte table. Return Delay Time is 0.
Here is a receive code snippet:
Code:
byte[] status = new byte[instruction.getReturnPacketLength()];
int bytesReceived = 0;
long startTime = System.nanoTime();
while (bytesReceived < status.length && (System.nanoTime() - startTime) < receiveTimeoutNanos) {
byte[] bytes = receive();
if (bytes != null) {
System.arraycopy(bytes, 0, status, bytesReceived, bytes.length);
bytesReceived += bytes.length;
}
}
I now include the duration in nanos, which I need for the timeout anway, in the status return object. This makes it easy to benchmark.
Here are the numbers.
USB2AX
fastestTimeMillis = 0.742871
slowestTimeMillis = 1.263982
averageTimeMillis = 1.0172655466666665
USB2Dynamixel
fastestTimeMillis = 1.008165
slowestTimeMillis = 16.386709
averageTimeMillis = 8.790776
These numbers validate the times that I had settled on using the send/wait/receive approach (2 millis for USB2AX and ~20 millis for USB2Dynamixel). And as Xevel mentioned, I should also state that the USB2Dynamixel driver has not been tuned.
Re: reading entire Dynamixel table in one packet
Here are the final numbers I am going to post on this for now. I'm moving on to other things, but I still want to benchmark SYNC_READ vs sequential reads when I have an actual robot to work with.
Setup: Windows 8, USB2AX, three AX-12+ on the bus, 1Mbps, Return Delay Time is 0.
address = 46, numAddresses = 1
fastestTimeMillis = 0.065299
slowestTimeMillis = 0.4827
averageTimeMillis = 0.3515166666666666
address = 30, numAddresses = 2
fastestTimeMillis = 0.292949
slowestTimeMillis = 0.534427
averageTimeMillis = 0.3676586999999999
address = 36, numAddresses = 8
fastestTimeMillis = 0.380014
slowestTimeMillis = 0.600494
averageTimeMillis = 0.4562476633333337
address = 0, numAddresses = 50
fastestTimeMillis = 0.910088
slowestTimeMillis = 1.144908
averageTimeMillis = 0.9971423900000002
As you can see, it's fast, but it gets slower the more data you read. There is not a big difference between reading, one, two or all eigth "Present" values. Reading the whole table takes more than twice as long to read than reading the eight Present values. I don't see any reason to read the whole table in one packet unless it's for logging or displaying purposes. Reading the chunk of Present values gives the most bang for the buck.
Reading multiple values as single bytes doesn't make much sense either. Reading the eight Present value addresses individually would take six times as long as reading them at once.
Also, it's interesting that reading a single byte can be crazy fast sometimes (0.065 millis!!).
I'm happy with these numbers. If they generally hold for a larger bus, I think it's plenty fast enough for communication speed to not affect robot performance. I was definitely a bit concerned with the times I was seeing for the USB2Dynamixel.
I will revisit later when I have something resembling a robot.
Re: reading entire Dynamixel table in one packet
Something is wrong in your test code, 65us to read a value is not possible.
Just to output the read_data packet (8 bytes) on the bus at 1mbps already takes 80us.
Could be that the function returned prematurely, or that the way time is measured is not accurate, or that there is something left in the input buffer... I did not really analyze the details of your piece of code so I don't know for sure, sorry.
Actually, a note on latency benchmarking: personally I find that the only reliable way to get acurate timing of fast operations is to measure them externally.
When I benchmark a piece of communication code like that, I look at the bus with a logic analyzer (I upgraded to the Saleae Logic Pro 16 at the beginning of the year, it's magical <3 ) and measure the actual time elapsed between the beginning of consecutive packets. But anyway, your test, even if quantitatively inacurate, led you to a valid conclusion.
Re: reading entire Dynamixel table in one packet
I don't think it's the my code per se, but there could be a glitch in the underlying nano time implementation. I added some logging, and it's only the very first read of the run that records a super fast time. Also, I get a valid response packet.
If I leave the first iteration out of the averaging, the single address read average nudges up a bit (~9us).
address = 46, numAddresses = 1
fastestTimeMillis = 0.039435
slowestTimeMillis = 0.58257
averageTimeMillis = 0.3609035622895622
address = 30, numAddresses = 2
fastestTimeMillis = 0.290133
slowestTimeMillis = 0.498576
averageTimeMillis = 0.3644584646464648
address = 36, numAddresses = 8
fastestTimeMillis = 0.349541
slowestTimeMillis = 0.624309
averageTimeMillis = 0.45329809090909085
address = 0, numAddresses = 50
fastestTimeMillis = 0.894724
slowestTimeMillis = 1.152591
averageTimeMillis = 1.010406363636364
1 Attachment(s)
Re: reading entire Dynamixel table in one packet
Good information,
FYI - I don't leave home without my Saleae Logic analyzer either :D
Attachment 6593
Helps to verify things, like above shows that simple read of 1 byte, the AX-BUSS takes about .176ms. Plus then you need to add in all of the additional overhead of USB...
On Odroid C1, if I run test to read in all registers of a servo, I currently read in 10 bytes at a time. If I measure the timings between the start of each read, I see something like 1.2ms... But the actual time on AX-Buss maybe is .2ms... Some of this is overhead in my code, some is libraries...
Re: reading entire Dynamixel table in one packet
Quote:
Originally Posted by
KurtEck
Good information,
FYI - I don't leave home without my Saleae Logic analyzer either :D
I just had an image of you at the grocery store with your Saleae in your hand.... :D
