reading entire Dynamixel table in one packet

[_ADAM_]

[Edit: This thread has evolved beyond this initial problem into a discussion of general Dynamixel bus read performance and strategies.]

Hi everyone,

I'm seeing some strange behavior, wanted to see if this is something that others have encountered before as well. I'm seeing extra data bytes in the params when I read the whole table at once. I don't have any problems reading one or two bytes at a time.

Here is some output of reading an AX-12+ table one address at a time:

12    [address=0,name=Model Number(L),initialValue=12,area=EEPROM,access=R,size=Word,index=0]
0    [address=1,name=Model Number(H),initialValue=0,area=EEPROM,access=R,size=Word,index=1]
24    [address=2,name=Version of Firmware,initialValue=-1,area=EEPROM,access=R,size=Byte,index=2]
3    [address=3,name=ID,initialValue=1,area=EEPROM,access=RW,size=Byte,index=3]
1    [address=4,name=Baud Rate,initialValue=1,area=EEPROM,access=RW,size=Byte,index=4]
0    [address=5,name=Return Delay Time,initialValue=250,area=EEPROM,access=RW,size=Byte,index=5]
200    [address=6,name=CW Angle Limit(L),initialValue=0,area=EEPROM,access=RW,size=Word,index=6]
0    [address=7,name=CW Angle Limit(H),initialValue=0,area=EEPROM,access=RW,size=Word,index=7]
55    [address=8,name=CCW Angle Limit(L),initialValue=255,area=EEPROM,access=RW,size=Word,index=8]
3    [address=9,name=CCW Angle Limit(H),initialValue=3,area=EEPROM,access=RW,size=Word,index=9]
70    [address=11,name=Highest Limit Temperature,initialValue=70,area=EEPROM,access=RW,size=Byte,index=10]
60    [address=12,name=Lowest Limit Voltage,initialValue=60,area=EEPROM,access=RW,size=Byte,index=11]
140    [address=13,name=Highest Limit Voltage,initialValue=140,area=EEPROM,access=RW,size=Byte,index=12]
255    [address=14,name=Max Torque(L),initialValue=255,area=EEPROM,access=RW,size=Word,index=13]
3    [address=15,name=Max Torque(H),initialValue=3,area=EEPROM,access=RW,size=Word,index=14]
2    [address=16,name=Status Return Level,initialValue=2,area=EEPROM,access=RW,size=Byte,index=15]
36    [address=17,name=Alarm LED,initialValue=36,area=EEPROM,access=RW,size=Byte,index=16]
36    [address=18,name=Alarm Shutdown,initialValue=36,area=EEPROM,access=RW,size=Byte,index=17]
0    [address=24,name=Torque Enable,initialValue=0,area=RAM,access=RW,size=Byte,index=18]
0    [address=25,name=LED,initialValue=0,area=RAM,access=RW,size=Byte,index=19]
1    [address=26,name=CW Compliance Margin,initialValue=1,area=RAM,access=RW,size=Byte,index=20]
1    [address=27,name=CCW Compliance Margin,initialValue=1,area=RAM,access=RW,size=Byte,index=21]
32    [address=28,name=CW Compliance Slope,initialValue=32,area=RAM,access=RW,size=Byte,index=22]
32    [address=29,name=CCW Compliance Slope,initialValue=32,area=RAM,access=RW,size=Byte,index=23]
248    [address=30,name=Goal Position(L),initialValue=-1,area=RAM,access=RW,size=Word,index=24]
1    [address=31,name=Goal Position(H),initialValue=-1,area=RAM,access=RW,size=Word,index=25]
0    [address=32,name=Moving Speed(L),initialValue=-1,area=RAM,access=RW,size=Word,index=26]
0    [address=33,name=Moving Speed(H),initialValue=-1,area=RAM,access=RW,size=Word,index=27]
255    [address=34,name=Torque Limit(L),initialValue=255,area=RAM,access=RW,size=Word,index=28]
3    [address=35,name=Torque Limit(H),initialValue=3,area=RAM,access=RW,size=Word,index=29]
248    [address=36,name=Present Position(L),initialValue=-1,area=RAM,access=R,size=Word,index=30]
1    [address=37,name=Present Position(H),initialValue=-1,area=RAM,access=R,size=Word,index=31]
0    [address=38,name=Present Speed(L),initialValue=-1,area=RAM,access=R,size=Word,index=32]
0    [address=39,name=Present Speed(H),initialValue=-1,area=RAM,access=R,size=Word,index=33]
0    [address=40,name=Present Load(L),initialValue=-1,area=RAM,access=R,size=Word,index=34]
0    [address=41,name=Present Load(H),initialValue=-1,area=RAM,access=R,size=Word,index=35]
123    [address=42,name=Present Voltage,initialValue=-1,area=RAM,access=R,size=Byte,index=36]
28    [address=43,name=Present Temperature,initialValue=-1,area=RAM,access=R,size=Byte,index=37]
0    [address=44,name=Registered,initialValue=0,area=RAM,access=R,size=Byte,index=38]
0    [address=46,name=Moving,initialValue=0,area=RAM,access=R,size=Byte,index=39]
0    [address=47,name=Lock,initialValue=0,area=RAM,access=RW,size=Byte,index=40]
32    [address=48,name=Punch(L),initialValue=32,area=RAM,access=RW,size=Word,index=41]
0    [address=49,name=Punch(H),initialValue=0,area=RAM,access=RW,size=Word,index=42]

Here is some output of reading all 43 addresses of an AX-12+ table at one time (comments below):

12    [address=0,name=Model Number(L),initialValue=12,area=EEPROM,access=R,size=Word,index=0]
0    [address=1,name=Model Number(H),initialValue=0,area=EEPROM,access=R,size=Word,index=1]
24    [address=2,name=Version of Firmware,initialValue=-1,area=EEPROM,access=R,size=Byte,index=2]
3    [address=3,name=ID,initialValue=1,area=EEPROM,access=RW,size=Byte,index=3]
1    [address=4,name=Baud Rate,initialValue=1,area=EEPROM,access=RW,size=Byte,index=4]
0    [address=5,name=Return Delay Time,initialValue=250,area=EEPROM,access=RW,size=Byte,index=5]
200    [address=6,name=CW Angle Limit(L),initialValue=0,area=EEPROM,access=RW,size=Word,index=6]
0    [address=7,name=CW Angle Limit(H),initialValue=0,area=EEPROM,access=RW,size=Word,index=7]
55    [address=8,name=CCW Angle Limit(L),initialValue=255,area=EEPROM,access=RW,size=Word,index=8]
3    [address=9,name=CCW Angle Limit(H),initialValue=3,area=EEPROM,access=RW,size=Word,index=9]
131    [address=11,name=Highest Limit Temperature,initialValue=70,area=EEPROM,access=RW,size=Byte,index=10]
70    [address=12,name=Lowest Limit Voltage,initialValue=60,area=EEPROM,access=RW,size=Byte,index=11]
60    [address=13,name=Highest Limit Voltage,initialValue=140,area=EEPROM,access=RW,size=Byte,index=12]
140    [address=14,name=Max Torque(L),initialValue=255,area=EEPROM,access=RW,size=Word,index=13]
255    [address=15,name=Max Torque(H),initialValue=3,area=EEPROM,access=RW,size=Word,index=14]
3    [address=16,name=Status Return Level,initialValue=2,area=EEPROM,access=RW,size=Byte,index=15]
2    [address=17,name=Alarm LED,initialValue=36,area=EEPROM,access=RW,size=Byte,index=16]
36    [address=18,name=Alarm Shutdown,initialValue=36,area=EEPROM,access=RW,size=Byte,index=17]
36    [address=24,name=Torque Enable,initialValue=0,area=RAM,access=RW,size=Byte,index=18]
0    [address=25,name=LED,initialValue=0,area=RAM,access=RW,size=Byte,index=19]
44    [address=26,name=CW Compliance Margin,initialValue=1,area=RAM,access=RW,size=Byte,index=20]
0    [address=27,name=CCW Compliance Margin,initialValue=1,area=RAM,access=RW,size=Byte,index=21]
206    [address=28,name=CW Compliance Slope,initialValue=32,area=RAM,access=RW,size=Byte,index=22]
3    [address=29,name=CCW Compliance Slope,initialValue=32,area=RAM,access=RW,size=Byte,index=23]
0    [address=30,name=Goal Position(L),initialValue=-1,area=RAM,access=RW,size=Word,index=24]
0    [address=31,name=Goal Position(H),initialValue=-1,area=RAM,access=RW,size=Word,index=25]
1    [address=32,name=Moving Speed(L),initialValue=-1,area=RAM,access=RW,size=Word,index=26]
1    [address=33,name=Moving Speed(H),initialValue=-1,area=RAM,access=RW,size=Word,index=27]
32    [address=34,name=Torque Limit(L),initialValue=255,area=RAM,access=RW,size=Word,index=28]
32    [address=35,name=Torque Limit(H),initialValue=3,area=RAM,access=RW,size=Word,index=29]
248    [address=36,name=Present Position(L),initialValue=-1,area=RAM,access=R,size=Word,index=30]
1    [address=37,name=Present Position(H),initialValue=-1,area=RAM,access=R,size=Word,index=31]
0    [address=38,name=Present Speed(L),initialValue=-1,area=RAM,access=R,size=Word,index=32]
0    [address=39,name=Present Speed(H),initialValue=-1,area=RAM,access=R,size=Word,index=33]
255    [address=40,name=Present Load(L),initialValue=-1,area=RAM,access=R,size=Word,index=34]
3    [address=41,name=Present Load(H),initialValue=-1,area=RAM,access=R,size=Word,index=35]
248    [address=42,name=Present Voltage,initialValue=-1,area=RAM,access=R,size=Byte,index=36]
1    [address=43,name=Present Temperature,initialValue=-1,area=RAM,access=R,size=Byte,index=37]
0    [address=44,name=Registered,initialValue=0,area=RAM,access=R,size=Byte,index=38]
0    [address=46,name=Moving,initialValue=0,area=RAM,access=R,size=Byte,index=39]
0    [address=47,name=Lock,initialValue=0,area=RAM,access=RW,size=Byte,index=40]
0    [address=48,name=Punch(L),initialValue=32,area=RAM,access=RW,size=Word,index=41]
123    [address=49,name=Punch(H),initialValue=0,area=RAM,access=RW,size=Word,index=42]

You can see that the first nine values are the same, then there is a 131 thrown in there. Another nine values match (but shifted down), and then there is this sequence inserted:

44    [address=26,name=CW Compliance Margin,initialValue=1,area=RAM,access=RW,size=Byte,index=20]
0    [address=27,name=CCW Compliance Margin,initialValue=1,area=RAM,access=RW,size=Byte,index=21]
206    [address=28,name=CW Compliance Slope,initialValue=32,area=RAM,access=RW,size=Byte,index=22]
3    [address=29,name=CCW Compliance Slope,initialValue=32,area=RAM,access=RW,size=Byte,index=23]
0    [address=30,name=Goal Position(L),initialValue=-1,area=RAM,access=RW,size=Word,index=24]

Then it matches again, but of course everything is shifted, and while I get 43 bytes, the last byte I get is address 42, not 49.

It's not that the data is getting randomly garbled either, because I get this byte sequence every time, even after power/port cycling. If I try a different AX-12+, I get the bytes in the same positions but different values. It's interesting that it's so consistent. I don't see anything in the documentation about this.

FYI, here is some data where I get a single address if it's a byte and two addresses if it's a word (only low address is printed but with combined word value):

12    [address=0,name=Model Number(L),initialValue=12,area=EEPROM,access=R,size=Word,index=0]
24    [address=2,name=Version of Firmware,initialValue=-1,area=EEPROM,access=R,size=Byte,index=2]
3    [address=3,name=ID,initialValue=1,area=EEPROM,access=RW,size=Byte,index=3]
1    [address=4,name=Baud Rate,initialValue=1,area=EEPROM,access=RW,size=Byte,index=4]
0    [address=5,name=Return Delay Time,initialValue=250,area=EEPROM,access=RW,size=Byte,index=5]
200    [address=6,name=CW Angle Limit(L),initialValue=0,area=EEPROM,access=RW,size=Word,index=6]
823    [address=8,name=CCW Angle Limit(L),initialValue=255,area=EEPROM,access=RW,size=Word,index=8]
70    [address=11,name=Highest Limit Temperature,initialValue=70,area=EEPROM,access=RW,size=Byte,index=10]
60    [address=12,name=Lowest Limit Voltage,initialValue=60,area=EEPROM,access=RW,size=Byte,index=11]
140    [address=13,name=Highest Limit Voltage,initialValue=140,area=EEPROM,access=RW,size=Byte,index=12]
1023    [address=14,name=Max Torque(L),initialValue=255,area=EEPROM,access=RW,size=Word,index=13]
2    [address=16,name=Status Return Level,initialValue=2,area=EEPROM,access=RW,size=Byte,index=15]
36    [address=17,name=Alarm LED,initialValue=36,area=EEPROM,access=RW,size=Byte,index=16]
36    [address=18,name=Alarm Shutdown,initialValue=36,area=EEPROM,access=RW,size=Byte,index=17]
0    [address=24,name=Torque Enable,initialValue=0,area=RAM,access=RW,size=Byte,index=18]
0    [address=25,name=LED,initialValue=0,area=RAM,access=RW,size=Byte,index=19]
1    [address=26,name=CW Compliance Margin,initialValue=1,area=RAM,access=RW,size=Byte,index=20]
1    [address=27,name=CCW Compliance Margin,initialValue=1,area=RAM,access=RW,size=Byte,index=21]
32    [address=28,name=CW Compliance Slope,initialValue=32,area=RAM,access=RW,size=Byte,index=22]
32    [address=29,name=CCW Compliance Slope,initialValue=32,area=RAM,access=RW,size=Byte,index=23]
504    [address=30,name=Goal Position(L),initialValue=-1,area=RAM,access=RW,size=Word,index=24]
0    [address=32,name=Moving Speed(L),initialValue=-1,area=RAM,access=RW,size=Word,index=26]
1023    [address=34,name=Torque Limit(L),initialValue=255,area=RAM,access=RW,size=Word,index=28]
504    [address=36,name=Present Position(L),initialValue=-1,area=RAM,access=R,size=Word,index=30]
0    [address=38,name=Present Speed(L),initialValue=-1,area=RAM,access=R,size=Word,index=32]
0    [address=40,name=Present Load(L),initialValue=-1,area=RAM,access=R,size=Word,index=34]
123    [address=42,name=Present Voltage,initialValue=-1,area=RAM,access=R,size=Byte,index=36]
31    [address=43,name=Present Temperature,initialValue=-1,area=RAM,access=R,size=Byte,index=37]
0    [address=44,name=Registered,initialValue=0,area=RAM,access=R,size=Byte,index=38]
0    [address=46,name=Moving,initialValue=0,area=RAM,access=R,size=Byte,index=39]
0    [address=47,name=Lock,initialValue=0,area=RAM,access=RW,size=Byte,index=40]
32    [address=48,name=Punch(L),initialValue=32,area=RAM,access=RW,size=Word,index=41]

Any thoughts?

[KurtEck]

I have found reading too many values at once can be problematic.

Did you check the status of your RX, to make sure that it was valid?

Kurt

[KevinO]

I agree with Kurt, reading the whole thing every time is never consistently clean. Never looked into it cause it wasn't something done in normal use cases. Let us know if you figure it out.

[Xevel]

Just checking: what do you use to read data (which interface, and how do you print the resulting data?)

And do you check the CRC on the packets?

[_ADAM_]

I was working on the read instruction this morning, then I posted this thread right before I had to leave for a trip. About an hour into the drive, I realized my mistake. The extra bytes are the gaps in the addresses that the control table uses (10, 19-23, 45). So I need to get 50 bytes starting from 0, not 43, and index by address. Silly me. Sorry if I wasted anyone's brain cells.

I will write a unit test when I get home Sunday night to confirm the behavior. I suspect that this will work fine.

[_ADAM_]

Right now I'm just prototyping and writing core classes and methods. Getting the whole table may not be that practical. But if I find that I'm getting several values from every joint in the bot, I think it would be useful. We'll see.

I will also stress test it on a bus of several dynamixels and make sure that there are no errors and the values are consistent.

[_ADAM_]

So I have this working reliably and also understand why an "unclean" result can be read.

For my test, I scan the bus, and for each ID found, I read the control table and put it in a map by ID. Then I loop over the IDs, get the control table, compare them, then sleep for half a second. Rinse and repeat. For the comparison, I test equality for non-variable fields, and for variable fields (voltage, etc) I test that the values are within a certain delta of each other.

The reliability is a function of how long you are waiting in between the serial port send and receive. The necessary wait time will likely vary from setup to setup.

Reading one or two bytes is faster than reading the whole table. I can wait 10ms and get a good result every time with 1-2 bytes. When reading the whole table, the shorter wait time is long enough more than 99% of the time, but it is not 100% reliable. I will occasionally get incomplete tables. I've found that the serial port receive will report bytes even if the results are not complete. You have to give it enough time for all the packet bytes to get into the buffer.

For my tests, using a fast Core i7 Windows 8 machine and USB2Dynamixel, I get 100% reliability with a 20ms wait time between send and receive. I'll test the USB2AX on Windows tomorrow, maybe it will be faster. Linux may give different results as well...

I think reading the whole table will be beneficial. It's not unreasonable to want all these values every cycle, maybe even more:

• Present Position(L)
• Present Position(H)
• Present Speed(L)
• Present Speed(H)
• Present Load(L)
• Present Load(H)
• Present Voltage
• Present Temperature
• Moving

If the robot has 20 Dynamixels, that's 9 values (or 6 if reading words as one packet). That's minimum 120 reads every cycle if reading each one separately. It starts to add up.

Actually, I think the fastest way to do this is just read the chunk of the table from about address 36 - 46 in one packet (for AX). Most of the table is static data anyway.

[KurtEck]

Sounds interesting and you are having some fun.

Sorry in advance if I stray a little off topic here. Also note that these are only my thoughts and other people may differ.

For me everything associated with communicating with the AX servos is a trade off.

That is with an infinite speed buss I would be grabbing all of the data all of the time, but it is not, so I need to prioritize.

With AX servos I have not had luck with the servos doing their own interpolation, to move from one poses position to the next, so instead I have the host do it. Which is what the standard bioloid library does. Many options here, but assume fixed time frame per iteration, example suppose 50 iterations per second. Again assume SYNC_WRITE, so the more precise that we can time these packets such that the checksum of each of these packets is sent out at 1/50th (20ms) of a second from the previous, the smoother the operation. Note: often we go for more iterations per second. I often use 100 per second, others have tried much higher.

So assuming the SYNC_WRITE includes all servos and just Goal Position, we have 60+ bytes being sent.

So then the next question is how much time do I have between each of these packets, to be able to reliably do reads from the servos. If worst case is 20ms per read, that already can screw up the timing, as we need at about 1ms to send out the SYNC_WRITE...
But lets assume maybe 5ms per read, then maybe we can do 3 reads per cycle...

So again need to prioritize. What data do I really need/want. For me, I don't care about Present Position/Speed/Moving as if my servo iteration loop is fast enough typically I don't move more than 1 or 2 units per iteration. The others maybe, as you can use them to detect overheating hopefully before damage, and maybe use to detect hitting something.. Likewise detect voltage to hopefully shut down before my battery dies. Do I need this from every servo on every servo iteration? Maybe not, maybe either round robin detect, or maybe again prioritize, depending on what each servo is doing. If I am lifting a leg maybe not important, but if lowering or moving forward more important.

Next question when you do a query, where is the time being spent? Host USB to send, Adapter USB to receive, AX Buss to send, Servo time to respond, AX Buss, Adapter USB to send, Host USB to receive... Note: there may be some other delays as well, example Arbotix Pro uses an FTDI chip, which then has to send UART data to processor... But simplify here, how much of the time is used up by USB versus all the rest. With some processors I have tried to measure this, by for example using an IO pin on the HOST, set high when I start the request and set low when the request finishes, and then use Logic Analyzer connected to this IO pin as well as AX buss. Also on some servo adapters, I have also setup to use IO pins to see when they do things as well...

One obvious place time is spent is on AX-BUSS, which you can reasonably calculate, but then how much time is spent by the servos?
To minimize some of this, you should check/set each of your servos AX_RETURN_DELAY_TIME to zero. Also note, if you watch the AX buss, you will often see servos sputtering out their response. That is don't assume that once it starts a response it will complete the whole packet one byte after another. Sometimes can be longer delays between bytes (like an ms)

What I have found is that a lot of this time is spent in the USB stuff. Which again is why some of the servo controllers give you ways to try to minimize how many USB transactions are needed. Examples: USB2AX has the SYNC_READ, Arobotix Pro, has BULK_READ, which again can read in values for multiple servos per USB transaction. Also even using these you may want to experiment as reading in more bytes per servo and/or more servos may result in more USB packets, as each packet has a fixed size...

Also found with some adapters, especially some FTDI chips, I could reduce the USB overhead if I could force the driver to flush(tcdrain) the data out after the host writes out a packet, but with other servo drivers, the addition of calling tcdrain added a large delay.

Again sorry if some of this was slightly off your topic. Looking forward to seeing what all you find out.

Side note, I was running into not reading everything reliably with one read earlier and played around with my AXServoTest program with my read and display all of the values for a servo, which currently looks like:

void PrintServoValues(void) {
#define MAX_BYTES_TO_READ 10
    // Version that does one read ...
    word wID;
    word wEndReg;
    word w;
    if (!FGetNextCmdNum(&wID))
        return;
    if (!FGetNextCmdNum(&wEndReg))
        wEndReg = 50;

    // Lets try breaking this up into reads of up to 10 bytes each time... 
    word first_byte_index = 0;
    word bytes_to_read = 10;
    int result;  
    while (wEndReg) {
        bytes_to_read = (wEndReg > MAX_BYTES_TO_READ)? MAX_BYTES_TO_READ : wEndReg;
        
        dxl_set_txpacket_id(wID);
        dxl_set_txpacket_instruction(INST_READ);
        dxl_set_txpacket_parameter(0, first_byte_index);
        dxl_set_txpacket_parameter(1, bytes_to_read);
        dxl_set_txpacket_length(4);
        dxl_txrx_packet();
        result = dxl_get_result();  
           
        if (result != COMM_RXSUCCESS) 
            break;  // abort this loop...
        for (int i = 0; i < bytes_to_read; i++) {
            printf("%d:%x ", first_byte_index + i, dxl_get_rxpacket_parameter(i));
            if ((i%10) == 9)
                printf("\n");
        }
        first_byte_index += bytes_to_read;
        wEndReg -= bytes_to_read;
    }    
        
    if (result != COMM_RXSUCCESS)  {
        printf("dxl bulk Read error: %d on index: %d\n", result, first_byte_index);
        wEndReg += first_byte_index;        // lets restore the last item
        for (int i = first_byte_index; i < wEndReg; i++) {
            printf("%d:", i);
            w = dxl_read_byte(wID, i);
            result = dxl_get_result();
            if (result != COMM_RXSUCCESS) {
                w = dxl_read_byte(wID, i);
                result = dxl_get_result();
            }
            if (result == COMM_RXSUCCESS)
                printf("%x ", w);
            else 
                printf("** ");
            if ((i%10) == 9)
                Serial.println("");
            Serial.flush();  // try to avoid any interrupts while processing.
        }    
    }  
    printf("\n");
}

Kurt

[jwatte]

It sounds like you do:

sendcommand
sleep
readfullcommand

This is not a great way of doing it, as you discover. Instead, a more robust structure is:

sendcommand
while (!receivedenough) {
  receivemore
}
parsecommand

In real code, it might look something like:

unsigned char buf[260]; // max possible cmd size
unsigned int ptr;
unsigned long timeout = millis() + 30; // or however long you will want to wait before giving up
while (millis() < timeout && (ptr < 4 || (ptr < buf[3]+2))) {
    while (SERIAL.available() && (ptr < 4 || (ptr < buf[3] + 2))) {
        buf[ptr++] = SERIAL.read();
        if (ptr <= 2 && buf[ptr-1] != 0xff) {
            // junk data found when expecting packet start!
            ptr = 0;
        }
    }
}
if (ptr < 4 || (ptr != buf[3] + 2)) {
    // timed out -- report error
}
calculate checksum, decode packet, etc

[_ADAM_]

Completely on topic and thanks for detailed response. You make a lot of good points. It's a lot to digest so I will think about it some more.

I'll be looking at this again tonight, plan is to test USB2AX and see if it is faster than USB2Dynamixel. I hope so.

One thing I've noticed is that the time it takes to get a response is variable. I can talk more about how to test for this if anyone is interested. I'm not sure where the variability is being introduced but as you say there are a lot of hoops the bytes have to jump through.

I think this topic is something that I will continue to learn about over time as I try to do more interesting things. For now I'm trying to get all of the instructions to work reliably, will optimize for different use cases later.