AX12-A not communicating after failed firmware recovery

[6L20]

Hi there.
I hope I am not asking again, I've been searching for a while and could not find quite a similar question.
I am using 18 dynamixel AX12-A on a customized hexapod (equipped with distance sensors for "autonomous behaviour") on a theatrical stage with actors and dancers. Great fun.
Thing is, 4 of my servos stopped responding during a show, my hexapod crawled ridiculously for a while ;-/. I dont know exactly why. Overload, bug in the firmware I wrote asking for an physically impossible position ? No idea.
Anyway, rebooting them did nothing, so I purchased a U2D2, and used Dynamixel Wizard 2.0 to try and find what was wrong.

When connecting to them, and activating the torque, well, as expected, no movement and the "overload" thingy going red.
I then tried the "Recovery" command, thinking maybe a firmware recovery might save my next show...

I did exactly the same operation on 2 of my 4 faulty AX12-A, and got twice exactly the same result : Firmware recovery fails at 70%, and my servo is now invisible. I tried to connect to them using every possible ID and baud rate (just in case something changed there), tried with my U2D2 and the dynamixel wizard, tried with dynamanager with the ros sketch uploaded on my arbotix (the way I set my Ids), nothing. As if the cable was unplugged. Of course, I tried connecting to other servos, to make sure the problem doesn't come from my computer or U2D2 or cable or power supply, everything works as expected.
Any idea what is wrong ? Did I destroy something physical ?
Thanks much !

[jwatte]

Overloading servos can die, for sure. Debugging this is going to require some effort!

You should add an Xbee or something that can send periodic status information. For example: Every five seconds, read the temperature, voltage, position, and status code of each servo, and send it to a collector; then dump that into a file you can later analyze. (I read out the status information from one of my servos each time through the main control/pose loop, and every so often, plus when any servo is "in trouble," I take a "snapshot")

If it turns out the servos are overheating, you may be able to customize the AX-12 with the heat sinks that the RTeam people developed (see other threads for dimensions and construction tips.) Or perhaps add torsion springs to reduce the load. Or both!

[tician]

Usually, the reason for an AX-12 failing to move when not under significant weight loading is some sort of mechanical interference like cables getting wrapped around frames or wedged into gaps between the servo face and frames. Overload conditions often require complete powerdown (disconnecting 12V) for a chunk of time (or a fan) to force the internal ICs (linear voltage regulator and MOSFETs for motor H-bridge) to actually cool off enough to clear the over-temperature conditions that triggered the shutdown. Actual damage to the motor or motor MOSFETs is usually extremely difficult to achieve with AX-12 servos, but it seems that firmware update failures for all types of dynamxiels have become an increasingly common topic here...

I bought a TQFP-32 programming socket/dongle thingy for ATmega88/168/328 from HobbyKing several years ago for programming stuff that did not have an easily accessible/documented programming port on board. If I ran into the corrupted bootloader and/or firmware issue, then I would probably use that to pull a complete copy of the FLASH/FUSES from the dud servo and a copy from a good servo. Compare the two, then try pushing the good copy onto the dud directly if there were any major differences in the FUSES or bootloader area of the FLASH. I have not dealt with ATmega88 or STM32 stuff in a while, but I seem to remember them being able to modify their own bootloader FLASH memory blocks under certain conditions (cannot remember if the STM32 in newer servos even has any dedicated, protected blocks of FLASH for the custom bootloader), so the firmware updater might be triggering them to overwrite the bootloader or otherwise send it out of bounds.

edit: actually, it was the Teensy-3.x that I was thinking about... Still not 100% certain about the STM32, but the Teensy-3.x definitely has no protected FLASH dedicated for a bootloader, hence the auxiliary IC that erases the Teensy and pushes over a USB bootloader using the JTAG/SWD port.