OpenCM 9.04 Freezes when using serial port

[xenoc]

I've been operating a turret with a AX12A via a serial connection from my PC using XBEES connected to to Serial 2 on the OpenCM. Python on the PC triggers the OpenCM to start a turret scan and return data from the board over the same serial port as a fairly continuous stream of Lidar distances and AX12A angles. It works fine for a while but after about 30 scans the firmware on the OpenCM seems to lock up. It no longer communicates via any serial port including the USB and I have to reset the board to recover. It's a bit random and doesn't always fail at the same point but never runs for more than 5-10mins. I'm running the connection at 115200 baud. I was previously using a RPI0 connected directly to serial port and had the same problem so it's not an issue specific to the XBees.
Any thoughts on how to debug this issue? If it's a memory leak how would I tell? I know the OpenCM boards tend to lock up if a lot of data goes through the USB but I'm not doing that.

[KurtEck]

Since you are talking about mostly Robotis products, I would first recommend that you post on their Forum, and/or enter a issue on their OpenCM github project...

Debugging:

a) I often will add a bunch of digitalWrite (or digitalWriteFast) around different key areas in the code. So if it hangs at a spot, you might be checking the state of the IO pins, on where the code may be. Sometimes also do it it with short Serial.print statements. Sometimes I then put in Serial.flush() to force it out.

b) Memory usage. A long time ago, I was debugging some of the memory issues on OpenCM... Had a couple of functions I added into the sketch to help me debug:

//=================================================================================
// Lets initialize our memory usage code, to get an idea of how much has been
// used
register uint8_t * stack_ptr asm("sp");
extern char end asm("end");

uint32_t g_end_stack_pointer;
uint32_t g_start_heap_pointer;

void initMemoryUsageTest()
{
  // Guess on start of stack. // probably using less than 100 bytes of stack space...
  g_end_stack_pointer = ((uint32_t)stack_ptr + 100) & 0xfffff000;

  // get the start of the heap ponter
  g_start_heap_pointer = (uint32_t)&end;

  // Print out some memory information
  Serial.printf("Estimated global data size: %d\n", g_start_heap_pointer & 0xffff);
  Serial.printf("starting Heap info: start: %x current: %x\n", g_start_heap_pointer, (uint32_t)_sbrk(0));
  Serial.printf("Start Stack info: end: %x current: %x\n", g_end_stack_pointer, (uint32_t)stack_ptr);
  Serial.println("Try to init memory");
  Serial.flush(); // make sure it has chance to write out.
  uint8_t *sp_minus = stack_ptr - 10;  // leave a little slop
  for (uint8_t *p = (uint8_t*)_sbrk(0); p < sp_minus; p++) *p = 0xff; // init to ff
  Serial.println("After init memory");
}

//=================================================================================
void printMemoryUsage()
{
  uint8_t *current_heap_ptr = (uint8_t*)_sbrk(0);
  Serial.printf("Heap ptr: %x  Usage: %d\n", (uint32_t)current_heap_ptr,
                 (uint32_t)current_heap_ptr - g_start_heap_pointer);

  // stack info
  uint8_t *sp_minus = stack_ptr - 10;  // leave a little slop
  uint8_t *p = current_heap_ptr;

  // try to find out how far the stack has been used
  while ((p < sp_minus) && (*p == 0xff)) p++;
  Serial.printf("Stack Max: %x, usage: %d\n", p, g_end_stack_pointer - (uint32_t)p);
  Serial.printf("Estimated unused memory: %d\n", (uint32_t)(p - current_heap_ptr));
}

My setup code called the first one:

void setup() 
{
  Serial.begin(57600);
  while(!Serial); // Open a Serial Monitor  
  initMemoryUsageTest();
..

I added a call to second one at some key location in loop()... In the particular case, I was testing Dynamixel Workbench test app, so it only printed it after a user entered a command... i.e. at the end of code that was within: if (Serial.available()) {

But the placement here is simply a trade off of how often is it called....

c) check for overflowing or under flowing arrays. Something as simple as you want to use an array of 100 items, so you allocate an array: uint8_t my_array[100]; And you try to access element 100 my_array[100] = 10... But array is 0 based so 0-99 are valid, 100 writes over next memory location.

[xenoc]

Thanks for those suggestions. I added some more serial prints and found that it was always dying when it executed dxl_wb.itemRead(dxl_id, "Moving", &moving_status, &log)
after a number of iterations. I then realised the preceding code which included the following functions could be moved to the setup section

result = dxl_wb.init(DEVICE_NAME, BAUDRATE, &log);

result = dxl_wb.ping(dxl_id, &model_number, &log);

result = dxl_wb.jointMode(dxl_id, 0, 0, &log);

Once I did that the problem disappeared. I had modified the example code supplied as part of the DynamixelWorkbench to get the moving status but didn't tidy it up properly. So it's working fine for 100s of scans now but I'm still not sure what caused the problem. I might add the memory usage code you provided to see if memory is still slowly being consumed. Any further thoughts? I'm not clear what the &log is for.