Page 1 of 3 123 LastLast
Results 1 to 10 of 29

Thread: robandix NXT Hero

  1. #1
    Join Date
    Dec 2010
    Hamburg, Germany
    Rep Power

    robandix NXT Hero


    as i own a Lego NXT and a Android HTC Hero i combined them to give my robot eyes , ears and some sense ..

    robandix NXT Hero

    Used parts:
    LEGO NXT 2.0
    Android HTC Hero

    Used programming languages:
    Android HTC Hero:
    Java (SDK)
    and good old C (NDK) for audio and video analysis

    Proof of concept: using Android Phone Sensors, Camera and Audio Input for NXT robot.

    Communication between NXT and HTC Hero via Bluetooth.

    The main objective was the simple coupling of robot and android smartphone.
    As the new generation android smartphones brings a lot of sensors
    (and video and audio capabilities of course) with (open source) documented programming capabilities,
    i want to interface my HTC Hero to my robots to enhance their senses.
    The main difficulties have been and are on the algorithmic side (voice recognition,
    machine vision,..). Although the new devices are very powerful they have limited resources
    concerning processing power etc. you have to consider.

    The next step (in software) will be a more general library for these tasks,
    with more flexible interfaces (as at the moment the base interface is limited
    to the NXT proprietary communication interface via bluetooth).
    The logical interface at the moment is in abstract a kind of mailbox communication.
    And beyond this i plan to make some experiments in machine learning.

    The next step (in hardware) is to extend the concept to new platforms like the bioloid robot kit.

    You may watch my experiments here:


    Recognizing the first words:

    Color Detection:

    COSAC - Color Seeking Audio Controlled

    Using Accelleration Sensor:

    Last edited by robandix; 12-24-2010 at 04:34 PM. Reason: minor changes

  2. #2
    Join Date
    Dec 2007
    Whidbey Island, WA
    Rep Power

    Re: robandix NXT Hero

    This is a really cool project. After the break I plan on sharing it with my students.


  3. #3
    Join Date
    Dec 2010
    Hamburg, Germany
    Rep Power

    Re: robandix NXT Hero

    Hi DB,

    i feel very honored and happy to read this from the maker of the famous hikari !


  4. Re: robandix NXT Hero

    Cool project indeed but i'm a little sorry to hear you have plans to replace the NXT Plateform.. i hope you will dazzle us with many more NXTandoidable robots before that day.

    PS good luck with the contest entry !

  5. #5
    Join Date
    Dec 2010
    Hamburg, Germany
    Rep Power

    Re: robandix NXT Hero

    Hi bazmarc,

    thank you very much.
    And don't worry, the next improvements will remain on Lego NXT.
    And "replace" was not the correct wording (i just corrected my original posing), because of the NXT its very easy to check some concepts.
    But nevertheless one day i want to see some humanoid carrying a backback containing my Android HTC Hero :-)


  6. Re: robandix NXT Hero

    This is fantastic robandix!
    I'm currently working on a university project where I'm creating a java applet that will communicate online with a server computer that will control a lego mindstorms robot.
    I would love to know if you have any further information about how to use the HTC hero as a camera as it would be perfect to stream video back to the computer.

  7. #7
    Join Date
    Feb 2009
    Mesa, AZ
    Rep Power

    Re: robandix NXT Hero

    Look for SECuRET livestream on the android market to use your phone as a wireless ip camera

  8. Re: robandix NXT Hero

    thanks mannyr7, that was cool except it only streams over a local network.
    I was looking for something more open source really, so I could integrate it to my nxt control website.

  9. #9
    Join Date
    Dec 2010
    Hamburg, Germany
    Rep Power

    Re: robandix NXT Hero

    hi entozoon,

    if you want to program it on your own, here some hints.

    In general you have to use the class which is well documented.
    A simple example you may find in the sdk samples:
    You have to extend it with a CallBack function, which is called on every frame:

    camera.setPreviewCallback(new PreviewCallback() {
    public void onPreviewFrame(byte[] _data, Camera _camera) {
    // TODO Do something with the preview image.

    Depending on you android version you may use setPreviewCallbackWithBuffer, which will avoid the annoying gc and improve performance.
    A workaround for android versions where the CallbackWithBuffer method is hidden (and a complete working example using the callback)
    you may find here:
    This example helped me a lot.

    But it depends on you needs of performance / fps you want to achive.

    In case of NXT Hero i am calling a native C function via JNI in the callback, because for an autonomous robot all processing has to be done locally.
    But if you do no analyizing but only forwarding to a server you may stay within java.

    You have to consider, that the buffer you get in the callback function is YUV420SP encoded (which you have to convert to a format of your needs).
    General information on this encoding you may find here:
    If you do a search on android yuv420sp you will find some example code too.

    And one last remark to possible fps:
    The max fps possible will be dependend on the device used and the camera builtin.
    And its very light-dependend. With a callback only counting frames (no more processing) i got 25 to 28 fps (with very good light).
    This goes down to about 5 to 10 on bad light.
    If you will process the frames , the fps rate decreases of course.
    Now you may optimize with the CallBackWithBuffer method and with NDK native C functions (as i did).
    As this - the video analysis - is not the only processing (i do audio analysis and i am using the other sensors too) and as i have to handle the bluetooth communication, the max fps rate i get at the moment is about 5 to 8 (on good light). This is sufficient for my experiments at the moment, but i will optimize code in future.


  10. Red face Re: robandix NXT Hero

    Wow robandix, I cant thank you enough!
    Im sure theres enough information there for me to figure it all out, thank you!

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Lego NXT Mindstorm with organizer and book
    By xero in forum Garage Sale
    Replies: 1
    Last Post: 12-11-2010, 12:12 PM
  2. Contest Entry Lego NXT Autonomous 'Wall-E' Robot
    By bazmarc in forum Project Showcase
    Replies: 11
    Last Post: 10-08-2008, 03:22 PM
  3. Discussion NXT Robot
    By scorch in forum Software and Programming
    Replies: 14
    Last Post: 09-24-2008, 02:58 PM
  4. Contest Entry Theodore, Lego Mindstorms NXT Robot
    By srobot in forum Project Showcase
    Replies: 2
    Last Post: 09-22-2008, 01:34 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts