System Description

Initial concept:

Vision is expected to play an important part in Leafís sensing. As such, a lot of processing capability will be necessary to achieve reasonable real time performance.  A laptop computer (or equivalent) is the primary processor for this robot. The Windows XP home operating system is used since so much software already exists that runs under it and the people currently working on Leaf are most acquainted with Windows. The software has been modified (slightly) to be compatible with Vista and now runs under either version of Windows.  The laptop display is used to display a face to show personality, emotions, reactions etc. and will also be available to monitor performance, debug and program.

Certain control applications (e.g. drive motor control, or arm position control) require a real time operating performance that can not be easily achieved with Windows. Hence, a Motorola microcontroller is used for real time control and as an interface device for sensor inputs and discrete I/O. The microcontroller communicates with the laptop via USB. More than one microcontroller can be used to accommodate additional sensors or motor controls etc.

Vision is  implemented using USB based cameras (initially, webcams). Sound acquisition and sound/speech generation will be done using the laptops built in sound capabilities.

A separate remote laptop (or desktop) may be used as a control/monitoring device. This laptop will communicate with the robot via 802.11b wireless.

Hardware Overview:

Note:  much of the following description is for the first prototype of Leaf (now called Rocky).  Subsequent robots have varied considerably in size and appearance while still using the basic Leaf system architecture.

The robot will be constructed as modularly as practical. Its motion will be provided by a round base driven by two differentially driven wheels The wheels will fit entirely within the approximately 20 inch diameter. The wheels will be 6 inches in diameter. Casters will be mounted fore and aft to ensure against tipping over. The rear caster will be mounted with a spring to provide some ability to get over minor floor irregularities (like edges of rugs) without high-centering.

The outer one inch of the base will be a bumper skirt with rubber padding to minimize impact damage to the robot or to the furniture. The bumper will also have switches to indicate to the robotís processor(s) when an impact has occurred.

The body of the robot will be built from lightweight aluminum angles connected at corners. Shelves will be added to the frame as required to mount components.

The battery will be located on top of the base at the rear within the body frame. This location may change to optimize the balance of the robot on the spring loaded rear caster.

A "head" sensor platform will be mounted on the top of the frame. This head will be driven by servos to pan left and right about 90 degrees and tilt up and down about 60 degrees. Sensors such as camera(s), microphones and range sensors may be mounted on this platform. The tilt pivot point will be near the front of the body so that by tilting down, the sensors can see the floor right in front of the robot.

A laptop computer will be attached to the front of the body. This laptop will be the main "brains" of the robot. Itís display may be used to show a face to portray robot expressions and emotions. It may also display text messages to interact with people. The display may also be used to display error messages to the operator and, in conjunction with the keyboard, may be used to reprogram or otherwise modify the robots programming. The laptop will be attached such that the keyboard may be swung up to a horizontal angle for ease of use.

While not shown, the front of the robots body will be protected by a bumper skirt mounted about 1 inch in front of the laptop. This skirt will sense collisions above the base bumper and will protect the laptop from such collisions. This bumper will be spring mounted and have switches to report collisions to the processor. There will not be a similar bumper on the rear as it is expected that if the robot is backing up, it has already passed through the area and hence the chance of collision is slight.

There will be provisions for one or even two arms. The arm will be designed to fit within the 20 inch diameter of the base when it is in a retracted position so that the robot can pivot without (usually) bumping the arm. The body will have sufficient height such that an arm with a shoulder pivot near the top of the body will be able to reach out straight onto the top of a 30 inch table. It will be desirable that the arm can also extend to reach objects on the floor in front. This height requirement will probably result in a robot with a total height of about 40 inches.


The laptop will be the primary processor. It will communicate with the outside world by various means including the microcontroller mentioned above, off the shelf USB (or other bus) sensors like cameras, RC servo controllers, X10 controllers, etc..

USB webcam(s) will be installed that will connect directly to the laptop USB port(s). (using USB hubs as necessary).

The "standard" PC microphone and speakers plugs may be used for sound and speech generation and speech recognition. We have found that an external USB microphone provides much better fidelity for voice recognition and external speakers may be added for increased speech volume.


Software functions in the Laptop may include:


Vision:    (note: use of RoboRealm and OpenCV provides far more than these sample capabilities)
Motion detection
Intensity change detection
Color blob detection
Image recognition

Speech and sound generation
Speech/sound recognition
Sound direction (someday?)

Bumpers (on base and body)
Position sensing of robot parts (e.g. arm)
laser scanning range sensors

Route planning

AI (Lisp):
Planning actions in response to sensory input



Software architecture:

Currently we are planning two applications/processes; mostly because most of the AI part will be written in Lisp and most of the functional/support software will be in C.

All software will run under Windows (we are using XP home edition). This is for reasons of general availability and knowledge. (Yes, it could be done in Linux also!!! Our choice)

Most major functions will run in their own threads. Threads will be as independent of each other as possible to minimize bugs. All communications between major functions will be via messages. Threads within the C package will pass messages through shared global memory using critical section flags. Interprocess communications will be done through shared memory using Mutexs.


Functionally, the laptop software will break down roughly as shown in the following diagram.

    The Lisp Artificial Intelligence software is the highest level control of the robot.  It has direct control of the Microsoft speech SDK  which provides speech recognition and generation; as well as the CSLU face animation which provides expression and lip synchronization to spoken words.

   The vision software generally runs in response to commands from the Lisp program.  It contains programs to capture frames from the webcam and to perform analysis on the captured pictures to achieve recognition and navigation objectives.

    The Navigation and Control software performs communications between the microcontroller and the laptop software.  It reports all sensor data from the micro and makes it available to Lisp.  Lisp can then send basic commands to the micro such as to go forward, turn or stop.   Nav and Control also can perform higher level navigation tasks so that Lisp can make commands like "go to the kitchen" and leave navigation to achieve that objective.