Abstract—When designing human robot communication interface, applying non-verbal interaction modalities such as robot motions is useful. Such modalities should be designed to augment verbal communication contents and set a mood for effective human robot interactions. Designing and implementing a set of useful robot motions, however, is a difficult problem. It requires developers to describe instructions for accurately controlling motors embedded in the robot. In this paper, the authors propose a method for generating a variety of robot motions using human gestures. The proposed method captures human gestures in real time by using motion tracking techniques, converting the acquired data to robot motion instructions, and applying them to a physical robot. We developed a prototype system targeted at supporting multiple motion tracking techniques and physical humanoid robots. We also conducted experiments for controlling the robots by human gestures to verify the effectiveness of the proposed method. It can promote better human robot communication environments with reducing labors in robot motion development.