This paper presents a human-robot interface system that enables a user to move a robot by moving his hand. It adopts the technique of the 3D measurement based on the affine invariants from multiple views. In this system, we can choose a reference frame to obtain the 3D hand motion between the user-centered frame and the world-fixed frame. With the former frame, the system interprets the user's relative indication with respect t o his body; with the latter frame, it interprets the indication with respect to the world. Selecting one of these two interpretations depending on the situation realizes easier operation of the robot. Experimental results confirm the usefulness of the system.