Control of unmanned aerial vehicles is a very active topic in research with lots of applications ranging from civilian to military. To control a UAV, its attitude is often controlled using gyroscopes, but to control its position, inertial sensors together with GPS are often used. However, obtaining accurate current position is difficult using inertial sensors because of the integration drift. GPS on the other hand is not functional in indoor applications since it cannot connect to GPS satellites. Since vision has been proved to be an inexpensive and consistent source of relative position information, vision-based control is getting more popular in UAVs recently, but then again, using vision in outdoor applications is challenging as the target can move fast and out of the vision sensor field of view. So, in order to keep the target inside the field of view, two algorithms are being developed and tested via simulation in this research. Using pan/tilt/zoom cameras or multi camera systems, the target is guaranteed to stay in vision system field of view and hence, the vision based pose estimation can provide the control system with proper relative position. Two case studies - vision-based mobile-target tracking of a quadrotor using a multi-camera vision sensor and vision-based mobile-target tracking of a tilting rotor aircraft equipped with a zooming camera - are presented in this research to show the applicability of these methods in UAV control.