RTL stands for Return-To-Launch. When switched into RTL mode the vehicle will turn and attempt to directly return to the position where it was last armed (unless the home position has been changed via the ground station). Once at the destination it will hold position if a surface vehicle, or loiter/circle around the destination if it is a boat.
The RTL_SPEED parameter can be used to set the speed (in meters/second) at which the vehicle will return to home. By default this parameter is zero meaning it will use the WP_SPEED (or CRUISE_SPEED for versions before Rover 4.0, if WP_SPEED is 0).