Thank you, shaunR!
But I still have some confusion. For example, how does this line fitting algorithm find the initial subset of points to start, and use what method to fit a line to the subset of points. Strangely enough, if I set the iterations value to 0 or a number more than 1, this VI will return almost best fit line, but when I set this value to 1, it return a line that seems to be fitted by the standard fit method.
(the red spots represent the original edge points, and the green line is the fit line returned by this VI)