I’m not going into too much technical detail. If you want to know more, contact me privately. Here is a self intersecting contour with just four points, that obviously has an unknown direction.
Yes, I see the problem with that example. If the first point is at the upper left corner, and the contour is clockwise, I would personally identify the 4th point as the culprit (the line connecting points 3 and 4 is the one that crosses the line connecting points 1 and 2). But I can see it could be difficult to come up with a general algorithm that would work with all problem contours.
The situation I am usually dealing with shouldn’t present that dilemma. It typically turns up when I use the glyph transformer to scale glyphs smaller, e.g. to create small caps.
The “contour with unknown direction” is usually due to the fact that just one point has been moved so it is on the wrong side of the next point or (at my usual zoom level) underneath it. Usually one point is on-curve and the other off-curve.
In fact I often find this same problem with some commercial fonts – even though I haven’t modified them in any way!