Divide a list of curves into groups based on the fact they are forming a poligon or not

Depending on your data set you could try some design script grouping of curves.

Are your points or curves all located in the same flat plane? Are you just looking to sort radially from base points at first? Would you be able to post an image of your workspace or what you have tried? As well as a test file?

There are a few statistical outliers removal algorithms out there (mainly for pointclouds) but your workflow probably only requires a simple recursive algorithm.