I have a huge CSV file containing GPS points of hotels in various cities. Sample:
CITY | HOTEL | LATITUDE | LONGITUDE
Chicago | Bellevue | 41.826 | -87.689
Chicago | SuperMt | 41.924 | -87.703
Chicago | Starhotel | 44.903 | -93.215
Chicago | BestW | 41.743 | -87.641
Tokyo | CityStay | 30.212 | 128.435
Is there a program that can detect outliers? For instance, Starhotel's latitude/longitude are clearly wrong, putting it hundreds of kilometers away from the other hotels of the same city.
Requirements:
- Outliers should be detected relative to the dispersion of the main cluster, for instance hotels in "California" will be rather far apart, whereas hotels in "East Village" will all be very close to each other. So "outlier" is relative to the dispersion of the whole group.
- Free, ideally open source
- Fast to configure
- Works with 300,000 lines 100 MB CSV file, or its equivalent RDF or OSM file
- Any OS. Ideally command-line. Online tool/API OK if it can handle the load.
- Longitude becomes less significant near the South/North Poles. Calculating distance in a naive way
sqrt(latitudeDelta²+longitudeDelta²)
is better than nothing, though, as the Poles don't have many hotels.
Final goal: catch probable errors, in order to send them to human reviewers. 100% accuracy not needed.