Hopkins statistic
Before clustering a dataset, we can test if there really are clusters. We need to test the hypothesis of the existence of patterns in the data against a uniformly distributed dataset (homogeneous distribution).
The Hopkins statistic is calculated as follows:
- Sample n points (p_i) from the dataset (D) uniformly and calculate the distance to their nearest neighbor (d(p_i))
- Generate n points (q_i) uniformly distributed in the dataset space and calculate their distance to the nearest neighbors in D (d (q_i))
- Calculate the quotient H:
If the data is evenly distributed, the value of H will be around 0.5.
