An open-source framework for the interactive exploration of Big Data: Applications in understanding health care

When analyzing data where the relationships between variables are not fully understood, it is typical to engage in visual exploration. However, this is slow and manually intensive, and interesting trends could potentially be missed. Ideally, we would like to focus on patterns and locations within the data that are interesting, such as significant clusters and outliers. We present a novel iterative k-means clustering algorithm to efficiently identify clusters in large datasets. This facilitates rapid visual exploration of new datasets. We illustrate our framework by performing a detailed analysis of open health care data released by the US Government and New York State. We apply our iterative k-means algorithm to identify clusters of trends in labor force participation to present a unique perspective by area of medical specialty over a 50-year period. Specialties such as nurse practitioners have seen a significant increase in the number of practitioners relative to internal medicine. Our code is open source and available on Github.