Many applications of geotagged content are predicated on the concept of localness (e.g., local restaurant recommendation, mining social media for local perspectives on an issue). However, definitions of who is a "local" in a given area are typically informal and ad-hoc and, as a result, approaches for localness assessment that have been used in the past have not been formally validated. In this paper, we begin the process of addressing these gaps in the literature. Specifically, we (1) formalize definitions of "local" using themes identified in a 30-paper literature review, (2) develop the first ground truth localness dataset consisting of 132 Twitter users and 58,945 place-tagged tweets, and (3) use this dataset to evaluate existing localness assessment approaches. Our results provide important methodological guidance to the large body of research and practice that depends on the concept of localness and suggest means by which localness assessment can be improved.