Back to Home

How Data are Collected and Scored

Robert Borofsky

Center for a Public Anthropology

Hawaii Pacific University

Altmetric, which provides the data for this project, monitors a range of online sources on a continuing basis to track and collate attention relating to academic content. It searches for mentions of scholarly outputs (items with DOI’s, arXiv, SSRN IDs, or unique URI identifiers) in order to disambiguate and present this information in a structured, coherent way. While this website focuses on mainstream news outlets, policy documents and blogs, it should be noted Altmetric also tracks other data sources such as Wikipedia, Twitter, Reddit, Sina Weibo, and Mendeley. For a full list of sources tracked by Altmetric refer to: http://support.altmetric.com/knowledgebase/articles/83335-which-data-sources-does-altmetric-track. To keep the project within manageable limits, especially in its initial stage, only the top 60 schools as ranked by their research funding (https://ncsesdata.nsf.gov/profiles/site?method=rankingBySource&ds=herd) are examined. Given the project’s focus on the social sciences, schools focused on medical research and technology are also not included.

The steps for tracking publications are discussed under “Explaining what is involved” in Step 2 (http://metrics.publicanthropology.org/steptwodescription.php) and Step 3 (http://metrics.publicanthropology.org/stepthreedescription.php).   When you click on the red or orange rectangles to the left of a publication, you are taken an Altmetric details page for that publication where you can investigate the specific references noted.

Collection of Data

Mainstream media news outlets: Altmetric receives a real-time feed of news stories via Moreover (part of LexisNexis). Each of the news stories is searched for links to scholarly outputs as well as text-mined (English language only) for mentions of an author name and journal title. These pieces of information are then cross-referenced with publications from CrossRef to determine if a publication, fitting these criteria, has been published within 15 days either side of the news item. If a positive match is made, the reference is included in Altmetric’s details page for that publication. For more information, including further details on the news sources tracked by Altmetric, refer to: http://www.altmetric.com/sources-news.php.

Public policy documents: Altmetric monitors a curated list of policy sources (such as the World Health Organization), and searches new documents daily as they appear online for references to published publications. These references are then added to a publication’s details page, usually within a day’s time. For further information on their public policy document tracking, refer to: http://www.altmetric.com/blog/new-source-alert-policy-documents/.

Blogs:  Altmetric scans RSS feeds from a manually curated list of blogs once a day, looking for links to published publications. When a mention is found, it is matched to the research output and then added to the appropriate details page within 24 hours.  For further information on the tracking of blogs, see:  http://www.altmetric.com/sources-blogs.php

Some Simplifying Rules:  Because of the complexities involved in compiling data on 1475 anthropologists from 60 different departments involved in the project, a few simplifying rules are used. First, the searches only include a department’s primary faculty. Secondary and affiliate faculty are excluded. Second, the exact names of faculty used in searches are drawn from their departmental websites. If an individual’s middle initial or name is listed on that website, it is included in the search. If only the first and last name are listed, then these are used in the search.

How Data Scored

A publication’s scores are displayed in the red and orange rectangles to the left of a publication. These numbers represent the times various news outlets, policy documents and/or blogs take note of that publication. The publications receiving the most attention (i.e. highest scores) are placed at the top. It is important to note that only publications referenced in news outlets and policy documents are included in departmental rankings. Blogs are not.

There are four reasons for not including blogs in a publication’s overall score and ranking. (a) The project focuses on publications that attract public attention. Generally speaking, news outlets and policy documents garner more attention than blogs in respect to two key audiences the project is concerned with – funders and university administrators. (b) A number of blogs are produced by journals as a way of publicizing their content. Consequently, a bias may exist when ranking a publication in terms of the number of blogs citing it. A publication’s blog score may depend on whether or not it is published in a journal that writes blogs to support its publications rather on the actual attention it receives.

(c) While blogs may be important, they vary widely in garnering public attention. Some blogs have large readerships, larger perhaps than some newspapers. But many, probably most, do not. (d) Trying to account for the differences between media outlets and blogs by giving them different scoring weights creates complications. Exactly how many points a news outlet should have vis-à-vis a blog is far from clear. Should the ratio between them, for example, be 10 to 2, 12 to 4, or 9 to 3? On what basis could one reasonably decide one ratio is more appropriate or more reasonable than another? Given the project’s focus on news outlets and policy documents, it seems best to side-step the problem by scoring each reference to a publication by a news outlet or policy document with one point while ignoring the publication’s blog count in the summarizing score that gives a publication its rank vis-à-vis other publications. While an imperfect solution, it creates fewer complications.

Missing mentions? If you believe that Altmetric has missed a mention of a publication, please let them know via this form: https://www.surveymonkey.com/s/missedmentions. : The Altmetric support team will review your submission and update the details page for the relevant publication where appropriate.

More information about why mentions of a publication are sometimes missed can be found at: http://support.altmetric.com/knowledgebase/articles/507497-why-has-this-mention-of-my-paper-been-missed.

A Pattern to Ponder:  Perusing the data, readers will note that archeologists and biological anthropologists tend to be cited in the media more than cultural anthropologists. One likely reason derives from the journals the discipline’s subfields publish in. Cultural anthropologists tend to publish in a set of sub-field journals. Archeologists and biological anthropologists tend to publish in more interdisciplinary journals leading, in turn, to a wider distribution and more attention paid to their publications. There is no reason why cultural anthropologists could not publish in PlusOne, Science, or Nature. But many prefer publishing in the American Anthropologist, American Ethnologist or Cultural Anthropology thereby attracting limited attention from those beyond their sub-field. Current Anthropology, which crosses the discipline’s sub-fields, tends to attract less attention than inter-disciplinary journals’, but comparatively more attention than the American Anthropological Associations journals, focused on specific sub-fields.

Back to Home