Skip to content
/ fuzzycat Public

Fuzzy matching publications for fatcat (wip).

License

Notifications You must be signed in to change notification settings

miku/fuzzycat

Folders and files

NameName
Last commit message
Last commit date

Latest commit

478d7d0 · Sep 24, 2021
Nov 3, 2020
Sep 24, 2021
Sep 21, 2021
Apr 15, 2021
Sep 24, 2021
Apr 17, 2021
Sep 13, 2021
Aug 12, 2020
Aug 12, 2020
Sep 13, 2021
Sep 13, 2021
Sep 13, 2021
Apr 14, 2021
Sep 21, 2021

Repository files navigation

fuzzycat: bibliographic fuzzy matching for fatcat.wiki

https://pypi.org/project/fuzzycat/

This Python library contains routines for finding near-duplicate bibliographic entities (primarily research papers), and estimating whether two metadata records describe the same work (or variations of the same work). Some routines are designed to work "offline" with batches of billions of sorted metadata records, and others are designed to work "online" making queries against hosted web services and catalogs.

fuzzycat was originally developed by Martin Czygan at the Internet Archive, and is used in the construction of a citation graph and to identify duplicate records in the fatcat.wiki catalog and scholar.archive.org search index.

DISCLAIMER: this tool is still under development, as indicated by the "0" major version. The interface, semantics, and behavior are likely to be tweaked.

Quickstart

Inside a virtualenv (or similar), install with pip:

pip install fuzzycat

The fuzzycat.simple module contains high-level helpers which query Internet Archive hosted services:

import elasticsearch
from fuzzycat.simple import *

es_client = elasticsearch.Elasticsearch("https://search.fatcat.wiki:443")

# parses reference using GROBID (at https://grobid.qa.fatcat.wiki),
# then queries Elasticsearch (at https://search.fatcat.wiki),
# then scores candidates against latest catalog record fetched from
#  https://api.fatcat.wiki
best_match = closest_fuzzy_unstructured_match(
    """Cunningham HB, Weis JJ, Taveras LR, Huerta S. Mesh migration following abdominal hernia repair: a comprehensive review. Hernia. 2019 Apr;23(2):235-243. doi: 10.1007/s10029-019-01898-9. Epub 2019 Jan 30. PMID: 30701369.""",
    es_client=es_client)

print(best_match)
# FuzzyReleaseMatchResult(status=<Status.EXACT: 'exact'>, reason=<Reason.DOI: 'doi'>, release={...})

# same as above, but without the GROBID parsing, and returns multiple results
matches = close_fuzzy_biblio_matches(
    dict(
        title="Mesh migration following abdominal hernia repair: a comprehensive review",
        first_author="Cunningham",
        year=2019,
        journal="Hernia",
    ),
    es_client=es_client,
)

A CLI tool is included for processing records in UNIX stdin/stdout pipelines:

# print usage
python -m fuzzycat

Features and Use-Cases

The refcat project builds on top of this library to build a citation graph by processing billions of structured and unstructured reference records extracted from scholarly papers (note: jfor performance critical parts, some code has been ported to Go, albeit the test suite is shared between the Python and Go implementations).

Automated imports of metadata records into the fatcat catalog use fuzzycat to filter new metadata which look like duplicates of existing records from other sources.

In conjunction with standard command-line tools (like sort), fatcat bulk metadata snapshots can be clustered and reduced into groups to flag duplicate records for merging.

Extracted reference strings from any source (webpages, books, papers, wikis, databases, etc) can be resolved against the fatcat catalog of scholarly papers.

Support and Acknowledgements

Work on this software received support from the Andrew W. Mellon Foundation through multiple phases of the "Ensuring the Persistent Access of Open Access Journal Literature" project (see original announcement).

Additional acknowledgements at fatcat.wiki.

About

Fuzzy matching publications for fatcat (wip).

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages