CompERBench: Complementing Entity Matching Benchmark Tasks

In an effort to enhance the reproducibility and comparability of matching methods, we complement existing benchmark tasks for entity matching with fixed development and test sets. In this page we provide 21 complete benchmark tasks for entity matching for public download. Additionally, we calculate and present baseline results for the 21 benchmark tasks using two standard classification methods. If you want to use our work, please cite our paper:

News

• 2020-31-07: Our paper with the title "Profiling Entity Matching Benchmark Tasks" was accepted in the Resource Track of CIKM 2020.

Contents

• 1. Motivation
• 2. Benchmark Matching Tasks - Download
• 3. Baseline Matching Method
• 4. Baseline Matching Results
• 5. References

1. Motivation

Entity matching is an important task for data integration and has been the focus of many research works. A large amount of entity matching tasks for benchmarking have been developed and made publicly available for evaluating, comparing, reproducing and showing the strengths of different matching methods. However, the lack of fixed development and test sets, correspondence sets including both matching and non-matching record pairs as well as baseline results, hinders reproducibility and comparability. We mitigate this problem by defining a standard execution procedure for complementing matching tasks, apply it in 20 benchmark matching tasks and establish baseline evaluation results.

2. Benchmark Matching Tasks - Download

3. Baseline Matching Method

Feature Creation

In a binary entity matching setting, the attribute values of single records need to be transformed to pairwise record features. We use data type specific similarity metrics and calculate symbolic features for the pair of records. For short strings we apply similarity metrics: Levenshtein, Jaccard on the token level, Jaccard with inner Levenshtein, exact and containment. For long strings, the similarity metrics used for short strings are applied while Jaccard is calculated on the word level and additionally thecosine similarity with tfidf weighting per feature is computed. For numeric attributes, the absolute difference is computed. Finally, an overall similarity score of each record pair is computed using cosine similarity with tfidf weighting over the concatenated values of all attributes.

Here is an example of the transformation of record attribute values to pairwise symbolic features:

Classification Methods

We apply supervised learning using two classifiers: SVM and Random Forest. We use the validation set of every task to optimize some parameters. We apply the optimized model to the test set and report the evaluation scores on the test set. For all non-optimized parameters we use the default values of the python sklearn library (version 0.22). For all our experiments we use random seeds (always set to 1) to allow the exact reproducibility of our results.

You can find the code for evaluating and profiling the benchmark matching task in the EntityMatchingTaskProfiler github repository. Please use the Jupyter notebook MatchingTaskProfiler.ipynb and activate the flag parameters summaryFeatures for basic data set related statistics, baselineResults for getting the SVM and RF baseline results and profilingFeatures for task related statistics including our profiling dimensions. You can find more details on the task-related profiling dimensions and their calculation in our paper.

4. Baseline Matching Results

5. References

1. Köpcke, H. et al.: Evaluation of entity resolution approaches on real-world match problems. Proceedings of the VLDB Endowment. 3, 1–2, 484–493 (2010).
2. Konda, P. et al.: Magellan: Toward building entity matching management systems. Proceedings of the VLDB Endowment. 9, 12, 1197–1208 (2016).
3. Petrovski, Petar, et al. The WDC gold standards for product feature extraction and product matching. International Conference on Electronic Commerce and Web Technologies. Springer, Cham, 2016.
4. Primpeli, A. et al.: The WDC Training Dataset and Gold Standard for Large-Scale Product Matching. In: Companion Proceedings of The 2019 World Wide Web Conference on - WWW ’19. pp. 381–386 ACM Press, San Francisco, USA (2019).
5. Draisbach, U. and Naumann, F.: DuDe: The Duplicate Detection Toolkit.In: QDB Workshop 2010.