Skip to main content
Back to E-commerce Dictionary

Data Deduplication

Data managementIntermediate Level

Product data deduplication finds and removes duplicate product records by matching on identifiers, names and attributes, so each product exists once.

Image by · CC BY 4.0

What is Data Deduplication?

Product data deduplication is the process of finding and removing duplicate product records, so each product exists once in your catalog. Deduping compares records on identifiers such as SKU, EAN or manufacturer part number, and on names and attributes, then keeps one correct record. Duplicates usually come from importing the same supplier file twice, combining feeds from several suppliers, or creating a product by hand that already existed. Exact matching catches records with the same identifier; fuzzy matching catches near matches such as "Nike Air Max 90 wht" and "Nike Air Max 90 White". Deduplication differs from data cleansing, which fixes errors inside records, although both are often done together. Our deduplication guide walks through the matching rules step by step.

Why Data Deduplication matters for e-commerce

Duplicate products split stock, reviews and sales data over several records. Shoppers see the same item twice at different prices, your reports undercount best sellers, and marketplaces may reject or hide listings. Data deduplication keeps one record per product, so every channel and report works from the same facts. The problem grows with every new supplier and every import. Preventing duplicates at the moment data enters the catalog is far cheaper than cleaning them up later.

Examples of Data Deduplication

  • 1Record A: EAN 8712345678906, Nike Air Max 90 White 42, Record B: EAN 8712345678906, Air Max 90 wht size 42, Match: Exact, same EAN
  • 2Record A: Bosch drill GSB 18V-55, Record B: Bosch GSB18V-55 cordless drill, Match: Fuzzy, same manufacturer part number written differently
  • 3Source: Supplier file imported twice, Result: 1,200 products created twice, Fix: Match on SKU and update instead of create
  • 4Source: Two suppliers deliver the same mug, Result: Two records with different titles, Fix: Keep one record and store both supplier codes
Image by · CC BY 4.0

How WISEPIM Helps

  • Match at import: WISEPIM product import can match incoming rows on an identifier such as SKU, so a second import updates existing products instead of creating copies.
  • Rollback Import: If an import still creates unwanted records, you can roll that import back.
  • Validate identifiers: Check EAN and GTIN codes with the free EAN and GTIN validator before you use them as match keys.
  • Quality checks: Quality Guard rules help you spot incomplete records that often turn out to be duplicates.

Common mistakes with Data Deduplication

  • Matching only on product names, which differ between suppliers.
  • Deleting duplicates without checking which record holds the best data or the order history.
  • Cleaning the catalog once and never preventing new duplicates at import.
  • Using fuzzy matching without a review step, so different variants get treated as one product.

Tips for Data Deduplication

  • Pick one match key per source, such as EAN or supplier SKU, and make it required.
  • Normalize data first: trim spaces, unify units and casing, then match.
  • Review fuzzy matches by hand before you remove or combine anything.
  • Keep supplier codes on the surviving record so future imports still match.

Trends around Data Deduplication

  • AI models that compare titles, attributes and images to find likely duplicates.
  • Matching on global identifiers such as GTIN as more suppliers deliver them.
  • Checking for duplicates at the moment data is imported instead of in yearly clean-ups.

Tools for Data Deduplication

  • WISEPIM: Product import with identifier matching, Rollback Import and Quality Guard rules.
  • Akeneo or Salsify: PIM and product experience platforms with data quality features.
  • OpenRefine: Open-source tool for clustering and cleaning messy data.
  • SQL scripts: Custom queries for finding duplicates directly in a database.

Related Terms

Attribute CollisionCanonical Data ModelCircular Economy Data ModelCross-Border Trade (CBT) AttributesData ContractData Decay

Also Known As

dedupingde-duplicationduplicate removalrecord matching

Frequently Asked Questions

Deduping is short for deduplication: removing duplicate records from a dataset. In e-commerce it usually means finding products that appear more than once in a catalog or feed and keeping one correct version. The term is also used for contact lists and backups.

Exact matching treats records as duplicates only when a field is identical, such as the same EAN. Fuzzy matching scores how similar records are, so it catches typos, abbreviations and different word order. Start with exact matching on identifiers and use fuzzy matching for records without a reliable code.

Duplicates split stock, reviews and sales history across two records, which confuses shoppers and your own reports. On marketplaces they can lead to rejected or suppressed listings. Duplicate product pages also compete with each other in search results.

The best moment is at import, before duplicates reach your catalog: match incoming rows on SKU or EAN and update the existing product instead of creating a new one. Run a periodic check on the whole catalog as well, because manual entries and new suppliers keep adding risk.

Still have questions?

Can't find the answer you're looking for? Please get in touch with our team.

Contact Support

Keep exploring

Hand-picked next steps to go deeper.