tag

#deduplication

2 pages tagged

Programming › Algorithm Cookbook › Chapter 5. Hashing and Maps ›

5.16 Hash-Based Deduplication

5.16 Hash-Based Deduplication Problem You need to remove duplicates from a dataset or stream efficiently. A naive approach compares each element with all previously seen elements, which leads to quadratic time. For large inputs, this is impractical. Solution Use a hash set to track seen elements. deduplicate(items): seen = empty set output = empty list for item in items: if not contains(seen, item): insert(seen, item) append item to output return...

Wiki › Algorithms › 02. Data Structures › 1. Core Data Structures and Operations › 1.1 Arrays and Dynamic Arrays ›

Array Deduplication

Array Deduplication Array deduplication removes repeated values from an array. The usual method keeps a set of values already seen and writes each new value once. You use it when the array may contain repeated elements and downstream work needs only distinct values. Problem Given an array $A$ of length $n$, produce an array containing each distinct value once. For stable deduplication, preserve the first occurrence order. Algorithm Use a...