Copy-Paste Deduplicator

What it does

Paste a list of lines (emails, URLs, names, IDs, anything one-per-line) and the tool removes duplicates with configurable handling: trim whitespace before comparing, case-insensitive matching, preserve original order vs sort alphabetically. Returns the deduplicated list plus stats — how many duplicates were found, total lines before and after, percentage reduction. Useful for cleaning up exports, merging lists, deduplicating email recipient lists before sending, or tidying any one-per-line dataset.

Common use cases: email recipient lists (consolidating multiple CSV exports for a one-off send); URL lists for batch processing (sitemap drafts, link audits); contact merging when consolidating multiple address books; data cleanup for a CSV column you extracted with grep / awk; SQL-result-to-application input(paste a SQL query result column, dedupe, paste into another query as IN clause).

The dedup options matter: trim whitespace catches lines that look identical to humans but differ by trailing spaces (a common copy-paste issue); case-insensitive treats “Foo” and “foo” as duplicates (right for emails which are case-insensitive at the local-part level by spec, wrong for case-sensitive identifiers like git SHAs); preserve order keeps the first occurrence of each duplicate at its original position (right when the list has meaningful order, e.g. chronological); sort alphabetically reorders for predictable output (right for logs, name lists, searchable references). Default settings work for most casual use; tune them for specific data types.

How to use it

1. Paste your list, one item per line. The tool handles \n, \r\n, and mixed line endings.
2. Pick options: trim whitespace (default on), case-insensitive (default off — turn on for emails), preserve order vs sort (default preserve).
3. Read the cleaned output and stats (lines before / after, duplicates removed, percentage reduction).
4. Click Copy to put the deduplicated list on your clipboard.
5. If the input has structure beyond one-per-line (CSV, JSON, key:value), use a more sophisticated tool — this is plain-text-line dedup only.

When to use this tool

• Cleaning up email recipient lists before bulk sends.
• Merging multiple text exports into one deduplicated set.
• Quick CSV column cleanup (extract column → dedupe → re-import).
• Sanity-checking a list you suspect has duplicates without manually scanning.

When not to use it

• CSV / structured data — use a spreadsheet (Google Sheets, Excel) which can dedupe by specific columns. Plain-text dedup loses column structure.
• Fuzzy matching (similar but not identical lines) — this is exact-match dedup. For fuzzy matching use OpenRefine or a string-similarity library.
• Massive lists (>1M lines) — browser memory is a bottleneck. Use shell tools: `sort uniq input > output` or `awk '!seen[$0]++' input > output`.
• Preserving duplicates with counts — this removes them. For 'count of each unique value' use `sort | uniq -c` or a spreadsheet pivot.

Common use cases

• Onboarding a colleague who needs the same calculation/conversion
• Verifying a number or output before passing it on
• Quick use during a typical workday
• Pre-decision sanity-check on inputs and outputs