Remove Duplicate Lines

Paste a list and this tool removes repeated lines while keeping the first occurrence of each and preserving the original order — ideal for cleaning email lists, deduping logs, tidying CSV columns, and merging lists without re-sorting.

First-seen wins: the dedupe rules

The tool reads your text line by line and tracks which lines it has already seen using a set. For each line it builds a comparison key based on your options: with case-insensitive on, the key is lowercased; with trim whitespace on, leading and trailing spaces are stripped before comparing. If the key has not been seen, the line is kept (exactly as you typed it) and the key is remembered; if it has been seen, the line is dropped. Because it walks top to bottom and keeps the first hit, the output preserves your original order. You can also choose whether blank lines are kept or removed.

Example

Input with case-insensitive and trim on:

Apple
banana
apple
  Banana
Cherry

Output:

Apple
banana
Cherry

The two later “apple” / “Banana” lines are removed as duplicates (matched ignoring case and padding), and the result reports 3 kept, 2 removed.

When to use each option

Case-insensitive matching is almost always the right call for natural-language lists — email addresses, names, tags, and keywords are all case-insensitive in practice. The exception is code, where NULL and null may genuinely mean different things in different contexts.

Trim whitespace is essential when data has been copied from spreadsheets, pasted from different sources, or exported from systems that pad fields. A trailing space is invisible to the eye but creates a false “unique” entry without trimming enabled. Turn it on by default and turn it off only if you are processing data where leading/trailing spaces carry meaning (for example, fixed-width file formats).

Keep blank lines depends on the format. For a plain list of values, removing blank lines cleans the output. For a structured text file where blank lines separate sections, keeping them preserves the document structure. If you are merging two keyword lists and want a clean flat output, remove blanks; if you are merging two outlines, keep them.

Common use cases

• Merging email lists from multiple exports — copy both lists into one block, run with case-insensitive on to catch duplicates across different capitalisations.
• Cleaning log output — paste repeated error lines and reduce a 500-line log to the unique messages only.
• Deduplicating CSV columns — copy a column from a spreadsheet, deduplicate, then paste back.
• Combining keyword lists for SEO or PPC — merge keyword suggestions from multiple tools and strip repeats before importing to your campaign.

One caveat for email lists specifically: deduplication here is textual, not semantic. [email protected] and [email protected] deliver to the same inbox on providers that support plus-addressing, yet they are different strings and both will be kept. Textual dedupe is the right first pass; address-level canonicalisation is a separate, deliberate step.

How this compares to sort -u and uniq

If you live in a terminal you may reach for the classic Unix pair: uniq only removes duplicates that are adjacent, so the standard idiom is sort file | uniq or simply sort -u — which necessarily destroys the original ordering. This tool behaves like the awk '!seen[$0]++' one-liner instead: it removes duplicates wherever they appear while keeping the first occurrence in place, which is usually what you want for lists whose order carries meaning (priority lists, playlists, CSV rows that must stay aligned with another column).

Line endings and invisible characters

Text copied between Windows, macOS and the web mixes line-ending styles: Windows uses CRLF (\r\n) while Unix and macOS use LF (\n). The tool normalises endings when splitting, so a CRLF line and an LF line with the same text are correctly treated as duplicates rather than two “different” lines.

The harder gotcha is invisible characters inside the line. A non-breaking space (U+00A0) pasted from a web page looks identical to a normal space but compares as different; the same goes for zero-width spaces and trailing tabs. If two visually identical lines survive deduplication, suspect an invisible character — run the text through a whitespace cleaner first, or enable trim whitespace to catch the leading/trailing cases.

The whole process runs in your browser and nothing is sent to a server.

A final habit worth adopting: keep the reported kept/removed count in your notes when cleaning important lists. If a merge of two 500-row exports reports only a handful removed, the lists probably did not overlap the way you assumed — the count is a built-in sanity check on the whole operation, not just decoration.