Duplicate Line Remover

What the tool does

The page has one input area and one output area. Paste your lines, click Remove Duplicates, and the output shows the unique lines in their original order unless you enable sorting. A stats row reports total lines, unique lines and duplicates removed.

Each line is treated as a separate record. The tool does not split on commas or tabs, so a CSV row stays one line. If you need to deduplicate a single column from a spreadsheet, copy that column rather than the whole sheet.

The output is read-only. Use Copy to put it on the clipboard, or Download .txt to save a plain-text file named unique-lines.txt. Paste and Clear shortcuts are provided for a quick workflow.

The options explained

• Case Insensitive: on by default. “Apple” and “apple” count as the same line. Turn it off when case is part of the identifier.
• Trim leading/trailing whitespace: on by default. Spaces or tabs at the edges are ignored when comparing and removed from the output.
• Remove Empty Lines: on by default. Blank lines are skipped instead of being kept as one empty entry.
• Sort alphabetically A-Z: off by default. When on, the unique lines are sorted with the browser's locale-aware comparison.

The first occurrence wins. If “Apple” appears before “apple” and case is ignored, the output keeps “Apple” in the position where it first appeared. That is useful when you want to preserve the original order and the original spelling of the first record.

How to clean a list

1. Paste your lines into the input box.
2. Decide whether case and whitespace should matter.
3. Leave Remove Empty Lines on if blank rows are noise.
4. Turn on Sort if you want a predictable alphabetical order.
5. Click Remove Duplicates.
6. Check the stats and scan the output for the values you expected.
7. Copy the result or download it as a .txt file.

If the output looks shorter than expected, check whether case-insensitive matching merged values that you consider different. Turn it off and run the list again.

Common uses

• Email lists: remove repeated addresses before an import.
• URL lists: deduplicate links collected from a crawl or a sitemap.
• Product SKUs: clean a supplier export.
• Tags and keywords: reduce a research list to unique entries.
• Logs: collapse repeated status lines while keeping the first occurrence.
• Writing: remove repeated names or phrases from a list.

For identifiers such as order numbers or coupon codes, keep case sensitivity on. A lowercase and uppercase version may be genuinely different values, and merging them can invalidate the list.

Blank lines, counts and line endings

The duplicate count is calculated as total lines minus output lines. If Remove Empty Lines is on, the blank lines that were removed are included in that number, so the “duplicates removed” figure can be larger than the number of repeated values. Read it as “lines removed”, not strictly “duplicate values”.

The tool splits on a newline character. Windows-style CRLF input may leave a carriage return at the end of a line when trimming is off. If a match is not working, turn trimming on or normalize the line endings in a text editor first.

A single line with no newline is a valid input only if it is not blank. A completely empty textarea shows a message instead of producing an empty result.

Large lists and privacy

Deduplication is synchronous. A very large paste can make the tab unresponsive while the browser builds the set and rewrites the output. For tens of thousands of lines, split the list into smaller batches or use a command-line tool such as sort -u.

The list stays in your browser. The page does not upload it, create an account or save a copy. The download is a local blob, and the clipboard copy is handled by the browser. Clear the clipboard if the list contains sensitive addresses.

The tool does not validate the content. It will keep an invalid email or a broken URL if it is unique. Clean the format separately after deduplicating.

Deduplicating exports and structured text

Many exports are not a clean one-value-per-line list. A CSV file has a header and comma-separated fields, and a log file may repeat a timestamp on every line. The tool compares whole lines, so two records that differ only in a timestamp are not duplicates.

For a CSV column, import the file into a spreadsheet, copy the column, and paste only that column into the tool. For a log, remove or normalize the timestamp first, then deduplicate the message lines. For a list with a trailing delimiter, decide whether the delimiter is part of the value before comparing.

If you need to preserve the header while deduplicating data rows, run the tool on the data rows only and add the header back afterward. Running it on the whole file could remove a data row that happens to match the header text.

Whitespace and line endings

Leading and trailing spaces are invisible in many interfaces, but they matter to a string comparison. Trimming is on by default for that reason. If you turn it off, “value” and “value ” become different lines, and the output may look like it failed to remove a duplicate.

Windows files often use CRLF line endings. The tool splits on LF, so a carriage return can remain at the end of each line. With trimming off, that hidden character can prevent a match. Turn trimming on, or convert the file to LF in a text editor before pasting.

Tabs inside a line are preserved. A tab-separated row is still one line, not multiple fields. Do not expect the tool to understand spreadsheet columns.

Verifying the result

After deduplicating, compare the total and unique counts. If the total is 100 and the unique count is 100, nothing was removed. If the count dropped by more than the number of visible repeats, check the empty-line and case options. A large drop can also mean the list contained many blank rows.

Scan the first and last few lines. The first occurrence is kept, so the order of the remaining values should match the original unless sorting is on. If a value appears with different capitalization, the retained spelling is the one from the first occurrence.

Download the result and keep it separate from the source. The page does not provide an undo, so a second run on the output is the only way to apply a different option set. Keeping the original lets you start over without reconstructing the list.

Frequently Asked Questions