Problem with uniq command

asked 2018-03-09 00:57:43 -0500

montasimmamun
37 ●1 ●6 http://binodonjagot.com/

updated 2018-03-09 00:58:15 -0500

Say I have a file name "reading.txt" with lots of duplicates:

book

paper

article

magazine

I want to remove the duplicates, so I typed this

"uniq reading.txt"

The output should come like this

book

paper

article

magazine

But it shows all elements of reading.txt file like this

[z@localhost Documents]$ cat reading.txt book

book

paper

article

magazine

magazine [z@localhost Documents]$ uniq reading.txt book

book

paper

article

magazine

magazine [z@localhost Documents]$

edit retag flag offensive close merge delete

add a comment

2

answered 2018-03-09 01:52:47 -0500

sideburns
7723 ●18 ●101 ●146

That's interesting, and I was able to duplicate it. Checking the man page, I found this:

Note: 'uniq' does not detect repeated lines unless they are adjacent.

I edited the text file to remove the blank lines, and found that the command worked as expected. I'm not sure how much help this will be, but at least it explains what's been happening.

edit flag offensive delete link

Comments

Try sort -u

villykruse ( 2018-03-09 01:57:42 -0500 )edit

1

You should have made this an answer. You have to sort your input before using uniq in most cases.

ssieb ( 2018-03-09 02:13:54 -0500 )edit

sort -u reading.txt | uniq should deal with empty lines (unless they've been only added here, because of messed up askbot's behaviour - ignoring singular new lines).

ozeszty ( 2018-03-09 02:14:42 -0500 )edit

I see "uniq" command works fine without any blank lines. Thank you all.

montasimmamun ( 2018-03-09 02:49:14 -0500 )edit

add a comment

1

answered 2018-03-09 04:19:13 -0500

Infoweb
51 ●5

uniq removes only duplicated lines that follow each other. You can use the sort -u command, it will remove duplicates, but it will sort the strings in alphabetical order. If you need to preserve the original order of strings, try this command:

awk '!($0 in dup) { dup[$0] = ""; print $0 }' reading.txt

edit flag offensive delete link

add a comment

Problem with uniq command

2 Answers

Comments

Question Tools

Stats

Related questions

Problem with uniq command edit

2 Answers

Comments

Question Tools

Stats

Related questions

Problem with uniq command