I always do something like cat file | sort -n | uniq to get a unique listing of a file to remove duplicates. the only problem with this is that it sorts the list so the ordering of the file is lost..
found a cool trick with AWK today of removing duplicate lines without destroying the order and yet its just a one liner!
awk '!a[$0]++' file
lets take an example
$ cat abc
ddd
bbb
eee
fff
aaa
ccc
bbb
kkk
aaa
zzz
xxx
yyy
now :
$ awk '!a[$0]++' abc
ddd
bbb
eee
fff
aaa
ccc
kkk
zzz
xxx
yyy
enjoy
