主页 / ubuntu / 问题

问题[uniq](ubuntu)

Martin Hope
Amin
Asked: 2019-08-27 04:29:19 +0800 CST
  • 1

没有唯一性：

amin@ubuntu:~/Desktop$ cut -f 1 info.log | tail -n +2 | head -n -1 | sort
Abol
Abol 
Ahmad
Akbar
Arash
Hadi 
Hamed
Mahmood
Maryam
Maryam
Mohsen
NIma
Rasool
Sadegh
Sepide
Sepide

使用独特的：

amin@ubuntu:~/Desktop$ cut -f 1 info.log | tail -n +2 | head -n -1 | sort | uniq
Abol
Abol 
Ahmad
Akbar
Arash
Hadi 
Hamed
Mahmood
Maryam
Mohsen
NIma
Rasool
Sadegh
Sepide
Sepide

如您所见，两者的结果相同，为什么？

text-processing uniq
Martin Hope
j0h
Asked: 2019-03-29 13:51:53 +0800 CST
  • 19

我有一个按 IP 地址排序的日志文件，我想查找每个唯一 IP 地址的出现次数。我怎么能用 bash 做到这一点？可能会在 ip 旁边列出出现次数，例如：

5.135.134.16 count: 5
13.57.220.172: count 30
18.206.226 count:2

等等。

以下是日志示例：

5.135.134.16 - - [23/Mar/2019:08:42:54 -0400] "GET /wp-login.php HTTP/1.1" 200 2988 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
5.135.134.16 - - [23/Mar/2019:08:42:55 -0400] "GET /wp-login.php HTTP/1.1" 200 2988 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
5.135.134.16 - - [23/Mar/2019:08:42:55 -0400] "POST /wp-login.php HTTP/1.1" 200 3836 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
5.135.134.16 - - [23/Mar/2019:08:42:55 -0400] "POST /wp-login.php HTTP/1.1" 200 3988 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
5.135.134.16 - - [23/Mar/2019:08:42:56 -0400] "POST /xmlrpc.php HTTP/1.1" 200 413 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
13.57.220.172 - - [23/Mar/2019:11:01:05 -0400] "GET /wp-login.php HTTP/1.1" 200 2988 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
13.57.220.172 - - [23/Mar/2019:11:01:06 -0400] "POST /wp-login.php HTTP/1.1" 200 3985 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
13.57.220.172 - - [23/Mar/2019:11:01:07 -0400] "GET /wp-login.php HTTP/1.1" 200 2988 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
13.57.220.172 - - [23/Mar/2019:11:01:08 -0400] "POST /wp-login.php HTTP/1.1" 200 3833 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
13.57.220.172 - - [23/Mar/2019:11:01:09 -0400] "GET /wp-login.php HTTP/1.1" 200 2988 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
13.57.220.172 - - [23/Mar/2019:11:01:11 -0400] "POST /wp-login.php HTTP/1.1" 200 3836 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
13.57.220.172 - - [23/Mar/2019:11:01:12 -0400] "GET /wp-login.php HTTP/1.1" 200 2988 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
13.57.220.172 - - [23/Mar/2019:11:01:15 -0400] "POST /wp-login.php HTTP/1.1" 200 3837 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
13.57.220.172 - - [23/Mar/2019:11:01:17 -0400] "POST /xmlrpc.php HTTP/1.1" 200 413 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
13.57.233.99 - - [23/Mar/2019:04:17:45 -0400] "GET / HTTP/1.1" 200 25160 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36"
18.206.226.75 - - [23/Mar/2019:21:58:07 -0400] "GET /wp-login.php HTTP/1.1" 200 2988 "https://www.google.com/url?3a622303df89920683e4421b2cf28977" "Mozilla/5.0 (Windows NT 6.2; rv:33.0) Gecko/20100101 Firefox/33.0"
18.206.226.75 - - [23/Mar/2019:21:58:07 -0400] "POST /wp-login.php HTTP/1.1" 200 3988 "https://www.google.com/url?3a622303df89920683e4421b2cf28977" "Mozilla/5.0 (Windows NT 6.2; rv:33.0) Gecko/20100101 Firefox/33.0"
18.213.10.181 - - [23/Mar/2019:14:45:42 -0400] "GET /wp-login.php HTTP/1.1" 200 2988 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
18.213.10.181 - - [23/Mar/2019:14:45:42 -0400] "GET /wp-login.php HTTP/1.1" 200 2988 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
18.213.10.181 - - [23/Mar/2019:14:45:42 -0400] "GET /wp-login.php HTTP/1.1" 200 2988 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
command-line bash sort uniq
Martin Hope
Suat Yazıcı
Asked: 2018-08-22 20:19:05 +0800 CST
  • 4

我有一个 xlsx 文件（110725x9 矩阵），我保存为文本类型（制表符删除），因为我不知道 Unix 是否有助于 xlsx 文件。重复行总是逐行连续。

例如，假设文本文件如下。您会看到第 3,4、第 7,8 和第 17,18 行是相同的。我想删除不总是更低的上部重复行。

2009,37214611872    2009    135 20  17,1    17,4    19,2    21,8    24,1
2009,37237442922    2009    135 22  16,5    14,5    12,6    11,2    10,5
2009,37260273973    2009    136 0   7,7     7,2     7,1     7,3     7,5
2009,37260273973    2009    136 0   7,7     7,2     7,0     7,2    7,4
2009,37488584475    2009    136 20  14,6    15,1    16,4    18,3    20,1
2009,37511415525    2009    136 22  15,9    14,6    12,8    10,9    9,4
2009,37534246575    2009    137 0   8,2     6,9     6,2     6,2     6,4
2009,37534246575    2009    137 0   8,1     6,8     6,1     6,0     6,3
2009,37557077626    2009    137 2   6,8     6,7     6,5     6,3     6,2
2009,37579908676    2009    137 4   5,8     5,6     5,4     5,4     5,7
2009,37602739726    2009    137 6   6,3     6,1     5,9     5,8     5,8
2009,37625570776    2009    137 8   4,5     5,2     6,0     6,6     7,2
2009,37648401826    2009    137 10  9,6     9,0     8,4     8,4     9,1
2009,37671232877    2009    137 12  11,4    11,7    12,4    13,4    14,4
2009,37694063927    2009    137 14  12,4    13,1    14,2    15,4    16,7
2009,37785388128    2009    137 22  15,5    14,0    12,2    10,3    8,7
2009,37808219178    2009    138 0   6,3     5,8     5,5     5,5     5,8
2009,37808219178    2009    138 0   6,2     5,7     5, 4    5,4     5,7

所以输出应该是这样的：

2009,37214611872    2009    135 20  17,1    17,4    19,2    21,8    24,1
2009,37237442922    2009    135 22  16,5    14,5    12,6    11,2    10,5
2009,37260273973    2009    136 0   7,7     7,2     7,0     7,2    7,4
2009,37488584475    2009    136 20  14,6    15,1    16,4    18,3    20,1
2009,37511415525    2009    136 22  15,9    14,6    12,8    10,9    9,4
2009,37534246575    2009    137 0   8,1     6,8     6,1     6,0     6,3
2009,37557077626    2009    137 2   6,8     6,7     6,5     6,3     6,2
2009,37579908676    2009    137 4   5,8     5,6     5,4     5,4     5,7
2009,37602739726    2009    137 6   6,3     6,1     5,9     5,8     5,8
2009,37625570776    2009    137 8   4,5     5,2     6,0     6,6     7,2
2009,37648401826    2009    137 10  9,6     9,0     8,4     8,4     9,1
2009,37671232877    2009    137 12  11,4    11,7    12,4    13,4    14,4
2009,37694063927    2009    137 14  12,4    13,1    14,2    15,4    16,7
2009,37785388128    2009    137 22  15,5    14,0    12,2    10,3    8,7
2009,37808219178    2009    138 0   6,2     5,7     5, 4    5,4     5,7

我怎么能不排序呢？

command-line duplicate text-processing uniq
Martin Hope
Spencer Hastings
Asked: 2018-07-06 12:39:37 +0800 CST
  • -1

我使用 vi 创建并编辑了一个名为 sort.txt 的文件，如下所示：

sentence one 
sentence two
sentence one
sentence three
sentence two
sentence two
sentence three
sentence one
sentence two
sentence one
sentence one
sentence one
sentence three
sentence four
sentence five

当我这样做时，sort sort.txt | uniq -c 我得到了这个：

  1 sentence five
  1 sentence four
  5 sentence one
  1 sentence one 
  3 sentence three
  4 sentence two

另外，当我写的时候sort sort.txt | uniq -u，“sentence one”也出现在那里。我不明白为什么“句子一”出现两次，好像内容不同，有人可以解释为什么吗？

command-line text-processing uniq