天天看点

数据清洗过程中常见的排序和去重操作

数据操作中排序和去重是比较常见的数据操作,本专题对排序和去重做专门介绍,并且给出一种不常用却比较有启发意义的示例:多列无序去重

目 录

1 排序

1.1 sort 单列排序返回值

1.2 order 单列排序返回索引

1.3 rank 单列排序返回“秩”

1.4 arrage 多列排序

1.5、reorder 用在绘图中

2 去重

2.1 unique 单向量/多列完全重复去重

2.2 duplicated函数

3  多列无序去重

说明:多列无序重复比较值得学习

正 文

1 排序

1.1 sort 单列排序返回值

总结:sort是直接对向量排序,返回原数值

#sort相关语法 
sort(x, decreasing = FALSE, ...)  
## Default S3 method: 
sort(x, decreasing = FALSE, na.last = NA, ...)  
sort.int(x, partial = NULL, na.last = NA, decreasing = FALSE, 
         method = c("auto", "shell", "quick", "radix"), index.return = FALSE)      

sort示例

> set.seed(416)  
> x <- round(runif(10,1,20))  
> x;sort(x) 
 [1]  9 13  7 13 20 16  4  1  6 17 
 [1]  1  4  6  7  9 13 13 16 17 20 #可以发现sort函数是对原始向量进行排序 
  
  
#如果遇到矩阵,sort函数会将矩阵转换为向量 
> set.seed(416)  
> x <- round(runif(10,1,20))  
> y <- matrix(x,nrow = 5)  
> y;sort(y) 
     [,1] [,2] 
[1,]    9   16  
[2,]   13    4  
[3,]    7    1  
[4,]   13    6  
[5,]   20   17  
 [1]  1  4  6  7  9 13 13 16 17 20 #sort(y)      

1.2 order 单列排序返回索引

总结:order先对数值排序,然后返回排序后各数值的索引

#order相关语法  
order(..., na.last = TRUE, decreasing = FALSE,  
      method = c("auto", "shell", "radix"))      

order示例

> set.seed(416)  
> x <- round(runif(10,1,20))  
> x 
 [1]  9 13  7 13 20 16  4  1  6 17 
> order(x) 
 [1]  8  7  9  3  1  2  4  6 10  5  #order返回x序列的索引值  
> sort(x)  
 [1]  1  4  6  7  9 13 13 16 17 20 
> x[order(x)]  
 [1]  1  4  6  7  9 13 13 16 17 20 #根据索引对x进行排序 
  
  
 #当遇到矩阵时,order将按列对原始矩阵进行排序,并且返回其索引向量 
 > set.seed(416)  
> x <- round(runif(10,1,20))  
> y <- matrix(x,nrow = 5)  
> y 
     [,1] [,2] 
[1,]    9   16  
[2,]   13    4  
[3,]    7    1  
[4,]   13    6  
[5,]   20   17  
> order(y) 
 [1]  8  7  9  3  1  2  4  6 10  5 #str(order(y)) 返回int  
> sort(y)  
 [1]  1  4  6  7  9 13 13 16 17 20 
> y[order(y)]  
 [1]  1  4  6  7  9 13 13 16 17 20      

1.3 rank 单列排序返回“秩”

总结:rank返回原数据各项排名(有并列的情况)

概念解释:秩是基于样本值的大小在全体样本中所占位次(秩)的统计量。

#rank语法 
rank(x, na.last = TRUE, 
     ties.method = c("average", "first", "last", "random", "max", "min"))      

rank示例

> set.seed(416)  
> x <- round(runif(10,1,20))  
> x 
 [1]  9 13  7 13 20 16  4  1  6 17 
> rank(x) #rank返回x中每个元素的秩  
 [1]  5.0  6.5  4.0  6.5 10.0  8.0  2.0  1.0  3.0  9.0      

1.4 arrage 多列排序

总结:arrange是dplyr包中的排序函数,可对数据框以列的形式进行因子排序

> library(dplyr) #加载dplyr    
> arrange(mtcars, cyl, disp) #对mtcars数据框按照cyl和disp升序排序   
    mpg cyl  disp  hp drat    wt  qsec vs am gear carb  
1  33.9   4  71.1  65 4.22 1.835 19.90  1  1    4    1  
2  30.4   4  75.7  52 4.93 1.615 18.52  1  1    4    2  
……  
6  22.8   4 108.0  93 3.85 2.320 18.61  1  1    4    1  
7  21.5   4 120.1  97 3.70 2.465 20.01  1  0    3    1  
……  
23 15.2   8 304.0 150 3.15 3.435 17.30  0  0    3    2  
……  
26 15.8   8 351.0 264 4.22 3.170 14.50  0  1    5    4  
27 18.7   8 360.0 175 3.15 3.440 17.02  0  0    3    2  
……  
32 10.4   8 472.0 205 2.93 5.250 17.98  0  0    3    4  
    
> arrange(mtcars, desc(disp)) #对mtcars数据框按照cyl升序和和disp降序排序   
    mpg cyl  disp  hp drat    wt  qsec vs am gear carb  
1  10.4   8 472.0 205 2.93 5.250 17.98  0  0    3    4  
2  10.4   8 460.0 215 3.00 5.424 17.82  0  0    3    4  
3  14.7   8 440.0 230 3.23 5.345 17.42  0  0    3    4  
……  
12 16.4   8 275.8 180 3.07 4.070 17.40  0  0    3    3  
13 17.3   8 275.8 180 3.07 3.730 17.60  0  0    3    3  
14 15.2   8 275.8 180 3.07 3.780 18.00  0  0    3    3  
15 21.4   6 258.0 110 3.08 3.215 19.44  1  0    3    1  
……  
27 22.8   4 108.0  93 3.85 2.320 18.61  1  1    4    1  
28 30.4   4  95.1 113 3.77 1.513 16.90  1  1    5    2  
……  
32 33.9   4  71.1  65 4.22 1.835 19.90  1  1    4    1      

1.5、reorder 用在绘图中

1.5.1 在graphics绘图系统中

require(graphics) 
  
bymedian <- with(InsectSprays, reorder(spray, count, median))  
boxplot(count ~ bymedian, data = InsectSprays,  
        xlab = "Type of spray", ylab = "Insect count",  
        main = "InsectSprays data", varwidth = TRUE,  
        col = "lightgray")      
数据清洗过程中常见的排序和去重操作

1.5.2 比如ggplot中绘条形图使x轴按y轴数值大小排序

说明:reorder函数具有对排序变量的因子化作用

> attach(mtcars) 
> str(reorder(gear,disp))  
 Factor w/ 3 levels "4","5","3": 1 1 1 3 3 3 3 1 1 1 ... 
 - attr(*, "scores")= num [1:3(1d)] 326 123 202  
  ..- attr(*, "dimnames")=List of 1 
  .. ..$ : chr [1:3] "3" "4" "5"  
> str(factor(gear))  
 Factor w/ 3 levels "3","4","5": 2 2 2 1 1 1 1 2 2 2 ... 
> detach(mtcars)      
library(ggplot2)  
data(mtcars)  
head(mtcars)  
ggplot(mtcars,aes(x=reorder(gear,disp), y= disp)) + geom_boxplot() + labs(title = "图1")   
ggplot(mtcars,aes(x=factor(gear), y= disp)) + geom_boxplot() + labs(title = "图2")      
数据清洗过程中常见的排序和去重操作
数据清洗过程中常见的排序和去重操作

2 去重

2.1 unique 单向量/多列完全重复去重

总结:unique中,R中默认的是fromLast=FALSE,即若样本点重复出现,则取首次出现的;否则去最后一次出现的。列名不变,去掉重复样本值之后的行名位置仍为原先的行名位置。

> df <- data.frame(x = c("A","B","C","D","E","B","C","B"), y = c("B","A","D","E","B","C","A","A"))  
> df  
  x y 
1 A B 
2 B A 
3 C D 
4 D E 
5 E B 
6 B C 
7 C A 
8 B A 
> unique(df) 
  x y 
1 A B 
2 B A 
3 C D 
4 D E 
5 E B 
6 B C 
7 C A 
> unique(df,fromLast = TRUE) 
  x y 
1 A B 
3 C D 
4 D E 
5 E B 
6 B C 
7 C A 
8 B A      

2.2 duplicated函数

总结:duplicated可对原数据框做单列或多列去重,并且返回波尔向量(索引)

> df <- data.frame(x = c("A","B","C","D","E","B","C","B"), y = c("B","A","D","E","B","C","A","A"))  
> df  
  x y 
1 A B 
2 B A 
3 C D 
4 D E 
5 E B 
6 B C 
7 C A 
8 B A 
> df_index <- duplicated(df$x) #构建一个布尔向量(索引)  
> df_index  
[1] FALSE FALSE FALSE FALSE FALSE  TRUE  TRUE  TRUE 
> df[!df_index,] #筛选 
  x y 
1 A B 
2 B A 
3 C D 
4 D E 
5 E B      

3  多列无序去重

总结:多列无序去重指,多列非按照独立列比较重复,而是指逐行比较每一行是否出现过此元素(不按照列顺序).

例如:matrix(c("a","b"),nrow = 1) 和 matrix(c("b","a"),nrow = 1)也是重复

> data.frame(matrix(c("a","b"),nrow = 1))  
  X1 X2 
1  a  b 
> data.frame(matrix(c("b","a"),nrow = 1))  
  X1 X2 
1  b  a      
#生成测试集 
> df <- data.frame(x = c("A","B","C","D","E","B","C","B"), y = c("B","C","D","E","B","C","A","A"),z = c(1:8)) 
#对数据集df[,c(1:2)]逐行操作排序,并将排序后结果合并  
> df$merge <- apply(df[,c(1:2)],1,function(x) paste(sort(x),collapse='')) 
#对逐行排序合并的结果进行去重,返回索引向量,然后(反向!)筛选  
> df_du<-df[!duplicated(df$merge),] 
> df 
  x y z merge 
1 A B 1    AB 
2 B C 2    BC 
3 C D 3    CD 
4 D E 4    DE 
5 E B 5    BE 
6 B C 6    BC 
7 C A 7    AC 
8 B A 8    AB 
> df_du  
  x y z merge 
1 A B 1    AB 
2 B C 2    BC 
3 C D 3    CD 
4 D E 4    DE 
5 E B 5    BE 
7 C A 7    AC