月度归档:2018年11月

PHP输出缓冲(output buffer)初级理解和应用

声明:不同的PHP版本,操作系统环境,Webserver, 参数默认值及配置可能会有差异。本文基于的环境如下:
PHP version: 7.2.8; Nginx version 1.14.0
Mac OS version: 10.10.5; Chrome version: 69.0;
Safari version: 8.0.8

第一部分 基础概念

计算机缓冲区(buffer)
计算机中暂时存放输出或输入信息的内存区域。缓和高速部件和低速部件之间通信速度不匹配的矛盾。缓冲(buffer)和缓存(cache)之间还是有本质区别的【点击查看区别】。
继续阅读

awk常用形式整理

这里并不打算介绍awk,也不是简明教程,如果想看教程,推荐耗子的文章《AWK 简明教程》

awk格式如下:

awk '{pattern + action}' {filenames}

以下展示为使用过的场景整理:
0、对齐输出部分列

$ cat 20180928.log
BANG|2018-09-28 20:00:02|||0|not same|topic 中国新说唱|md5 70a33ddbaae24c83d9bf1b18786d17cf|old read 6743582553 | new read 6743903396 | old mention 34576698| new mention 34576697 |read_inc: 320843| menton_inc: -1
BANG|2018-09-28 20:00:02|||0|not same|topic 明日之子|md5 470c7ea352169bc4cf766cc10126e3d9|old read 19442104721 | new read 19442126259 | old mention 87717229| new mention 87979643 |read_inc: 21538| menton_inc: 262414
BANG|2018-09-28 20:00:02|||0|the same|topic 综艺ELEVEN|md5 61eeebb92bc7a9d216633e1e1604a0a3|old read 0 | new read 0 | old mention 0| new mention 0
BANG|2018-09-28 20:00:02|||0|not same|topic 加油好身材|md5 b0e3dcf03ab2c4bdb916f083c3ecd861|old read 90588874 | new read 92111031 | old mention 104225| new mention 104224 |read_inc: 1522157| menton_inc: -1
BANG|2018-09-28 20:00:02|||0|not same|topic 这就是灌篮|md5 d4da257e8be25298e2c4b245e47632b8|old read 2694756739 | new read 2694765957 | old mention 5403696| new mention 5403696 |read_inc: 9218| menton_inc: 0
BANG|2018-09-28 20:00:02|||0|not same|topic 我在大理寺当宠物|md5 ca932540077f8c959187f06c2ecf9131|old read 71751204 | new read 71750514 | old mention 89286| new mention 89285 |read_inc: -690| menton_inc: -1
BANG|2018-09-28 20:00:02|||0|not same|topic 优酷秋集|md5 edca63655c351a1a503c1bd4d90b4505|old read 906746383 | new read 913171740 | old mention 5868201| new mention 5958567 |read_inc: 6425357| menton_inc: 90366
BANG|2018-09-28 20:00:02|||0|not same|topic 凉生,我们可不可以不忧伤|md5 c47f14a0d063e103d3cfbab61ac29231|old read 502732993 | new read 502732992 | old mention 502829| new mention 502829 |read_inc: -1| menton_inc: 0
BANG|2018-09-28 20:00:02|||0|not same|topic 橙红年代|md5 b5bef8afd1e7fa2fd6effd1d7b69947c|old read 859229830 | new read 859229076 | old mention 2828826| new mention 2828826 |read_inc: -754| menton_inc: 0
BANG|2018-09-28 20:00:02|||0|not same|topic 相声有新人|md5 222d028fd017cb630fd58f951d956f8f|old read 346645090 | new read 347368858 | old mention 521064| new mention 521065 |read_inc: 723768| menton_inc: 1

使用awk展示部分列,并对齐输出

$ awk -F '|' '{print $2, $7, $13, $14}' 20180928.log |column -t

1、多个分隔符

$ cat access.log
2017-03-24 15:23:18|x aaa|AA\n
2017-03-24 15:24:15|x bbb|BB\n
2017-03-25 22:18:55|x ccc|CC\n
$ grep '2017-03-24' | awk -F ['=|'] '{print $3}' awk.log
aaa
bbb
ccc

2、按照某一列排序

cat /tmp/debug_vote_log/20190319.log | grep '2118' | grep '2019-03-19 22:00'
debug_vote_log|2019-03-19 22:00:02|10.77.40.52|10.73.14.110|6033294916|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656415
debug_vote_log|2019-03-19 22:00:13|10.85.55.62|10.73.14.110|6033294916|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656415
debug_vote_log|2019-03-19 22:00:18|172.16.143.158|10.73.14.110|6033294916|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656417
debug_vote_log|2019-03-19 22:00:21|10.22.3.50|10.73.14.110|6033294916|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656419
debug_vote_log|2019-03-19 22:00:24|10.131.235.5|10.73.14.110|6033294916|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656419
debug_vote_log|2019-03-19 22:00:26|10.131.227.167|10.73.14.110|6033294916|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656420
debug_vote_log|2019-03-19 22:00:42|172.16.88.211|10.73.14.110|6978645170|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656420
debug_vote_log|2019-03-19 22:00:43|10.41.21.69|10.73.14.110|2520938221|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656420
debug_vote_log|2019-03-19 22:00:44|172.16.190.92|10.73.14.110|2520938221|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656422
debug_vote_log|2019-03-19 22:00:44|10.41.20.109|10.73.14.110|2520938221|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656424
debug_vote_log|2019-03-19 22:00:44|10.41.21.122|10.73.14.110|2520938221|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656425
debug_vote_log|2019-03-19 22:00:45|10.85.55.245|10.73.14.110|2520938221|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656427
debug_vote_log|2019-03-19 22:00:46|10.22.3.21|10.73.14.110|2520938221|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656432
debug_vote_log|2019-03-19 22:00:46|10.131.234.154|10.73.14.110|2520938221|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656434
debug_vote_log|2019-03-19 22:00:47|172.16.36.175|10.73.14.110|2520938221|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656436
debug_vote_log|2019-03-19 22:00:48|10.131.238.83|10.73.14.110|2520938221|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656438
debug_vote_log|2019-03-19 22:00:48|10.85.48.65|10.73.14.110|2520938221|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656438
debug_vote_log|2019-03-19 22:00:49|10.85.48.112|10.73.14.110|2520938221|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656442
debug_vote_log|2019-03-19 22:00:50|10.41.21.159|10.73.14.110|2520938221|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656446
debug_vote_log|2019-03-19 22:00:50|10.22.1.42|10.73.14.110|2520938221|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656447
debug_vote_log|2019-03-19 22:00:51|10.22.3.51|10.73.14.110|2520938221|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656449
debug_vote_log|2019-03-19 22:00:52|10.41.25.167|10.73.14.110|6978645170|page_id: 1745| tab_id: 2000| oid: 2118 | count: 656450

按照票数倒序,票数相同的按照时间升序,并打印出time,oid,和count

cat /tmp/debug_vote_log/20190319.log | grep '2118' | grep '2019-03-19 22:00' | awk -F '|' '{print $2, $8, $9}' | sort -t ' ' -k 6 -rn -k 2
2019-03-19 22:00:52  oid: 2118   count: 656450
2019-03-19 22:00:51  oid: 2118   count: 656449
2019-03-19 22:00:50  oid: 2118   count: 656447
2019-03-19 22:00:50  oid: 2118   count: 656446
2019-03-19 22:00:49  oid: 2118   count: 656442
2019-03-19 22:00:48  oid: 2118   count: 656438
2019-03-19 22:00:48  oid: 2118   count: 656438
2019-03-19 22:00:47  oid: 2118   count: 656436
2019-03-19 22:00:46  oid: 2118   count: 656434
2019-03-19 22:00:46  oid: 2118   count: 656432
2019-03-19 22:00:45  oid: 2118   count: 656427
2019-03-19 22:00:44  oid: 2118   count: 656425
2019-03-19 22:00:44  oid: 2118   count: 656424
2019-03-19 22:00:44  oid: 2118   count: 656422
2019-03-19 22:00:43  oid: 2118   count: 656420
2019-03-19 22:00:42  oid: 2118   count: 656420
2019-03-19 22:00:26  oid: 2118   count: 656420
2019-03-19 22:00:24  oid: 2118   count: 656419
2019-03-19 22:00:21  oid: 2118   count: 656419
2019-03-19 22:00:18  oid: 2118   count: 656417
2019-03-19 22:00:13  oid: 2118   count: 656415
2019-03-19 22:00:02  oid: 2118   count: 656415

关于linux sort命令参数:

-f  :忽略大小写的差异,例如 A 与 a 视为编码相同;
-b  :忽略最前面的空格符部分;
-M  :以月份的名字来排序,例如 JAN, DEC 等等的排序方法;
-n  :使用『纯数字』进行排序(默认是以文字型态来排序的);
-r  :反向排序;
-u  :就是 uniq ,相同的数据中,仅出现一行代表;
-t  :分隔符,默认是用 [tab] 键来分隔;
-k  :以那个区间 (field) 来进行排序的意思, 数字表示列

完整用法参考《linux sort 多列正排序,倒排序》