注:本文描述的调用方和被调用方都是同一家公司内部。
一、背景
作为服务端研发,我们经常需要去调用公司内部其他部门提供的服务,发起HTTP请求调用API,印象中被调用的API默认都是http协议,也从来没有在这个协议上出现过任何问题,今天却遇到了问题,最后查名跟协议有关。
二、问题描述
上游的服务通过PHP的curl调用我提供的API服务,今天突然收到大量的报警提示调用API timeout了,上游服务端错误日志如下
curl server is error: Operation timed out after 3001 milliseconds with 0 out of -1 bytes received errorInfo: {"url":"https:\/\/office-online.niliu.me\/oos\/batchCall","content_type":null,"http_code":0, "header_size":0,"request_size":1144,"filetime":-1,"ssl_verify_result":20,"redirect_count":0, "total_time":3.001267,"namelookup_time":0.006121,"connect_time":0.009759,"pretransfer_time":0.022562, "size_upload":977,"size_download":0,"speed_download":0,"speed_upload":325,"download_content_length":-1, "upload_content_length":977,"starttransfer_time":0,"redirect_time":0,"redirect_url":"", "primary_ip":"112.34.116.16","certinfo":[],"primary_port":443,"local_ip":"192.168.255.37","local_port":37024}
错误记录的很明显,就是我的接口3秒超时了。但是查我自身服务的日志,请求根本没有过来, 从http_code为0也可以看出。
三、解决思路
1、搞清楚超时时间3秒,在哪里设置的
看了对方代码,是在curl请求的时候设置了3秒超时时间, 核心代码如下
$curl_session = curl_init(); curl_setopt($curl_session, CURLOPT_URL, $url); if ($this->_connectTimeOut) { curl_setopt($curl_session, CURLOPT_CONNECTTIMEOUT, $this->_connectTimeOut); } $http_response = curl_exec($curl_session); if (curl_errno($curl_session) != 0) { BPIT_Log::warning('curl server is error: ' . curl_error($curl_session) . ' errorInfo: ' . json_encode(curl_getinfo($curl_session))); return false; } curl_close($curl_session); return $http_response;
2、分析curl_getinfo返回的结果
发现ssl_verify_result为20,根据文档返回20的含义如下
unable to get local issuer certificate the issuer certificate of a locally looked up certificate could not be found. This normally means the list of trusted certificates is not complete.
大意是使用HTTPS的请求,但是找不到本地证书
问题同: PHP:cURL error 60: SSL certificate unable to get local issuer certificate
四、解决办法
1、调用接口不校验证书
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
2、添加证书地址(调用公司外部服务的时候会用到,详细查看这里)
下载证书:
https://curl.haxx.se/ca/cacert.pem
配置证书:(参考手册)
配置php.ini [curl] ; A default value for the CURLOPT_CAINFO option. This is required to be an ; absolute path. curl.cainfo = 【你的绝对路径】
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, true); curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, true); curl_setopt($curl,CURLOPT_CAINFO,dirname(__FILE__).'/cacert.pem');
3、把https请求修改为http请求
五、Linux下curl命令模拟
1、连接超时
curl --connect-timeout 2 --url http://xxx.com curl: (28) Connection timed out after 2004 milliseconds
2、请求超时
curl --max-time 2 --url http://niliu.me curl: (28) Operation timed out after 2002 milliseconds with 0 bytes received
3、curl请求https接口
Linux系统一般都安装了ca-certificates,证书一般在目录/etc/ssl/certs/ca-certificates.crt,请求的时候默认会从这里获取证书,也可以显示带上
curl --cacert
4、curl正常https请求
curl -v https://office-online.baidu.com/oos/call -w "time_connect: %{time_connect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n" * Trying 220.181.33.218... * TCP_NODELAY set * Connected to office-online.baidu.com (220.181.33.218) port 443 (#0) * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * successfully set certificate verify locations: * CAfile: /home/wangchuanbo/.jumbo/etc/ssl/certs/ca-certificates.crt CApath: none * TLSv1.2 (OUT), TLS header, Certificate Status (22): * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Client hello (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Client hello (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * ALPN, server accepted to use http/1.1 * Server certificate: * subject: C=CN; ST=beijing; L=beijing; OU=service operation department; O=Beijing Baidu Netcom Science Technology Co., Ltd; CN=baidu.com * start date: Oct 20 06:57:18 2020 GMT * expire date: Jul 26 05:31:02 2021 GMT * subjectAltName: host "office-online.baidu.com" matched cert's "*.baidu.com" * issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Organization Validation CA - SHA256 - G2 * SSL certificate verify ok. > GET /oos/call HTTP/1.1 > Host: office-online.baidu.com > User-Agent: curl/7.61.0 > Accept: */* > < HTTP/1.1 200 OK < Content-Length: 97 < Content-Type: application/json; charset=utf-8 < Date: Tue, 01 Jun 2021 06:36:39 GMT < Logid: 2199494315 < P3p: CP=" OTI DSP COR IVA OUR IND COM " < Server: Apache < Set-Cookie: BAIDUID=273B77B378FAE61C1D115F76F9342232:FG=1; expires=Wed, 01-Jun-22 06:36:39 GMT; max-age=31536000; path=/; domain=.baidu.com; version=1 < Tracecode: 21994943151640409280060114 < * Connection #0 to host office-online.baidu.com left intact {"errno":200001,"errmsg":"\u8bf7\u6c42\u53c2\u6570\u9519\u8bef","result":"","logid":"2199494315"}time_connect: 0.030292 time_starttransfer: 0.140660 time_total: 0.140972
5、使用curl -k忽略校验证书
从结果看,还是检查了证书
curl -k -v https://office-online.baidu.com/oos/call -w "time_connect: %{time_connect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n" * Trying 39.156.68.99... * TCP_NODELAY set * Connected to office-online.baidu.com (39.156.68.99) port 443 (#0) * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * successfully set certificate verify locations: * CAfile: /home/wangchuanbo/.jumbo/etc/ssl/certs/ca-certificates.crt CApath: none * TLSv1.2 (OUT), TLS header, Certificate Status (22): * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Client hello (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Client hello (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * ALPN, server accepted to use http/1.1 * Server certificate: * subject: C=CN; ST=beijing; L=beijing; OU=service operation department; O=Beijing Baidu Netcom Science Technology Co., Ltd; CN=baidu.com * start date: Oct 20 06:57:18 2020 GMT * expire date: Jul 26 05:31:02 2021 GMT * issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Organization Validation CA - SHA256 - G2 * SSL certificate verify ok. > GET /oos/call HTTP/1.1 > Host: office-online.baidu.com > User-Agent: curl/7.61.0 > Accept: */* > < HTTP/1.1 200 OK < Content-Length: 97 < Content-Type: application/json; charset=utf-8 < Date: Tue, 01 Jun 2021 06:39:04 GMT < Logid: 2344927726 < P3p: CP=" OTI DSP COR IVA OUR IND COM " < Server: Apache < Set-Cookie: BAIDUID=21B053641A8E18CA7A8B33FBD0B8486A:FG=1; expires=Wed, 01-Jun-22 06:39:04 GMT; max-age=31536000; path=/; domain=.baidu.com; version=1 < Tracecode: 23449277261640409280060114 < * Connection #0 to host office-online.baidu.com left intact {"errno":200001,"errmsg":"\u8bf7\u6c42\u53c2\u6570\u9519\u8bef","result":"","logid":"2344927726"}time_connect: 0.024712 time_starttransfer: 0.113276 time_total: 0.113541
6、把证书删除【或者把证书内容篡改了,一样的问题】
不忽略证书错误,无法正常返回
mv ca-certificates.crt ca-certificates.crt.bk curl -v https://office-online.baidu.com/oos/call -w "time_connect: %{time_connect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n" * Trying 220.181.33.218... * TCP_NODELAY set * Connected to office-online.baidu.com (220.181.33.218) port 443 (#0) * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * error setting certificate verify locations: CAfile: /home/wangchuanbo/.jumbo/etc/ssl/certs/ca-certificates.crt CApath: none * Closing connection 0 time_connect: 0.043693 time_starttransfer: 0.000000 time_total: 0.044306 curl: (77) error setting certificate verify locations: CAfile: /home/wangchuanbo/.jumbo/etc/ssl/certs/ca-certificates.crt CApath: none
注意⚠️:这个时候请求不会到服务端,并且观察time_starttransfer为0,尽管time_total不为0
忽略证书错误,可以得到正常结果
curl -k -v https://office-online.baidu.com/oos/call -w "time_connect: %{time_connect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n" * Trying 220.181.33.218... * TCP_NODELAY set * Connected to office-online.baidu.com (220.181.33.218) port 443 (#0) * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * error setting certificate verify locations, continuing anyway: * CAfile: /home/wangchuanbo/.jumbo/etc/ssl/certs/ca-certificates.crt CApath: none * TLSv1.2 (OUT), TLS header, Certificate Status (22): * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Client hello (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Client hello (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * ALPN, server accepted to use http/1.1 * Server certificate: * subject: C=CN; ST=beijing; L=beijing; OU=service operation department; O=Beijing Baidu Netcom Science Technology Co., Ltd; CN=baidu.com * start date: Oct 20 06:57:18 2020 GMT * expire date: Jul 26 05:31:02 2021 GMT * issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Organization Validation CA - SHA256 - G2 * SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway. > GET /oos/call HTTP/1.1 > Host: office-online.baidu.com > User-Agent: curl/7.61.0 > Accept: */* > < HTTP/1.1 200 OK < Content-Length: 97 < Content-Type: application/json; charset=utf-8 < Date: Tue, 01 Jun 2021 06:41:31 GMT < Logid: 2491174444 < P3p: CP=" OTI DSP COR IVA OUR IND COM " < Server: Apache < Set-Cookie: BAIDUID=8E3F6DC84EDFF55D87C915EFD43BD7D0:FG=1; expires=Wed, 01-Jun-22 06:41:31 GMT; max-age=31536000; path=/; domain=.baidu.com; version=1 < Tracecode: 24911744440476686528060114 < * Connection #0 to host office-online.baidu.com left intact {"errno":200001,"errmsg":"\u8bf7\u6c42\u53c2\u6570\u9519\u8bef","result":"","logid":"2491174444"}time_connect: 0.025137 time_starttransfer: 0.110727 time_total: 0.110990
注意:观察返回结果中的SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway
典型的证书不存在的问题:curl: (77) error – Stack Overflow
7、curl -k小结
-k 并不是不坚持SSL证书,只是证书错误的时候,会忽略错误。在请求https的资源时,遇到证书不匹配的问题,一般的工具都有不进行https证书验证的选项
man curl -k, --insecure (TLS) By default, every SSL connection curl makes is verified to be secure. This option allows curl to proceed and operate even for The server connection is verified by making sure the server's certificate contains the right name and verifies successfully using the cert store. See this online resource for further details: https://curl.haxx.se/docs/sslcerts.html See also --proxy-insecure and --cacert.
六、总结
1、记录日志的时候最好把curl_errno的结果记录上,我猜测这里大概是60
2、完全没有解决思路的时候,就去逐字看分析错误日志
3、服务端对服务端的HTTP请求最好要用HTTP协议,而不是HTTPS协议【不常规的操作,总是会引发一些莫名其妙的问题】
七、TODO
1、把这个结论给调用方的时候,对方问我确认是这个问题么,为什么有些请求可以,为什么突然出现这个问题。我说确认,但是为什么有些请求可以我还不知道
2、为什么我的服务端没有日志呢
3、请求触发了调用方设置的超时时间,整个请求过程发生了什么
理论上会返回499的状态码,并且我的服务端会记录日志
4、我的服务端日志中异常请求header中boundary都不完整,为什么
content-type":"multipart\/form-data; boundary=------------------------548d87d20c6bd049"
参考:
SSL Certificate Verification
https://www.php.net/manual/zh/function.curl-getinfo.php
curl: (60) SSL certificate problem: unable to get local issuer certificate