CloudWatchLogs request errors

[jimmyt857]

When calling CreateLogGroup, CreateLogStream, DeleteLogStream, or PutLogEvents, I am getting curl error 35 (ssl connect error). Any of the first 3 will just fail after ~10 attempts, but for some reason PutLogEvents always succeeds on the 7th attempt. Here is an example log trace:

[INFO] 2016-07-15 03:29:18 CurlHandleContainer [139736288929792] Initializing CurlHandleContainer with size 25
[TRACE] 2016-07-15 03:29:18 AWSClient [139736288929792] Found body, but content-length has not been set, attempting to compute content-length
[DEBUG] 2016-07-15 03:29:18 AWSAuthV4Signer [139736288929792] Calculated sha256 ed5df988d44e451e18fedf10d89327e33aa4ec818bf5eac665b2679d77c22dee for payload.
[DEBUG] 2016-07-15 03:29:18 AWSAuthV4Signer [139736288929792] Canonical Header String: content-length:31
content-type:application/x-amz-json-1.1
host:logs.us-west-2.amazonaws.com
user-agent:aws-sdk-cpp/0.13.4 Linux/3.13.0-74-generic x86_64
x-amz-content-sha256:ed5df988d44e451e18fedf10d89327e33aa4ec818bf5eac665b2679d77c22dee
x-amz-date:20160715T032918Z
x-amz-target:Logs_20140328.CreateLogGroup

[DEBUG] 2016-07-15 03:29:18 AWSAuthV4Signer [139736288929792] Signed Headers value:content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-target
[DEBUG] 2016-07-15 03:29:18 AWSAuthV4Signer [139736288929792] Canonical Request String: POST
/

content-length:31
content-type:application/x-amz-json-1.1
host:logs.us-west-2.amazonaws.com
user-agent:aws-sdk-cpp/0.13.4 Linux/3.13.0-74-generic x86_64
x-amz-content-sha256:ed5df988d44e451e18fedf10d89327e33aa4ec818bf5eac665b2679d77c22dee
x-amz-date:20160715T032918Z
x-amz-target:Logs_20140328.CreateLogGroup

content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-target
ed5df988d44e451e18fedf10d89327e33aa4ec818bf5eac665b2679d77c22dee
[DEBUG] 2016-07-15 03:29:18 AWSAuthV4Signer [139736288929792] Final String to sign: AWS4-HMAC-SHA256
20160715T032918Z
20160715/us-west-2/logs/aws4_request
6bde1138796478370ca249eb333f7dc6fb618b90cb07f5874081196b130dd249
[DEBUG] 2016-07-15 03:29:18 AWSAuthV4Signer [139736288929792] Final computed signing hash: ca9443049e51356c14eba7fb8025aa0ebc589d4c7b4324df833df4f8c4300651
[DEBUG] 2016-07-15 03:29:18 AWSAuthV4Signer [139736288929792] Signing request with: AWS4-HMAC-SHA256 Credential=AKIAIMTDWTMSLPNECNJA/20160715/us-west-2/logs/aws4_request, SignedHeaders=content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-target, Signature=ca9443049e51356c14eba7fb8025aa0ebc589d4c7b4324df833df4f8c4300651
[DEBUG] 2016-07-15 03:29:18 AWSClient [139736288929792] Request Successfully signed
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] Making request to https://logs.us-west-2.amazonaws.com
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] Including headers:
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] authorization: AWS4-HMAC-SHA256 Credential=AKIAIMTDWTMSLPNECNJA/20160715/us-west-2/logs/aws4_request, SignedHeaders=content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-target, Signature=ca9443049e51356c14eba7fb8025aa0ebc589d4c7b4324df833df4f8c4300651
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] content-length: 31
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] content-type: application/x-amz-json-1.1
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] host: logs.us-west-2.amazonaws.com
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] user-agent: aws-sdk-cpp/0.13.4 Linux/3.13.0-74-generic x86_64
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] x-amz-content-sha256: ed5df988d44e451e18fedf10d89327e33aa4ec818bf5eac665b2679d77c22dee
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] x-amz-date: 20160715T032918Z
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] x-amz-target: Logs_20140328.CreateLogGroup
[DEBUG] 2016-07-15 03:29:18 CurlHandleContainer [139736288929792] Attempting to acquire curl connection.
[DEBUG] 2016-07-15 03:29:18 CurlHandleContainer [139736288929792] No current connections available in pool. Attempting to create new connections.
[DEBUG] 2016-07-15 03:29:18 CurlHandleContainer [139736288929792] attempting to grow pool size by 2
[INFO] 2016-07-15 03:29:18 CurlHandleContainer [139736288929792] Pool successfully grown by 2
[DEBUG] 2016-07-15 03:29:18 CurlHandleContainer [139736288929792] Returning connection handle 0x42746a0
[DEBUG] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] Obtained connection handle 0x42746a0
[ERROR] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] Curl returned error code 35
[DEBUG] 2016-07-15 03:29:18 CurlHandleContainer [139736288929792] Releasing curl handle 0x42746a0
[DEBUG] 2016-07-15 03:29:18 CurlHandleContainer [139736288929792] Notifying waiting threads.
[DEBUG] 2016-07-15 03:29:18 AWSClient [139736288929792] Request returned error. Attempting to generate appropriate error codes from response
[WARN] 2016-07-15 03:29:18 AWSClient [139736288929792] Request failed, now waiting 0 ms before attempting again.

[JonathanHenson]

This actually looks like an TLS issue (error code 35). Can you turn on the verbose line in the curl client so I can see the TLS handshake output?

[jimmyt857]

Sure; do you mean these two lines?

[JonathanHenson]

Just the verbose line.

Sent from my iPhone

On Jul 17, 2016, at 1:48 PM, jimmyt857 notifications@github.com wrote:

Sure; do you mean these two lines?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[jimmyt857]

Here is the output from several attempts to connect. It's mostly this "Illegar parameter" error, but there is 1 Decrypt error instead.

• Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
• Hostname was NOT found in DNS cache
• Trying 54.240.253.17...
• Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 ([Question] Default MemorySystemInterface? #6)
• found 173 certificates in /etc/ssl/certs/ca-certificates.crt
• gnutls_handshake() failed: Illegal parameter
• Closing connection 6
• Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
• Hostname was found in DNS cache
• Trying 54.240.253.17...
• Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (Error compiling: fatal error: curl/curl.h: No such file or directory #7)
• found 173 certificates in /etc/ssl/certs/ca-certificates.crt
• gnutls_handshake() failed: Illegal parameter
• Closing connection 7
• Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
• Hostname was found in DNS cache
• Trying 54.240.253.17...
• Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (error: comparison between signed and unsigned integer expressions [-Werror=sign-compare] #8)
• found 173 certificates in /etc/ssl/certs/ca-certificates.crt
• gnutls_handshake() failed: Illegal parameter
• Closing connection 8
• Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
• Hostname was found in DNS cache
• Trying 54.240.253.17...
• Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (Iterator-based APIs #9)
• found 173 certificates in /etc/ssl/certs/ca-certificates.crt
• gnutls_handshake() failed: Decrypt error
• Closing connection 9
• Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
• Hostname was found in DNS cache
• Trying 54.240.253.17...
• Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (This repo is too big #10)
• found 173 certificates in /etc/ssl/certs/ca-certificates.crt
• gnutls_handshake() failed: Illegal parameter
• Closing connection 10
• Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
• Hostname was found in DNS cache
• Trying 54.240.253.17...
• Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (Fixed build errors and failed unit tests with MSVC 2015 #11)
• found 173 certificates in /etc/ssl/certs/ca-certificates.crt
• gnutls_handshake() failed: Decrypt error
• Closing connection 11
• Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
• Hostname was found in DNS cache
• Trying 54.240.253.17...
• Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (aws-cpp-sdk-example1 broken ? #12)
• found 173 certificates in /etc/ssl/certs/ca-certificates.crt
• gnutls_handshake() failed: Illegal parameter
• Closing connection 12

[JonathanHenson]

Ah... I think you are using the wrong libcurl. You need the one linked and compiled against OpenSSL. GnuTLS isn't compatible with our load balancers.

Sent from my iPhone

On Jul 18, 2016, at 11:22 AM, jimmyt857 notifications@github.com wrote:

Here is the output from several attempts to connect. It's mostly this "Illegar parameter" error, but there is 1 Decrypt error instead.

Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
Hostname was NOT found in DNS cache
Trying 54.240.253.17...
Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (#6)
found 173 certificates in /etc/ssl/certs/ca-certificates.crt
gnutls_handshake() failed: Illegal parameter
Closing connection 6
Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
Hostname was found in DNS cache
Trying 54.240.253.17...
Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (#7)
found 173 certificates in /etc/ssl/certs/ca-certificates.crt
gnutls_handshake() failed: Illegal parameter
Closing connection 7
Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
Hostname was found in DNS cache
Trying 54.240.253.17...
Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (#8)
found 173 certificates in /etc/ssl/certs/ca-certificates.crt
gnutls_handshake() failed: Illegal parameter
Closing connection 8
Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
Hostname was found in DNS cache
Trying 54.240.253.17...
Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (#9)
found 173 certificates in /etc/ssl/certs/ca-certificates.crt
gnutls_handshake() failed: Decrypt error
Closing connection 9
Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
Hostname was found in DNS cache
Trying 54.240.253.17...
Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (#10)
found 173 certificates in /etc/ssl/certs/ca-certificates.crt
gnutls_handshake() failed: Illegal parameter
Closing connection 10
Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
Hostname was found in DNS cache
Trying 54.240.253.17...
Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (#11)
found 173 certificates in /etc/ssl/certs/ca-certificates.crt
gnutls_handshake() failed: Decrypt error
Closing connection 11
Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
Hostname was found in DNS cache
Trying 54.240.253.17...
Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (#12)
found 173 certificates in /etc/ssl/certs/ca-certificates.crt
gnutls_handshake() failed: Illegal parameter
Closing connection 12
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[jimmyt857]

My curl --version is outputting
curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
so I think it is using openssl. Also, it is only these cloudwatch commands that have the problem.

Do you have a suggestion for how to install curl? I tried this repository https://github.com/curl/curl but I get compile issues when I try to build the aws library afterwards.

[jimmyt857]

Ok, I was able to get this working. I was originally getting the following errors when building:

aws-sdk-cpp/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:123:5: error: invalid conversion from ‘void*’ to ‘CURL* {aka Curl_easy*}’ [-fpermissive]
curl_easy_setopt(handle, CURLOPT_NOSIGNAL, 1L);
^
aws-sdk-cpp/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:124:5: error: invalid conversion from ‘void*’ to ‘CURL* {aka Curl_easy*}’ [-fpermissive]
curl_easy_setopt(handle, CURLOPT_TIMEOUT_MS, m_requestTimeout);
^
aws-sdk-cpp/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:125:5: error: invalid conversion from ‘void*’ to ‘CURL* {aka Curl_easy*}’ [-fpermissive]
curl_easy_setopt(handle, CURLOPT_CONNECTTIMEOUT_MS, m_connectTimeout);

After adding the casts, I was able to build and it worked. Could we add these casts to the library?

[JonathanHenson]

We don't get those errors when we build, so to evaluate this I need to know versions of the library and header files you are using.

curl --version is a fully built and linked binary. This is very much different from what you've actually built and linked against. There are multiple variants of the libcurl header files, some which use openssl, some gnutls, and some that use other dependencies.

I am certain, assuming the verbose output you showed is correct, that you were building against the gnutls variant. It may work against some of our load balancers, but it is also known widely to not work in most cases, so you shouldn't use it.

[jimmyt857]

I am using the current version of https://github.com/curl/curl, building as follows:

autoreconf -i
./configure --with-ssl=/usr/lib/ssl --libdir=/usr/lib/x86_64-linux-gnu
make
sudo make install

[ambasta]

Hi,

Piggybacking on this issue, I too am getting:

In file included from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/include/aws/core/http/curl/CurlHandleContainer.h:23:0,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:16:
/home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp: In member function ‘void Aws::Http::CurlHandleContainer::SetDefaultOptionsOnHandle(void*)’:
/home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:123:5: error: invalid conversion from ‘void*’ to ‘CURL* {aka Curl_easy*}’ [-fpermissive]
     curl_easy_setopt(handle, CURLOPT_NOSIGNAL, 1L);
     ^
In file included from /usr/include/curl/curl.h:2420:0,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/include/aws/core/http/curl/CurlHandleContainer.h:23,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:16:
/usr/include/curl/easy.h:29:22: note:   initializing argument 1 of ‘CURLcode curl_easy_setopt(CURL*, CURLoption, ...)’
 CURL_EXTERN CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...);
                      ^
In file included from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/include/aws/core/http/curl/CurlHandleContainer.h:23:0,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:16:
/home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:124:5: error: invalid conversion from ‘void*’ to ‘CURL* {aka Curl_easy*}’ [-fpermissive]
     curl_easy_setopt(handle, CURLOPT_TIMEOUT_MS, m_requestTimeout);
     ^
In file included from /usr/include/curl/curl.h:2420:0,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/include/aws/core/http/curl/CurlHandleContainer.h:23,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:16:
/usr/include/curl/easy.h:29:22: note:   initializing argument 1 of ‘CURLcode curl_easy_setopt(CURL*, CURLoption, ...)’
 CURL_EXTERN CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...);
                      ^
In file included from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/include/aws/core/http/curl/CurlHandleContainer.h:23:0,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:16:
/home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:125:5: error: invalid conversion from ‘void*’ to ‘CURL* {aka Curl_easy*}’ [-fpermissive]
     curl_easy_setopt(handle, CURLOPT_CONNECTTIMEOUT_MS, m_connectTimeout);
     ^
In file included from /usr/include/curl/curl.h:2420:0,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/include/aws/core/http/curl/CurlHandleContainer.h:23,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:16:
/usr/include/curl/easy.h:29:22: note:   initializing argument 1 of ‘CURLcode curl_easy_setopt(CURL*, CURLoption, ...)’
 CURL_EXTERN CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...);
                      ^
make[2]: *** [aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/build.make:1164: aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/source/http/curl/CurlHandleContainer.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:86: aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/all] Error 2
make: *** [Makefile:128: all] Error 2

I've curl compiled w/ openssl

net-misc/curl-7.50.0::gentoo USE="ldap ssl -adns -http2 -idn -ipv6 -kerberos -metalink -rtmp -samba -ssh -static-libs {-test} -threads" CURL_SSL="openssl -axtls -gnutls -libressl -mbedtls -nss -polarssl (-winssl)"

[jimmyt857]

Since I have to make this edit every time I update my version of the library, I've been looking at this file some more; CURL* seems to be used already in several other places in the file. In fact, both instances of this function being called in this file pass a CURL*, and it isn't called from anywhere else.

[singku]

@jimmyt857 @ambasta Hello, it seems you have fixed the original problem after compiling curl with openssl. Is the casting problem still exists?