crash with the SDK library

[msinha31]

I am sending the stack trace also code is attached
build options

-Wall -std=c++0x -g -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -mtune=core2 -l aws_dynamo -pedantic-errors -Wall -Wno-long-long -O1 -L/usr/lib -lstdc++ -lm -laws-cpp-sdk-dynamodb -laws-cpp-sdk-core -laws-cpp-sdk-sdb -DAWS_CUSTOM_MEMORY_MANAGEMENT=1"

Stack trace:

Program terminated with signal 11, Segmentation fault.
#0 std::basic_string<char, std::char_traits, Aws::Allocator >::_Rep::_S_create (__capacity=__capacity@entry=5,

__old_capacity=__old_capacity@entry=0, __alloc=...) at /usr/include/c++/4.8.3/bits/basic_string.tcc:609

609 __p->_M_capacity = __capacity;
Missing separate debuginfos, use: debuginfo-install glibc-2.17-55.145.amzn1.x86_64 gperftools-libs-2.0-11.5.amzn1.x86_64 keyutils-libs-1.5.8-3.12.amzn1.x86_64 krb5-libs-1.13.2-10.39.amzn1.x86_64 libcom_err-1.42.12-4.40.amzn1.x86_64 libevent-2.0.18-1.11.amzn1.x86_64 libselinux-2.1.10-3.22.amzn1.x86_64 openssl-1.0.1k-13.88.amzn1.x86_64 zlib-1.2.8-7.18.amzn1.x86_64
(gdb) where
#0 std::basic_string<char, std::char_traits, Aws::Allocator >::_Rep::_S_create (__capacity=__capacity@entry=5,

__old_capacity=__old_capacity@entry=0, __alloc=...) at /usr/include/c++/4.8.3/bits/basic_string.tcc:609

#1 0x0000000000823f91 in _S_construct<char const*> (__a=..., __end=,

__beg=0x7f4ccb11a5e8 <vtable for std::_Sp_counted_ptr_inplace<Aws::Client::DynamoDBErrorMarshaller, Aws::Allocator<Aws::Client::DynamoDBErrorMarshaller>, (__gnu_cxx::_Lock_policy)2>+40> "Linux")
at /usr/include/c++/4.8.3/bits/basic_string.tcc:138

#2 _S_construct_aux<char const*> (__a=..., __end=,

__beg=0x7f4ccb11a5e8 <vtable for std::_Sp_counted_ptr_inplace<Aws::Client::DynamoDBErrorMarshaller, Aws::Allocator<Aws::Client::DynamoDBErrorMarshaller>, (__gnu_cxx::_Lock_policy)2>+40> "Linux") at /usr/include/c++/4.8.3/bits/basic_string.h:1725

#3 _S_construct<char const*> (__a=..., __end=,

__beg=0x7f4ccb11a5e8 <vtable for std::_Sp_counted_ptr_inplace<Aws::Client::DynamoDBErrorMarshaller, Aws::Allocator<Aws::Client::DynamoDBErrorMarshaller>, (__gnu_cxx::_Lock_policy)2>+40> "Linux") at /usr/include/c++/4.8.3/bits/basic_string.h:1746

#4 std::basic_string<char, std::char_traits, Aws::Allocator >::basic_string (this=0x7f4cbee59b60,

__s=<optimized out>, __a=...) at /usr/include/c++/4.8.3/bits/basic_string.tcc:215

#5 0x00007f4ccac06d2b in Aws::Utils::StringUtils::LTrim(char const*) ()

from /home/boss/awsnew/aws-cpp-sdk-core/libaws-cpp-sdk-core.so
#6 0x00007f4ccac06f5c in Aws::Utils::StringUtils::Trim(char const*) ()

from /home/boss/awsnew/aws-cpp-sdk-core/libaws-cpp-sdk-core.so
#7 0x00007f4ccac05415 in getSysCommandOutput(char const*) () from /home/boss/awsnew/aws-cpp-sdk-core/libaws-cpp-sdk-core.so
#8 0x00007f4ccac054ae in Aws::Utils::OSVersionInfo::ComputeOSVersionString() const ()

from /home/boss/awsnew/aws-cpp-sdk-core/libaws-cpp-sdk-core.so
#9 0x00007f4ccabf1b5e in Aws::Client::ComputeUserAgentString() ()

from /home/boss/awsnew/aws-cpp-sdk-core/libaws-cpp-sdk-core.so
#10 0x00007f4ccabf1eec in Aws::Client::ClientConfiguration::ClientConfiguration() ()

from /home/boss/awsnew/aws-cpp-sdk-core/libaws-cpp-sdk-core.so
#11 0x00000000008201ae in boss::Dig_CP::getScores::retrieveScore (this=this@entry=0x7f4cbee5a770, tableName="good_wtp",

wtp="e2e40c9a-95a8-47c3-a746-e82cbecd2e42")
at /home/boss/boss_deployment/CP_source_new_v2/source_updated/C++/Algorithms/Dig_CP/src/Dig_CP_getScores.cpp:81

#12 0x0000000000821e7a in boss::Dig_CP::getScores::getScoresWRTWTP (this=this@entry=0x7f4cbee5a770,

wtp="e2e40c9a-95a8-47c3-a746-e82cbecd2e42")

Dig_CP_getScores.txt

[JonathanHenson]

-fno-builtin-malloc is the likely problem. The default memory handler if you are using custom memory management does malloc under the hood. You can either build the library with custom memory management turned off, or provide a memory allocation strategy that goes through your allocators (what I'm guessing you'll want to do, otherwise you wouldn't have turned malloc off).

Info is in the readme. Also, I do a live coding demo on this in the video posted in the readme.

[msinha31]

I am unable to fix this
step I did

1. cloned github
2. cmake CMakeList.txt
3. make
4. sudo make install

then

CMakelist.txt has
set(CMAKE_LIBRARY_PATH /usr/local/lib/linux/intel64/Release
)

set(CMAKE_CXX_FLAGS "-Wall -std=c++0x -g -mtune=core2 -l aws_dynamo -pedantic-errors -Wall -Wno-long-long -O1 -L/usr/lib -lstdc++ -lm -laws-cpp-sdk-dynamodb -laws-cpp-sdk-core -laws-cpp-sdk-sdb -DAWS_CUSTOM_MEMORY_MANAGEMENT=1")

Core was generated by `./../../build/bin/Dig_CP_1 conf/Dig_CP_MainConf.xml'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f5408f693db in void std::vectorstd::shared_ptr<Aws::Auth::AWSCredentialsProvider, Aws::Allocatorstd::shared_ptr<Aws::Auth::AWSCredentialsProvider > >::_M_emplace_back_auxstd::shared_ptr<Aws::Auth::AWSCredentialsProvider const&>(std::shared_ptrAws::Auth::AWSCredentialsProvider const&) ()
from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so
Missing separate debuginfos, use: debuginfo-install glibc-2.17-55.145.amzn1.x86_64 gperftools-libs-2.0-11.5.amzn1.x86_64 keyutils-libs-1.5.8-3.12.amzn1.x86_64 krb5-libs-1.13.2-10.39.amzn1.x86_64 libcom_err-1.42.12-4.40.amzn1.x86_64 libevent-2.0.18-1.11.amzn1.x86_64 libselinux-2.1.10-3.22.amzn1.x86_64 openssl-1.0.1k-13.88.amzn1.x86_64 zlib-1.2.8-7.18.amzn1.x86_64
(gdb) where
#0 0x00007f5408f693db in void std::vectorstd::shared_ptr<Aws::Auth::AWSCredentialsProvider, Aws::Allocatorstd::shared_ptr<Aws::Auth::AWSCredentialsProvider > >::_M_emplace_back_auxstd::shared_ptr<Aws::Auth::AWSCredentialsProvider const&>(std::shared_ptrAws::Auth::AWSCredentialsProvider const&) ()
from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so
#1 0x00007f5408f68cb3 in Aws::Auth::DefaultAWSCredentialsProviderChain::DefaultAWSCredentialsProviderChain() ()
from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so
#2 0x00007f5409241ae3 in Aws::DynamoDB::DynamoDBClient::DynamoDBClient(Aws::Client::ClientConfiguration const&) ()
from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-dynamodb.so
#3 0x0000000000820183 in boss::Dig_CP::getScores::retrieveScore (
this=this@entry=0x7f53ff9e5770, tableName="good_wtp",
wtp="cd27e640-6a82-474f-addb-7747c0d88986")
at /home/boss/boss_deployment/CP_source_new_v2/source_updated/C++/Algorithms/Dig_CP/src/Dig_CP_getScores.cpp:81

[msinha31]

DO you have any suggestion for this ?
-Manisha

[JonathanHenson]

I'm looking now.

[JonathanHenson]

-Wno-long-long .... we use long long, I doubt that's the issue but something to be aware of.

It looks like it's failing in the construction of the default credentials provider chain.
Two things, can you turn logging on and upload the log? directions are in the readme.
Also, can you run valgrind on the code and see where it shows the error?

[msinha31]

Thanks for your suggestion but I am still unable to solve the issue
attaching the log
Moreover I have removed the -Wno-long-long . from compilation.

#0 0x0000000000000000 in ?? ()
#1 0x00007fa87343ecda in Aws::Utils::Logging::FormattedLogSystem::Log(Aws::Utils::Logging::LogLevel, char const_, char const_, ...) ()
from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so
#2 0x00007fa873413a06 in Aws::Auth::DefaultAWSCredentialsProviderChain::DefaultAWSCredentialsProviderChain() () from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-core.so
#3 0x00007fa8736ecae3 in Aws::DynamoDB::DynamoDBClient::DynamoDBClient(Aws::Client::ClientConfiguration const&) ()
from /usr/local/lib/linux/intel64/Release/libaws-cpp-sdk-dynamodb.so
#4 0x0000000000820d43 in boss::Dig_CP::getScores::retrieveScore (this=this@entry=0x7fa867111770, tableName="good_wtp", wtp="d089be3a-a7d3-4543-a07a-d280c0237e89")
at /home/boss/boss_deployment/CP_source_new_v2/source_updated/C++/Algorithms/Dig_CP/src/Dig_CP_getScores.cpp:87
#5 0x0000000000822b01 in boss::Dig_CP::getScores::getScoresWRTWTP (this=this@entry=0x7fa867111770, wtp="d089be3a-a7d3-4543-a07a-d280c0237e89")
at /home/boss/boss_deployment/CP_source_new_v2/source_updated/C++/Algorithms/Dig_CP/src/Dig_CP_getScores.cpp:143
#6 0x000000000081cb01 in boss::Dig_CP::Dig_CP_LineItemBasic::bid (this=0x2437a80, request=...)
aws_sdk_2016-01-13-13.txt

[JonathanHenson]

A couple of thoughts.

a.) It looks like you ran 9 minutes before the segfault (that would have been useful to know as I thought it was failing on the first call).
b.) It is fairly expensive to allocate these clients and I would just store one inside your class as a member variable, or a global pointer somewhere. That should dramatically improve your performance as you can take advantage of connection pooling, not needing to hit disk to grab credentials etc... The clients are thread safe.
c.) It appears to be failing in a different place everytime which makes me wonder if your stack is getting corrupted elsewhere in your code base and failing inside this code. Any chance you could run valgrind --tool=memcheck on this and post the output?

[bretambrose]

Additionally, this appears to be a run of a Release build. Perhaps things would be clearer running a Debug build.

[msinha31]

running valgrind is not possible we tried some time back with no success
you want sdk to be debug build or my code

[msinha31]

i am using GNU Make 3.82

[JonathanHenson]

Without running some sort of memory analyzer, we will be looking for a needle in a haystack.... I can try pulling it down and running valgrind if you would like. All it takes is one buffer being overwritten or one bad alias and we would get this behavior--and the code base is large. Valgrind will find it in about 30 seconds.

[msinha31]

valgrind ./../../build/bin/Dig_CP_1 conf/Dig_CP_MainConf.xml --tool=memcheck -v
vex amd64->IR: unhandled instruction bytes: 0x66 0xF 0x1B 0x4 0x24 0x66 0xF 0x1B
vex amd64->IR: REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
vex amd64->IR: VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=0F
vex amd64->IR: PFX.66=1 PFX.F2=0 PFX.F3=0
==13663== valgrind: Unrecognised instruction at address 0x4015237.
==13663== at 0x4015237: _dl_runtime_resolve (in /lib64/ld-2.17.so)
==13663== by 0x6BEF2F8: __exp_finite (in /lib64/libm-2.17.so)
==13663== by 0x400B75A: _dl_relocate_object (in /lib64/ld-2.17.so)
==13663== by 0x4003AD9: dl_main (in /lib64/ld-2.17.so)
==13663== by 0x40167D4: _dl_sysdep_start (in /lib64/ld-2.17.so)
==13663== by 0x4004CC0: _dl_start (in /lib64/ld-2.17.so)
==13663== by 0x4001437: ??? (in /lib64/ld-2.17.so)
==13663== by 0x3: ???
==13663== by 0xFFF00045E: ???
==13663== by 0xFFF000479: ???
==13663== by 0xFFF000492: ???
==13663== by 0xFFF0004A2: ???
==13663== Your program just tried to execute an instruction that Valgrind
==13663== did not recognise. There are two possible reasons for this.
==13663== 1. Your program has a bug and erroneously jumped to a non-code
==13663== location. If you are running Memcheck and you just saw a
==13663== warning about a bad jump, it's probably your program's fault.
==13663== 2. The instruction is legitimate but Valgrind doesn't handle it,
==13663== i.e. it's Valgrind's fault. If you think this is the case or
==13663== you are not sure, please let us know and we'll try to fix it.
==13663== Either way, Valgrind will now raise a SIGILL signal which will
==13663== probably kill your program.
==13663==
==13663== Process terminating with default action of signal 4 (SIGILL)
==13663== Illegal opcode at address 0x4015237
==13663== at 0x4015237: _dl_runtime_resolve (in /lib64/ld-2.17.so)
==13663== by 0x6BEF2F8: __exp_finite (in /lib64/libm-2.17.so)
==13663== by 0x400B75A: _dl_relocate_object (in /lib64/ld-2.17.so)
==13663== by 0x4003AD9: dl_main (in /lib64/ld-2.17.so)
==13663== by 0x40167D4: _dl_sysdep_start (in /lib64/ld-2.17.so)
==13663== by 0x4004CC0: _dl_start (in /lib64/ld-2.17.so)
==13663== by 0x4001437: ??? (in /lib64/ld-2.17.so)
==13663== by 0x3: ???
==13663== by 0xFFF00045E: ???
==13663== by 0xFFF000479: ???
==13663== by 0xFFF000492: ???
==13663== by 0xFFF0004A2: ???
==13663== Jump to the invalid address stated on the next line
==13663== at 0x576: ???
==13663== by 0x49CCDF: ??? (in /home/boss/boss_deployment/CP_source_new_v2/build/bin/Dig_CP_1)
==13663== by 0x4003AD9: dl_main (in /lib64/ld-2.17.so)
==13663== by 0x40167D4: _dl_sysdep_start (in /lib64/ld-2.17.so)
==13663== by 0x4004CC0: _dl_start (in /lib64/ld-2.17.so)
==13663== by 0x4001437: ??? (in /lib64/ld-2.17.so)
==13663== by 0x3: ???
==13663== by 0xFFF00045E: ???
==13663== by 0xFFF000479: ???
==13663== by 0xFFF000492: ???
==13663== by 0xFFF0004A2: ???
==13663== Address 0x576 is not stack'd, malloc'd or (recently) free'd
==13663==
==13663==
==13663== Process terminating with default action of signal 11 (SIGSEGV)
==13663== Bad permissions for mapped region at address 0x576
==13663== at 0x576: ???
==13663== by 0x49CCDF: ??? (in /home/boss/boss_deployment/CP_source_new_v2/build/bin/Dig_CP_1)
==13663== by 0x4003AD9: dl_main (in /lib64/ld-2.17.so)
==13663== by 0x40167D4: _dl_sysdep_start (in /lib64/ld-2.17.so)
==13663== by 0x4004CC0: _dl_start (in /lib64/ld-2.17.so)
==13663== by 0x4001437: ??? (in /lib64/ld-2.17.so)
==13663== by 0x3: ???
==13663== by 0xFFF00045E: ???
==13663== by 0xFFF000479: ???
==13663== by 0xFFF000492: ???
==13663== by 0xFFF0004A2: ???
==13663==
==13663== HEAP SUMMARY:
==13663== in use at exit: 0 bytes in 0 blocks
==13663== total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==13663==
==13663== All heap blocks were freed -- no leaks are possible
==13663==
==13663== For counts of detected and suppressed errors, rerun with: -v
==13663== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)
Segmentation fault

[msinha31]

==9793== Memcheck, a memory error detector
==9793== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==9793== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==9793== Command: ./../../build/bin/Dig_CP_1 conf/Dig_CP_MainConf.xml --tool=memcheck
==9793==
==9793== brk segment overflow in thread #1: can't grow to 0x4a24000
./../../build/bin/Dig_CP_1 starts
to load xml tree /home/boss/boss_deployment/CP_source_new/deploy/Dig_CP/conf/Dig_CP_ReadLineItemValFromAdBoard.xml
to load xml tree /home/boss/boss_deployment/CP_source_new/deploy/Dig_CP/conf/Dig_CP_ServerConf.xml
to load xml tree /home/boss/boss_deployment/CP_source_new/deploy/Dig_CP/conf/Dig_CP_data_conf.xml
to load xml tree /home/boss/boss_deployment/CP_source_new/deploy/Dig_CP/conf/Dig_CP_log4cppPath.xml
Dig_CP_LineItemFactory::create() LineItemID=1888121476 not supported, will create Dig_CP_LineItemBasic
Dig_CP_LineItemFactory::create() LineItemID=1888117390 not supported, will create Dig_CP_LineItemBasic
Dig_CP_LineItemFactory::create() LineItemID=1888121476 not supported, will create Dig_CP_LineItemBasic

[msinha31]

Hi
The logs and valgrind output
ignore the previous ones
valgrind ./../../build/bin/Dig_CP_1 conf/Dig_CP_MainConf.xml --tool=memcheck -v
aws_sdk_2016-01-15-08.txt
val.txt

Thanks
Manisha

[JonathanHenson]

Looks like we have a bug in the logger. Have you ever run this with logging turned off?