Core dump at ClientConfiguration::ClientConfiguration()

[BanghuiLuo]

Confirm by changing [ ] to [x] below:

• [ x] I've gone though Developer Guide and API reference
• [x ] I've searched for previous similar issues and didn't find any solution

Platform/OS/Hardware/Device
What are you running the sdk on?
Create EC2MetadataClient

Describe the question
Got below core dump

#4 0x0000000000000000 in ?? ()
#5 0x000000ad9c87ad0b in Aws::Client::ClientConfiguration::ClientConfiguration() ()
from /tmp/debug-uw.H7NRyYLx/lib64/libaws-cpp-sdk-core.so
#6 0x000000ad9c8873ce in Aws::Internal::AWSHttpResourceClient::AWSHttpResourceClient(char const*) () from /tmp/debug-uw.H7NRyYLx/lib64/libaws-cpp-sdk-core.so
#7 0x000000ad9c887508 in Aws::Internal::EC2MetadataClient::EC2MetadataClient(char const*)
() from /tmp/debug-uw.H7NRyYLx/lib64/libaws-cpp-sdk-core.so

Logs/output
If applicable, add logs or error output.
Some logs before core dump

2021-06-23T07:27:27.941Z info VmkCtlAwsLogger: Aws_Init_Cleanup Initiate AWS SDK for C++ with Version:1.8.186
…
2021-06-23T07:27:27.943Z info VmkCtlAwsLogger: Aws::Config::AWSConfigFileProfileConfigLoader Unable to open config file /.aws/config for reading.
2021-06-23T07:27:27.943Z info VmkCtlAwsLogger: Aws::Config::AWSProfileConfigLoader Failed to reload configuration.
2021-06-23T07:27:27.944Z info VmkCtlAwsLogger: ClientConfiguration ClientConfiguration will use SDK Auto Resolved profile: [default] if not specified by users.
2021-06-23T07:27:27.944Z info VmkCtlAwsLogger: ClientConfiguration Retry Strategy will use the default max attempts.
2021-06-23T07:27:27.944Z info VmkCtlAwsLogger: EC2MetadataClient Creating AWSHttpResourceClient with max connections 2 and scheme HTTP
2021-06-23T07:27:27.944Z info VmkCtlAwsLogger: CurlHandleContainer Initializing CurlHandleContainer with size 2

2021-06-23T07:27:27.944Z info VmkCtlAwsLogger: ClientConfiguration ClientConfiguration will use SDK Auto Resolved profile: [default] if not specified by users.
2021-06-23T07:27:27.944Z info VmkCtlAwsLogger: ClientConfiguration Retry Strategy will use the default max attempts.

Then the system got a core dump.

I tried to add some debug logs locally. I see it goes into line 129 but didn't call into GetCurrentRegion() at all. So I guess the core dump is supposed to be caused at line 130
https://github.com/aws/aws-sdk-cpp/blob/main/aws-cpp-sdk-core/source/client/ClientConfiguration.cpp

125    if (Aws::Utils::StringUtils::ToLower(Aws::Environment::GetEnv("AWS_EC2_METADATA_DISABLED").c_str()) != "true")
126   {
127        auto client = Aws::Internal::GetEC2MetadataClient();
128        if (client)
129        {
130            region = client->GetCurrentRegion();  <-----------------
131        }
132    }

Seems it failed to invoke the function GetCurrentRegion(). It looks really weird. Please AWS experts help with it. Thanks.

SDK version number:
1.8.186

[KaibaLopez]

So, an easy workaround for this could be to set the environment variable AWS_EC2_METADATA_DISABLED to true. That should avoid the whole call.

But I'd like to know what's going on there, could you show me a small code sample that reproduces this?

[BanghuiLuo]

Thanks Gilberto a lot for your timely reply and workaround. I also tried to similar workaround to pypass the call, it works. But since it changes the way to get current region, we’d like to know if it has any impact? Here is how we call it, 1. Firstly, we init an AWS sdk helper ``` 1295 AwsKmsSdkHelper::AwsKmsSdkHelper() 1296 { … 1307 Aws::InitAPI(_options); 1308 } ``` 2. Then save a static instance for use. ``` 1185 static AwsKmsSdkHelper& Instance() { 1186 static AwsKmsSdkHelper instance; 1187 return instance; 1188 } ``` 3. Create a EC2MetadataClient in one of AwsKmsSdkHelper’s functions AwsKmsSdkHelper::Encrypt() { 1318 shared_ptr<Aws::Internal::EC2MetadataClient> metadataClient; … 1329 metadataClient = Aws::MakeShared<Aws::Internal::EC2MetadataClient>( 1330 _awsAllocationTag, metaServer.c_str()); } 4. The function was invoked in below way ``` 8760 AwsKmsSdkHelper& awsKms = AwsKmsSdkHelper::Instance(); 8761 awsKms.Encrypt(); ``` As described in the issue, a core dump was generated when new the EC2MetadataClient at step 3. Hope it would be helpful. Please feel free to let me know if any other information requires. Really appreciate your time.

…

-- Thanks, Banghui Luo From: Gilberto Lopez Zayas ***@***.***> Reply-To: aws/aws-sdk-cpp ***@***.***> Date: Friday, June 25, 2021 at 12:41 AM To: aws/aws-sdk-cpp ***@***.***> Cc: Banghui Luo ***@***.***>, Author ***@***.***> Subject: Re: [aws/aws-sdk-cpp] Core dump at ClientConfiguration::ClientConfiguration() (#1694) So, an easy workaround for this could be to set the environment variable AWS_EC2_METADATA_DISABLED to true. That should avoid the whole call. But I'd like to know what's going on there, could you show me a small code sample that reproduces this? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhub.490633.xyz%2Faws%2Faws-sdk-cpp%2Fissues%2F1694%23issuecomment-867791215&data=04%7C01%7Cbanghuil%40vmware.com%7C43d79571e87f471320ee08d9372edb1f%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637601496656804371%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=shagvVSSHKbuO%2Bq6RLMpOHPM0ON5LbIG3equVGmFoVw%3D&reserved=0>, or unsubscribe<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhub.490633.xyz%2Fnotifications%2Funsubscribe-auth%2FAUTIAQGIMIGWDEZM2VKJANTTUNNZ5ANCNFSM47HUCQ2A&data=04%7C01%7Cbanghuil%40vmware.com%7C43d79571e87f471320ee08d9372edb1f%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637601496656804371%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=N1FRJ7WlvbTBy9lF%2BFAsJc2TWStTIgdIGtQ%2BUsyyrLU%3D&reserved=0>.

[BanghuiLuo]

In addition, I also tried to set the environment variable AWS_EC2_METADATA_DISABLED

#include <aws/core/platform/Environment.h>

   AwsKmsSdkHelper::AwsKmsSdkHelper()
    {
        …
        Aws::InitAPI(_options);
        Aws::Environment::SetEnv("AWS_EC2_METADATA_DISABLED", "true", 1);
    }

Unfortunately, met below compiling error

error: 'SetEnv' is not a member of 'Aws::Environment'
Aws::Environment::SetEnv("AWS_EC2_METADATA_DISABLED", "true", 1);
^~~

By checking https://github.com/aws/aws-sdk-cpp/blob/main/aws-cpp-sdk-core/include/aws/core/platform/Environment.h , seems it doesn’t provide a SetEnv API.

Could you tell me the right way to set AWS_EC2_METADATA_DISABLED? Thanks.

[KaibaLopez]

Ok so still trying to reproduce your error, but in the mean time, to add Aws::Environment::SetEnv("AWS_EC2_METADATA_DISABLED", "true", 1); you'll need:

#include <aws/testing/platform/PlatformTesting.h>

[KaibaLopez]

ah I just re-read your previous comment and no, using the environment variable has no other implications, it is pretty much there for this, it will simply not try to get the region from the EC2 metadata, so you'd have to provide it in some other way, either hard coded, on the aws config file (usually in .aws/config )or on the environment variable AWS_DEFAULT_REGION.

[BanghuiLuo]

Thanks so much for your analysis.

Trying to include the header, seems we cann't refer to this header file.

fatal error: aws/testing/platform/PlatformTesting.h: No such file or directory
#include <aws/testing/platform/PlatformTesting.h>
^

Listing files under the sdk's built-in directory, the platform directory is found. Is this the right way to check the files?

ls cayman_aws-sdk-cpp/sb-47859133/linux-centos72-gc32/lin64+glibc211+gcc9/include/aws/testing/
aws_test_allocators.h aws_test_harness.h

Nevertheless, I am not sure if it is proper to introduce a testing API to our releasing product code. I think it's better to implement the API locally, but refer to aws cpp sdk's API.

Besides, I'd like to know if you could reproduce the core dump issue in your local env? I am still a little bit curious about why the core dump happened. Thanks.

[BanghuiLuo]

Oops, I tried to set AWS_EC2_METADATA_DISABLED to true, initializing ClientConfiguration() passed, but it got a core dump again at destroy due to segment fault.

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000000d15ac2cad0 in vtable for Aws::Internal::EC2MetadataClient ()
from /tmp/debug-uw.YeztY5nD/lib64/libaws-cpp-sdk-core.so
(gdb) bt
#0 0x000000d15ac2cad0 in vtable for Aws::Internal::EC2MetadataClient ()
from /tmp/debug-uw.YeztY5nD/lib64/libaws-cpp-sdk-core.so
#1 0x000000d1581596b0 in destroyAws::Internal::EC2MetadataClient (__p=, this=)
at /build/mts/sharedcompcache/cayman_esx_toolchain/ob-15599702/linux64/usr/x86_64-vmk-linux-gnu/include/c++/6.4.0/ext/new_allocator.h:124
#2 destroyAws::Internal::EC2MetadataClient (__p=, __a=...)
at /build/mts/sharedcompcache/cayman_esx_toolchain/ob-15599702/linux64/usr/x86_64-vmk-linux-gnu/include/c++/6.4.0/bits/alloc_traits.h:487
#3 std::_Sp_counted_ptr_inplace<Aws::Internal::EC2MetadataClient, std::allocatorAws::Internal::EC2MetadataClient,(__gnu_cxx::_Lock_policy)2>::_M_dispose (this=)
at /build/mts/sharedcompcache/cayman_esx_toolchain/ob-15599702/linux64/usr/x86_64-vmk-linux-gnu/include/c++/6.4.0/bits/shared_ptr_base.h:529
#4 0x000000d157aca665 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0xd1161fd860)
at /build/mts/release/sb-47758574/compcache/cayman_esx_toolchain/ob-15599702/linux64/usr/x86_64-vmk-linux-gnu/include/c++/6.4.0/bits/shared_ptr_base.h:150
#5 0x000000d15a9f3bae in Aws::ShutdownAPI(Aws::SDKOptions const&) ()
from /tmp/debug-uw.YeztY5nD/lib64/libaws-cpp-sdk-core.so

I guess it failed in CleanupEC2MetadataClient when referring s_ec2metadataClient, which has similar sympathy to the first core dump.
https://github.com/aws/aws-sdk-cpp/blob/main/aws-cpp-sdk-core/source/internal/AWSHttpResourceClient.cpp

        void CleanupEC2MetadataClient()
        {
            if (!s_ec2metadataClient)
            {
                return;
            }
            s_ec2metadataClient = nullptr;
        }

Please help to check further. Really appreciate your timely help.

[BanghuiLuo]

Continued, I made a workaround to remove refers to s_ec2metadataClient.

         void InitEC2MetadataClient()
         {
-            if (s_ec2metadataClient)
-            {
-                return;
-            }
-            s_ec2metadataClient = Aws::MakeShared<EC2MetadataClient>(EC2_METADATA_CLIENT_LOG_TAG);
         }

         void CleanupEC2MetadataClient()
         {
-            if (!s_ec2metadataClient)
-            {
-                return;
-            }
-            s_ec2metadataClient = nullptr;
         }

Then everything is fine to go.

Likely something wrong with s_ec2metadataClient.