Invalid free() at program exit if custom memory manager is used

[PaavoHelde]

Describe the bug

We have code for starting up and shutting down the aws-cpp-sdk library with Aws::InitAPI() and Aws::ShutdownAPI(), with a custom memory manager (redirecting all memory allocations to tbbmalloc). Alas, it appears there is a relatively recent regression in aws-cpp-sdk which causes runtime abort at the very end of the program, in atexit handlers. Commenting out the installation of custom memory manager gets rid of the problem.

Expected Behavior

Program exits normally.

Current Behavior

Program is aborted by SIGABRT.

Reproduction Steps

    // Use Intel's TBB allocator
    #define __TBBMALLOC_NO_IMPLICIT_LINKAGE 1
    #include <tbb/scalable_allocator.h>

    #include <aws/core/Aws.h>
    #include <aws/core/auth/AWSCredentialsProvider.h>
    #include <aws/core/config/AWSProfileConfigLoader.h>
    #include <aws/core/client/ClientConfiguration.h>
    #include <aws/s3/S3Client.h>


    class MyMemoryManager : public Aws::Utils::Memory::MemorySystemInterface {
        virtual void Begin() override {}
        virtual void End() override {}
        virtual void* AllocateMemory(std::size_t blockSize, std::size_t alignment, const char* allocationTag = nullptr) override {
	        return scalable_aligned_malloc(blockSize, alignment);
        }
        virtual void FreeMemory(void* memoryPtr) override {
	        scalable_aligned_free(memoryPtr);
        }
    };

class AwsLibrary {
public:
	AwsLibrary() {
		options_.memoryManagementOptions.memoryManager = &myMemoryManager_;
		Aws::InitAPI(options_);
	}
	~AwsLibrary() {
		Aws::ShutdownAPI(options_);
	}
private:
	MyMemoryManager myMemoryManager_;
	Aws::SDKOptions options_;
};

    int main() {
       AwsLibrary  x;
    }

Possible Solution

No response

Additional Information/Context

free(): invalid size

Thread 1 "acapella" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007ffff528c535 in __GI_abort () at abort.c:79
#2 0x00007ffff52e3518 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff53ed28d "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3 0x00007ffff52e9c3a in malloc_printerr (str=str@entry=0x7ffff53eb453 "free(): invalid size") at malloc.c:5359
#4 0x00007ffff52eb47c in _int_free (av=, p=0x7fffb6cf6b70, have_lock=) at malloc.c:4184
#5 0x00007fffb5aff883 in Aws::Delete<std::vector<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface >, Aws::Crt::StlAllocator<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface > > > > (pointerToT=)
at /usr/include/c++/8/ext/new_allocator.h:86
#6 Aws::Deleter<std::vector<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface >, Aws::Crt::StlAllocator<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface > > > >::operator() (this=,
pointerToT=0x7fffb6cf6b80)
at /srv/paavo/DataAnalysis/Acapella/trunk/Production/Intermediates/ThirdParty/src/aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/utils/memory/AWSMemory.h:246
#7 Aws::Deleter<std::vector<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface >, Aws::Crt::StlAllocator<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface > > > >::operator() (
this=0x7fffb62c1ad8 Aws::Monitoring::s_monitors, pointerToT=0x7fffb6cf6b80)
at /srv/paavo/DataAnalysis/Acapella/trunk/Production/Intermediates/ThirdParty/src/aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/utils/memory/AWSMemory.h:243
#8 std::unique_ptr<std::vector<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface >, Aws::Crt::StlAllocator<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface > > >, Aws::Deleter<std::vector<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface >, Aws::Crt::StlAllocator<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface > > > > >::reset (__p=0x7fffb6cf6b80, this=0x7fffb62c1ad8 Aws::Monitoring::s_monitors)
at /usr/include/c++/8/bits/unique_ptr.h:382
#9 Aws::UniquePtrSafeDeleted<std::vector<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface >, Aws::Crt::StlAllocator<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface > > >, Aws::Deleter<std::vector<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface >, Aws::Crt::StlAllocator<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface > > > > >::forceReset (this=0x7fffb62c1ad8 Aws::Monitoring::s_monitors)
at /srv/paavo/DataAnalysis/Acapella/trunk/Production/Intermediates/ThirdParty/src/aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/utils/memory/AWSMemory.h:282
#10 Aws::UniquePtrSafeDeleted<std::vector<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface >, Aws::Crt::StlAllocator<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface > > >, Aws::Deleter<std::vector<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface >, Aws::Crt::StlAllocator<std::unique_ptr<Aws::Monitoring::MonitoringInterface, Aws::DeleterAws::Monitoring::MonitoringInterface > > > > >::~UniquePtrSafeDeleted (this=0x7fffb62c1ad8 Aws::Monitoring::s_monitors, __in_chrg=)
at /srv/paavo/DataAnalysis/Acapella/trunk/Production/Intermediates/ThirdParty/src/aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/utils/memory/AWSMemory.h:291
#11 0x00007ffff52a3d8c in __run_exit_handlers (status=0, listp=0x7ffff5424718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true,
run_dtors=run_dtors@entry=true) at exit.c:108
#12 0x00007ffff52a3eba in __GI_exit (status=) at exit.c:139
#13 0x00007ffff528e0a2 in __libc_start_main (main=0x5555555c098d <main(int, char**)>, argc=2, argv=0x7fffffffe968, init=, fini=,
rtld_fini=, stack_end=0x7fffffffe958) at ../csu/libc-start.c:342
#14 0x000055555559f3ca in _start ()

AWS CPP SDK version used

current trunk 58b715c

Compiler and Version used

gcc 8.3

Operating System and version

Debian GNU/Linux 10 (buster)

[jmklix]

Did you notice which recent version of the sdk caused this regression?

[PaavoHelde]

Not quite sure. I think it has happened after Oct 5, but not 100% sure.

[jmklix]

Does this still happen when you wrap all sdk calls in {}?
i.e. like this:

#include <aws/core/Aws.h>
int main(int argc, char** argv)
{
   Aws::SDKOptions options;
   Aws::InitAPI(options);
   {
      // make your SDK calls here.
   }
   Aws::ShutdownAPI(options);
   return 0;
}

[PaavoHelde]

Hi @jmklix In the original bug demo source file there were no other SDK calls, so nothing to wrap in {}. In the real usage the code is in a demand-loaded shared object, so there is no main(), but I believe all other cleanup is done properly before calling Aws::ShutdownAPI().

Anyway, I now added the custom memory manager setup back in the code, and it seems the bug has been fixed meanwhile, no crashes at exit any more. At the moment we are using: https://github.com/aws/aws-sdk-cpp.git branch=main tag=1.10.57

Thanks for getting back to us!

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.