Fix GLIBCXX Errors From Serverless Framework And AWS Lambda

While I haven’t done too much with Serverless Framework and Functions as a Service (Faas) recently, I did in the past and it isn’t something that I’ve forgotten. In the past I demonstrated how to deploy Node.js functions to Amazon Web Services (AWS) Lambda that contain native dependencies. While not a necessity for all Lambda functions, it is for functions that use libraries for specific operating systems and architectures. For example, my previous article titled, Use AWS Lambda and API Gateway with Node.js and Couchbase NoSQL, fell into this situation. Making use of an EC2 instance or a Docker container with Amazon Linux will help most of the time, but there are scenarios where a little bit extra must be done to accomplish the task.

In certain circumstances everything may package and deploy correctly, but still throw errors. For example, a common error is around libstdc++ and a version of GLIBCXX not being found.

In this tutorial we’re going to see how to resolve library errors that might not be caught in a typical packaging and deployment scenario with Serverless Framework and AWS Lambda.

For this example we’re going to reference my previous Couchbase example, but we’re not going to think too hard on the code. Instead, lets imagine that we’ve packaged the functions and deployed them as I recommended previously.

When we try to run our functions, either from an API Gateway endpoint or directly as an invocation, we might come across this error in our CloudWatch logs:

module initialization error: Error
    at Object.Module._extensions..node (module.js:681:18)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at bindings (/var/task/node_modules/bindings/bindings.js:84:48)
    at Object.<anonymous> (/var/task/node_modules/couchbase/lib/binding.js:213:36)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)

Great, the above error is not at all helpful to us. For whatever reason a module didn’t initialize and all we can see is that Couchbase was involved somehow. To be fair, I was getting this error when I looked at my CloudWatch dashboard, not when I tried to invoke the functions directly.

If you search the web for the error in question, you get a lot of results that indicate that you probably have native dependencies and you just need to build them using the same flavor and architecture of Linux that AWS Lambda uses. Well, that is great, but if you followed my previous tutorials you’ll know that we downloaded the amazonlinux image on Docker and downloaded our Node.js modules through that container deployment.

So we have to dig a bit deeper into the problem.

If we invoke the function directly, without API Gateway or some other trigger, we get a new error:

{
    "errorMessage": "/var/lang/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /var/task/node_modules/couchbase/build/Release/couchbase_impl.node)",
    "errorType": "Error",
    "stackTrace": [
        "Module.load (module.js:565:32)",
        "tryModuleLoad (module.js:505:12)",
        "Function.Module._load (module.js:497:3)",
        "Module.require (module.js:596:17)",
        "require (internal/module.js:11:18)",
        "bindings (/var/task/node_modules/bindings/bindings.js:84:48)",
        "Object.<anonymous> (/var/task/node_modules/couchbase/lib/binding.js:213:36)",
        "Module._compile (module.js:652:30)",
        "Object.Module._extensions..js (module.js:663:10)"
    ]
}

To invoke your function you can either use serverless invoke -f <function-name> with the Serverless CLI, or through the AWS Lambda dashboard. In this new error, we have information regarding libstdc++ and GLIBCXX. If you’re like me, you have absolutely no idea what that is. While I develop using numerous technologies, C and C++ is not one of them.

After doing some searching around the web, people are saying numerous things about what the problem could be. Here is an aggregate of some of the things that came back in my search:

• The dependencies were not compiled using the correct Amazon Linux flavor.
• The version of libstdc++ in the Docker image didn’t match the actual version on AWS Lambda.
• The libstdc++ application or library wasn’t truly installed.

So what do I do with this information?

In the first scenario, I’m pretty sure my dependencies were installed using the correct Amazon Linux flavor. I had mapped my project directory with Docker and installed my Node.js modules from within my Docker container. Life should be good because it worked the last time I tried about a year ago. In the second scenario, I have no idea how to check what version of libstdc++ I’m using in my Docker container or on AWS Lambda. I’m only hoping that the versions are matched up. In the third scenario, I did find this file in the container, so I was hoping that was enough.

If you can believe it, my Couchbase Node.js SDK was not compiled using the correct Amazon Linux flavor. When I downloaded this dependency from within my Docker container, it was using a pre-built version of the SDK which didn’t match what I truly needed. To resolve this, I needed to build the SDK from source.

To build my Node.js module from source so that the GLIBCXX errors went away, I needed to first download the build essentials to my Amazon Linux container. From within my container, I could execute the following:

yum -y install git
yum -y groupinstall "Development Tools"

The source to the Couchbase Node.js SDK was on GitHub so I needed to install the NPM dependency from GitHub rather than the standard NPM repository. For this to work git needed to be available. Also, since the Amazon Linux instance is very basic, the appropriate compilers and build tools needed to be available. These tools were in the “Developer Tools” group.

After installing the proper OS tools, we could run the following:

npm install git+https://github.com/couchbase/couchnode.git#v2.6.2 --save

The above command will install the 2.6.2 release of the SDK, but it will also build it from source. By building it from source we can be sure that the proper operating system and architecture were used. After deploying this to AWS Lambda and AWS API Gateway, everything worked fine.

In case you’d like to create your own Docker image and save some of the hassle, you could use the following Dockerfile file:

FROM amazonlinux:1

RUN yum -y install tar
RUN yum -y install gzip
RUN yum -y install git
RUN yum -y groupinstall "Development Tools"

RUN curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.34.0/install.sh | bash
RUN /bin/bash -c "source /root/.nvm/nvm.sh; nvm install 8.10.0"

CMD /bin/bash -c "source /root/.nvm/nvm.sh; nvm use 8.10.0; bash"

The above blueprint will install the necessary applications, and choose the correct version of Node.js to match what AWS Lambda uses. Using Docker, you could run something like the following:

docker build -t custom-amazon-linux /path/to/Dockerfile

With the custom image available, you could deploy it as a container and map your project directory on the host, similar to how we did it in the previous tutorial.

Conclusion

You just saw how to solve GLIBCXX version errors related to the libstdc++ package, something that is a surprisingly common issue when it comes to function deployment on AWS Lambda. To be clear, these issues are not specific to the Couchbase Node.js SDK. These issues come up due to native dependencies being deployed after having been built using the incorrect operating system or architecture. Some Node.js and Python dependencies use native libraries and these native libraries need to match what Amazon Linux uses on AWS Lambda.

Nic Raboy

Nic Raboy is an advocate of modern web and mobile development technologies. He has experience in C#, JavaScript, Golang and a variety of frameworks such as Angular, NativeScript, and Unity. Nic writes about his development experiences related to making web and mobile development easier to understand.