feat: use vendored version of requests package in layer.

[purple4reina]

What does this PR do?

Removes the requests package from the zipped lambda layer. Adds /path/to/pip/_vendor to sys.path so requests can instead be imported from pip._vendor.requests.

Motivation

Removing requests and all its dependencies from the layer reduces the package size (unzipped) from 15,212 KB to 13,904 KB -- a roughly 8.5% reduction!! (Data gathered running PYTHON_VERSION=3.12 ARCH=arm64 ./scripts/build_layers.sh)

Potential Drawbacks

Some customers may already be relying on the presence of the requests package in our python layer. This change could crash their lambda functions if requests is imported before datadog_lambda. This would only be the case if they are using the @datadog_lambda_wrapper decorator and not the automatic handler redirection using DD_LAMBDA_HANDLER.

For example, prior to this change, assuming a customer adds just our python layer and nothing else, this handler will fail to initialize.

import requests

from datadog_lambda.wrapper import datadog_lambda_wrapper

@datadog_lambda_wrapper
def handler(event, context):
    resp = requests.get('https://example.com')
    resp.raise_for_status()
    return 'ok'

Therefore, we absolutely must include something about the import order importance in the release notes.

Testing Guidelines

Additional Notes

If for some reason the importing of requests doesn't work (say pop._vendor.requests isn't installed on the container) then the only situation in which the lambda function would raise an exception is when attempting to submit metrics via the datadog api. We do this when either the metric has a post dated timestamp or there is no extension available. Therefore, a pretty narrow use case.

Types of Changes

• Bug fix
• New feature
• Breaking change
• Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

• This PR's description is comprehensive
• This PR contains breaking changes that are documented in the description
• This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
• This PR impacts documentation, and it has been updated (or a ticket has been logged)
• This PR's changes are covered by the automated tests
• This PR collects user input/sensitive content into Datadog
• This PR passes the integration tests (ask a Datadog member to run the tests)

[purple4reina]

Creating a custom metric with a timestamp forces our instrumentation to use the datadog api package instead of sending the metric to the extension.

[purple4reina]

Creating a custom metric with a timestamp forces our instrumentation to use the datadog api package instead of sending the metric to the extension.

[datadog-datadog-prod-us1]

🟠 Code Vulnerability

No explicit permissions set for at the workflow level (...read more)

Datadog’s GitHub organization defines default permissions for the GITHUB_TOKEN to be restricted (contents:read, metadata:read, and packages:read).

Your repository may require a different setup, so consider defining permissions for each job following the least privilege principle to restrict the impact of a possible compromise.

You can find the list of all possible permissions in Workflow syntax for GitHub Actions - GitHub Docs. They can be defined at the job or the workflow level.

    

[datadog-datadog-prod-us1]

🟠 Code Vulnerability

Workflow depends on a GitHub actions pinned by tag (...read more)

When using a third party action, one needs to provide its GitHub path (owner/project) and can eventually pin it to a Git ref (a branch name, a Git tag, or a commit hash).

No pinned Git ref means the action uses the latest commit of the default branch each time it runs, eventually running newer versions of the code that were not audited by Datadog. Specifying a Git tag is better, but since they are not immutable, using a full length hash is recommended to make sure the action content is actually frozen to some reviewed state.

Be careful however, as even pinning an action by hash can be circumvented by attackers still. For instance, if an action relies on a Docker image which is itself not pinned to a digest, it becomes possible to alter its behaviour through the Docker image without actually changing its hash. You can learn more about this kind of attacks in Unpinnable Actions: How Malicious Code Can Sneak into Your GitHub Actions Workflows. Pinning actions by hash is still a good first line of defense against supply chain attacks.

Additionally, pinning by hash or tag means the action won’t benefit from newer version updates if any, including eventual security patches. Make sure to regularly check if newer versions for an action you use are available. For actions coming from a very trustworthy source, it can make sense to use a laxer pinning policy to benefit from updates as soon as possible.

    

[datadog-datadog-prod-us1]

🟠 Code Vulnerability

Workflow depends on a GitHub actions pinned by tag (...read more)

When using a third party action, one needs to provide its GitHub path (owner/project) and can eventually pin it to a Git ref (a branch name, a Git tag, or a commit hash).

No pinned Git ref means the action uses the latest commit of the default branch each time it runs, eventually running newer versions of the code that were not audited by Datadog. Specifying a Git tag is better, but since they are not immutable, using a full length hash is recommended to make sure the action content is actually frozen to some reviewed state.

Be careful however, as even pinning an action by hash can be circumvented by attackers still. For instance, if an action relies on a Docker image which is itself not pinned to a digest, it becomes possible to alter its behaviour through the Docker image without actually changing its hash. You can learn more about this kind of attacks in Unpinnable Actions: How Malicious Code Can Sneak into Your GitHub Actions Workflows. Pinning actions by hash is still a good first line of defense against supply chain attacks.

Additionally, pinning by hash or tag means the action won’t benefit from newer version updates if any, including eventual security patches. Make sure to regularly check if newer versions for an action you use are available. For actions coming from a very trustworthy source, it can make sense to use a laxer pinning policy to benefit from updates as soon as possible.

    

[purple4reina]

We have tried this idea before, though using the requests version vendored with botocore. However there were major problems and we reverted. See

• Remove patching for botocore.vendored.requests #22
• DO NOT import requests from botocore datadog-serverless-functions#424

Since we do not have any control over the lambda runtime, if for some reason the requests version changes or goes away, we have little recourse in trying to solve it. There is potential that it could cause every lambda everywhere to start failing, which is an incredibly high risk, even if chances of it happening are very low.