feat(potel): Make tracing APIs use OTel in the background (#3242)

* Skeletons for new components

* Add simple scope management whenever a context is attached

* create a new otel context `_SCOPES_KEY` that will hold a tuple of
  `(curent_scope, isolation_scope)`
* the `current_scope` will always be forked (like on every span creation/context update in practice)
  * note that this is on `attach`, so not on all copy-on-write context
    object creation but only on apis such as
    [`trace.use_span`](https://github.com/open-telemetry/opentelemetry-python/blob/ba22b165471bde2037620f2c850ab648a849fbc0/opentelemetry-api/src/opentelemetry/trace/__init__.py#L547)
    or [`tracer.start_as_current_span`](https://github.com/open-telemetry/opentelemetry-python/blob/ba22b165471bde2037620f2c850ab648a849fbc0/opentelemetry-api/src/opentelemetry/trace/__init__.py#L329)
  * basically every otel `context` fork corresponds to our `current_scope` fork
* the `isolation_scope` currently will not be forked
  * these will later be updated, for instance when we update our top
    level scope apis that fork isolation scope, that will also have a
    corresponding change in this `attach` function

* Don't parse DSN twice

* wip

* Skeletons for new components

* Skeletons for new components

* Add simple scope management whenever a context is attached

* create a new otel context `_SCOPES_KEY` that will hold a tuple of
  `(curent_scope, isolation_scope)`
* the `current_scope` will always be forked (like on every span creation/context update in practice)
  * note that this is on `attach`, so not on all copy-on-write context
    object creation but only on apis such as
    [`trace.use_span`](https://github.com/open-telemetry/opentelemetry-python/blob/ba22b165471bde2037620f2c850ab648a849fbc0/opentelemetry-api/src/opentelemetry/trace/__init__.py#L547)
    or [`tracer.start_as_current_span`](https://github.com/open-telemetry/opentelemetry-python/blob/ba22b165471bde2037620f2c850ab648a849fbc0/opentelemetry-api/src/opentelemetry/trace/__init__.py#L329)
  * basically every otel `context` fork corresponds to our `current_scope` fork
* the `isolation_scope` currently will not be forked
  * these will later be updated, for instance when we update our top
    level scope apis that fork isolation scope, that will also have a
    corresponding change in this `attach` function

* mypy fixes

* working span processor

* lint

* Port over op/description/status extraction

* defaultdict

* naive impl

* wip

* fix args

* wip

* remove extra docs

* Add simple scope management whenever a context is attached (#3159)

Add simple scope management whenever a context is attached

* create a new otel context `_SCOPES_KEY` that will hold a tuple of
  `(curent_scope, isolation_scope)`
* the `current_scope` will always be forked (like on every span creation/context update in practice)
  * note that this is on `attach`, so not on all copy-on-write context
    object creation but only on apis such as
    [`trace.use_span`](https://github.com/open-telemetry/opentelemetry-python/blob/ba22b165471bde2037620f2c850ab648a849fbc0/opentelemetry-api/src/opentelemetry/trace/__init__.py#L547)
    or [`tracer.start_as_current_span`](https://github.com/open-telemetry/opentelemetry-python/blob/ba22b165471bde2037620f2c850ab648a849fbc0/opentelemetry-api/src/opentelemetry/trace/__init__.py#L329)
  * basically every otel `context` fork corresponds to our `current_scope` fork
* the `isolation_scope` currently will not be forked
  * these will later be updated, for instance when we update our top
    level scope apis that fork isolation scope, that will also have a
    corresponding change in this `attach` function

* Implement new POTel span processor (#3223)

* only acts on `on_end` instead of both `on_start/on_end` as before
* store children spans in a dict mapping `span_id -> children`
* new dict only stores otel span objects and no sentry transaction/span objects so we save a bit of useless memory allocation
* I'm not using our current `Transaction/Span` classes at all to build the event because when we add our APIs later, we'll need to rip these out and we also avoid having to deal with the `instrumenter` problem
* if we get a root span (without parent), we recursively walk the dict and find the children and package up the transaction event and send it 
  * I didn't do it like JS because I think this way is better
  *  they [group an array of `finished_spans`](https://github.com/getsentry/sentry-javascript/blob/7e298036a21a5658f3eb9ba184165178c48d7ef8/packages/opentelemetry/src/spanExporter.ts#L132) every time a root span ends and I think this uses more cpu than what I did
  * and the dict like I used it doesn't take more space than the array either
* if we get a span with a parent we just update the dict to find the span later
* moved the common `is_sentry_span` logic to utils

* Basic test cases for potel (#3286)

* Proxy POTelSpan.set_data to underlying otel span attributes (#3297)

* ref(tracing): Simplify backwards-compat code (#3379)

With this change, we aim to simplify the backwards-compatibility code
for POTel tracing.

We do this as follows:
  - Remove `start_*` functions from `tracing`
  - Remove unused parameters from `tracing.POTelSpan.__init__`.
  - Make all parameters to `tracing.POTelSpan.__init__` kwarg-only.
  - Allow `tracing.POTelSpan.__init__` to accept arbitrary kwargs,
    which are all ignored, for compatibility with old `Span` interface.
  - Completely remove `start_inactive_span`, since inactive spans can
    be created by setting `active=False` when constructing a
    `POTelSpan`.

* New Scope implementation based on OTel Context (#3389)

* New `PotelScope` inherits from scope and reads the scope from the otel context key `SENTRY_SCOPES_KEY`
* New `isolation_scope` and `new_scope` context managers just use the context manager forking and yield with the scopes living on the above context key
  * isolation scope forking is done with the `SENTRY_FORK_ISOLATION_SCOPE_KEY` boolean context key

* Fix circular imports (#3431)

* Random tweaks (#3437)

* Origin improvements (#3432)

* Tweak OTel timestamp utils (#3436)

* Create spans on scope (#3442)

* Fill out more property/method stubs (#3441)

* Cleanup origin handling and defaults (#3445)

* add note to migration guide

* Attribute namespace for tags, measurements (#3448)

---------

Co-authored-by: Neel Shah <neel.shah@sentry.io>
Co-authored-by: Neel Shah <neelshah.sa@gmail.com>
Co-authored-by: Daniel Szoke <7881302+szokeasaurusrex@users.noreply.github.com>

Author: 3 people

MIGRATION_GUIDE.md

@@ -9,22 +9,26 @@ Looking to upgrade from Sentry SDK 2.x to 3.x? Here's a comprehensive list of wh
 
 ### Changed
 
+- The SDK now supports Python 3.7 and higher.
+- `sentry_sdk.start_span` now only takes keyword arguments.
 - The `Span()` constructor does not accept a `hub` parameter anymore.
 - `Span.finish()` does not accept a `hub` parameter anymore.
 - The `Profile()` constructor does not accept a `hub` parameter anymore.
 - A `Profile` object does not have a `.hub` property anymore.
 
 ### Removed
 
+- When setting span status, the HTTP status code is no longer automatically added as a tag.
 - Class `Hub` has been removed.
 - Class `_ScopeManager` has been removed.
 - The context manager `auto_session_tracking()` has been removed. Use `track_session()` instead.
 - The context manager `auto_session_tracking_scope()` has been removed. Use `track_session()` instead.
-- Utility function `is_auto_session_tracking_enabled()` has been removed. There is no public replacement. There is a private `_is_auto_session_tracking_enabled()` (if you absolutely need this function) It accepts a `scope` parameter instead of the previously used `hub` parameter. 
+- Utility function `is_auto_session_tracking_enabled()` has been removed. There is no public replacement. There is a private `_is_auto_session_tracking_enabled()` (if you absolutely need this function) It accepts a `scope` parameter instead of the previously used `hub` parameter.
 - Utility function `is_auto_session_tracking_enabled_scope()` has been removed. There is no public replacement. There is a private `_is_auto_session_tracking_enabled()` (if you absolutely need this function)
 
 ### Deprecated
 
+- `sentry_sdk.start_transaction` is deprecated. Use `sentry_sdk.start_span` instead.
 
 ## Upgrading to 2.0
 

sentry_sdk/api.py

@@ -2,10 +2,15 @@
 
 from sentry_sdk import tracing_utils, Client
 from sentry_sdk._init_implementation import init
-from sentry_sdk.scope import Scope, new_scope, isolation_scope
-from sentry_sdk.tracing import NoOpSpan, Transaction, trace
+from sentry_sdk.tracing import POTelSpan, Transaction, trace
 from sentry_sdk.crons import monitor
 
+# TODO-neel-potel make 2 scope strategies/impls and switch
+from sentry_sdk.integrations.opentelemetry.scope import (
+    PotelScope as Scope,
+    new_scope,
+    isolation_scope,
+)
 
 from sentry_sdk._types import TYPE_CHECKING
 
@@ -227,22 +232,40 @@ def flush(
     return get_client().flush(timeout=timeout, callback=callback)
 
 
-@scopemethod
 def start_span(
+    *,
+    span=None,
+    custom_sampling_context=None,
     **kwargs,  # type: Any
 ):
-    # type: (...) -> Span
-    return get_current_scope().start_span(**kwargs)
+    # type: (...) -> POTelSpan
+    """
+    Start and return a span.
+
+    This is the entry point to manual tracing instrumentation.
+
+    A tree structure can be built by adding child spans to the span.
+    To start a new child span within the span, call the `start_child()` method.
+
+    When used as a context manager, spans are automatically finished at the end
+    of the `with` block. If not using context managers, call the `finish()`
+    method.
+    """
+    # TODO: Consider adding type hints to the method signature.
+    return get_current_scope().start_span(span, custom_sampling_context, **kwargs)
 
 
-@scopemethod
 def start_transaction(
     transaction=None,  # type: Optional[Transaction]
     custom_sampling_context=None,  # type: Optional[SamplingContext]
     **kwargs,  # type: Unpack[TransactionKwargs]
 ):
-    # type: (...) -> Union[Transaction, NoOpSpan]
+    # type: (...) -> POTelSpan
     """
+    .. deprecated:: 3.0.0
+        This function is deprecated and will be removed in a future release.
+        Use :py:meth:`sentry_sdk.start_span` instead.
+
     Start and return a transaction on the current scope.
 
     Start an existing transaction if given, otherwise create and start a new
@@ -271,8 +294,10 @@ def start_transaction(
         constructor. See :py:class:`sentry_sdk.tracing.Transaction` for
         available arguments.
     """
-    return get_current_scope().start_transaction(
-        transaction, custom_sampling_context, **kwargs
+    return start_span(
+        span=transaction,
+        custom_sampling_context=custom_sampling_context,
+        **kwargs,
     )
 
 
@@ -311,10 +336,8 @@ def get_baggage():
     return None
 
 
-def continue_trace(
-    environ_or_headers, op=None, name=None, source=None, origin="manual"
-):
-    # type: (Dict[str, Any], Optional[str], Optional[str], Optional[str], str) -> Transaction
+def continue_trace(environ_or_headers, op=None, name=None, source=None, origin=None):
+    # type: (Dict[str, Any], Optional[str], Optional[str], Optional[str], Optional[str]) -> Transaction
     """
     Sets the propagation context from environment or headers and returns a transaction.
     """

sentry_sdk/integrations/asgi.py

@@ -96,9 +96,9 @@ def __init__(
         unsafe_context_data=False,
         transaction_style="endpoint",
         mechanism_type="asgi",
-        span_origin="manual",
+        span_origin=None,
     ):
-        # type: (Any, bool, str, str, str) -> None
+        # type: (Any, bool, str, str, Optional[str]) -> None
         """
         Instrument an ASGI application with Sentry. Provides HTTP/websocket
         data to sent events and basic handling for exceptions bubbling up

sentry_sdk/integrations/boto3.py

@@ -3,7 +3,6 @@
 import sentry_sdk
 from sentry_sdk.consts import OP, SPANDATA
 from sentry_sdk.integrations import Integration, DidNotEnable
-from sentry_sdk.tracing import Span
 
 from sentry_sdk._types import TYPE_CHECKING
 from sentry_sdk.utils import (
@@ -19,6 +18,8 @@
     from typing import Optional
     from typing import Type
 
+    from sentry_sdk.tracing import Span
+
 try:
     from botocore import __version__ as BOTOCORE_VERSION  # type: ignore
     from botocore.client import BaseClient  # type: ignore

sentry_sdk/integrations/opentelemetry/consts.py

@@ -1,7 +1,23 @@
 from opentelemetry.context import create_key
 
 
+# propagation keys
 SENTRY_TRACE_KEY = create_key("sentry-trace")
 SENTRY_BAGGAGE_KEY = create_key("sentry-baggage")
+
+# scope management keys
+SENTRY_SCOPES_KEY = create_key("sentry_scopes")
+SENTRY_FORK_ISOLATION_SCOPE_KEY = create_key("sentry_fork_isolation_scope")
+
 OTEL_SENTRY_CONTEXT = "otel"
 SPAN_ORIGIN = "auto.otel"
+
+
+class SentrySpanAttribute:
+    # XXX not all of these need separate attributes, we might just use
+    # existing otel attrs for some
+    DESCRIPTION = "sentry.description"
+    OP = "sentry.op"
+    ORIGIN = "sentry.origin"
+    MEASUREMENT = "sentry.measurement"
+    TAG = "sentry.tag"

sentry_sdk/integrations/opentelemetry/contextvars_context.py

@@ -1,26 +1,32 @@
-from opentelemetry.context import Context, create_key, get_value, set_value
+from opentelemetry.context import Context, get_value, set_value
 from opentelemetry.context.contextvars_context import ContextVarsRuntimeContext
 
-from sentry_sdk.scope import Scope
-
-
-_SCOPES_KEY = create_key("sentry_scopes")
+import sentry_sdk
+from sentry_sdk.integrations.opentelemetry.consts import (
+    SENTRY_SCOPES_KEY,
+    SENTRY_FORK_ISOLATION_SCOPE_KEY,
+)
 
 
 class SentryContextVarsRuntimeContext(ContextVarsRuntimeContext):
     def attach(self, context):
         # type: (Context) -> object
-        scopes = get_value(_SCOPES_KEY, context)
+        scopes = get_value(SENTRY_SCOPES_KEY, context)
+        should_fork_isolation_scope = context.pop(
+            SENTRY_FORK_ISOLATION_SCOPE_KEY, False
+        )
 
         if scopes and isinstance(scopes, tuple):
             (current_scope, isolation_scope) = scopes
         else:
-            current_scope = Scope.get_current_scope()
-            isolation_scope = Scope.get_isolation_scope()
+            current_scope = sentry_sdk.get_current_scope()
+            isolation_scope = sentry_sdk.get_isolation_scope()
 
-        # TODO-neel-potel fork isolation_scope too like JS
-        # once we setup our own apis to pass through to otel
-        new_scopes = (current_scope.fork(), isolation_scope)
-        new_context = set_value(_SCOPES_KEY, new_scopes, context)
+        new_scope = current_scope.fork()
+        new_isolation_scope = (
+            isolation_scope.fork() if should_fork_isolation_scope else isolation_scope
+        )
+        new_scopes = (new_scope, new_isolation_scope)
 
+        new_context = set_value(SENTRY_SCOPES_KEY, new_scopes, context)
         return super().attach(new_context)

sentry_sdk/integrations/opentelemetry/integration.py

@@ -6,7 +6,12 @@
 
 from sentry_sdk.integrations import DidNotEnable, Integration
 from sentry_sdk.integrations.opentelemetry.propagator import SentryPropagator
-from sentry_sdk.integrations.opentelemetry.span_processor import SentrySpanProcessor
+from sentry_sdk.integrations.opentelemetry.potel_span_processor import (
+    PotelSentrySpanProcessor,
+)
+from sentry_sdk.integrations.opentelemetry.contextvars_context import (
+    SentryContextVarsRuntimeContext,
+)
 from sentry_sdk.utils import logger
 
 try:
@@ -46,9 +51,14 @@ def setup_once():
 
 def _setup_sentry_tracing():
     # type: () -> None
+    import opentelemetry.context
+
+    opentelemetry.context._RUNTIME_CONTEXT = SentryContextVarsRuntimeContext()
+
     provider = TracerProvider()
-    provider.add_span_processor(SentrySpanProcessor())
+    provider.add_span_processor(PotelSentrySpanProcessor())
     trace.set_tracer_provider(provider)
+
     set_global_textmap(SentryPropagator())
 
 

sentry_sdk/integrations/opentelemetry/potel_span_processor.py

@@ -5,14 +5,16 @@
 from opentelemetry.sdk.trace import Span, ReadableSpan, SpanProcessor
 
 from sentry_sdk import capture_event
+from sentry_sdk.tracing import DEFAULT_SPAN_ORIGIN
 from sentry_sdk.integrations.opentelemetry.utils import (
     is_sentry_span,
-    convert_otel_timestamp,
+    convert_from_otel_timestamp,
+    extract_span_attributes,
     extract_span_data,
 )
 from sentry_sdk.integrations.opentelemetry.consts import (
     OTEL_SENTRY_CONTEXT,
-    SPAN_ORIGIN,
+    SentrySpanAttribute,
 )
 from sentry_sdk._types import TYPE_CHECKING
 
@@ -107,21 +109,21 @@ def _root_span_to_transaction_event(self, span):
         # type: (ReadableSpan) -> Optional[Event]
         if not span.context:
             return None
-        if not span.start_time:
-            return None
-        if not span.end_time:
+
+        event = self._common_span_transaction_attributes_as_json(span)
+        if event is None:
             return None
 
         trace_id = format_trace_id(span.context.trace_id)
         span_id = format_span_id(span.context.span_id)
         parent_span_id = format_span_id(span.parent.span_id) if span.parent else None
 
-        (op, description, status, _) = extract_span_data(span)
+        (op, description, status, _, origin) = extract_span_data(span)
 
         trace_context = {
             "trace_id": trace_id,
             "span_id": span_id,
-            "origin": SPAN_ORIGIN,
+            "origin": origin or DEFAULT_SPAN_ORIGIN,
             "op": op,
             "status": status,
         }  # type: dict[str, Any]
@@ -135,47 +137,68 @@ def _root_span_to_transaction_event(self, span):
         if span.resource.attributes:
             contexts[OTEL_SENTRY_CONTEXT] = {"resource": dict(span.resource.attributes)}
 
-        event = {
-            "type": "transaction",
-            "transaction": description,
-            # TODO-neel-potel tx source based on integration
-            "transaction_info": {"source": "custom"},
-            "contexts": contexts,
-            "start_timestamp": convert_otel_timestamp(span.start_time),
-            "timestamp": convert_otel_timestamp(span.end_time),
-        }  # type: Event
+        event.update(
+            {
+                "type": "transaction",
+                "transaction": description,
+                # TODO-neel-potel tx source based on integration
+                "transaction_info": {"source": "custom"},
+                "contexts": contexts,
+            }
+        )  # type: Event
 
         return event
 
     def _span_to_json(self, span):
         # type: (ReadableSpan) -> Optional[dict[str, Any]]
         if not span.context:
             return None
-        if not span.start_time:
-            return None
-        if not span.end_time:
+
+        span_json = self._common_span_transaction_attributes_as_json(span)
+        if span_json is None:
             return None
 
         trace_id = format_trace_id(span.context.trace_id)
         span_id = format_span_id(span.context.span_id)
         parent_span_id = format_span_id(span.parent.span_id) if span.parent else None
 
-        (op, description, status, _) = extract_span_data(span)
+        (op, description, status, _, origin) = extract_span_data(span)
 
-        span_json = {
-            "trace_id": trace_id,
-            "span_id": span_id,
-            "origin": SPAN_ORIGIN,
-            "op": op,
-            "description": description,
-            "status": status,
-            "start_timestamp": convert_otel_timestamp(span.start_time),
-            "timestamp": convert_otel_timestamp(span.end_time),
-        }  # type: dict[str, Any]
+        span_json.update(
+            {
+                "trace_id": trace_id,
+                "span_id": span_id,
+                "op": op,
+                "description": description,
+                "status": status,
+                "origin": origin or DEFAULT_SPAN_ORIGIN,
+            }
+        )
 
         if parent_span_id:
             span_json["parent_span_id"] = parent_span_id
+
         if span.attributes:
             span_json["data"] = dict(span.attributes)
 
         return span_json
+
+    def _common_span_transaction_attributes_as_json(self, span):
+        # type: (ReadableSpan) -> Optional[dict[str, Any]]
+        if not span.start_time or not span.end_time:
+            return None
+
+        common_json = {
+            "start_timestamp": convert_from_otel_timestamp(span.start_time),
+            "timestamp": convert_from_otel_timestamp(span.end_time),
+        }  # type: dict[str, Any]
+
+        measurements = extract_span_attributes(span, SentrySpanAttribute.MEASUREMENT)
+        if measurements:
+            common_json["measurements"] = measurements
+
+        tags = extract_span_attributes(span, SentrySpanAttribute.TAG)
+        if tags:
+            common_json["tags"] = tags
+
+        return common_json