Telemetry Processors: The Middle of the OTel Pipeline

WHY IT EXISTS: The OpenTelemetry Collector is designed to provide a flexible pipeline for observability data. A structured pipeline with distinct stages is necessary to manage the flow of telemetry from collection to storage. Processors exist as a dedicated stage in this pipeline for handling data after it's received but before it's exported.

THE MENTAL MODEL: Think of an OpenTelemetry Collector as an assembly line for your telemetry data. Receivers are the loading docks where raw materials (data) arrive. Exporters are the shipping department sending finished goods (processed data) out. Processors are the workstations on the assembly line itself, where the data is handled, modified, or inspected as it passes through.

HOW IT WORKS: A processor is defined as a component in the Collector's config.yaml file. The configuration has a top-level section called processors where you define one or more specific processor instances and their settings. However, defining a processor does not activate it. To make it part of the data flow, you must reference it by name in the pipelines section, which is located under the service section. A pipeline connects receivers, a chain of processors, and exporters for a specific data type like traces or metrics.

WHEN TO USE IT: According to the OpenTelemetry Collector's design, you use a processor whenever you need to act on telemetry data that has been received but not yet exported. It is the designated component type for any in-flight data manipulation. The configuration structure requires you to place any such logic within a processor component and enable it in a pipeline.

WHEN NOT TO USE IT: A processor is not used for collecting data from external sources; that is the role of a Receiver. It is also not used for sending data to a backend or another system; that is the role of an Exporter. If you only need to forward data from a receiver to an exporter without any intermediate steps, you can define a pipeline with an empty or non-existent processor list.

ONE CANONICAL EXAMPLE: While the source does not provide a specific processor example, it details the configuration structure. A configuration file would have a top-level processors: key. Below this, you would name and configure your processor, for example, my_processor:. Then, in the service: section, you would define a pipeline like traces: which lists your receivers, processors, and exporters: pipelines: traces: receivers: [otlp_receiver] processors: [my_processor] exporters: [otlp_exporter]. This structure ensures the my_processor is applied to data from otlp_receiver before being sent to otlp_exporter.