Interface CheckpointData

Legend

Namespace
Variable
Function
Function with type parameter
Type alias
Type alias with type parameter

Interface
Interface with type parameter
Property

Enumeration

Class

Used to manually checkpoint in a pipeline step. It's only rarely used in more advanced cases where one waits until the end of a pipeline to checkpoint manually.

Let's assume that we are reading from an upstream queue that we want to checkpoint to and keep track of where we've read to. Let's assume that we are aggregating events from the upstream queue and turing say 10 events from the upstream queue into just 1 event that we write to the downstream queue. In this case, we would probably wait until we've done the aggregation. When we checkpoint, we would set the units to 10, we'd set the started_timestampand ended_timestamp to the first/last event timestamps of the 10 respectively and we'd set the `start_eid to the first event event IDs respectively.

see: StreamUtil.stats A function that can be called to set the checkpoint.
see: StatsStream Creates a pipeline step that you can use to get/set checkpoints manually

Hierarchy

CheckpointData

Index

Properties

eid

eid: string

The exact event ID to checkpoint

Optional ended_timestamp

ended_timestamp?: number

Read the comment on this interface to understand what this is

Optional source_timestamp

source_timestamp?: number

The timestamp of the very first event that made it to a queue of the bus that then flowed down the bus to get to here. This allows the bus to compute source lag times. Each derivative event that flows down the bus should track this and pass it along as events flow/are derived and flow down the bus.

Optional start_eid

start_eid?: string

Read the comment on this interface to understand what this is

Optional started_timestamp

started_timestamp?: number

Read the comment on this interface to understand what this is

Optional units

units?: number

Read the comment on this interface to understand what this is