- Preparing search index...
- The search index is not available
The type of the event read from the source inQueue
This governs micro-batching events that have been received from the source inQueue before they
are sent to your transform function, allowing that function to receive events in batches instead
of one at a time. This can be useful when your transform function will reach out and hit an external
resource such as a database. Hitting a database for every single event that flows through a pipe can
be very detrimental to performance. So, it's common to micro-batch say 100 or 1000 or so and then
construct a single query to a database to read/write all data as a single DB operation.
If this is a number, it's just the number of events to micro-batch up.
If true, connect to multiple S3 files simultaneously to pre-fetch files when reading from a queue that has events stored in S3. This is a new feature and so is not on by default, though it can dramatically improve read performance. It is expected to be made the default in Q3 2022.
When using the ReadOptions.fast_s3_read feature, this specifies how many bytes of s3 data we want
to prefetch. The default usually is correct.
When using the ReadOptions.fast_s3_read feature, this specifies how many retries for S3 timeouts
before giving up on preconnecting.
The default usually is correct.
The name of the bot that this code is acting as. The SDK will use it to query to the bot Dynamo DB table to pull checkpoints and to checkpoint for you.
The source queue from which events will be read
The limit of the number of records, events, to read in total from the queue before closing the read stream.
The max number of times the SDK will query the stream for new events before it shuts down the read stream. Consider that each query by the SDK to retrieve events from DynamoDB could return many, many events and that an "event" may actually be a pointer to an S3 file full of events.
It is uncommon for developers to have to set this, much less know it exists.
The largest event ID that you should read, exclusive. So, stop reading when you arrive at this event ID.
Use startTime and this option to read a range of events from a queue.
The duration of time the to read for before closing the read stream. It is common to set this to 75% to 80% of the time remaining before the lambda is shut down to give the lambda sufficient time to finish processing. Of course, different types of processing will differ.
Note, this type is any one of the valid durations the Moment JS library can take: Duration | number | string | FromTo | DurationInputObject.
The read stream will shutdown as soon as one of the constraints is met: runTime, loops, limit, size,
stopTime.
The limit of the number of bytes to read in total from the queue before closing the read stream.
The event ID of the starting position to read from with in the queue. It is common to not provide this because each queue is stateful in that it remembers the last read position of a bot. Then, as bots read they make a call back to the RStreams Bus to update the read position.
Usually, the SDK just handles this for you. So, if the start isn't provided, the SDK will just use the bot's last read position as the starting point. So, as bots are invoked, read some events and do some processing, they automatically update back how far they've read to and then the bot shuts down after a period of time. When the bot starts back up to read again, it knows where it last read from and just continues.
When to stop reading as a time since the epoch.
The max number of records, events, the SDK should retrieve each time it retrieves events from the RStreams Bus' Dynamo DB events table.
The SDK will invoke this function after reading events from the inQueue where you can do your processing.
Generated using TypeDoc
These options for an
RStreamsSdk.offloadpipeline step. This reads events from a queue and allows for the processing of the data. Note this inherits all the useful options fromReadOptionswhich control reading frominQueue.This inherits from
OffloadOptionsand adds the ability to batch the data read from the source inQueue before it is sent to thetransformfunction which means your transform function will recieve an array of type T which is the batched events instead of just a single event of type T.RStreamsSdk.offloadRStreamsSdk.offloadEvents