Message investigation guide

Kafka Streamyard: Message Playback and Live Feed

Learn how to inspect records already stored in Kafka, follow new messages as they arrive, narrow both workflows with filters, and turn the result grid into a focused investigation workspace.

Lesson purpose

Historical playback and live monitoring in one workflow

Message Playback answers “What messages are already in this topic?” Live Feed answers “What is arriving now?” Both begin with the same Result Control panel and lead into the same result grid and Message Details tools.

Result Control

Define the investigation before reading Kafka

The Result Control panel sits above the message grid. Select partitions, add an optional filter, choose a Function, then start finite playback or a live feed. The same toolbar also controls result counts, views, column sizing, import, export, and wrapping.

For the matching in-app reference, open Help and search for Message Playback. Related Help topics include Offset Function Help, SL (Streamyard Language) reference guide, Import / Export, and Result View Customisation.

Finite investigation

Use Message Playback for records already in Kafka

Playback performs a finite read from a defined log position or time window. Use it for historical incidents, known customer journeys, filter testing, exact offsets, and controlled exports.

Choose partitions, Filter, and Function, then press Play. Playback stops automatically when the requested read completes. Press Stop when a range is larger than expected or you already have enough evidence.

Continuous monitoring

Use Live Feed for newly arriving records

Live Feed starts at the latest position and follows new records on the selected partitions. It is ideal for test runs, deployments, active incidents, producer checks, and following one request through a system.

Prepare partitions, Filter, and View first. While live monitoring is active, the controls that define the feed are protected from accidental changes. Stop the feed before changing its scope.

Safe controls

Clear is not delete

Clear removes displayed rows and resets visible counters. During a live feed it gives you a clean screen while monitoring continues. It never removes records from Kafka.

Stop ends active playback or monitoring. Existing rows remain available for inspection until they are cleared or replaced.

Partitions and ordering

Choose where to look before choosing how to read

All or a subset

Select the relevant partitions

Choose All, one partition, or any useful subset. All partitions are best when you do not know where a key landed or want a topic-wide chronological investigation. A single-partition topic disables the selector because there is no choice to make.

Partition-local positions

Understand Kafka log offsets

Offsets are local to each partition. Partition 0 offset 42 and partition 1 offset 42 are unrelated records. Direct offset Functions can still target several partitions: Streamyard applies the same numeric position independently to each selected partition.

Correct sorting

Timestamp across partitions, offset within one

With several partitions selected, sort by Timestamp for a useful chronological merge. With one partition selected, sort by Offset for strict Kafka record order.

Playback Functions

Control the read window without memorising every command

The Function defines finite playback. Type a canonical expression directly, focus the field for suggestions and recent values, or use the clock builder for guided absolute time, Time offset, and Log offset choices.

Recent log data

`tail=2000`

Read the latest 2,000 records from each selected partition. This is a practical default because it provides recent context without scanning the whole retained log.

Use ranges such as tail=1000..2000 when you need a bounded slice relative to each partition’s latest offset.

Earliest records

`head=100` and `beginning`

head=100 reads the first 100 retained records per selected partition. beginning starts at the earliest available offsets and scans forward, so use it carefully on large topics.

A head range such as head=10..20 selects positions relative to each partition’s earliest retained offset.

Shared time window

`since=15m`

Time Functions are useful across partitions because they describe one shared period. Try since=1h, since=24h, or since=7d.

Use since=48h..24h to inspect a historical window that starts 48 hours ago and ends 24 hours ago.

Calendar selection

Absolute local or UTC time

The clock builder creates absolute ranges from calendar and time fields. Choose UTC or Local explicitly. A single start such as utc=250220T10:20 or abs=250220T10:20 reads from that instant through now.

A complete range such as utc=250220T10:20..250224T02:00 provides exact start and end boundaries.

Exact log positions

`offset=100..200`

Read the inclusive offset range 100 through 200 independently from every selected partition. Selecting one partition remains the clearest choice when you know exactly which partition owns the records.

offset=1000.. starts each selected partition at offset 1,000 and reads forward up to the configured result limits.

Fast reuse

Suggestions, history, and reset

Focus Function to see common examples and recently used commands. The embedded X first restores the active connection’s configured default. Select X again when that default is displayed to empty the field.

Open Offset Function Help in the app for all canonical forms and aliases.

Filtering

Reduce a large read to the records that answer your question

Message, key, and headers

Start with readable SL expressions

Use m for message content, k for the key, and h for headers. Examples include m.status == 'FAILED', m.customer.id == 'C-42', k.orderId == 'ORD-1001', and h.environment == 'test'.

Autocomplete

Learn fields from real topic data

After Streamyard has seen representative records, autocomplete can suggest message fields, key fields, headers, nested paths, operators, and functions. This reduces typing and helps avoid incorrect paths.

Reusable investigations

Recent history and saved filters

Press Down Arrow on an empty Filter to open recent history, or Ctrl+Space when text is present. Use the chevron for the multiline editor and the pencil for named, tagged, testable saved filters.

Result status and inspection

Understand the match, then open the complete record

Filter selectivity

Read the result status as evidence

A status such as Messages found: 12/2000 means 2,000 records were examined and 12 matched. A broad ratio such as 950/1000 and a selective ratio such as 12/1000 tell very different stories.

Zero matches can mean no matching data, the wrong partitions, the wrong Function window, or an incorrect filter. Elapsed milliseconds can be shown or hidden in Settings, and descriptive labels collapse before counts when toolbar space is constrained.

Message Details

Click a row for the full Kafka record

The right- or bottom-docked Message Details pane shows the complete value, key, headers, partition, offset, timestamp, formats, sizes, copy actions, samples, validation, and message-production tools.

Focus the grid and use Up/Down Arrow, Home, or End to move through results while Message Details follows the selected row.

Timezone clarity

Display message time as Local or UTC

Settings can display Kafka message timestamps in machine-local time or UTC. Result headers and Message Details labels identify the selected timezone without changing the underlying Kafka timestamp.

Result views

Put the useful fields directly into the grid

Focused layouts

Choose built-in or saved views

A useful result view reduces how often you need to open every record. Display Kafka metadata, headers, key and message paths, arrays, mappings, and calculated values in the order that supports the investigation.

Fast sizing

Auto fit displayed content

Choose Auto fit from View to measure displayed headers and values and persist the resulting widths. Wide content uses horizontal scrolling; with Wrap enabled, a long final column targets approximately three lines.

Long payloads

Toggle Wrap

Wrap off keeps rows compact for scanning. Wrap on creates taller rows that reveal more JSON, XML, or text. It changes presentation only and never modifies Kafka data or playback.

Moving data

Export evidence or deliberately produce imported records

Export

Use loaded results or a server-side task

Copy or download loaded rows as a JSON array or NDJSON. Optional envelopes can preserve message value, key, headers, and partition. A CSV report uses the columns in the currently selected result view.

Server-side export is better for substantial ranges because it can process data without first loading every record into the UI.

Import

Import produces new Kafka records

Import accepts JSON arrays, JSON objects, NDJSON, and XML. Records can include values, keys, headers, and partition information. Streamyard validates source partitions against the destination topic when you choose to preserve them.

Import is not a temporary UI operation: it produces records to the selected Kafka topic. Always review the preview, destination, key, header, and partition options before applying it.

Suggested workflows

Put the controls together

Historical failure

Find a recent failed event

Select all partitions, use since=1h, add m.status == 'FAILED', and press Play. Compare found versus consumed, sort by Timestamp, then inspect each matching Message Details record.

Active journey

Follow one customer live

Select relevant partitions, add a customer or correlation filter, choose a view containing identity, event type, and status, then start Live Feed. Clear between test runs without stopping monitoring.

Precise log slice

Inspect exact offsets

Select the known partition when possible, use a range such as offset=1000..1100, press Play, sort by Offset, and step through rows with the keyboard.

Repeatable operations

Build a reusable view and filter

Play representative data, create a custom result view, Auto fit or resize its columns, save a filter preset, and reuse both for later playback and live monitoring.

Summary

Start with the question you need Kafka to answer

Use Message Playback for finite historical reads and Live Feed for newly arriving records. Choose partitions, Filter, Function, and View first; then use the result status and Message Details to turn matching rows into evidence.

Try Kafka Streamyard Read the topic filtering guide Back to blog