Searchable product docs

Kafka Streamyard docs

Browse the same help topics that power the desktop app, rendered as HTML for search engines and easy linking.

Product help
Choose a topic from the left or search for a feature
Selected topic
Overviewoverview.screenTour

Screen tour

layoutscreenpanestopic listresult controlmessage playbacklive controlsresult viewmessage panelhelp
This is a guided tour of the main Kafka Streamyard screen, using focused screenshots of the main areas you work in most often.

Main screen

This is the main screen. It gives you the overall working layout with topic and navigation tools on the left, active working controls in the middle, and the message panel docked to the right.
In this example you can see a topic selected from the list, matching results returned to the UI, and the selected message opened in the details panel for inspection and testing.
Kafka Streamyard main screen showing the left navigation and topic list, the center message playback and live controls with results table, and the message details panel on the right.

Top-left navigation

At the top-left, the static navigation links provide quick access to tools such as Cluster Dashboard, All Consumers, Topic Information and the important Playbooks and Data Samples - also will show Schema Registry editor when configured.
Kafka Streamyard top-left static navigation links including dashboard, consumers, topic information, playbooks, and data samples.

Quick connection bar

This is a quick connection access bar for your most used connections.
Hover over to pop-out an additional screen to the same connection, which is useful if you are doing lots of work and testing on the same connection, live view feeds showing on one screen and you send samples on the other against various topics.
Kafka Streamyard quick connection access bar for frequently used connections.

Hover over to show popout clickable glyph
Kafka Streamyard quick connection pop-out additional connection.

Icons & dropdown

This shows other connections which is for connections you don't use so often, a task icon to show running tasks like copy topic data, repartitioning, playbooks etc, the searchable help, and the settings / connection manager icon.
Kafka Streamyard top-right icons and dropdown showing other connections, tasks, help, and settings.

Topic list

The topic list offers advanced saveable filters that are easily selectable.
Green tabs mean data has been searched and returned to UI and is persisted until cleared.
A blue flash tab shows data is in live feed mode and will show number returned in square brackets.
The number next to topic in orange is number of partitions.
Everything is asynchronous so you can find data on one topic and look for data on another and any will show when data results are returned with the green tab as the indicator.
At the bottom of the topic list shows the filtered vs total amount of topics on this connection.
Kafka Streamyard topic list showing topic filters, tab indicators, partition counts, and topic activity status.

Topic list context menu

On the list is a context menu with many functions available directly from the selected topic.
Kafka Streamyard topic list context menu showing topic actions such as create, import, export, copy, repartition, delete, and playbook tools.

Views

Bottom left shows the Views these are live feeds of 1 or more related topics chronologically returned to the UI.
You can have 30 topics per view and can filter on each one individually using saved filters or immediately added filters.
Right-click the topic list and choose Add visible topics to a new or existing view... to bulk-add the currently filtered topics.
Kafka Streamyard views area showing multi-topic live views in the bottom-left of the screen.

Message Playback & Live Controls and result list

At the top is partition selection, filter (press down key for history and pencil icon to save and select saved filters from any connection or topic).
The Offset function provides targeted functions to select from various offsets.
There is also a live feed button here: Live feed.
You can import or export data from this top area and even customize the result display to make it more user friendly which you can export then as CSV making it great to share with the business.
For filtering on a message with the aid of autocomplete, m. means message, k. means key and h. means header, once the data is 'learnt', meaning you've played some messages back, autocomplete is available on key, value and header.
Kafka Streamyard Message Playback and Live Controls showing partition selection, filter history, offset functions, live feed, and import/export actions.

Message panel and samples

The all important message panel which can be docked right or bottom. Shows all the info about a topic and it's message, value key and header.
You can trace selected history here which is great when you want a message back but can't remember the offset.
You can copy and paste the whole message in KS json format (holds all the info about a message, partition, offset, timestamp, etc.) or individual parts of the message.
The stand out feature of KS is being able to send samples to the topic for manual testing. We'll do a blog post on this subject as this is one of the main drivers for creating this software.
You can see at the bottom you can have various sample data organised in folders and the order you might want to send them in.
You can also save samples directly here if you have a message that needs investigate and later replaying once a fix is in.
You can create templated messages and assign values automatically and is used in conjunction with Data Samples seen top left.
Kafka Streamyard message panel showing message details, history, and sample playback tools.
Overviewoverview.connectionSettings

Connection settings (Connections Editor)

connectionsconnection settingsconnections editorbootstrap serverssecurityschema registrykafka connectpermissions
The Connections Editor defines how Streamyard connects to Kafka (and optional services like Schema Registry and Kafka Connect). Use the popovers next to fields for quick explanations.

Settings tab

Cluster name
Streamyard’s internal name for this connection. Used in tabs, tasks, and quick links. Must be unique.
Bootstrap servers
One or more host:port pairs, comma-separated.
Default function
Default query/time range when opening topics. Example: tail=2000 or since=24h.
Partitioning
Controls how produced records are assigned to partitions when you don’t explicitly choose one.
Default replication factor
Convenience default used by Streamyard when creating new topics (does not change existing topics).
Key/message format
How Streamyard parses and displays message keys/values (Auto is best-effort; explicit formats reduce mis-detection).
Quick links + color
Favorites control whether the connection appears in the top-bar quick links. Optional colors help visually tag environments.

Security tab (Kafka)

Security option selects the authentication mode (Plaintext / SSL / SASL / Confluent / AWS MSK / AWS MSK IAM).
SSL typically requires trust store / key store paths (JKS) and matching passwords.
SASL requires a security protocol (SASL_PLAINTEXT or SASL_SSL) plus a mechanism (PLAIN / SCRAM) and credentials.
AWS MSK IAM uses IAM authentication over TLS (SASL_SSL). See the dedicated AWS MSK IAM (SASL_SSL) setup help entry.

Schema tab (Schema Registry)

Configure Schema Registry only if you use schema-aware formats. Streamyard can validate/format produce payloads using schemas.
Security for Schema Registry is configured separately from Kafka security (often different URLs and credentials).

Connect tab (Kafka Connect REST)

Optional: set the Kafka Connect REST base URL so Streamyard can list connectors and show status.

Permissions tab (guardrails)

These toggles are Streamyard-side defaults that help prevent accidental destructive actions.
You can disable deleting topics, emptying topics, or producing/sending messages even if your Kafka credentials would allow it.

Advanced tab (performance tuning)

Consumer profiles and flush settings trade off throughput vs UI responsiveness. If you see laggy UI or high memory use during Live feeds, adjust these.
This depends heavily on your connection throughput and latency: if the connection is slow / low throughput, you may notice little to no performance difference when changing these.
Overviewoverview.tasks

Tasks (background jobs)

tasksbackgroundjobscopytopic copyingplaybookssample loaderrandomizedrandomiserprogresswaitdelayburstquantity-1until stopped
Tasks are long-running operations executed by the backend. The Tasks window lets you monitor progress and review details/output.

Why Tasks matter

Tasks let you start heavy operations and come back later instead of blocking your main investigation window.
This is useful for long-running topic copies, imports, playbooks, repartitioning, and sample-loader runs where you still want to keep browsing topics or validating results elsewhere in the app.
A shared task history also makes it easier to explain what ran, whether it succeeded, and where it failed.

What runs under Tasks

Topic copying: background copy operations that move/replicate data from one topic to another.
Playbooks running: submitted playbook scripts execute as tasks and stream logs/output as they run.
Randomised data sample loading: sample loader runs can use fixed/random waits, burst sizes like 10 to 20, or -1 quantity to keep producing until you stop the task.
Overviewoverview.awsMskIam

AWS MSK IAM (SASL_SSL) setup

awsmskiamsasl_sslaws_msk_iamprofilecredentialstailscalevpc9098
This guide explains how to connect Streamyard to an AWS MSK cluster that uses IAM authentication (AWS_MSK_IAM) over TLS (SASL_SSL).

1) AWS prerequisites

Your MSK cluster must have IAM authentication enabled and you must use the IAM-SASL bootstrap brokers on port 9098.
Your IAM identity must have permission to access the cluster (for example: list topics, describe cluster, read/write topics as needed).

2) Network prerequisites

IAM-SASL brokers are often private VPC endpoints. Your laptop must be able to route to the VPC.
If you are using Tailscale, make sure you have joined the correct tailnet and are accepting routes (--accept-routes).

3) Configure AWS credentials locally

Streamyard uses the standard AWS credential chain. The easiest approach is to create a named profile (example: oso_prod) in ~/.aws/credentials.
# ~/.aws/credentials
[oso_prod]
aws_access_key_id=...
aws_secret_access_key=...
Sanity check:
aws sts get-caller-identity --profile oso_prod
Security note: treat access keys as secrets and rotate them if they are exposed.

4) Streamyard connection settings

In Connection settings:
Servers
Comma-separated brokers, including port 9098.
Security option
AWS MSK IAM
AWS Region
Example: eu-west-2
AWS Profile
Optional, but recommended (example: oso_prod).
Use Test connection to verify the backend can authenticate and reach the brokers.
Overviewoverview.cloudSync

Cloud Sync

cloud syncbunnycloudflarer2s3bucketstorage zoneaccesskeybackupmultiple computerssaved datafilter historysession filter historycriteria history

What is synchronized

Cloud Sync synchronizes saved working data between the local ~/.kafka-streamyard folder and either Bunny Storage or Cloudflare R2:
Criteria presetsReusable SL filters for Kafka records.Topic-list filtersUUID-based saved filters used to find and group topics.Filter historiesValid filters used from Message Playback & Live Controls, scoped by connection and topic, plus valid Session filters used from Live Views, scoped by connection and View.Message-list viewsSaved result columns and calculated fields.PlaybooksSaved Kafka playbook definitions.Data samplesSample messages and their reusable folders.Live ViewsView topics, saved-filter references, custom tab views, visual order, and Flow Charts.
Connections, credentials, application settings, task history, result caches, current Session filter text, and window layout remain local to each computer.

Set up Bunny Storage

Open Settings → Saved Data Cloud Sync, enable synchronization, then enter the Storage Zone name and its Storage Zone AccessKey.
The endpoint defaults to https://storage.bunnycdn.com. An optional folder keeps this app's files separate from other objects in the zone.
Save the settings before using Sync now. The AccessKey is stored using the operating system's secure Electron storage and is never uploaded with the saved data.

Set up Cloudflare R2

Select Cloudflare R2, then enter the account S3 endpoint, bucket, Access Key ID, and Secret Access Key. A pasted endpoint ending in the bucket name is separated automatically.
Use R2's S3 credentials rather than a Cloudflare API token. Both credentials are encrypted using the operating system's secure Electron storage.
Bunny Storage and Cloudflare R2 are mutually exclusive; switching provider preserves the inactive provider's saved credentials for later use.

Same-item conflict handling

Wait until Cloud Sync reports Synced before switching computers. Unrelated UUID records merge independently; if the same record is edited on two computers, the latest UTC update wins. Cloud changes are checked at startup, periodically, when the app regains focus, and when Sync now is pressed.

Local-first behavior

Saves always complete against the local files first. Only the added, updated, or deleted UUID item is uploaded; deletions use tombstones so another computer cannot resurrect stale data. Failed uploads remain pending and are retried later. Disabling Cloud Sync immediately returns Kafka Streamyard to normal local-only operation without removing local data.
Live FeedsliveFeeds.overview

Live Feeds (real-time streaming)

live feedlivestreamfiltercriteriaresult controlmessage playbacklive controlstopic listresult view

What is a Live Feed?

Live mode keeps a stream open and continuously adds newly received records to the current result list.
You enable Live mode from the Message Playback & Live Controls for the currently selected topic.

Why Live Feeds are useful

Live Feeds are useful when you need to watch Kafka while something is actively happening: a tester clicks a button, a user retries a payment, or a service is retrying in the background.
Instead of repeatedly re-running reads, you can keep one filtered stream open and let matching records arrive in real time.
This is especially helpful for production triage, manual verification, and proving whether a message was produced at all versus being produced with the wrong payload.

Common examples

- Follow one customer or order while a tester exercises the UI.
- Watch only retry/error events during an incident.
- Confirm that a new deployment is still emitting the fields and headers you expect.

Start / stop Live mode

In Message Playback & Live Controls:
Start: click Live feed.
Stop: click Stop.
While Live (or Play) is running, controls like time range / since are disabled to keep the stream consistent.

Entering the filter (criteria / SL)

The filter is typed into the Filter input in Message Playback & Live Controls.
Criteria uses the same language as the SL reference guide (see help: SL).
m.type == 'order' and m.total >= 100
@partition == 0 and k.customerId.isPresent()

Counts vs. what you see

Topic list live counter: in TopicList, a topic in Live mode shows a live indicator and a bracketed [count] representing how many records have been received for that live session.
Result grid rows: the ResultView only renders rows you are currently viewing (selected topic / selected partitions) and that pass the current Message Playback & Live Controls criteria.
If a message is received by the Live feed but fails the criteria filter, it will not appear in the result list.
Live ViewsliveViews.overview

Live Views (multi-topic chronological stream)

live viewsviewslivestreammultiple topicschronologicalfiltercriteriatopic flowdiagramorientationhorizontalverticalannotationstext labelarrow messagefeedback arrowloop-backcyclic flowstate storepush pullbidirectional arrowdouble arrowshift selectmulti-selectundoredominimapoverview map4k viewultra widepreferred orderbulk addmicro serviceapi gatewaysession filtercorrelation idlive tracingpost tracingconjunctionlogical and@topicplaybackhistorical playbackforensicspost-event analysisflow chartpulsereceivedshownshow sincedisplay topic resultsprepare filterstest flaguser idtenant idrequest idreduce noisebusy topics

What is a Live View?

A Live View combines messages from multiple topics into one timestamp-ordered timeline. It is useful for following business and technical flows that cross topic boundaries.
Despite the name, a Live View supports both live monitoring and playback. Live feed follows new messages as they arrive, while Play reads a selected historical period and then stops.
Historical playback is especially useful for post-event analysis and incident forensics. A bounded range such as 1h-2h reconstructs what happened across every included topic between two hours ago and one hour ago, in chronological order.
The results are shown together in ResultView. Each row includes its sourceTopic so you can identify where the message originated while reading the complete cross-topic sequence.

Flow Chart as a live visualization

The Flow Chart turns the documented architecture into an operational view of the running message flow. Topic nodes sit alongside the services, connectors, APIs, stores, and annotations that explain how the complete process is intended to work.
While Live feed is running, each topic node displays its Received and Shown counts. When new messages arrive, the topic and its connected arrows pulse; higher activity produces a stronger pulse, making active parts of the process immediately visible.
Used with a Session filter, this provides a focused visual trace: the combined ResultView supplies the exact chronological records, while the Flow Chart shows where those active topics belong in the wider, visually documented process.
Pulsing indicates activity on a connected topic rather than proving that one individual Kafka record traversed every arrow. The chart communicates architectural context and current activity; timestamps, topic names, keys, headers, and payloads in ResultView remain the evidence for detailed tracing.

Why Live Views are useful

Live Views are useful when a business flow spans several topics and you want one investigation timeline instead of jumping between tabs.
They make cross-topic debugging much easier because events from related services interleave naturally in chronological order.
This is often the fastest way to understand whether a record was published, transformed, retried, dead-lettered, or consumed by a downstream step.

Common examples

- Trace an order from command topic to validation topic to fulfillment topic.
- Follow one customer journey across audit, event, and notification topics.
- Watch a retry queue alongside the main topic to understand recovery behavior.

View Result Control

When a View is selected, the toolbar switches to View Result Control.
Use Live feed to start a real-time stream for the view. Show since accepts one lookback value such as 15m; the view loads that recent context and then continues receiving new messages.
Use Play for finite playback. A single value such as 15m reads messages from that cutoff onward, while a closed range such as 1h-2h reads only that historical window.
Show since supports duration units s, m, h, d, and w, plus an absolute UTC timestamp ending in Z. Relative values are calculated when Play or Live is pressed.
Use Stop to stop streaming.
Use Reset to clear the current Live View results and counters without removing the view, topics, or filters. This is useful while a feed is active if you want to start watching from a clean slate.
Use the Wrap toggle to switch line wrapping on or off in the combined results.
The summary below the controls shows how many messages were received and how many are currently shown in the active time window.

Chronological ordering

Live Views are intended to be read as a single timeline across topics.
The grid is ordered using message timestamps (and uses offsets/partitions as tie-breakers), so events from different topics interleave naturally.

Selecting topics to display

In View Result Control, use Display topic results to choose which topics are currently shown.
Selecting or deselecting a topic is a temporary display choice only. Hide topics to reduce visual noise and focus the combined timeline on the part of an investigation that matters right now.
Hiding a topic does not remove it from the View, change its filters, or stop its data collection. Messages continue to arrive in the background, and selecting the topic again restores its available results to the combined grid.

Preparing topic filters before using a Live View

A useful Live View often needs some initial preparation, especially when its topics carry more traffic than the messages you want to study. Prepare each topic filter before assembling the final multi-topic view.
Open the appropriate topic in Message Playback & Live Controls, write the filter against that topic's key, message, or headers, and test it with Play until the results consistently match your expectations. Once you are satisfied, save the filter with the appropriate connection and topic tags so it is ready to select in Edit view.
Topic filters reduce noise before records enter the combined View results. Busy topics may contain unrelated users, requests, environments, automated jobs, or other teams' traffic; ignoring those records makes the event sequence easier to understand and reduces unnecessary processing and display volume.
Look for an identifier you can use or co-opt for testing and investigations: a dedicated test flag, user id, tenant id, request id, or correlation id in the key, payload, or message headers. For example, h.testRun == "checkout-42" or h.userId == "1234" can isolate your messages from normal topic traffic.
Filters are configured per topic because schemas and identifier locations may differ across a flow. Testing each one independently avoids an incorrect filter silently hiding an important stage when the topics are later combined chronologically.

Session filter

The Session filter groups related messages for live tracing or post-event tracing. A common example is following one correlation id across every topic in a View.
The Session filter is a conjunction (logical AND) with each hydrated topic filter. Both expressions must pass for a message to be shown. A topic without its own filter uses the Session filter by itself.
Not every topic necessarily stores the correlation id in the same place. Use header presence checks and @topic conditions to describe those differences safely.
(h.correlationId.isPresent() && h.correlationId == "1234")
|| (@topic == "pri-ord-order-changelog" && m.correlationId == "1234")
@topic is the Kafka topic currently being checked by the server. The topic-specific branch above can only pass for pri-ord-order-changelog.
The current Session filter is temporary and is not stored in the View definition. After a valid filter is used with Play or Live, it is added to the Session filter history; Cloud Sync makes that history available on your other computers so you can reuse the expression and change only values such as the correlation id.

Per-topic filters

Live Views support one saved filter per topic. Open Edit view and choose a filter from each topic’s Topic filter list.
Test and save the criteria from the topic results first. Views keep only the saved-filter reference, and resolve its latest criteria whenever Play or Live starts.
A mini filter icon appears next to topics that have a filter selected. Choose No filter to remove it.
Only messages that pass each topic’s resolved criteria appear in the combined results. To filter every topic, select a saved filter for every topic in the view.
View settings can only be changed while the Live View feed is inactive. Stop the feed before editing the view.
Aggregate criteria functions count(...) and countUnique(...) are not available in Live Views.
# Example: keep only errors on one topic
errorsTopic: m.level == 'ERROR'

# Example: track one customer across another topic
ordersTopic: k.customerId == '123'

Adding topics and creating Views

Topics can be added to or removed from a View only when the Live View is not active. If Live feed is running, stop it before editing the View.
Add one topic by dragging it onto an inactive View or directly onto its Topic Flow diagram. Dropping onto the diagram registers the topic in the View and places a topic node at the drop position.
To add the current filtered Topic List in bulk, right-click the list and choose Add visible topics to a new or existing view.... This action appears in its own separated context-menu section.
The picker can add to an existing View or create a new View from a name you enter. Existing destinations show their current, new, and resulting topic counts; duplicate topics are ignored.
Each View supports up to 30 topics. Bulk creation with 25 visible topics is valid, while an operation producing 31 topics is prevented with Cannot add more than 30 topics to a view.

Edit View: preferred topic order

Open Edit view to add, remove, or arrange topics in the wider editor. Position numbers show the saved preferred visual order.
Drag a topic by its grip and drop it on an insertion edge, including above the first topic or below the last topic. The red edge shows where it will be inserted. The up and down arrow buttons provide the same ordering control without dragging.
Preferred order is a visual aid only: it keeps topic markers, menus, and diagram presentation logically consistent, but message rows remain ordered chronologically by timestamp as data arrives.
An In diagram badge identifies topics already represented by a topic node in the Topic Flow diagram.
Choosing Remove opens a confirmation first. Confirming and then saving removes the topic from both the View and its flow diagram.

Build and edit the Topic Flow diagram

Use Topic Flow to document how topics, applications, and infrastructure relate. The sidebar keeps remaining View topics and alphabetically ordered Process nodes in separate scrollable areas, followed by an Annotations section. Drag the vertical splitter to resize the sidebar, then drag topics, processes, or text labels onto the canvas.
Join nodes by dragging from a source arrow handle to a target handle. The arrows document the direction in which records move. Topic nodes show live received/shown counters and expose their topic filter; process nodes can be resized and edited in place.
Process categories and the process-type dropdown are alphabetical. Alongside consumers, producers, Kafka Streams, connectors, databases, and general services, the palette includes Micro service, API Gateway, Schema Registry, Cache, Data Warehouse, Object Storage, Observability, Data Lake, and Third-party API.
Process nodes support title, description, environment, category, connector type, color, background shade, and visible-field controls. The color and visibility menu closes when you click or drag away, select another node, deselect the node, or lock the diagram.
Removing a topic node with its diagram trash button removes only that visual node. To remove topic membership from the Live View, use Edit view and confirm the removal there.

State stores, annotations, and flow arrows

Drag Text label from the sidebar’s Annotations section to place a free-position comment anywhere on the canvas. Edit its text in place, use the small grip to reposition it, and use its trash action to remove it. Text labels are documentary only and do not have connection handles.
To describe what an arrow carries or why the connection exists, select any standard or feedback arrow and type in the inline Arrow message input. Press Enter or click away to save; clear the input to remove the message. Saved text remains visible on the arrow when it is not selected.
Use the Store control on a topic node to mark it as a State store. Every arrow connected to that topic then shows an arrowhead at both ends to communicate push/write and pull/read activity with one uncluttered bidirectional line. This is useful for a Task state store or a compacted backing topic. Turning the option off restores ordinary one-way arrows.
Draw a connection against the current diagram orientation to create a feedback arrow: connect from a node on the right back to one on the left in horizontal mode, or from a lower node back to an upper node in vertical mode. Kafka Streamyard routes the loop outside the diagram with rounded corners, making retry and cyclic flows clear without duplicating the original service or topic node.
Text labels, arrow messages, state-store options, and feedback routing are saved with the View. Locking prevents accidental edits, while Undo and Redo include these diagram changes.

Select, move, and orient the diagram

Click a node to select it. To select any number of nodes, hold Shift and drag the mouse across empty canvas to draw a selection box around them. Drag any selected node to move the whole selected group together.
Use the orientation toggle in the bottom-right controls to choose horizontal or vertical arrow joins. Horizontal mode joins the right side of a source to the left side of a target; vertical mode joins the bottom of a source to the top of a target.
React Flow draws each edge between its source and target handles, so orientation controls where arrows leave and enter a node. There is no single correct direction: it is a presentation choice for making the topology easier to read and does not change the Kafka flow or the meaning of an edge.
Choose horizontal for a mostly sequential pipeline that reads naturally from left to right—for example producer → input topic → stream processor → output topic → consumer. It often makes best use of an ultrawide monitor and keeps a long chain on one visual line.
Choose vertical for a top-to-bottom overview, a narrower diagram pane, or a flow with several fan-out and fan-in branches. Stacking stages can make parallel consumers, retry paths, and downstream services easier to compare without spreading the chart too far across the screen.
Changing orientation moves the connection handles and redraws the arrows, but it does not automatically rearrange your nodes. Your manual node positions remain intact, so you can arrange the same flow from left to right or from top to bottom and switch back without losing the layout.

Bottom-right controls and overview map

The bottom-right control stack provides zoom in, zoom out, fit-to-view, diagram orientation, lock/unlock, Undo, and Redo. Undo and Redo cover diagram edits such as adding, removing, connecting, moving, resizing, changing orientation, editing text labels, and adding arrow messages.
Lock the diagram when you want to navigate or monitor without accidentally editing it. A View active in another window is also protected from editing.
The overview map below the controls shows the whole flow and your current viewport. Drag inside this map to pan around a large diagram and use its zoom interaction to move between an overview and a detailed area; use fit-to-view whenever you want every node back on screen.

4K View and Ultra Wide layouts

These tabs combine Topic Flow, the live chronological results, and message details in one workspace. They are designed for two monitor types: a conventional high-resolution 4K display and a wide ultrawide display. Choose the tab that matches your screen; you do not need both monitor types.
4K View stacks Topic Flow above the results area and keeps message details to the right of the results. It suits a tall 16:9 4K workspace where vertical pixels are available.
Ultra Wide places Topic Flow beside the results area and docks message details below the results. It suits 21:9 and 32:9 monitors where horizontal space is the main advantage.
Drag the divider between Topic Flow and the result/details workspace to resize the split; Kafka Streamyard remembers the split for that View and layout. A 4K-class display is recommended for Kafka Streamyard generally because its dense topic, result, and details panes benefit from the extra resolution.

Show since and memory usage

Use Show since to define the time window for messages shown in the View. It accepts relative values such as 15m, 6h, and 2d, as well as absolute ISO timestamps.
While Live feed is running, older rows outside that window are dropped from the in-memory View buffer so the grid focuses on recent activity.
On very busy topics, using too large a window may still lead to high memory use because many records can arrive in a short time. Prefer a smaller window for high-volume streams.

Hover preview

Hover over a View in the Views list to open a quick preview.
The preview shows the current status of the View, how many topics it contains, total received and shown counts, and a per-topic breakdown.
This is useful when you want to inspect a View without switching the main results area away from your current selection.

Details pane and layout

For Live Views, the preferred details-pane dock is bottom by default.
This works well because ResultView is usually wider in Live Views due to the included topic name column.
You can still dock the details pane to the right if that better suits your screen or workflow.

Column layout in Live Views

Live Views do not currently support the same column-layout customisation that is available for single-topic result views.
This is mainly because combining multiple topics into one chronological grid makes column selection and formatting more complex.
We may improve this in the future, for example by allowing a View to adopt a compatible layout from one of its topics.
Streamingstreaming.importExport

Import / Export (Message Playback & Live Controls)

importexportdownloadndjsoncsvjsonxmlreportresult controlmessage playbacklive controls
Import/Export lives in the Message Playback & Live Controls toolbar (download/upload icons).

Why this is useful

Import/Export helps you move between investigation, reporting, and replay without leaving the current workflow.
CSV reports are useful when you want to share selected columns with testers, analysts, or support teams who do not need the raw Kafka envelope.
JSON and NDJSON exports are useful when you want to archive a slice of data, replay it elsewhere, or use it as seed input for automated test flows.

Export

Export opens a preview modal where you can copy the data or download it as a file.
Formats:
JSON
Exports a JSON array. If you enable envelope options, each row is exported as an object like { messageValue, key, headers, partition } (some fields optional). XML payloads remain unchanged inside messageValue.
NDJSON
Newline-delimited JSON (one JSON value or envelope per line). Uses the same row structure as JSON export but without the outer array wrapper. Unwrapped XML is represented as a JSON string so multiline XML remains safe.
CSV report (selected view)
Exports a report using the currently selected message list view (the same columns you see in the grid). This is intended for spreadsheets and reporting (not re-import).

Import

Import reads messages from a file and produces them to Kafka for the currently selected topic.
You can import JSON arrays, JSON objects, NDJSON, or a raw XML document. A raw XML file becomes one Kafka message. For multiple XML messages, use the JSON/NDJSON export container so each original XML payload, key, headers, and optional partition can round-trip without modification.
You can also provide a separate key file (including a raw XML key), or generate keys from a template.
If the import file includes a partition field, Streamyard will validate whether those partitions exist on the target topic. If they do, it can preserve partitions; otherwise partitioning is automatic.
Streamingresult.sl

SL (Streamyard Language) reference guide

filtercriteriasloperatorsjsonxmlnamespaceattributesheadersheader countheaders existautocompletehistoryaicontainsmultiple substringsage betweenstatus choicesbetweenarray filtersame elementfindflattenexistsall@.deep telemetryhistogramdataPointsbyte.ratecontainsAllcountWherenoneisBlankcoalescehasKeygetsumavgminmaxlikereplacesplitcsv countparse integerparse decimalnumeric textcompound idnegative indexfirst array itemfirst functionlast array itemlast functionend elementnumeric segment@topictopic metadata
SL (Streamyard Language) uses the same compact expression syntax for JSON and XML. Payload is available as m, key as k, and headers as h. XML paths omit namespace prefixes, attributes are under @attributes, and mixed text is under @text. Use h.count() to filter on header presence or the number of header keys.Methods must include parentheses, e.g. toLower() (not toLower).

XML filtering essentials

XML keeps its root element in the SL path. Namespace prefixes are omitted because prefixes may change between producers: <ns:PersonEvent> is addressed as m.PersonEvent.
Leaf element content behaves as text, so ordinary string methods work directly. For example: m.PersonEvent.Person.Email.endsWith('@example.com').
Attributes are grouped under @attributes. For example: m.PersonEvent.@attributes.eventType == 'person.created'. Direct mixed text on an element containing child elements is available through @text.
Repeated XML siblings are stable collections. Direct paths and string comparisons work without adding [*] for singleton elements. Use count(), exists(...), find(...), or all(...) when collection behavior matters.
Use exists(...) with @ when several conditions must match the same repeated element. XML numeric text can be compared numerically or converted explicitly with parseInteger() or parseDecimal().
Do not write XPath. Do not include namespace prefixes, text(), //, or XPath attribute syntax in SL. Malformed XML and unsafe DTD/entity payloads fail closed instead of being treated as structured content.

Why teams use SL

SL is useful because it lets you isolate exactly the messages you care about without writing code or reconfiguring consumers.
In the UI, autocomplete helps you discover fields and headers quickly, filter history helps you bring back useful queries, and optional AI assistance can help draft a starting expression that you then refine.
Typical uses include isolating one customer, finding only failed events, checking that a new field is present, or narrowing a noisy topic down to one business case.
AI builder prompt examples
When asking AI, say what field you mean and what kind of match you want: exact choice values, contains-style text matching, or a numeric range. For inclusive ranges, prefer asking for between-style logic.
Only customers aged between 30 and 40 inclusive
m.age.between(30, 40)
Only ticket statuses OPEN, PENDING, or CLOSED
m.ticketStatus in ('OPEN', 'PENDING', 'CLOSED')
Hobbies contain cooking or cycling, ignoring case
m.hobbies.toLower().contains('cooking', 'cycling')
Combined example: age between 30 and 50, hobbies mention music or gaming, and country is France or Germany
m.person.identity.age.between(30, 50) and m.person.hobbies.toLower().contains('music', 'gaming') and m.person.address.country in ('France', 'Germany')
Included by the filter
{
  "person": {
    "hobbies": "music, gaming, cycling, reading",
    "address": {
      "country": "France"
    },
    "identity": {
      "age": "35"
    }
  }
}
Age is inside the range, hobbies contain music/gaming, and country is one of the allowed values.
Excluded by the filter
{
  "person": {
    "hobbies": "cooking, hiking, reading",
    "address": {
      "country": "Spain"
    },
    "identity": {
      "age": "27"
    }
  }
}
This one fails all three checks: age is outside the range, hobbies do not mention music/gaming, and country is not France or Germany.
Operators & syntax
==
Equality
m.country == 'Brazil'
!=
Inequality
m.country != 'Brazil'
=
Equality (single = is normalized to ==)
m.country = 'Brazil'
>
Comparison
m.age > 18
>=
Comparison
m.age >= 18
<
Comparison
m.age < 18
<=
Comparison
m.age <= 18
String comparisons use lexicographic ordering
m.someDate > '20250101T1020Z'
and / &&
Boolean logic
@partition == 2 and @offset > 10
@partition == 2 && @offset > 10
or / ||
Boolean logic
m.country == 'BR' or m.country == 'US'
m.country == 'BR' || m.country == 'US'
!
Boolean negation
!m.error.contains('ignored')
!m.status.toLower().in('done', 'cancelled')
in (...) / .in(...)
Exact membership. The method form is convenient after another method call.
m.country in ('BR', 'US', 'ZA')
m.person.forename.toLower().in('john', 'thomas')
.
Field access
m.project.detail.id == 'FR'
[n] / [-n]
Array index. Zero selects the first item; negative indexes count from the end, so [-1] is the last item and [-2] is the second-last. first() and last() are the readable method equivalents of [0] and [-1].
m.orders[0].price > 100
m.orders[-1].price > 100
[*]
Array wildcard. Selects every array element and applies the following field access or method chain to each value.
m.orders[*].price > 100
m.lines[*].description.toLower().contains('training')
find(predicate)
Returns the first matching array item so you can continue to one of its fields. Returns missing when no item matches.
m.order.lines.find(@.sku == 'SKU-3003').description
flatten()
Flattens one nested array level after navigating through a parent array.
m.resourceMetrics.resource.attributes.flatten().find(@.key == 'service.name').value.stringValue
exists(predicate)
Boolean array check. Inside the predicate, @ is the current array element.
m.characteristics.exists(@.name == 'color' and @.value == 'red')
m.lines.exists(@.sku == 'RED' and @.tags.exists(@.code == 'PRIMARY'))
all(predicate)
Boolean array check that requires every array element to match.
m.orders.all(@.price > 100)
[?(...)]
Array element filter. Use this when you need the filtered array itself so you can keep navigating or count matches.
m.characteristics[?(@.name == 'color' and @.value == 'red')].count() > 0
m.lines[?(@.sku == 'RED' and @.tags[?(@.code == 'PRIMARY')].count() > 0)].count() > 0
[start:length]
Slice (start index + length)
m.name[1:3] == 'att'
Complex examples
m.currency == "EUR" && m.lines[*].description.toLower().contains("training")
m.characteristics.exists(@.name == 'color' and @.value == 'red')
m.orders.all(@.price > 100)
@partition == 0 && (k.customerId.isPresent() || h.correlationId.isPresent())
Methods
isPresent()
Missing field checks
m.optionalField.isPresent()
isAbsent()
Missing field checks
m.optionalField.isAbsent()
between(min, max)
Inclusive numeric range check. Bounds may be supplied in either order; an array matches when any numeric element is in range.
m.order.orderTotal.between(100.00, 999.99)
m.person.identity.age.between(31, 49)
m.measurements.between(100, 300)
isEmpty()
Empty string/list checks
m.name.trim().isEmpty()
isNullOrEmpty()
Empty string/list checks (also true for null/missing)
m.optionalName.isNullOrEmpty()
isBlank()
True for an empty or whitespace-only string
m.note.isBlank()
isNullOrBlank()
Also true for null or missing strings
m.note.isNullOrBlank()
type()
Type check: returns string, number, boolean, object, array, null, missing, unknown
m.person.forename.type() == 'string'
toString()
Convert to string
h.someHeader.toString() == '123'
toJson()
Parse escaped embedded JSON
h.someHeader.toJson().id == '123'
toLower()
String normalization helpers
m.name.trim().toLower() == 'matthew'
toUpper()
String normalization helpers
m.name.trim().toUpper() == 'MATTHEW'
trim()
String normalization helpers
m.name.trim().toLower() == 'matthew'
parseInteger()
Converts signed whole-number text to a number. Leading zeroes are allowed; decimals or other invalid text return missing. After [*], conversion and between() apply across all positions and match when any parsed number is in range.
m.code.split('-')[2].parseInteger() == 1
m.customerId.split('-')[0].parseInteger().between(700000, 799999)
m.customerId.split('-').first().parseInteger().between(700000, 799999)
m.customerId.split('-')[-1].parseInteger().between(700000, 799999)
m.customerId.split('-').last().parseInteger().between(700000, 799999)
m.customerId.split('-')[*].parseInteger().between(700000, 799999)
parseDecimal()
Converts signed decimal text to a number. Invalid numeric text returns missing.
m.amount.substring(4).parseDecimal() >= 42.75
contains(substr1[, substr2, ...])
String match helpers. Returns true when any supplied substring matches.
m.name.contains('att')
m.person.hobbies.toLower().contains('gaming', 'cooking')
m.lines[*].description.toLower().contains('training')
containsAll(substr1, substr2, ...)
True only when every supplied substring occurs. On string arrays, matches may occur across different elements.
m.summary.toLower().containsAll('payment', 'timeout')
startsWith(prefix1[, prefix2, ...])
True when any supplied prefix matches
m.code.startsWith('ERR_', 'WARN_')
endsWith(suffix1[, suffix2, ...])
True when any supplied suffix matches
m.email.endsWith('@example.com', '@example.net')
like(pattern)
Safe wildcard matching: * matches any text and ? matches one character
m.code.like('WARN_*')
replace(old, new)
Literal string replacement for normalization
m.formattedId.replace('-', '') == 'AB1234'
substring(start, len) / substr(start, len)
Substring with optional length; substr is an alias
m.code.substring(1, 3) == 'BCD'
now()
Current epoch time
@timestamp >= now(-1, week)
toMillis(text)
Parse a UTC datetime string to epoch milliseconds
m.createdMillis > toMillis('20250101T102030123Z')
m.createdMillis > toMillis('2025-01-01T10:20:30.123Z')
toSeconds(text)
Parse a UTC datetime string to epoch seconds
m.createdSeconds > toSeconds('20250101T102030Z')
toDays(text)
Parse a UTC date/datetime string to epoch days
m.epochDays >= toDays('2025-01-01')
toMicros(text)
Parse a UTC datetime string to epoch microseconds
m.ts_us >= toMicros('2025-01-01T10:20:30.123Z')
toNanos(text)
Parse a UTC datetime string to epoch nanoseconds
m.ts_ns >= toNanos('2025-01-01T10:20:30.123Z')
now(-1h)
Current epoch time with relative offset
@timestamp >= now(-1h)
now(-1, day)
Current epoch time with relative offset
@timestamp >= now(-1, day)
count()
List helpers; on h, returns the number of header keys
m.tags.count() > 0
h.count() > 0
h.count() > 4
first()
Returns the first array element so method or field chaining can continue. Equivalent to [0]; an empty array returns missing.
m.orders.first().status == 'PENDING'
m.customerId.split('-').first().parseInteger()
last()
Returns the final array element so method or field chaining can continue. Equivalent to [-1]; an empty array returns missing.
m.orders.last().status == 'COMPLETE'
m.customerId.split('-').last().parseInteger()
split(delimiter)[index]
Splits a string on a literal delimiter and returns an array without regular expressions. Use first() or [0] for the first element and last() or [-1] for the last. Prefer first() or last() when they best express the intent. [*] selects every split element. With [*], following methods run for each element; between() is true if any parsed number is in range.
m.person.hobbies.split(',').count() > 3
m.customerId.split('-')[0].parseInteger().between(700000, 799999)
m.customerId.split('-').first().parseInteger().between(700000, 799999)
m.customerId.split('-')[-1].parseInteger().between(700000, 799999)
m.customerId.split('-').last().parseInteger().between(700000, 799999)
m.customerId.split('-')[*].parseInteger().between(700000, 799999)
length()
Number of characters in a string or elements in an array
m.customerName.length() > 3
size()
Array-size alias; prefer count() in new filters
m.tags.size() > 0
find(predicate)
Returns the first matching array item so you can continue to one of its fields. Returns missing when no item matches.
m.order.lines.find(@.sku == 'SKU-3003').description
flatten()
Flattens one nested array level after navigating through a parent array.
m.resourceMetrics.resource.attributes.flatten().find(@.key == 'service.name').value.stringValue
exists(predicate)
Array boolean helper. Returns true when at least one array element matches the predicate.
m.characteristics.exists(@.name == 'color' and @.value == 'red')
m.lines.exists(@.sku == 'RED' and @.tags.exists(@.code == 'PRIMARY'))
all(predicate)
Array boolean helper. Returns true when every array element matches the predicate.
m.orders.all(@.price > 100)
none(predicate)
True when no array element matches; also true for an empty array
m.orders.none(@.status == 'CANCELLED')
countWhere(predicate)
Counts matching array elements without constructing a filtered array
m.orders.countWhere(@.status == 'FAILED') >= 2
join(delim)
List helpers
m.tags.join(',') == 'A,B'
sum() / avg() / min() / max()
Numeric array aggregates; null, missing, and non-numeric elements are ignored
m.measurements.avg().between(100, 300)
get(key)
Reads an exact object or header key, including keys containing dots or slashes
m.labels.get('app.kubernetes.io/name') == 'payments'
hasKey(key)
Checks whether an object or header contains an exact key
h.hasKey('trace.parent')
coalesce(value1, value2, ...)
Returns the first value that is neither missing nor null
coalesce(m.customer.id, m.customerId, k.customerId) == 'C-42'
count(expr) / countUnique(expr)
Top-level result aggregations. countUnique accepts a value expression, not a comparison.
count(m.status == 'FAILED')
countUnique(m.customerId)
Convenience methods sample
This anonymized event can be used to adapt the matching, normalization, array, literal-key, fallback, and numeric aggregate examples above.
{
  "code": "WARN_TIMEOUT",
  "summary": "Payment timeout after retry",
  "email": "alerts@example.net",
  "note": "   ",
  "formattedId": "AB-1234",
  "customerId": "C-42",
  "labels": {
    "app.kubernetes.io/name": "payments"
  },
  "orders": [
    { "status": "FAILED", "total": 120 },
    { "status": "READY", "total": 240 },
    { "status": "FAILED", "total": 360 }
  ],
  "measurements": [120, 240, 360]
}
Unlike regular expressions, like(...) only supports the safe * and ? wildcards and uses linear-time matching.
Array element filters
Use exists(...) when multiple checks must apply to the same object inside an array and you only need a boolean answer.
For example, m.characteristics[*].name == 'color' and m.characteristics[*].value == 'red' can match different array elements. Prefer m.characteristics.exists(@.name == 'color' and @.value == 'red') when both checks must be true on one element.
Use all(...) when every array element must pass, e.g. m.orders.all(@.price > 100).
Use none(...) when no element may match and countWhere(...) when the number of matching elements matters.
You can nest the same pattern for arrays inside arrays, e.g. m.lines.exists(@.sku == 'RED' and @.tags.exists(@.code == 'PRIMARY')).
Keep [?(...)] for cases where you need the filtered array itself so you can continue navigating or count the matches, e.g. m.characteristics[?(@.name == 'color' and @.value == 'red')].count() > 0.
Deep telemetry arrays
SL can query JSON at any depth. For OpenTelemetry-style payloads, keep nesting exists(...) as you move through resourceMetrics, scopeMetrics, metrics, dataPoints, and inner attributes arrays.
Each exists(...) moves into one array level. Put related checks in the same predicate when they must match within the same parent element.
For payloads this nested, it is usually easier to ask AI assist to draft the filter first, then refine the generated SL instead of typing the whole path by hand.
{
  "resourceMetrics": [
    {
      "resource": {
        "attributes": [
          { "key": "service.name", "value": { "stringValue": "telemetry-agent" } },
          { "key": "network.test.target", "value": { "stringValue": "probe.example.net" } }
        ]
      },
      "scopeMetrics": [
        {
          "scope": { "name": "network-monitoring" },
          "metrics": [
            {
              "name": "network.jitter",
              "histogram": {
                "dataPoints": [
                  {
                    "attributes": [
                      { "key": "network.type", "value": { "stringValue": "IPV6" } },
                      { "key": "byte.rate", "value": { "longValue": 2448 } }
                    ]
                  },
                  {
                    "attributes": [
                      { "key": "network.type", "value": { "stringValue": "IPV4" } },
                      { "key": "byte.rate", "value": { "longValue": 9579 } }
                    ]
                  }
                ]
              }
            }
          ]
        }
      ]
    }
  ]
}
Find an IPV4 histogram data point whose byte rate is between 8192 and 16384, inclusive:
m.resourceMetrics
  .exists(
    @.scopeMetrics
      .exists(
        @.metrics
          .exists(
            @.histogram.dataPoints
              .exists(
                @.attributes
                  .exists(
                    @.key == 'network.type' and
                    @.value.stringValue == 'IPV4'
                  ) and
                @.attributes
                  .exists(
                    @.key == 'byte.rate' and
                    @.value.longValue.between(8192, 16384)
                  )
              )
          )
      )
  )
The second histogram data point matches because both required attributes occur in that same data point: network.type is IPV4, and byte.rate is 9579. The first data point is excluded.
Common fields
m / message
Message payload (structured object for JSON/XML topics; scalar or raw string otherwise)
k / key
Key payload (structured object for JSON/XML keys; scalar or raw string otherwise)
h / headers
Header map by key (first header value wins); h.count() returns the number of header keys
@topic
Kafka topic currently being evaluated
@timestamp / @ts / timestamp
Record timestamp (epoch ms)
@partition / @part / partition
Partition id
@offset / offset
Record offset
Streamingresult.offsetFunction

Offset Function help

functiontailheadsinceabsutctodayyesterdayoffset
This function determines where the stream starts reading from. Some functions are offset-based (tail/head/offset=...) and some are time-based (since/range/today/yesterday). Direct offset functions (offset=... and o=...) can only operate on a single selected partition.
Examples
FunctionMeaningOperates On
tail=2000Tail: latest 2000 records per selected partition (offset-based)All Partition
tail=1000..2000Tail range: latest-2000 to latest-1000 per selected partitionAll Partition
head=100From beginning: first 100 records per selected partitionAll Partition
head=10..20Head range: offsets 10 to 20 from the earliest offset per selected partitionAll Partition
beginningScan forward from the earliest available offsets across all selected partitionsAll Partition
0Alias for beginning across all selected partitions (auto-normalizes on field exit)All Partition
since=24hTime range: from now - 24h to nowAll Partition
since=48h..24hTime range: start = now - 48h, end = now - 24hAll Partition
abs=250220..250224Absolute time range: Feb 20 to Feb 24 (local time, midnight to midnight)All Partition
abs=250220T10:20..250224T02:00Absolute time range: local time with minutesAll Partition
utc=250220..250224Absolute time range: UTC midnight to midnightAll Partition
today-utcTime range: start of today in UTC, end = nowAll Partition
yesterdayTime range: previous dayAll Partition
offset=1000..Offset start: from offset 1000, then read up to max items (requires 1 partition)1 Selected Partition
offset=1000Alias for offset=1000.. (single offset start)1 Selected Partition
o=1000Alias for offset=10001 Selected Partition
s=24hAlias for since=24hAll Partition
t=-24hAlias for since=24hAll Partition
Topicstopics.filter

Topic list and filter help

topictopic listfilterfilter topicssaved filters menudropdownreload topicsrefresh topicsmark all topics as seennew topicnew badgeshow record countshow data sizesort topicsbuildertabsAI topic filter assistantFilter samplespresets@wildcards*?single characterandornotinternal__connectorskafka connectmirrormakerparenthesesgrouping&&||aliasesmetadata@@partitions@@prt@@cleanupPolicy@@cp@@cleanupPolicy=both@@size@@size>0b@@size>=10mb@@size>1mb && @@size<8mb@@ignoreInternals@@ii@@ignoreConnectors@@ikc@@ignoreConfluent@@ic_confluent-*confluentinexamplessamplesrecipespartitionscleanupcompactdeletebothcompact,deletesizebyteskbmbgbtbboolean

Topic list controls

Filter topics
Filters the current connection immediately as you type. The footer shows how many topics match out of the total loaded for that connection.
Clear filter
Use the X button or choose Clear filter from the Saved filters menu to show all loaded topics again.
Reload topics
Use the reload button to fetch the latest topics for the active connected environment. The same action is available as Reload topics in the Saved filters menu and Refresh topics in the topic context menu. A blue spinning refresh icon is shown while the request runs, followed briefly by a green check on success or a red X on failure. Reload errors also appear in a notification.
New topics
Topics discovered after connecting are marked new. Selecting one topic marks that topic as seen. Use Mark all topics as seen to remove every new badge from the active connection tab without deleting topics or messages.
Sorting
Select the Topic, Partitions, Records, or Size column heading to sort. Select the same heading again to reverse the direction.
Metric columns
Record count and data size columns can be shown independently. Drag their shared divider to resize the metric area.
Topic actions
Right-click a topic row for create, copy, import, export, repartition, empty, delete, playbook, and other topic-specific actions.

Saved filters menu

Topic filter builder…
Open the visual filter editor with the current filter already loaded.
Clear filter
Remove the current filter expression.
Reload topics
Fetch the latest topic list for the connected environment. This uses the same reload action and status indicator as the button beside the filter box.
Mark all topics as seen
Clear all new badges for the active connection tab, similar to marking all messages as read.
Show record count
Show or hide the Records column.
Show data size
Show or hide the retained Size column.
Ignore internals
Silently hide Kafka internal topics in addition to the visible filter expression. This preference is equivalent to adding @@ignoreInternals or @@ii.
Ignore Confluent
Silently hide topics beginning with _confluent- in addition to the visible filter expression. This preference is equivalent to adding @@ignoreConfluent or @@ic.
Help…
Open this topic-list and filtering guide.
Saved presets
Select a preset to apply it. Use the pencil to modify it or the trash button to delete it. Use Arrow Up/Down, Home/End, and Enter or Tab to navigate and activate menu entries from the keyboard.
Topic filters use a boolean expression language. Filters are case-insensitive. - has no special meaning. Allowed: letters/digits, ., _, -, spaces, grouping with (), list commas inside in(...), operators &&/and, ||/or, !/not, wildcards */?, preset @, metadata @@, and comparisons =, ==, !=, >, >=, <, <=.
The topic filter matches topic names and selected topic metadata such as partitions, cleanup policy, retained size in bytes, and built-in ignore predicates. The old topic-filter rules based on space/comma for OR and + for AND should be considered retired for topic filtering.

How to think about the language

Build filters in layers: start with a term, combine terms with and / or, remove unwanted matches with not, and add metadata checks with @@... when needed.
Think of it like plain-English logic: orders and not retry, (crm or cust) and @@partitions>=3, @@ii and @@size>0b, @billing and not dlq.
Operator precedence is not first, then and, then or. When a filter mixes AND and OR, use parentheses so the intention is obvious.

Saving

Open Topic filter builder… from the Saved filters menu. The current topic-list filter is loaded into the editor automatically, where you can refine and save it as a preset.

Topic Filter Builder tabs

AI topic filter assistant
Describe the topics you want in plain language. The assistant uses the visible topics, filter syntax, metadata rules, and saved presets to generate a filter in the editor.
Filter samples
Browse grouped, ready-to-adapt examples for wildcards, AND/OR/NOT, nesting, in(...), saved presets, metadata aliases, and complete recipes. Each expression is shown beside an explanation of what it does.

Using presets in the filter

You can reference a saved preset by name using @presetName. Type @ to open autocomplete, then press Enter or Tab to insert.
Presets can be combined with other terms and operators (e.g. @orders and pri) and can be negated (e.g. not @billing or !@billing).
Good preset examples: @core_finance and @@cleanupPolicy=compact, @payments and not in(dlq,retry,error), (@orders or @billing) and @@partitions>2, @billing and @@size>10mb.

Internal topics

The simplest way to hide Kafka internal topics is @@ignoreInternals. It keeps normal topics and filters out internal ones such as __consumer_offsets and __transaction_state.
Name-based filters still work too. Kafka internal topics commonly start with __, so not __* or !__* can also hide them.
To show only internal topics, use __*. Example: __* and not __transaction_state keeps only internal topics but removes one noisy internal topic.
For common Kafka Connect and MirrorMaker system topics, add @@ignoreConnectors or its short alias @@ikc. This is intentionally conservative and targets known connector system-topic naming patterns rather than guessing arbitrary business topics produced by connectors.
To hide Confluent-created topics whose names begin with _confluent-, use @@ignoreConfluent or its short alias @@ic. It is equivalent to the name pattern !_confluent-*.

Basic matching

A plain term matches anywhere in the topic name. orders matches pri-ord-orders, team-orders-live, and orders-dlq.
Terms are not quoted strings and do not need regex syntax. Just type the meaningful fragments that appear in your topic names.
orders
pub-fin
crm
__*

Boolean building blocks

OR
billing or payments or billing || payments matches either alternative.
AND
pri and orders or pri && orders matches only when both terms are present.
NOT
not invoices or !invoices excludes matching topics.
Grouping with parentheses
Use parentheses whenever a filter mixes AND and OR. Example: (crm or cust) and not retry.
Operator precedence
The parser evaluates not first, then and, then or. Even though that works, explicit parentheses are still easier to read and safer to maintain.
Examples
orders and not retry
billing or payments
(crm or cust) and not (dlq or retry)
((crm and cust) or (pub-fin* and payments)) and @@partitions=3
Important: standalone and, or, and not are operators.

Wildcards

* matches any number of characters, including none. ? matches exactly one character, so ???? matches exactly four characters.
Wildcards work anywhere inside terms. For example, pri-ord-*-create requires both the pri-ord- prefix and -create suffix, with any text between them—the same intent as pri-ord-* and *-create.
Use prefix* for prefix-style matching, *middle* for contains-style matching, *suffix for suffix-style matching, and patterns such as order-???? when an exact character count matters.
If you want to match the literal words and, or, or not in topic names, use wildcard terms such as *and*, *or*, or *not*.
orders*
pri-ord-*-create
order-????
*dlq*
*retry
*or* and *and*

Compact lists with in(...)

in(a,b,c) is shorthand for (a or b or c). It is especially useful when excluding several similar topic fragments.
Wildcards are allowed inside the list, so in(ledger,cashflow,*acct*) is valid.
in(crm,cust,payments)
not in(dlq,retry,error)
-fin- and not in(ledger,cashflow,*acct*)
(inv or payments) and not in(calc,line,test)

Metadata filters

Use metadata variables when you want to constrain by topic properties, not just the name.
Comparison metadata variables are @@partitions, @@cleanupPolicy, and @@size. Standalone boolean predicates are @@ignoreInternals, @@ignoreConnectors, and @@ignoreConfluent.
Short aliases are @@prt for @@partitions, @@cp for @@cleanupPolicy, @@ii for @@ignoreInternals, @@ikc for @@ignoreConnectors, and @@ic for @@ignoreConfluent.
Supported comparisons are =, ==, !=, >, >=, <, and <=.
Cleanup policy values are delete for delete-only topics, compact for compact-only topics, and both for topics configured with both compact and delete cleanup. @@cleanupPolicy=both matches the Kafka values compact,delete and delete,compact.
@@size compares retained topic size in bytes. Bare numbers default to bytes, and explicit units such as b, kb, mb, gb, tb, and pb are supported.
@@recs is intentionally not available yet. Unlike @@size, retained record counts would need per-topic background requests, results would settle progressively instead of appearing in one batch, and compacted topics would still only produce approximate counts rather than exact retained-record totals.
@@partitions=3
@@prt > 1
@@cleanupPolicy==compact
@@cleanupPolicy=both
@@cp=both
@@size>0
@@size>0b
@@size>=10mb
@@size>1mb && @@size<8mb
@@ignoreInternals
@@ignoreConnectors
@@ikc
@@ignoreConfluent
@@ic
@@ii and @@ikc and @@ic
orders and @@cleanupPolicy=delete
@billing and @@ignoreInternals and @@partitions>=3

Common mistakes to avoid

Do not use legacy + for AND. Write and or && instead.
Do not use square brackets [] for grouping. Use parentheses ().
Do not use commas as general OR separators outside in(...).
Do not use quotes, regex syntax, SQL fragments, or CEL-style expressions. Keep filters as simple topic-filter expressions.
Use this: orders and not retry
Not this: orders + !retry
Use this: (crm or cust) and not dlq
Not this: [crm cust] !dlq

Filter samples

Learn the filter language from complete examples. Expressions are on the left; their effect is described on the right.
Topic names and wildcards
orders
Matches any topic name containing “orders”. Matching is case-insensitive.
orders*
Matches topic names that start with “orders”.
*orders
Matches topic names that end with “orders”.
*orders*
Same effect as “orders”: matches any topic name containing “orders”, but makes the contains wildcard explicit.
pri-ord-*-create
Matches names starting with “pri-ord-” and ending with “-create”, with any number of characters between them.
order-????
Matches “order-” followed by exactly four characters.
*
Matches every topic.
AND, OR, and NOT
orders and prod
Matches topics whose names contain both “orders” and “prod”.
orders && prod
Symbol form of AND; equivalent to “orders and prod”.
orders or payments
Matches topics containing either “orders” or “payments”.
orders || payments
Symbol form of OR; equivalent to “orders or payments”.
orders AND (payments OR billing) AND NOT retry
AND, OR, and NOT are case-insensitive. Capitalizing them is optional, but makes the filter logic easier to scan.
not retry
Excludes topic names containing “retry”.
!retry
Symbol form of NOT; equivalent to “not retry”.
orders and not (retry or dlq)
Keeps order topics while excluding retry and dead-letter queue topics.
Grouping and nesting
(orders or payments) and prod
Requires “prod” and at least one of “orders” or “payments”.
(prod or stage) and not (retry or dlq)
Selects production or staging topics, then removes retry and DLQ topics.
((orders or payments) and (prod or stage)) and not (retry or dlq)
A nested filter combining two inclusion groups with a grouped exclusion.
Compact lists with in(...)
in(orders,payments,billing)
Shorthand for “orders or payments or billing”.
in(ord*,pay*,*invoice*)
An OR list whose entries can each use wildcards.
prod and in(orders,payments) and not dlq
Requires “prod”, accepts either business area, and excludes DLQ topics.
Saved preset references
@billing
Uses the saved topic-filter preset named “billing”.
@billing and not retry
Reuses a saved preset and adds a local exclusion.
(@billing or @orders) and @@ignoreInternals
Combines saved presets and removes Kafka internal topics.
Topic metadata
@@partitions=3
Matches topics with exactly three partitions.
@@partitions>=3
Matches topics with three or more partitions.
@@cleanupPolicy=compact
Matches topics whose cleanup policy is compact only.
@@cleanupPolicy=both
Matches topics configured with both compact and delete cleanup policies.
@@cp=both
Short alias form of “@@cleanupPolicy=both”.
@@size>0
Matches topics with more than 0 bytes. Bare numbers default to bytes.
@@size>0b
Equivalent explicit-byte form of “@@size>0”.
@@size>=10mb
Matches topics whose retained size is at least 10 MB.
@@size>1mb && @@size<8mb
Matches topics whose retained size falls between 1 MB and 8 MB.
@@cleanupPolicy!=delete
Excludes topics whose cleanup policy is delete only.
@@ignoreInternals
Keeps non-internal topics and removes Kafka internal topics.
@@ignoreConnectors
Removes recognized Kafka Connect and MirrorMaker system topics.
@@ikc
Short alias form of “@@ignoreConnectors”.
@@ignoreConfluent
Removes topics whose names start with “_confluent-”.
@@ii and @@ic
Short alias form of “@@ignoreInternals and @@ignoreConfluent”.
@@prt>=3
Short alias form of “@@partitions>=3”.
@@ignoreInternals and @@ignoreConnectors and @@partitions>=3
Keeps larger user topics while excluding internal and recognized connector topics.
Literal operator words
*or*
Matches the literal text “or” inside a topic name instead of using OR.
*and*
Matches the literal text “and” inside a topic name instead of using AND.
*not*
Matches the literal text “not” inside a topic name instead of using NOT.
*or* and *and*
Requires both literal words to appear in the topic name.
Combined recipes
(prod and in(orders,payments)) and not (retry or dlq)
Production order or payment topics without retry or DLQ variants.
(@billing or finance) and @@cleanupPolicy=compact
Billing-preset or finance topics that use compact cleanup.
@@ii && @@size>0b
Keeps non-internal topics that currently retain any data.
@@ignoreInternals and @@ignoreConnectors and (auth or user)
User-facing auth topics without Kafka internal or recognized connector topics.
Topicstopics.contextMenu

Topic list context menu

topic listcontext menuright clickempty topicdelete topiccopy topic datapaste topic dataplaybook builderwizard
Right-click a topic row (or the empty area below the list) to open the context menu.

Common actions

Refresh topics
Reload the topic list for the current connection.
Create topic…
Create a new topic (opens the Create Topic modal).
Create based on…
When invoked on a topic row, opens the create dialog using that topic as a template.
Playbook builder wizard…
Open the wizard to generate a playbook based on selected topic(s) and common operations.

Create Topic modal notes

Create topic… starts from defaults; Create based on… pre-fills partitions, replication factor, and configs from the source topic.
Replication factor is validated against the current broker count (if available) and is capped/validated to 1..brokerCount.
Topic configs can be edited either as rows or as JSON. In JSON mode, values are normalized to strings where possible and *.msvalues accept convenience durations.
cleanup.policy is normalized and must be one of delete, compact, or compact,delete.

Live feed action

Stop live feed appears only for topics that are currently in Live mode.

Copy / paste topic data

Copy topic data stores a source reference (connection + topic).
Paste topic data into existing topic copies from the last copied topic into the selected destination topic. It is disabled if nothing has been copied yet, or if you try to paste into the same topic.
If you right-click the empty area (not a specific topic), you can use Create new topic and paste data… to create a destination topic and paste into it.

Danger zone

Empty topic
Clear topic data (confirmation required). This may be disabled by settings.
Delete topic
Delete the topic (confirmation required). This may be disabled by settings.

Copy helpers

Copy topic name
Copy the selected topic name.
Copy topic name & config (JSON)
Copy a JSON object containing the topic name and known configs.
Copy all visible topic names
Copy the currently visible topic names (after topic list filtering), one per line.
Copy all visible topic names & config (JSON)
Copy a JSON array for all currently visible topics.
Topicstopics.acls

Kafka ACLs (Access Control Lists)

aclaclssecurityauthorizationprincipalresourceoperationallowdenyprefixedliteral
Kafka ACLs are server-side authorization rules. They determine whether a principal (user/service) is allowed to perform an operation on a resource (topic, group, cluster, etc.).

How to read an ACL row

Principal
The identity (often User:alice or a service account) that the rule applies to.
Resource
What is being protected (Topic / Group / Cluster / TransactionalId). Topics and groups are the most common.
Name + Pattern
Resource name with a pattern type. Literal matches exactly; Prefixed matches any name that starts with the prefix.
Host
Client host/IP constraint (* means any host).
Operation
The action (Read / Write / Describe / Create / Alter / Delete / All, etc.).
Permission
Allow or Deny. If any matching Deny applies, the request is denied. If no matching Allow exists, the request is denied.

Using the ACLs pane

The ACLs pane lists ACLs for the active connection and supports filtering by principal/resource/operation.
+ Create ACL opens a form to add an ACL entry. Creating ACLs usually requires admin-level privileges on the cluster.
Topicstopics.kafkaConnectors

Kafka Connectors (Kafka Connect REST)

kafka connectconnectorsconnectorrestpauseresumerestartdeletecreatetemplatevalidate
The Kafka Connectors pane is a UI for listing and managing Kafka Connect connectors via the Kafka Connect REST API.

Prerequisite: Connect URL

Configure the Connect REST base URL in Connection settings (Kafka Connect tab). Typically this is port 8083.
If your Connect endpoint is protected, set the optional API key/secret (HTTP Basic auth).

Reading the table

Status is shown for the connector and its tasks (running/paused/failed). Failed tasks are the most common indicator that the connector is unhealthy.
Use the search box to filter by name, type, plugin class, or status.

Common actions

Pause / Resume
Stop/start work without deleting the connector.
Stop
Stop a connector (if supported by the Connect API / configuration).
Restart
Restart connector and tasks (useful after fixing a config or external dependency).
Delete
Remove the connector from Kafka Connect.

Creating connectors (Create modal)

The Create modal lets you pick a connector plugin and provide Config JSON. The selected plugin supplies connector.class automatically.
Use load template to populate a starting config (when available), and validate to run server-side validation before creating.
Validation may report errors but still allow creation; Kafka Connect may still reject the final request depending on plugin requirements.

Common errors

401/403: Connect REST authentication is wrong or missing.
Connection refused / timeout: Connect URL is wrong, unreachable, or blocked by network rules.
Validation errors: required fields for the selected plugin are missing or have invalid values.
Topicstopics.topicSettings

Topic Settings (per-topic overrides)

topic settingsguardrailspermissionsallow deleteallow emptyallow sendsamplespersistconfirm before sendpartitioningformatschema
Topic Settings lets you override connection defaults for a single topic. This is useful for applying stricter guardrails or different performance tuning on one topic without changing the entire connection.

Guardrails (destructive actions)

Settings like Delete topic, Empty topic, and Send messages can be overridden per topic.
Most toggles support inherit so the topic follows the connection defaults.

Consumer profiles & tuning

Consumer tuning values (poll time, max poll records, and timeouts) can be overridden per topic.
Larger profiles improve throughput but can reduce UI responsiveness and increase memory use during Live feeds.

Partitioning & formats

Topic Settings can control partition calculation/partitioner behavior, plus Key/Message formats. This affects how messages are displayed and how produced records choose partitions.
If you use schema formats, set the Key/Message schema ids for the topic.
Topicstopics.partitionInfo

Partition Information (offsets, counts, and disk usage)

partition infopartitionoffsetsearliestlatestcountdisk usagecompact
Partition Information is a per-topic breakdown showing per-partition offsets and (where supported) record counts and disk usage.

Offsets (earliest vs latest)

Earliest is the lowest available offset; Latest is the end offset (the next offset that would be written).
For non-compacted topics, an approximate message count per partition is often latest - earliest.

Compacted topics

On compacted topics, offset deltas do not represent current record count reliably (compaction removes older keys).
In that case, the UI may use scan count to compute a more accurate count.

Disk usage

Disk usage is loaded per topic and partition when supported by the backend. Some connection types (for example Confluent) may not provide disk usage.
Topicstopics.information

Topic Information (audit topics at a glance)

topic informationtopicspartitionsreplication factorimbalanceskewoffsetsrecordsdiskconfigscolumnssortingfilterhot partitions
Topic Information is a quick “audit table” for your cluster: a sortable, filterable list of topics with partitions, offsets, record estimates, disk usage (if available), and selected topic config values.

Common uses

- Check what topics are populated (records/offsets) and how many partitions they have
- Spot unbalanced partitions (Imbal / hot partitions highlight)
- Quick eyeballing of key parameters (partitions, RF, cleanup policy, retention, min ISR, etc.)
- Filter topics by selected config values and derived metrics using the column filter
- Sort to find inconsistencies across configs, partitions, replication factor, records, or disk usage
- Estimate average message size (Avg msg) to spot unusually large payload topics

Filtering

Filter topics narrows the topic list by name. It supports the same preset-style filtering as the topic list (type @ to autocomplete saved presets).
Filter columns matches against the currently visible columns (offsets/records/disk and your selected config columns). This is useful for finding outliers by value.

Sorting & columns

Click any column header to sort (click again to toggle asc/desc). Use Columns to add/remove topic config columns (for example retention.ms or min.insync.replicas).

Partition breakdown

Expand a topic row to see a per-partition breakdown (begin/end offsets, record deltas, and disk usage where available). Hot partitions are highlighted when they exceed the average load.
Imbal is a quick skew factor:maxPartitionDelta / avgPartitionDelta, wheredelta = latestOffset - earliestOffset per partition. 1.0x is perfectly balanced; higher values indicate a few partitions hold much more data.
Avg msg is computed as totalBytes / recordCount when both inputs are available.
For compact topics, recordCount is not reliably represented by offset deltas (because compaction can remove older records). In those cases, you may see scan count, which runs a scan to compute a more accurate current record-count estimate for the topic.
For compact topics, Imbal is computed from disk-by-partition skew (because offsets are not a reliable proxy for current records after compaction).
Topicstopics.allConsumers

All Consumers (consumer groups across topics)

consumersconsumer groupsoffsetslagcommittedendclienthostnavigatetopic consumers
All Consumers shows consumer groups across all topics in the selected connection. It’s a quick way to spot lagging groups and see where they are consuming.

Navigation to Topic Consumers

Click a Topic value to open that topic and jump directly to its Consumers tab.

Filtering & grouping

Use the filter to match group id, topic, partition, state, client id, and host. Filtering is case-insensitive.
Use Group to group rows by consumer group and reduce repeated group ids.

Offsets and lag

Committed is the last committed offset for the group; End is the current partition end offset; Lag is the difference.
Deleting a consumer group is only possible when it is empty (no members attached).
Topicstopics.topicConsumers

Topic Consumers (per-topic offsets and lag)

topic consumersconsumersconsumer groupsoffsetslagcommittedendearliestlatestlockedclient attached
Topic Consumers shows consumer group offsets for a single topic, broken down by partition.

Why offsets can be locked

An offset can only be changed when there is no client attached to the consumer group (i.e. there is no active application consuming).
When a client is attached, the row is shown as locked and the Apply action is disabled.

Changing offsets (earliest / latest / manual)

Use earliest / latest to jump quickly, or type a specific offset and press Apply.
Use the + / − buttons to nudge the draft offset and to load an example record into the Inspect message panel.
Messagesmessages.playback

Message Playback

message playbackresult controlresult viewpartitionfilterfilter historysaved filterscriteria presetsfunctionoffsetmessages foundmessage detailssortingtimestampimportexportwrap
Message Playback combines the controls above the grid with the result list below it. Use the controls to choose what Kafka records to read, then click any result row to inspect its key, value, headers, metadata, and send/test tools in the Message Details pane.

A typical playback

1. Choose one or more partitions, or choose All.
2. Add an optional Filter to keep only matching keys, messages, or headers.
3. Set the all-important Function to decide where playback starts and how much data is read.
4. Press Play for a finite read, or Live feed to continue receiving new records. Stop ends the current read and Clear removes the displayed results.
5. Click a row in the result grid. The selected row is highlighted and its full content opens in the right- or bottom-docked Message Details pane. If that pane is collapsed, use its edge control to expand it.

Partitions

On a multi-partition topic, open Partition and select All, one partition, or any useful subset. At least one partition must be selected before playback can start.
On a single-partition topic the selector is disabled because there is no choice to make.
Functions such as tail=2000, head=100, and since=24h operate across the selected partitions. Direct offset ranges such as offset=1000.. require exactly one selected partition because Kafka offsets only have meaning inside their own partition.

Filters, recent history, and saved presets

Type an SL expression into Filter. Autocomplete can use fields learned from previously loaded messages, and clicking the Filter label opens the full SL reference.
With an empty filter field focused, press the keyboard Down Arrow to open recent filters for the current connection and topic. Ctrl+Space opens that history even when the field already contains text. Choose an item to reuse it; recent history is ideal for quick investigation changes.
The chevron button beside the field opens the larger multiline filter editor. The pencil button opens Saved filters, where important criteria can be named, tested, tagged, edited, and reused as presets instead of being left only in recent history.

The Function controls the read window

The Function is as important as the filter: it controls whether Streamyard reads the latest records, starts at the beginning, uses a time window, or reads a precise offset range. Focus the field to see suggestions and recently used functions.
See Offset Function Help for every supported form, aliases, time-zone behavior, and single-partition restrictions.

Messages found and opening details

Messages found reports the number displayed. When a filter is active it can also show found versus consumed counts, helping you judge how selective the filter is; finite playback includes elapsed timing, while a live feed reports messages received.
Clicking a line selects that exact topic/partition/offset record and reveals it in Message Details. You can also focus the grid and use the arrow keys to move through results while keeping the details pane in sync.

Show the relevant information quickly

Use the view selector above the grid to choose a built-in or saved result layout. Custom views can expose the metadata, headers, key/message fields, array summaries, mappings, and calculated values that matter to your investigation, in the order and widths you prefer.
See Result View Customisation for creating, previewing, resizing, saving, and applying reusable views.

Sorting Kafka results

Across multiple selected partitions, only Timestamp is sortable. An offset is local to its partition, so partition 0 offset 42 and partition 1 offset 42 have no global ordering relationship. Timestamp gives the useful chronological merge across partitions.
For a topic with one partition, or when exactly one partition is selected, only Offset is sortable. Within a partition, offsets define Kafka record order unambiguously. Click the active sortable heading to toggle ascending or descending order.

Import, export, and wrap

Export works with the result rows currently in the list: copy or download JSON/NDJSON, or create a CSV report using the selected result view. Server-side export can read a larger range without first loading every row into the UI.
Import reads JSON, NDJSON, or XML and produces those messages to the selected Kafka topic, with options for keys, headers, and partitions where available.
Wrap switches long cell values between a compact single-line grid and taller wrapped rows. It only changes presentation; it does not alter message content or playback.
Messagesmessages.resultViewCustomisation

Result View Customisation

messagesMESSAGESresult viewview customisationsaved viewsmessage list viewcolumnscolumn widthscolumn orderpreviewJSON pathsArray dataJSON mappingsCustom columnsAI Assistconnection tagstopic tagsGUIDduplicate namesorigin topicdragresize

What this modal is for

Customize result views lets you save reusable message-grid layouts: which columns are shown, what order they appear in, and which connection/topic tags must match before the view appears in a topic.
It edits a draft first. Nothing is committed until you press Save or Save & Use.

What is persisted, and where

Saved views
Saved presets are stored by the Streamyard API in ~/.kafka-streamyard/message-list-view-presets.json with an immutable GUID plus the display name, original connection/topic, selected columns, applicability tags, and update time. The GUID lets different records reuse the same display name. The repository boundary allows this file-backed store to be replaced by a shared cloud store later.
Current topic state
Only the active preset GUID and display name are cached per connection/topic, so reopening that topic can request the exact server record even when another view has the same name. Saved columns and tags are not duplicated into local storage.
Column widths
Widths are persisted by ResultView using a key that includes connection, topic, and the preset GUID (or built-in view name), so same-named saved views keep independent column sizing.

Changing topics, connections, and restarts

When you switch to another topic, Streamyard restores that topic's cached result-view state. That means one topic can remember a different active view from another.
If the active saved view no longer matches the current connection/topic tags, Streamyard automatically falls back to the default view instead of applying a mismatched preset.
After an app restart, saved views are reloaded from the server, topic-specific active-view state is restored from the local UI cache, and column widths come back as long as the same connection/topic/preset identity is used again.

Preview sample and available columns

The preview tries to use a small sample of real messages from the current topic so the modal can show live column data while you edit.
If preview rows are already available, they are reused immediately. Otherwise the modal requests a preview load and shows Attempting to load data sample....
The preview uses only a limited sample and derives the available column picker groups from those rows, including metadata, headers, key JSON, and message JSON fields.
If no data can be found yet, the modal explains that column requirements cannot be determined until some results exist.

Choosing an Available columns tab

JSON paths
Choose ordinary metadata, headers, key JSON fields, and message JSON fields. Click a field to add or remove it from the selected view.
Array data (N)
N is the number of distinct array locations found in the loaded samples, not the number of items inside an array. Use this tab for array counts, numeric aggregates, or finding one matching item and returning one of its fields.
JSON mappings (N)
N is the number of columns currently produced by the selected mapping. Use mappings for repeated key/value or name/value objects where one field supplies the column name and another supplies its value.
Custom columns
Enter a heading and an SL value expression for lookups, transformations, counts, or calculations that cannot be represented by a normal JSON path.
AI Assist
Let AI inspect up to five loaded samples and propose normal and calculated columns. Suggestions are reviewable and are not added until you select and apply them.

Columns, drag reorder, and width editing

Add columns from the Available columns side. The selected list on the left is the exact order used by the preview and the saved view.
Drag rows in Selected columns to reorder them. The preview updates immediately because it is driven from the current draft config.
Resize widths directly in the preview header. Those width changes are stored by the ResultView persistence key, so widths can survive closing the modal and restarting the app.

Saving and applying

A saved view is only shown for topics whose connection tags and topic tags both match. Names are labels rather than identities, so several records may share a name such as All Cols; the left list shows each record's origin connection/topic.
If the saved tags match the current context, saving also applies the view immediately. If they do not match, the view is still saved but will not appear in the current topic's view dropdown.
Deleting a currently active saved view reverts the topic back to the default result view.
Messagesmessages.details

Message panels (details, editing, samples, and sending)

messagedetailssendtesttombstonesamplesfavoriteshistorycopypasteformatautosession varssession variablesvariablesparent samplerecalcautoIncautoCycleautoDateIncclear sessionreset stateful@@s
The Message panel exists in two layouts (right dock and bottom dock), but both support the same actions for inspecting a selected message and using it as a base for producing new messages.

Why this panel is useful

The Message panel is where investigation turns into action: you can inspect what happened, copy the exact payload shape, adjust it, and replay or produce a new message.
It is especially useful for recreating a bug, testing a downstream fix, or building a reusable library of sample flows for your team.
Common examples include tweaking one field in a real message, saving a good sample as a favorite, or applying a template repeatedly with different session variables.

Data Sample syntax lint and testing

Key and Message each have an independent Syntax lint selector beside the find control.
NONE treats the field as unrestricted text and shows no format status. JSON and XML enable the matching syntax highlighting and validation.
Press Test to resolve templates first and validate the resolved template output for key and message independently. The preview reports valid or invalid JSON/XML and includes parser details for invalid content. Testing does not advance stateful values.
The Templates menu follows the selected message format, including XML starter samples when XML is active.
Sample Loader automatically saves and validates the current sample before opening. If the sample cannot be saved, the loader stays closed and the validation or persistence error remains visible.

Selection history

UseBack/Forwardto navigate recently selected messages.
UseRecentto open a menu of recent selections (typically the last ~10). This is handy for reusing a message as a sendable sample.

Formats (Key/Message type)

The panel shows the currently selected Key/Message format. Click the if the format is wrong, it will try to detect the correct format.

Editing, copy/paste, and templates

The panel has tabs for Message, Key, and Headers. You can copy and paste per-tab.
Template resolution happens in a couple of layers. One layer is @@s.* session vars (described below), and another layer is the general template engine for tokens like @@guid(...), @@rnd(...), etc. Use Test to preview the fully-resolved payload.

Session vars (sample variables) and @@s.someVar

Session vars are stored on a data sample (in the Sample Editor under variables). When you pick an “Apply sample” in the Message panel, those session vars become available to the editor.
Use @@s.someVar anywhere in:
Headers
Header values can include @@s.*.
Key
The key editor can include @@s.*.
Message
The message editor can include @@s.*.
{
  "orderId": "@@s.orderId",
  "customerId": "@@s.customerId"
}
If a session var is missing, Streamyard replaces it with ??@@s.someVar?? and shows a warning listing the missing tokens.

Parent samples (inherit variables)

Samples can have a Parent Sample. Session vars are resolved across the chain:
Order
Parent vars are applied first, then the child sample overrides any keys with the same name.
Scope
This inheritance is for variables. Key/value/headers do not automatically merge from parent samples.
Circular parent chains are blocked by the Sample Editor.

Recalc (stable vs. regenerated values)

Each session var can be marked Recalc in the Sample Editor.
Recalc off
The computed value is cached (per renderer session). This is useful for IDs that should stay consistent while you test/send repeatedly.
Recalc on
The value is recomputed every time session vars are resolved. This is useful for values that should change each send (e.g. timestamps or random numbers).
Session var values can also reference other session vars, so you can compose values like @@s.traceId inside@@s.correlationId.
If a non-recalc session var uses generated content such as @@person.*, the resolved value stays stable for reuse until you deliberately clear the session cache or use a recalc-enabled variable.
The Clear Session button in the favorites header clears cached non-recalc session values for the current renderer session. For autoCycle(...), that also resets the in-memory cycle index back to0. autoInc(...) and autoDateInc(...) keep their current persisted values.

When are session vars applied?

In both layouts (right dock and bottom dock), the flow is the same:
1) Apply session vars
Replace any @@s.* tokens in headers/key/value using the selected “Apply sample” (including its parent chain).
2) Resolve remaining templates
The template engine resolves other @@ tokens.
3) Use the resolved output
Test shows the resolved output. Send produces the resolved output. Calc partition (when available) also uses the resolved key/headers so the partition calculation matches what will actually be sent.

Samples & favorites

You can load a saved sample to overwrite the current Key/Message/Headers, make its session variables available in the Message Details tabs, save the current content as a sample, and mark samples as favorites for quick access.
Favorite sample
Favorited samples are scoped by connection/topic context (tags) and are ideal for maintaining a reusable “send library”.
In both the favorites row and the sample dropdown menu, there is a small <> (code) action next to each sample. Clicking it loads the raw sample text into the editor (no resolution of @@ tokens, and no substitution of @@s.* session vars). This is useful when you want to edit templates in their original form.
The Message Details Session Vars tab shows the applied sample’s effective variables, including inherited parent values. You can edit local values, add variables, change recalc behavior, or edit an inherited value to create an override in the applied sample.
Clear Session clears cached generated values and opens that tab for the applied sample. The Message Details state reset control and the Sample Editor Reset Stateful action rewrite persisted autoInc(...), @@inc(...), autoCycle(...), and autoDateInc(...) definitions back to their starting position before saving.

Test / Send / Tombstone

Use Test to resolve templates and optionally validate JSON before sending.
SendTombstone
If your message uses template tokens like @@, use Test to preview the fully resolved payload before sending.
Test
Use Send to produce to the current topic. If the topic is compacted, you can also send a tombstone (null value) for a key.
If the selected sample provides a session var whose value is autoInc(start,current,step), a successful manual send from the Message panel resolves it to current and then persists the sample with the next value. Test previews the current value but does not advance it.
autoCycle(...) uses the current session index, then advances that in-memory index after send so the next parent/child sample resolve sees the next cycle item.
autoDateInc(...) follows the same persisted-next-value model as autoInc(...).
You can also pick a destination partition. Default lets the producer choose the partition (typically key-based when a key is present, otherwise the producer may distribute records). Manual lets you force an explicit partition number (0..N-1).

Copy full message / reuse elsewhere

Use Copy all to copy the full message details as JSON (meta + headers + key/value). This is useful for creating a data sample, or pasting into other tools.
Copy all
You can also copy individual tabs, and paste into another connection/topic workflow using the “paste into new topic” tools.
Messagesmessages.partitioning

Partitioning (producing messages)

partitionpartitioningpartitionersendproducedefaultpartitionerroundrobinrandommodulocrc32fnvjavahash
Partitioning controls which topic partition a produced record is written to. In Streamyard, there are two ways a partition can be chosen:
Manual: you explicitly pick a partition number.
Automatic: the producer/partitioner decides based on the selected strategy.

Manual partition (always deterministic)

If you pick Partition: N, Streamyard sets the record partition explicitly.
This overrides any partitioner strategy.

Automatic strategies (when you do not pick a partition)

DefaultPartitioner
Kafka default partitioner. When a key is present, it is key-based. When the key is missing, Kafka may distribute records in a way that is not strictly predictable (producer-dependent).
RoundRobinPartitioner
Uses org.apache.kafka.clients.producer.RoundRobinPartitioner. Records are spread across partitions over time; the exact partition for a given key is not deterministic.
Random
Chooses a random partition. This is indeterministic and cannot be calculated ahead of time.
Numeric Modulo
Interprets the key as an integer (up to a long/big int) and uses key % partitionCount (supports negative keys). If the key is not numeric, Streamyard cannot calculate a partition.
JavaHash
Uses Java string hashCode() of the key, then maps it to a partition.
CRC32
Uses CRC32 of the key bytes, then maps it to a partition.
FNV-1a
Uses FNV-1a of the key bytes, then maps it to a partition.

Current Key Calc

For calculatable strategies, Streamyard can compute the expected destination partition for the current key.
Clicking Current Key Calc will calculate the partition and switch the UI to a manual partition.
Strategies like RoundRobinPartitioner and Random are indeterministic and cannot be calculated.
Data toolsdata.sampleLoaderPartitioning

Sample Loader help (wait time, quantity, partitioning)

sample loaderpartitioningpartitionermodulusnumeric modulomanualtemplates@@m@@keywaitdelayburstquantity-1until stoppedmssh
Sample Loader produces messages as a background task. Pressing Sample Loader automatically saves and validates the current sample before opening the task setup. Keys and values may contain templates (for example @@rnd(...) or @@m.id) which are resolved by the backend per produced record.

Why teams use Sample Loader

Sample Loader is useful when you want realistic traffic without hand-producing every record one by one.
It can simulate steady background activity, bursty workloads, or endless test data streams while you validate consumers, dashboards, alerts, and retry behaviour.
Common uses include load-testing a dev workflow, generating many variants of a business object, or keeping a topic active while a tester validates end-to-end behaviour.

Wait time + quantity

Waits are optional. Leave the wait checkbox off when you want Sample Loader to send as fast as possible.
Burst from and Burst to control how many records are sent together before the next optional wait. The default is 1 to 1.
Wait from and Wait to accept ms, s, m, and h.
Use one value for a fixed cadence (for example 1000ms, 1s, or 5m), or set a range such as 10s to 15s for a random wait between produced messages.
Set burst mode to values like 10 and 20 to send a random batch of 10..20 messages together each time.
Enter -1 in Quantity to load to keep producing until you stop/delete the task from the Tasks window.

Numeric Modulo

Numeric Modulo requires the resolved key to be a valid signed 64-bit integer (Java long).
If your key is templated (example: @@m.id), it is validated after template resolution. If the resolved value is not numeric, the task will fail and the error will be shown in the Tasks window.
# Example
value: {"id": @@rnd(10000000,99999999)}
key: @@m.id  # OK (id resolves to digits)

Manual

Manual partitioning requires an explicit partition number (0..N-1). Sample Loader will validate the partition exists on the target topic.

Default / Round-robin / Random

These strategies do not require numeric keys. Random and Round-robin are indeterministic and cannot be predicted ahead of time.
Data toolsdata.variableSubstitution

Variable Substitution Help

sampledatasubstitutionrandomguidheaderskeyvaluepaymentshipmentproductorganizationaccountsubscriptionthingsvehiclesaircraftfinancialscurrenciesfxequitiesbondscommoditiesaccountingappliancesclothescountriescodesindustriesscienceforRndrndTextUniquewithout replacementrandom arraysloopsinline templateslessonsautoIncautoCycleautoDateIncsession varsmanual send

Overview

You can use @@ template tokens inside headers, key, and value. They are resolved by the backend endpoint POST /api/message-template/resolve.
The goal is to quickly generate synthetic test data and to reuse values across fields (e.g. use the same generated GUID in the key and in the JSON value).

Why this is useful

Variable substitution is useful when you need realistic but disposable business data for testing, demos, or replay scenarios.
It helps you keep related fields in sync, such as using the same generated id in the key, headers, and message body.
Typical examples include generating people, orders, invoices, payments, shipments, and products without maintaining a giant folder of hand-written JSON files.

Resolution order (important)

Resolution happens in passes so you can reference other fields:
Pass 1: Generators (randoms / timestamps / composite generators like @@person.*, @@organization.*, @@payment.*, @@vehicles.*, @@financials.*, @@household.*, @@people.clothes.*, @@countries.*, @@codes.*, @@industries.*, and @@science.*).
Pass 2: Context references (read values from headers / key / message JSON): @@h.*, @@k.*, @@m.*, @@keyRaw.
Conditional pass: @@if(...) is resolved and then the chosen branch is re-resolved (generators + references again). This is repeated a few times to catch nested expressions.
If a reference cannot be resolved (unknown token / missing header / invalid JSON), the token is usually left as-is (so you can see what didn’t resolve).

Quick examples (copy/paste)

Generate an id once, reuse everywhere
key: order-@@guid(1)
value: {"id":"@@guid(1)","createdAt":"@@now.toIsoDate()"}
Pick from a list
value: {"env":"@@rndText('dev','stage','prod')"}
Counter for sample playback
key: order-@@count(1000,1)
value: {"seq":@@count(1,0.0001)}
Reference a header
headers: {"x-correlation-id":"@@guid()"}
value: {"correlationId":"@@h.x-correlation-id"}
Conditional based on message JSON
value: {"kind":"@@if(m.type,'invoice','invoice','order')","ref":"@@if(m.type,'invoice', rnd(10), rnd(8))"}

Stateful sample vars for manual sends and Sample Loader

These are used in a sample session var, then referenced from headers, key, or message with @@s.someVar. Manual sends use the renderer session; Sample Loader sends the merged definitions to a task-local backend session.
autoInc(start,current,step)
Numeric incrementer that emits the stored current value and adds step after a successful manual send or Sample Loader write.
@@inc(0,1,1000)
Bounded incrementer: start at 0, add 1 after each successful send or write, include 1000, then wrap to 0. Manual sends persist an internal current value; Sample Loader keeps it in task memory. Negative steps and descending limits are supported.
autoCycle('NEW','PAID','SHIPPED')
Cycles through quoted text values in order using an in-memory session index. The stored sample text is left unchanged.
autoDateInc('2025-01-01','2025-01-01','P1D')
Date/time incrementer using a quoted start, quoted current value, and a quoted ISO-8601 duration such as 'P1D' or 'PT5M'.
# Sample variables
orderSeq = autoInc(1000,1000,1)
boundedSeq = @@inc(0,1,1000)
status = autoCycle('NEW','PAID','SHIPPED')
eventTime = autoDateInc('2025-01-01T00:00:00Z','2025-01-01T00:00:00Z','PT5M')

# Message panel usage
key: order-@@s.orderSeq
value: {
  "sequence": "@@s.orderSeq",
  "boundedSequence": "@@s.boundedSeq",
  "status": "@@s.status",
  "eventTime": "@@s.eventTime"
}
In the Message panels, a successful manual Send resolves the current value and then persists the next value back onto the sample. Test previews the current value without advancing it.
Sample Loader creates an isolated in-memory session for each task. It merges parent and child definitions, starts autoInc and autoDateInc from their stored current values, and advances state only after a successful Kafka write. Task state is discarded when the task ends and is not written back to the UI sample.
Non-recalc generated values remain stable for the task, recalc values refresh per message, nested @@s.* references are supported, and missing or circular references fail the task instead of silently producing incorrect data.
@@count(start,step) is separate from session vars. It can be placed directly in headers, key, or message, or inserted through a referenced session value. Identical expressions share one value within a message and emit start + step × messageIndex for each following task message. Decimal and negative steps are supported.

Generators

@@guid()
Random UUID string
@@guid(1)
Stable slot UUID (same index produces the same UUID within one resolve call)
@@guid().toLower()
Format UUID casing (also supports .toUpper())
@@rnd(6)
Random digits (0 .. 10^n-1)
@@rnd(100,999)
Random inclusive numeric range (min/max order doesn’t matter)
@@rnd(6).toPadded(6)
Pad with zeros (useful for fixed-width ids)
@@now
Current time as ISO instant string
@@now.toMillis()
Unix epoch milliseconds (also toSeconds() and toIsoDate())
@@rndText('a','b','c')
Pick one value at random (also accepted as alias: @@rnd('a','b'))
@@rndTextUnique('a','b','c')
Chooses without replacement within one message resolution. Identical choice lists share a pool; after every value has been used, the pool refills and starts another random cycle.
@@count(start,step)
Stateful counter for sample playback. Starts at start and increments by step per produced message (stable within one message). Sample Data Test previews the initial start value without advancing the task counter.
@@forRnd(10) { ... }
Creates a random 1–10 item JSON array from the following object, array, string, number, boolean, or null value. The inline shape is resolved independently for every item and loops may be nested.
@@rndChars(12).toMixed()
Random token-like string (also supports toUpper() / toLower())
@@rndAlphas(8)
Random alphanumeric string (no hyphen/underscore)
{
  "batchId": "@@guid(1)",
  "items": @@forRnd(5) {
    "id": "@@guid(1)",
    "sku": "SKU-@@rndChars(8).toUpper()",
    "quantity": @@rnd(1,10),
    "tags": @@forRnd(3) "@@rndTextUnique('new','priority','bulk')"
  }
}

Random data generators

Random data generators create a shared context the first time they are used in a resolve call, so related fields stay consistent. All generated values are synthetic and random, and any resemblance to real people or companies is purely coincidental.
These JSON examples can be pasted into a key, message, or a header value. You can also use individual fields directly as tokens such as @@person.fullName or @@shipment.status.
Person
{
  "name": "@@person.fullName",
  "email": "@@person.email",
  "country": "@@person.country",
  "phone": "@@person.phone"
}
Order / Invoice
{
  "orderNo": "@@order.number",
  "lines": @@order.lines,
  "invoiceNo": "@@invoice.number"
}
Payment
{
  "id": "@@payment.id",
  "amount": "@@payment.amount",
  "currency": "@@payment.currency",
  "payer": "@@payment.payerEmail"
}
Shipment
{
  "tracking": "@@shipment.trackingNumber",
  "status": "@@shipment.status",
  "carrier": "@@shipment.carrier",
  "events": @@shipment.events
}
Product
{
  "sku": "@@product.sku",
  "name": "@@product.name",
  "price": "@@product.price",
  "tags": @@product.tags
}
Device
{
  "type": "@@device.type",
  "hostname": "@@device.hostname",
  "ip": "@@device.ip",
  "mac": "@@device.mac",
  "os": "@@device.os",
  "related": @@device.related,
  "interfaces": @@device.interfaces
}
Support ticket
{
  "id": "@@ticket.id",
  "status": "@@ticket.status",
  "priority": "@@ticket.priority",
  "reporter": "@@ticket.reporterName",
  "tags": @@ticket.tags
}
Organization
{
  "id": "@@organization.id",
  "name": "@@organization.name",
  "industry": "@@organization.industry",
  "hq": @@organization.headquarters
}
Account
{
  "id": "@@account.id",
  "number": "@@account.number",
  "owner": "@@account.ownerEmail",
  "tags": @@account.tags
}
Subscription
{
  "id": "@@subscription.id",
  "plan": "@@subscription.planName",
  "amount": "@@subscription.amount",
  "features": @@subscription.features
}

Composite generator fields (full reference)

Note: @@gen.<field> is accepted as a backwards-compatible alias of @@person.<field>.
@@person.*
@@person.title
@@person.forename
@@person.firstname
@@person.first
@@person.surname
@@person.lastname
@@person.last
@@person.name
@@person.fullName
@@person.hobby
@@person.hobbies
@@person.profession
@@person.employmentStatus
@@person.employmentHistory
@@person.employment_history
@@person.workHistory
@@person.work_history
@@person.careerHistory
@@person.career_history
@@person.address1
@@person.address_line1
@@person.line1
@@person.street
@@person.address2
@@person.address_line2
@@person.line2
@@person.city
@@person.town
@@person.county
@@person.region
@@person.state
@@person.province
@@person.postcode
@@person.postal
@@person.zip
@@person.country
@@person.phone
@@person.phone_number
@@person.email
@@person.username
@@person.user
@@person.login
@@person.lat
@@person.latitude
@@person.long
@@person.lng
@@person.longitude
@@person.countryCode
@@person.country_code
@@person.dob
@@person.date_of_birth
@@person.dateOfBirth
@@person.age
@@person.age_years
@@person.ageYears
@@person.gender
@@person.ssn
@@person.national_id
@@person.nationalId
@@person.driver_license
@@person.driverLicense
@@person.driving_license
@@person.drivingLicense
@@person.passport_number
@@person.passportNumber
@@person.passport
@@person.vat
@@person.vat_number
@@person.vatNumber
@@person.tax
@@person.tax_id
@@person.tax_number
@@person.taxNumber
@@person.employmentHistory resolves to a JSON array string with entries like [{"type":"employment","status":"employed","fromDate":"2022-03-01","toDate":"2024-11-30","title":"Software Engineer","organization":"Northstar Systems","employmentType":"full-time","salaryPeriod":"annual","salary":84250}].
@@organization.*
@@organization.id
@@organization.organizationId
@@organization.organization_id
@@organization.orgId
@@organization.name
@@organization.organizationName
@@organization.organization_name
@@organization.orgName
@@organization.legalName
@@organization.legal_name
@@organization.industry
@@organization.sector
@@organization.segment
@@organization.domain
@@organization.website
@@organization.url
@@organization.country
@@organization.countryCode
@@organization.country_code
@@organization.phone
@@organization.phone_number
@@organization.registrationNumber
@@organization.registration_number
@@organization.companyNumber
@@organization.company_number
@@organization.vatNumber
@@organization.vat_number
@@organization.vat
@@organization.taxNumber
@@organization.tax_number
@@organization.taxId
@@organization.currency
@@organization.employeeCount
@@organization.employee_count
@@organization.employees
@@organization.annualRevenue
@@organization.annual_revenue
@@organization.revenue
@@organization.tags
@@organization.tagsJson
@@organization.tags_json
@@organization.headquarters
@@organization.hq
@@organization.tags resolves to a JSON array string, and @@organization.headquarters resolves to a JSON object string.
@@account.*
@@account.id
@@account.accountId
@@account.account_id
@@account.number
@@account.accountNumber
@@account.account_number
@@account.externalId
@@account.external_id
@@account.crmId
@@account.status
@@account.state
@@account.type
@@account.accountType
@@account.account_type
@@account.segment
@@account.region
@@account.lifecycleStage
@@account.lifecycle_stage
@@account.stage
@@account.openedAt
@@account.opened_at
@@account.createdAt
@@account.created_at
@@account.renewalAt
@@account.renewal_at
@@account.nextBillingAt
@@account.next_billing_at
@@account.currency
@@account.balance
@@account.outstandingBalance
@@account.outstanding_balance
@@account.creditLimit
@@account.credit_limit
@@account.paymentTermsDays
@@account.payment_terms_days
@@account.paymentTerms
@@account.riskScore
@@account.risk_score
@@account.ownerName
@@account.owner_name
@@account.primaryContactName
@@account.primary_contact_name
@@account.ownerEmail
@@account.owner_email
@@account.primaryContactEmail
@@account.primary_contact_email
@@account.organizationName
@@account.organization_name
@@account.orgName
@@account.organizationId
@@account.organization_id
@@account.orgId
@@account.tags
@@account.tagsJson
@@account.tags_json
@@account.tags resolves to a JSON array string.
@@subscription.*
@@subscription.id
@@subscription.subscriptionId
@@subscription.subscription_id
@@subscription.number
@@subscription.accountNumber
@@subscription.account_number
@@subscription.account
@@subscription.organizationName
@@subscription.organization_name
@@subscription.orgName
@@subscription.planCode
@@subscription.plan_code
@@subscription.planName
@@subscription.plan_name
@@subscription.plan
@@subscription.status
@@subscription.state
@@subscription.billingPeriod
@@subscription.billing_period
@@subscription.term
@@subscription.channel
@@subscription.source
@@subscription.quantity
@@subscription.seats
@@subscription.autoRenew
@@subscription.auto_renew
@@subscription.startedAt
@@subscription.started_at
@@subscription.startDate
@@subscription.start_date
@@subscription.trialEndsAt
@@subscription.trial_ends_at
@@subscription.trialEnd
@@subscription.trial_end
@@subscription.renewalAt
@@subscription.renewal_at
@@subscription.nextBillingAt
@@subscription.next_billing_at
@@subscription.currency
@@subscription.amount
@@subscription.grossAmount
@@subscription.gross_amount
@@subscription.value
@@subscription.mrr
@@subscription.monthlyRecurringRevenue
@@subscription.monthly_recurring_revenue
@@subscription.features
@@subscription.featuresJson
@@subscription.features_json
@@subscription.features resolves to a JSON array string.
@@order.*
@@order.number
@@order.date
@@order.country
@@order.countryCode
@@order.currency
@@order.vatRate
@@order.subtotalNet
@@order.totalVat
@@order.totalGross
@@order.lines
@@order.lines resolves to a JSON array string.
@@invoice.*
@@invoice.number
@@invoice.date
@@invoice.dueDate
@@invoice.country
@@invoice.countryCode
@@invoice.currency
@@invoice.vatRate
@@invoice.subtotalNet
@@invoice.totalVat
@@invoice.totalGross
@@invoice.lines
@@invoice.lines resolves to a JSON array string.
@@payment.*
@@payment.id
@@payment.paymentId
@@payment.payment_id
@@payment.number
@@payment.createdAt
@@payment.created_at
@@payment.date
@@payment.processedAt
@@payment.processed_at
@@payment.settledAt
@@payment.settled_at
@@payment.capturedAt
@@payment.status
@@payment.state
@@payment.method
@@payment.paymentMethod
@@payment.provider
@@payment.gateway
@@payment.currency
@@payment.amount
@@payment.grossAmount
@@payment.feeAmount
@@payment.fee
@@payment.netAmount
@@payment.reference
@@payment.paymentReference
@@payment.authCode
@@payment.authorizationCode
@@payment.authorisationCode
@@payment.cardLast4
@@payment.last4
@@payment.maskedPan
@@payment.masked_pan
@@payment.payerName
@@payment.payer_name
@@payment.customerName
@@payment.payerEmail
@@payment.payer_email
@@payment.customerEmail
@@shipment.*
@@shipment.id
@@shipment.shipmentId
@@shipment.shipment_id
@@shipment.number
@@shipment.tracking
@@shipment.trackingNumber
@@shipment.tracking_number
@@shipment.carrier
@@shipment.service
@@shipment.shippingService
@@shipment.status
@@shipment.state
@@shipment.shippedAt
@@shipment.shipped_at
@@shipment.dispatchAt
@@shipment.estimatedDeliveryAt
@@shipment.estimated_delivery_at
@@shipment.eta
@@shipment.destinationCountry
@@shipment.destination_country
@@shipment.destinationCountryCode
@@shipment.destination_country_code
@@shipment.country
@@shipment.countryCode
@@shipment.warehouse
@@shipment.warehouseCode
@@shipment.warehouse_code
@@shipment.packageCount
@@shipment.package_count
@@shipment.packages
@@shipment.weightKg
@@shipment.weight_kg
@@shipment.weight
@@shipment.shippingCost
@@shipment.shipping_cost
@@shipment.cost
@@shipment.currency
@@shipment.events
@@shipment.trackingEvents
@@shipment.tracking_events
@@shipment.events resolves to a JSON array string.
@@product.*
@@product.sku
@@product.id
@@product.productId
@@product.product_id
@@product.name
@@product.title
@@product.category
@@product.subcategory
@@product.sub_category
@@product.type
@@product.brand
@@product.vendor
@@product.manufacturer
@@product.status
@@product.availability
@@product.currency
@@product.price
@@product.unitPrice
@@product.unit_price
@@product.listPrice
@@product.list_price
@@product.taxRate
@@product.tax_rate
@@product.vatRate
@@product.inventoryLevel
@@product.inventory_level
@@product.stock
@@product.onHand
@@product.reorderLevel
@@product.reorder_level
@@product.reorderPoint
@@product.warehouseBin
@@product.warehouse_bin
@@product.bin
@@product.location
@@product.barcode
@@product.ean
@@product.gtin
@@product.tags
@@product.tagsJson
@@product.tags_json
@@product.tags resolves to a JSON array string.
@@vehicles.cars.*
@@vehicles.cars.id
@@vehicles.cars.name
@@vehicles.cars.manufacturer
@@vehicles.cars.model
@@vehicles.cars.trim
@@vehicles.cars.modelYear
@@vehicles.cars.bodyStyle
@@vehicles.cars.vehicleClass
@@vehicles.cars.countryOfOrigin
@@vehicles.cars.engine
@@vehicles.cars.engineType
@@vehicles.cars.displacementLitres
@@vehicles.cars.cylinders
@@vehicles.cars.powerHp
@@vehicles.cars.powerKw
@@vehicles.cars.torqueNm
@@vehicles.cars.fuelType
@@vehicles.cars.transmission
@@vehicles.cars.drivetrain
@@vehicles.cars.batteryCapacityKwh
@@vehicles.cars.electricRangeKm
@@vehicles.cars.doors
@@vehicles.cars.seats
@@vehicles.cars.vin
@@vehicles.cars.registration
@@vehicles.cars.colour
@@vehicles.cars.odometerKm
@@vehicles.cars.condition
Fields resolve from one car profile per message, so manufacturer, model, engine, fuel, transmission, drivetrain, and specifications remain consistent.
Things catalogue namespaces
@@vehicles.aircraft.*: id, name, manufacturer, model, type, engineManufacturer, engine, engineType, engineCount, totalThrustKn, passengers, rangeKm, cruiseSpeedKph, serviceCeilingFt, lengthMetres, wingspanMetres, countryOfOrigin, registration
@@vehicles.motorcycles.*: id, name, manufacturer, model, type, engine, displacementCc, cylinders, powerHp, torqueNm, transmission, fuelCapacityLitres, weightKg, seatHeightMm, countryOfOrigin, registration
@@vehicles.boats.*: id, name, manufacturer, model, type, hull, lengthMetres, beamMetres, engine, engineCount, powerHp, capacity, fuelCapacityLitres, countryOfOrigin, registration
@@vehicles.trains.*: id, name, manufacturer, model, type, powerType, powerKw, maxSpeedKph, tractiveEffortKn, wheelArrangement, lengthMetres, weightTonnes, voltageSystem, countryOfOrigin
@@vehicles.bicycles.*: id, name, manufacturer, model, type, frameMaterial, wheelSize, tireWidthMm, groupset, brakes, weightKg, gears, countryOfOrigin
@@financials.currencies.*: name, code, numericCode, symbol, minorUnits, region, centralBank
@@financials.fx.*: pair, baseCurrency, baseCurrencyName, quoteCurrency, quoteCurrencyName, rate, inverseRate, bid, ask, spread, asOf
@@financials.markets.*: name, mic, city, country, countryCode, timezone, currency, benchmarkIndex
@@financials.equities.*: company, ticker, isin, sector, industry, exchange, mic, country, currency, previousClose, open, high, low, last, change, changePercent, volume, sharesOutstanding, marketCap, peRatio, dividendYield, asOf
@@financials.bonds.*: isin, issuer, type, currency, faceValue, couponRate, issueDate, maturityDate, termYears, creditRating, yield, price, couponFrequency
@@financials.commodities.*: name, symbol, category, unit, exchange, currency, price, asOf
@@financials.crypto.*: name, symbol, network, consensus, decimals, maxSupply, circulatingSupply, currency, price, marketCap, asOf
@@financials.transactions.*: id, type, status, direction, amount, fee, netAmount, currency, account, counterparty, channel, timestamp
@@financials.accounting.*: entryId, debitAccountCode, debitAccountName, debitAccountClass, debitAmount, creditAccountCode, creditAccountName, creditAccountClass, creditAmount, currency, description, postingDate, period
@@financials.statements.*: fiscalYear, currency, revenue, costOfRevenue, grossProfit, operatingExpenses, operatingIncome, taxExpense, netIncome, assets, liabilities, equity, operatingCashFlow, investingCashFlow, financingCashFlow, netCashFlow, asOf
@@household.appliances.refrigerator.*: manufacturer, model, type, capacityLitres, freezerCapacityLitres, energyRating, finish, feature, widthCm, countryOfOrigin
@@household.appliances.washingMachine.*: manufacturer, model, type, loadKg, spinRpm, energyRating, colour, feature, countryOfOrigin
@@household.appliances.oven.*: manufacturer, model, type, capacityLitres, fuelType, maximumTemperatureC, finish, feature, countryOfOrigin
@@household.appliances.vacuumCleaner.*: manufacturer, model, type, powerWatts, bagless, runtimeMinutes, colour, countryOfOrigin
@@household.appliances.coffeeMaker.*: manufacturer, model, type, pressureBar, waterCapacityLitres, colour, feature, countryOfOrigin
@@people.clothes.jeans.*: manufacturer, model, fit, material, colour, size, waistInches, inseamInches, gender, countryOfOrigin
@@people.clothes.jackets.*: manufacturer, model, fit, material, colour, size, gender, countryOfOrigin
@@people.clothes.shoes.*: manufacturer, model, type, material, colour, sizeEu, countryOfOrigin
@@countries.*: name, capital, code, alpha3, continent, currency, currencyCode, callingCode, language, domain, population, timezone
@@codes.vat.*: number, country, countryCode, uk, germany, france, finland, sweden, netherlands, italy, spain
@@codes.bank.*: name, country, countryCode, iban, bic, swift, accountNumber, sortCode, routingNumber, sic
@@codes.industry.*: sic, naics, nace, isic
@@codes.product.*: ean13, upc, isbn13, sku, hsCode
@@codes.business.*: lei, duns, companyNumber
@@codes.logistics.*: containerCode, trackingNumber, sscc18
@@codes.medical.*: icd10, snomed
@@industries.*: name, sector, subsector, sic, naics, nace, product, process, material, occupation, metric
@@science.chemistry.elements.*: name, symbol, atomicNumber, atomicWeight, category, state, discoveryYear, discoveredBy
@@science.biology.species.*: commonName, scientificName, kingdom, class, habitat, diet, lifespanYears, conservationStatus, continent
@@science.materials.*: name, category, densityKgM3, meltingPointC, thermalConductivityWmK, electricalConductivitySm, hardness, commonUse
@@science.astronomy.stars.*: name, spectralType, distanceLy, constellation, massSolar, radiusSolar, temperatureK, luminositySolar
@@science.astronomy.planets.*: name, distanceFromSunAu, diameterKm, massKg, dayLengthEarthDays, moons, type, averageTemperatureC, maximumTemperatureC, minimumTemperatureC
@@science.units.*: quantity, name, symbol, siBase, conversionFactor
@@science.measurements.*: value, quantity, unit, symbol
Every namespace is stable within one resolved message. Related fields come from one selected profile, and generated IBAN, routing, EAN, UPC, ISBN, VIN, container, and SSCC values include their appropriate check digits.
@@device.*
@@device.type
@@device.deviceType
@@device.device_type
@@device.category
@@device.vendor
@@device.manufacturer
@@device.make
@@device.brand
@@device.model
@@device.os
@@device.osName
@@device.os_name
@@device.osVersion
@@device.os_version
@@device.hostname
@@device.host
@@device.name
@@device.domain
@@device.fqdn
@@device.mac
@@device.macAddress
@@device.mac_address
@@device.ip
@@device.ipv4
@@device.ipv4Private
@@device.privateIp
@@device.private_ip
@@device.lanIp
@@device.lan_ip
@@device.ipv4Public
@@device.publicIp
@@device.public_ip
@@device.wanIp
@@device.wan_ip
@@device.ipv6
@@device.subnet
@@device.subnetCidr
@@device.subnet_cidr
@@device.cidr
@@device.gateway
@@device.gatewayIp
@@device.gateway_ip
@@device.protocol
@@device.port
@@device.serial
@@device.serialNumber
@@device.serial_number
@@device.assetTag
@@device.asset_tag
@@device.asset
@@device.imei
@@device.cpu
@@device.cpuModel
@@device.cpu_model
@@device.cores
@@device.cpuCores
@@device.cpu_cores
@@device.memoryGb
@@device.memory_gb
@@device.ramGb
@@device.ram_gb
@@device.storageGb
@@device.storage_gb
@@device.diskGb
@@device.disk_gb
@@device.rack
@@device.datacenter
@@device.dc
@@device.related
@@device.interfaces
@@device.related resolves to a JSON object string, and @@device.interfaces resolves to a JSON array string.
@@ticket.*
@@ticket.id
@@ticket.ticketId
@@ticket.ticket_id
@@ticket.number
@@ticket.createdAt
@@ticket.created_at
@@ticket.created
@@ticket.openedAt
@@ticket.opened_at
@@ticket.updatedAt
@@ticket.updated_at
@@ticket.updated
@@ticket.lastUpdated
@@ticket.last_updated
@@ticket.dueAt
@@ticket.due_at
@@ticket.due
@@ticket.status
@@ticket.state
@@ticket.priority
@@ticket.severity
@@ticket.category
@@ticket.type
@@ticket.source
@@ticket.channel
@@ticket.queue
@@ticket.team
@@ticket.tags
@@ticket.tagsJson
@@ticket.tags_json
@@ticket.subject
@@ticket.title
@@ticket.description
@@ticket.body
@@ticket.reporterName
@@ticket.reporter_name
@@ticket.requester
@@ticket.requesterName
@@ticket.reporterEmail
@@ticket.reporter_email
@@ticket.requesterEmail
@@ticket.assigneeName
@@ticket.assignee_name
@@ticket.agent
@@ticket.agentName
@@ticket.assigneeEmail
@@ticket.assignee_email
@@ticket.agentEmail
@@ticket.sla
@@ticket.slaMinutes
@@ticket.sla_minutes
@@ticket.tags resolves to a JSON array string.

References (read from key/value/headers)

Use these to reuse already-generated values (or values you typed in yourself).
@@keyRaw
Raw key string
@@h.x-foo
Header reference (recursive: if the header value contains tokens, it is resolved too)
@@k.id
Read a field from key JSON (supports nested paths: @@k.customer.id)
@@m.type
Read a field from message JSON (supports nested paths: @@m.person.identity.age)
Note: JSON references require the key/value to be valid JSON objects. If parsing fails, the token is left unchanged.

Conditionals

@@if(ref, expected, thenExpr, elseExpr?)
ref must be a reference like m.<path>, k.<path>, or h.<header>.
expected can be an exact match, or comparisons like >=65, !=foo. Date comparisons also work for ISO dates (YYYY-MM-DD).
Branch expressions can contain generators; inside branches you can use short form like rnd(8) (it is treated as @@rnd(8)).
value: {"age":"@@person.age","tier":"@@if(m.age,'>=65','pension',@@if(m.age,'>=18','adult','minor'))"}
Automationautomation.playbooks

Playbook help

playbooktaskscreate-topicdelete-topicscopy-dataimport-datandjsonjsonburstdelay

Overview

A playbook is a line-based script for running bulk operations (create/copy/delete/etc.) against Kafka topics.
Each non-empty line is a single command. Validation happens before running.
You can also use the Playbook builder wizard to generate scripts. It is available from the topic list context menu and from the playbook editor (New with Wizard).
Use Dry run in the playbook editor to preview the expanded script and validation output before submitting a task.
Playbooks can now also load message envelopes from disk using import-data, reusing the same import/export envelope shape you see in the topic import modal.
Important: playbooks are not designed to set up a production environment. Production provisioning should be handled via CI/CD and IaC (Infrastructure as Code). Playbooks are intended for controlling your own environment (or a slice of an environment) and for quick refreshes to prepare for manual testing with Kafka Streamyard. For example, if you have a Docker environment or a development environment, playbooks help you quickly get it back into a known state that is ready for testing again.

MCP-backed AI assistant

When an AI provider API key is configured, the playbook editor and wizard show an AI playbook assistant side pane.
The assistant uses read-only MCP-style tools from the backend to inspect list_topics, describe_topics, latest_schemas, and a limited sample_topic_messages preview before it suggests a draft.
Ask it for full playbooks, import flows, topic copy/reset plans, repeat blocks, or for-each-topic examples.
Suggested drafts are not auto-run. Use Replace draft, Append draft, Validate, and Dry run to review the output before you save or submit a task.

Script basics

Comments: lines starting with # are ignored.
One command per line: playbooks are line-based, so import-group, each import-data, and end-group must all be on separate lines.
Targets: most commands take <connectionName>:<topicPattern>.
Repeat blocks: you can repeat an indented block using repeat <count>. The block is defined by indentation (spaces/tabs).
For-each blocks: you can expand a topic pattern into concrete topic names and run an indented block for each topic using for-each-topic <connectionName>:<topicPattern> as <var>. Inside the block, @@<var> is replaced with the matched topic name.
Import groups: wrap multiple import-data commands in import-group / end-group to run them in parallel.
Import group rules: only import-data commands are allowed inside a group, and nested groups are rejected during validation.
# Send the same operation 3 times
repeat 3
  echo "Round"
  copy-data dev:orders* stage
for-each-topic <connectionName>:<topicPattern> as <var>
  <indented commands using @@<var>>
# Clear matching topics one by one
for-each-topic dev:orders* as t
  echo Clearing @@t
  clear-topics dev:@@t

Topic pattern matching

Patterns are evaluated as:
AND: split by +. Every + part must match.
Exclude: within each + part, you can exclude with *-* (example: *foo*-*bar* means “include foo, exclude bar”).
Wildcards: * is a glob wildcard. If there is no *, the token behaves like a “contains” match.
orders*
*dlq*
orders*+prod
*-*.tmp*
*orders*-*test*

Command reference

echo
Log a message to the task output (quotes are optional).
create-topic
Create one topic on one connection (requires partitions + replication-factor).
create-topics
Create topics on destination based on the source topic specs (optional overrides).
clear-topics
Empty matching topics (does not delete/recreate; clears messages).
delete-topics
Delete matching topics (waits for disappearance).
update-configs
Copy configs from source to destination, or apply fixed configs to a target.
copy-data
Copy records from source topics to destination topics (creates destination topic if missing).
slow-copy
Like copy-data, but sleeps up to a max delay between produced records (maxDelay like 5s or 250ms).
import-data
Load exported message envelopes from a file and produce them to a topic with optional delay and burst controls.
import-group
Run multiple import-data commands in parallel until each file is exhausted.
end-group
Closes an import-group block.

Import-data details

Syntax: import-data "<filePath>" -topic <connection:topic> [-delay <duration | min-max>] [-burst <count | min-max>]
-delay accepts ms, s, m, and hvalues, for example 250ms, 10s, 2m, 1h, or a random range like 1s-5s.
-burst controls how many records are sent each time before the next delay; it accepts a fixed count like 10 or a range like 5-10.
import-group is also line-based: do not place multiple import-data commands on the same line as import-group or end-group.
-topic must always use the explicit connection:topic form, just like the other playbook commands.
The backend checks that the file exists before sending anything. If the file is missing, unreadable, or invalid, the task logs the error and stops.

Accepted file formats

Files can be a single JSON object, a JSON array, or NDJSON. The preferred envelope shape is: { headers, key, messageValue, partition }
headers can be an array from export/import modal format or a simple object map; headers and key may be omitted.
messageValue can be a JSON payload string, XML payload string, or plain text. payload is also accepted.
If a row is just a plain JSON object without envelope fields, it is sent as the message body with no key, no headers, and automatic partitioning.
{"headers":[{"key":"dataType","valueUtf8":"hr-candidate"}],"messageValue":"{\"id\":87831168}","key":"87831168","partition":0}

Examples (copy/paste)

Echo
echo Starting migration
echo "Copying topics..."
Create a single topic
create-topic dev:orders.v1 -partitions=12 -replication-factor=3 -configs cleanup.policy=compact,retention.ms=7d
Notes: -configs is a comma-separated k=v list. For *.ms keys you can use durations like 7d, 1h, 30s.
cleanup.policy must be one of: delete, compact, compact,delete. The UI normalizes delete,compact to compact,delete, but playbook -configs should use the canonical compact,delete explicitly.
Create topics on destination (copy topic specs)
create-topics dev:orders* stage
Optional overrides: -partitions=<n>, -replication-factor=<n>
create-topics dev:orders* stage -partitions=6 -replication-factor=2
Clear topics (empty messages)
clear-topics dev:*dlq*
Delete topics
delete-topics dev:*tmp*
Update configs (copy from source to destination)
update-configs dev:orders* stage
Update configs (apply fixed config values)
update-configs dev:orders* -configs cleanup.policy=compact,retention.ms=7d,replication.factor=3
Tip: replication.factor is handled specially and may be adjusted to the destination broker count.
Tip: cleanup.policy should be provided as compact,delete (not delete,compact).
Copy data
copy-data dev:orders* stage
Slow copy data
slow-copy dev:orders* stage 5s
maxDelay supports ms and s only (example: 250ms, 5s).
Import exported messages from a file
import-data "c:/data/orders.ndjson" -topic stage:pri-ord-orders -delay 1s-5s
-topic must include the connection name, for example stage:pri-ord-orders.
Files can be a single JSON object, a JSON array, or NDJSON in the same envelope format used by the import/export modal.
Import bursts with a fixed batch size
import-data "c:/data/warehouse.json" -topic stage:pri-whr-warehouse -delay 1s -burst 10
Run multiple imports in parallel
import-group
  import-data "c:/data/people.ndjson" -topic pri-hr-employees -delay 10s
  import-data "c:/data/orders.json" -topic pri-ord-orders -delay 1s-5s
  import-data "c:/data/warehouse.json" -topic pri-whr-warehouse -delay 1s-2s -burst 5-10
end-group
One full “migration” playbook (create + configs + data)
# 1) Create destination topics from source specs
create-topics dev:orders* stage

# 2) Ensure configs match (or re-apply)
update-configs dev:orders* stage

# 3) Copy records (will create missing topics too)
copy-data dev:orders* stage
Automationautomation.playbookWizard

Playbook builder wizard

playbookwizardbuildergeneratecreate topicscopy datadelete topics
The Playbook builder wizard helps generate a playbook script from a guided UI, so you don’t have to remember command syntax.

Where to open it

Open it from the Topic list context menu (Playbook builder wizard…) or from the playbook editor (New with Wizard).

What it generates

Depending on the chosen action, the wizard produces commands like create-topics, copy-data, clear-topics, and delete-topics, targeting the topics you selected.
You can save the generated script as a playbook and then run it like any other playbook.
Technicaltechnical.backendApiPort

Backend API port and startup

backendapiportstartupdebug logloopback8082KAFKA_STREAMYARD_API_PORTenvironment variable
Kafka Streamyard runs a local Java backend that listens only on the loopback interface. Development uses port 8082 by default. Packaged applications ask the operating system for an available loopback port each time they start, avoiding conflicts with other software.

Find the current port

Open Settings → Debug → Show Debug Log dock.
When the backend is ready, the log contains an entry like:
Backend API started on port 62614
The number is the actual port in use for the current application session.

Override the port

Set KAFKA_STREAMYARD_API_PORT before launching Kafka Streamyard to use a fixed port in either development or a packaged application. Valid values are integers from 1 to 65535.
macOS / Linux
export KAFKA_STREAMYARD_API_PORT=9090

Windows PowerShell
$env:KAFKA_STREAMYARD_API_PORT="9090"
Restart the application after changing the environment variable. If the configured port is already in use, startup fails with a port-in-use message instead of connecting to the other service.
Unset the variable to restore port 8082 in development or automatic port selection in packaged applications.