Searchable product docs

Kafka Streamyard docs

Browse the same help topics that power the desktop app, rendered as HTML for search engines and easy linking.

Product help
Choose a topic from the left or search for a feature
Selected topic
Overviewoverview.screenTour

Screen tour

layoutscreenpanestopic listresult controlresult viewmessage panelhelp
This is a guided tour of the main Kafka Streamyard screen, using focused screenshots of the main areas you work in most often.

Main screen

This is the main screen. It gives you the overall working layout with topic and navigation tools on the left, active working controls in the middle, and the message panel docked to the right.
In this example you can see a topic selected from the list, matching results returned to the UI, and the selected message opened in the details panel for inspection and testing.
Kafka Streamyard main screen showing the left navigation and topic list, the center result controls and results table, and the message details panel on the right.

Top-left navigation

At the top-left, the static navigation links provide quick access to tools such as Cluster Dashboard, All Consumers, Topic Information and the important Playbooks and Data Samples - also will show Schema Registry editor when configured.
Kafka Streamyard top-left static navigation links including dashboard, consumers, topic information, playbooks, and data samples.

Quick connection bar

This is a quick connection access bar for your most used connections.
Hover over to pop-out an additional screen to the same connection, which is useful if you are doing lots of work and testing on the same connection, live view feeds showing on one screen and you send samples on the other against various topics.
Kafka Streamyard quick connection access bar for frequently used connections.

Hover over to show popout clickable glyph
Kafka Streamyard quick connection pop-out additional connection.

Icons & dropdown

This shows other connections which is for connections you don't use so often, a task icon to show running tasks like copy topic data, repartitioning, playbooks etc, the searchable help, and the settings / connection manager icon.
Kafka Streamyard top-right icons and dropdown showing other connections, tasks, help, and settings.

Topic list

The topic list offers advanced saveable filters that are easily selectable.
Green tabs mean data has been searched and returned to UI and is persisted until cleared.
A blue flash tab shows data is in live feed mode and will show number returned in square brackets.
The number next to topic in orange is number of partitions.
Everything is asynchronous so you can find data on one topic and look for data on another and any will show when data results are returned with the green tab as the indicator.
At the bottom of the topic list shows the filtered vs total amount of topics on this connection.
Kafka Streamyard topic list showing topic filters, tab indicators, partition counts, and topic activity status.

Topic list context menu

On the list is a context menu with many functions available directly from the selected topic.
Kafka Streamyard topic list context menu showing topic actions such as create, import, export, copy, repartition, delete, and playbook tools.

Views

Bottom left shows the Views these are live feeds of 1 or more related topics chronologically returned to the UI.
You can have 20 topics per view and can filter on each one individually using saved filters or immediately added filters.
Kafka Streamyard views area showing multi-topic live views in the bottom-left of the screen.

Result controls and result list

At the top is partition selection, filter (press down key for history and pencil icon to save and select saved filters from any connection or topic).
The Offset function provides targeted functions to select from various offsets.
There is also a live feed button here: Live feed.
You can import or export data from this top area and even customize the result display to make it more user friendly which you can export then as CSV making it great to share with the business.
For filtering on a message with the aid of autocomplete, m. means message, k. means key and h. means header, once the data is 'learnt', meaning you've played some messages back, autocomplete is available on key, value and header.
Kafka Streamyard result controls showing partition selection, filter history, offset functions, live feed, and import/export actions.

Message panel and samples

The all important message panel which can be docked right or bottom. Shows all the info about a topic and it's message, value key and header.
You can trace selected history here which is great when you want a message back but can't remember the offset.
You can copy and paste the whole message in KS json format (holds all the info about a message, partition, offset, timestamp, etc.) or individual parts of the message.
The stand out feature of KS is being able to send samples to the topic for manual testing. We'll do a blog post on this subject as this is one of the main drivers for creating this software.
You can see at the bottom you can have various sample data organised in folders and the order you might want to send them in.
You can also save samples directly here if you have a message that needs investigate and later replaying once a fix is in.
You can create templated messages and assign values automatically and is used in conjunction with Data Samples seen top left.
Kafka Streamyard message panel showing message details, history, and sample playback tools.
Overviewoverview.connectionSettings

Connection settings (Connections Editor)

connectionsconnection settingsconnections editorbootstrap serverssecurityschema registrykafka connectpermissions
The Connections Editor defines how Streamyard connects to Kafka (and optional services like Schema Registry and Kafka Connect). Use the ? popovers next to fields for quick explanations.

Settings tab

Cluster name
Streamyard’s internal name for this connection. Used in tabs, tasks, and quick links. Must be unique.
Bootstrap servers
One or more host:port pairs, comma-separated.
Default function
Default query/time range when opening topics. Example: tail=2000 or since=24h.
Partitioning
Controls how produced records are assigned to partitions when you don’t explicitly choose one.
Default replication factor
Convenience default used by Streamyard when creating new topics (does not change existing topics).
Key/message format
How Streamyard parses and displays message keys/values (Auto is best-effort; explicit formats reduce mis-detection).
Quick links + color
Favorites control whether the connection appears in the top-bar quick links. Optional colors help visually tag environments.

Security tab (Kafka)

Security option selects the authentication mode (Plaintext / SSL / SASL / Confluent / AWS MSK / AWS MSK IAM).
SSL typically requires trust store / key store paths (JKS) and matching passwords.
SASL requires a security protocol (SASL_PLAINTEXT or SASL_SSL) plus a mechanism (PLAIN / SCRAM) and credentials.
AWS MSK IAM uses IAM authentication over TLS (SASL_SSL). See the dedicated AWS MSK IAM (SASL_SSL) setup help entry.

Schema tab (Schema Registry)

Configure Schema Registry only if you use schema-aware formats. Streamyard can validate/format produce payloads using schemas.
Security for Schema Registry is configured separately from Kafka security (often different URLs and credentials).

Connect tab (Kafka Connect REST)

Optional: set the Kafka Connect REST base URL so Streamyard can list connectors and show status.

Permissions tab (guardrails)

These toggles are Streamyard-side defaults that help prevent accidental destructive actions.
You can disable deleting topics, emptying topics, or producing/sending messages even if your Kafka credentials would allow it.

Advanced tab (performance tuning)

Consumer profiles and flush settings trade off throughput vs UI responsiveness. If you see laggy UI or high memory use during Live feeds, adjust these.
This depends heavily on your connection throughput and latency: if the connection is slow / low throughput, you may notice little to no performance difference when changing these.
Overviewoverview.tasks

Tasks (background jobs)

tasksbackgroundjobscopytopic copyingplaybookssample loaderrandomizedrandomiserprogresswaitdelayburstquantity-1until stopped
Tasks are long-running operations executed by the backend. The Tasks window lets you monitor progress and review details/output.

Why Tasks matter

Tasks let you start heavy operations and come back later instead of blocking your main investigation window.
This is useful for long-running topic copies, imports, playbooks, repartitioning, and sample-loader runs where you still want to keep browsing topics or validating results elsewhere in the app.
A shared task history also makes it easier to explain what ran, whether it succeeded, and where it failed.

What runs under Tasks

Topic copying: background copy operations that move/replicate data from one topic to another.
Playbooks running: submitted playbook scripts execute as tasks and stream logs/output as they run.
Randomised data sample loading: sample loader runs can use fixed/random waits, burst sizes like 10 to 20, or -1 quantity to keep producing until you stop the task.
Overviewoverview.awsMskIam

AWS MSK IAM (SASL_SSL) setup

awsmskiamsasl_sslaws_msk_iamprofilecredentialstailscalevpc9098
This guide explains how to connect Streamyard to an AWS MSK cluster that uses IAM authentication (AWS_MSK_IAM) over TLS (SASL_SSL).

1) AWS prerequisites

Your MSK cluster must have IAM authentication enabled and you must use the IAM-SASL bootstrap brokers on port 9098.
Your IAM identity must have permission to access the cluster (for example: list topics, describe cluster, read/write topics as needed).

2) Network prerequisites

IAM-SASL brokers are often private VPC endpoints. Your laptop must be able to route to the VPC.
If you are using Tailscale, make sure you have joined the correct tailnet and are accepting routes (--accept-routes).

3) Configure AWS credentials locally

Streamyard uses the standard AWS credential chain. The easiest approach is to create a named profile (example: oso_prod) in ~/.aws/credentials.
# ~/.aws/credentials
[oso_prod]
aws_access_key_id=...
aws_secret_access_key=...
Sanity check:
aws sts get-caller-identity --profile oso_prod
Security note: treat access keys as secrets and rotate them if they are exposed.

4) Streamyard connection settings

In Connection settings:
Servers
Comma-separated brokers, including port 9098.
Security option
AWS MSK IAM
AWS Region
Example: eu-west-2
AWS Profile
Optional, but recommended (example: oso_prod).
Use Test connection to verify the backend can authenticate and reach the brokers.
Live FeedsliveFeeds.overview

Live Feeds (real-time streaming)

live feedlivestreamfiltercriteriaresult controltopic listresult view

What is a Live Feed?

Live mode keeps a stream open and continuously adds newly received records to the current result list.
You enable Live mode from the Result Control for the currently selected topic.

Why Live Feeds are useful

Live Feeds are useful when you need to watch Kafka while something is actively happening: a tester clicks a button, a user retries a payment, or a service is retrying in the background.
Instead of repeatedly re-running reads, you can keep one filtered stream open and let matching records arrive in real time.
This is especially helpful for production triage, manual verification, and proving whether a message was produced at all versus being produced with the wrong payload.

Common examples

- Follow one customer or order while a tester exercises the UI.
- Watch only retry/error events during an incident.
- Confirm that a new deployment is still emitting the fields and headers you expect.

Start / stop Live mode

In Result Control:
Start: click Live feed.
Stop: click Stop.
While Live (or Play) is running, controls like time range / since are disabled to keep the stream consistent.

Entering the filter (criteria / SL)

The filter is typed into the Criteria input in Result Control.
Criteria uses the same language as the SL reference guide (see help: SL).
m.type == 'order' and m.total >= 100
@partition == 0 and k.customerId.isPresent()

Counts vs. what you see

Topic list live counter: in TopicList, a topic in Live mode shows a live indicator and a bracketed [count] representing how many records have been received for that live session.
Result grid rows: the ResultView only renders rows you are currently viewing (selected topic / selected partitions) and that pass the current Result Control criteria.
If a message is received by the Live feed but fails the criteria filter, it will not appear in the result list.
Live ViewsliveViews.overview

Live Views (multi-topic chronological stream)

live viewsviewslivestreammultiple topicschronologicalfiltercriteria

What is a Live View?

A Live View is a combined stream for multiple topics. It is useful for debugging and monitoring flows that span several topics.
The results are shown together in ResultView. Each row includes its sourceTopicso you can see where it came from.

Why Live Views are useful

Live Views are useful when a business flow spans several topics and you want one investigation timeline instead of jumping between tabs.
They make cross-topic debugging much easier because events from related services interleave naturally in chronological order.
This is often the fastest way to understand whether a record was published, transformed, retried, dead-lettered, or consumed by a downstream step.

Common examples

- Trace an order from command topic to validation topic to fulfillment topic.
- Follow one customer journey across audit, event, and notification topics.
- Watch a retry queue alongside the main topic to understand recovery behavior.

View Result Control

When a View is selected, the toolbar switches to View Result Control.
Use Live feed to start a real-time stream for the view.
Use Stop to stop streaming.
Use Reset to clear the current Live View results and counters without removing the view, topics, or filters. This is useful while a feed is active if you want to start watching from a clean slate.
Use the Wrap toggle to switch line wrapping on or off in the combined results.
The summary on the right shows how many messages were received and how many are currently shown in the active time window.

Chronological ordering

Live Views are intended to be read as a single timeline across topics.
The grid is ordered using message timestamps (and uses offsets/partitions as tie-breakers), so events from different topics interleave naturally.

Selecting topics to display

In View Result Control, use Display topic results to choose which topics are currently shown.
This is a visual aid only. It changes what is displayed in the combined grid, but it does not stop data from arriving into the view.
If a topic belongs to the view and passes its filter, data will still continue to arrive in the background even when that topic is hidden from the current display.

Per-topic filters

Live Views support individual filter criteria per topic. Use Add topic filter… to add a filter row for a specific topic in the view.
It is recommended to first test the filter criteria at the topic results level and, ideally, save it as a filter before applying it in a Live View. This makes it easier to confirm you are matching the results you expect.
To filter all topics in a view, add a filter row for each topic. Each topic keeps its own criteria.
Once a filter row exists, you can type criteria directly, open Filter history, clear the criteria with Clear filter, open Saved filters, or remove that topic filter row with Remove topic filter.
You can also press Down Arrow while focused in a filter input to reopen previously used filters for that topic.
Only messages that pass each topic’s criteria will appear in the combined results.
Filtering can only be changed while the Live View feed is inactive. Stop the feed first if you want to add filters or edit criteria.
Aggregate criteria functions count(...) and countUnique(...) are not available in Live Views.
# Example: keep only errors on one topic
errorsTopic: m.level == 'ERROR'

# Example: track one customer across another topic
ordersTopic: k.customerId == '123'

Topics in a Live View

Topics can be added to or removed from a View only when the Live View is not active. If Live feed is running, stop it before editing the View.
You can edit topics from the Edit view action, or add topics by dragging them into the View when the View is inactive.
Each view can support 20 topics.

Show since and memory usage

Use Show since to define the time window for messages shown in the View. It accepts relative values such as 15m, 6h, and 2d, as well as absolute ISO timestamps.
While Live feed is running, older rows outside that window are dropped from the in-memory View buffer so the grid focuses on recent activity.
On very busy topics, using too large a window may still lead to high memory use because many records can arrive in a short time. Prefer a smaller window for high-volume streams.

Hover preview

Hover over a View in the Views list to open a quick preview.
The preview shows the current status of the View, how many topics it contains, total received and shown counts, and a per-topic breakdown.
This is useful when you want to inspect a View without switching the main results area away from your current selection.

Details pane and layout

For Live Views, the preferred details-pane dock is bottom by default.
This works well because ResultView is usually wider in Live Views due to the included topic name column.
You can still dock the details pane to the right if that better suits your screen or workflow.

Column layout in Live Views

Live Views do not currently support the same column-layout customisation that is available for single-topic result views.
This is mainly because combining multiple topics into one chronological grid makes column selection and formatting more complex.
We may improve this in the future, for example by allowing a View to adopt a compatible layout from one of its topics.
Streamingstreaming.importExport

Import / Export (Result Control)

importexportdownloadndjsoncsvjsonreportresult control
Import/Export lives in the Result Control toolbar (download/upload icons).

Why this is useful

Import/Export helps you move between investigation, reporting, and replay without leaving the current workflow.
CSV reports are useful when you want to share selected columns with testers, analysts, or support teams who do not need the raw Kafka envelope.
JSON and NDJSON exports are useful when you want to archive a slice of data, replay it elsewhere, or use it as seed input for automated test flows.

Export

Export opens a preview modal where you can copy the data or download it as a file.
Formats:
JSON
Exports a JSON array. If you enable envelope options, each row is exported as an object like { messageValue, key, headers, partition } (some fields optional).
NDJSON
Newline-delimited JSON (one JSON object per line). Uses the same row structure as JSON export but without the outer array wrapper.
CSV report (selected view)
Exports a report using the currently selected message list view (the same columns you see in the grid). This is intended for spreadsheets and reporting (not re-import).

Import

Import reads messages from a file and produces them to Kafka for the currently selected topic.
You can import JSON arrays, JSON objects, or NDJSON. You can also provide a separate key file (same number of rows), or generate keys from a template.
If the import file includes a partition field, Streamyard will validate whether those partitions exist on the target topic. If they do, it can preserve partitions; otherwise partitioning is automatic.
Streamingresult.sl

SL (Streamyard Language) reference guide

filtercriteriasloperatorsjsonheadersautocompletehistoryai
SL (Streamyard Language) uses a compact expression syntax. For JSON messages, payload is available as m, key as k, and headers as h.Methods must include parentheses, e.g. toLower() (not toLower).

Why teams use SL

SL is useful because it lets you isolate exactly the messages you care about without writing code or reconfiguring consumers.
In the UI, autocomplete helps you discover fields and headers quickly, filter history helps you bring back useful queries, and optional AI assistance can help draft a starting expression that you then refine.
Typical uses include isolating one customer, finding only failed events, checking that a new field is present, or narrowing a noisy topic down to one business case.
Operators & syntax
==
Equality
m.country == 'Brazil'
!=
Inequality
m.country != 'Brazil'
=
Equality (single = is normalized to ==)
m.country = 'Brazil'
>
Comparison
m.age > 18
>=
Comparison
m.age >= 18
<
Comparison
m.age < 18
<=
Comparison
m.age <= 18
String comparisons use lexicographic ordering
m.someDate > '20250101T1020Z'
and / &&
Boolean logic
@partition == 2 and @offset > 10@partition == 2 && @offset > 10
or / ||
Boolean logic
m.country == 'BR' or m.country == 'US'm.country == 'BR' || m.country == 'US'
in (...)
Membership (operator, not a method)
m.country in ('BR', 'US', 'ZA')m.person.forename.toLower() in ('john', 'thomas')
.
Field access
m.project.detail.id == 'FR'
[n]
Array index
m.orders[0].price > 100
[*]
Array wildcard
m.orders[*].price > 100m.lines[*].description.toLower().contains('training')
[start:length]
Slice (start index + length)
m.name[1:3] == 'att'
Complex examples
m.currency == "EUR" && m.lines[*].description.toLower().contains("training")
@partition == 0 && (k.customerId.isPresent() || h.correlationId.isPresent())
Methods
isPresent()
Missing field checks
m.optionalField.isPresent()
isAbsent()
Missing field checks
m.optionalField.isAbsent()
between(min, max)
Inclusive numeric range check
m.order.orderTotal.between(100.00, 999.99)
isEmpty()
Empty string/list checks
m.name.trim().isEmpty()
isNullOrEmpty()
Empty string/list checks (also true for null/missing)
m.optionalName.isNullOrEmpty()
type()
Type check: returns string, number, boolean, object, array, null, missing, unknown
m.person.forename.type() == 'string'
toString()
Convert to string
h.someHeader.toString() == '123'
toJson()
Parse escaped embedded JSON
h.someHeader.toJson().id == '123'
toLower()
String normalization helpers
m.name.trim().toLower() == 'matthew'
toUpper()
String normalization helpers
m.name.trim().toUpper() == 'MATTHEW'
trim()
String normalization helpers
m.name.trim().toLower() == 'matthew'
contains(substr)
String match helpers
m.name.contains('att')m.lines[*].description.toLower().contains('training')
startsWith(prefix)
String match helpers
m.name.startsWith('Mat')
endsWith(suffix)
String match helpers
m.name.endsWith('hew')
substring(start, len)
Substring with optional length
m.code.substring(1, 3) == 'BCD'
now()
Current epoch time
@timestamp >= now(-1, week)
toMillis(text)
Parse a UTC datetime string to epoch milliseconds
m.createdMillis > toMillis('20250101T102030123Z')m.createdMillis > toMillis('2025-01-01T10:20:30.123Z')
toSeconds(text)
Parse a UTC datetime string to epoch seconds
m.createdSeconds > toSeconds('20250101T102030Z')
toDays(text)
Parse a UTC date/datetime string to epoch days
m.epochDays >= toDays('2025-01-01')
toMicros(text)
Parse a UTC datetime string to epoch microseconds
m.ts_us >= toMicros('2025-01-01T10:20:30.123Z')
toNanos(text)
Parse a UTC datetime string to epoch nanoseconds
m.ts_ns >= toNanos('2025-01-01T10:20:30.123Z')
now(-1h)
Current epoch time with relative offset
@timestamp >= now(-1h)
now(-1, day)
Current epoch time with relative offset
@timestamp >= now(-1, day)
count()
List helpers
m.tags.count() > 0
join(delim)
List helpers
m.tags.join(',') == 'A,B'
Common fields
m / message
Message payload (JSON object for JSON topics; raw string otherwise)
k / key
Key payload (JSON object for JSON keys; raw string otherwise)
h / headers
Header map by key (first header value wins)
@timestamp / timestamp
Record timestamp (epoch ms)
@partition / partition
Partition id
@offset / offset
Record offset
Streamingresult.offsetFunction
Offset Function help
functiontailheadsinceabsutctodayyesterdayoffset
This function determines where the stream starts reading from. Some functions are offset-based (tail/head/offset=...) and some are time-based (since/range/today/yesterday).
Examples
tail=2000
Tail: latest 2000 records per selected partition (offset-based)
tail=1000..2000
Tail range: latest-2000 to latest-1000 per selected partition
head=100
From beginning: first 100 records per selected partition
head=10..20
Head range: offsets 10 to 20 from the earliest offset per selected partition
beginning
Scan forward from the earliest available offsets across all selected partitions
0
Alias for beginning across all selected partitions (auto-normalizes on field exit)
since=24h
Time range: from now - 24h to now
since=48h..24h
Time range: start = now - 48h, end = now - 24h
abs=250220..250224
Absolute time range: Feb 20 to Feb 24 (local time, midnight to midnight)
abs=250220T10:20..250224T02:00
Absolute time range: local time with minutes
utc=250220..250224
Absolute time range: UTC midnight to midnight
today-utc
Time range: start of today in UTC, end = now
yesterday
Time range: previous day
offset=1000..
Offset start: from offset 1000, then read up to max items (requires 1 partition)
offset=1000
Alias for offset=1000.. (single offset start)
o=1000
Alias for offset=1000
s=24h
Alias for since=24h
t=-24h
Alias for since=24h
Topicstopics.filter
Topic filter help
topicfilterbuildertabsAI topic filter assistantFilter samplespresets@wildcardsandornotinternal__connectorskafka connectmirrormakerparenthesesgrouping&&||aliasesmetadata@@partitions@@prt@@cleanupPolicy@@cp@@cleanupPolicy=both@@ignoreInternals@@ii@@ignoreConnectors@@icinexamplessamplesrecipespartitionscleanupcompactdeletebothcompact,deleteboolean
Topic filters use a boolean expression language. Filters are case-insensitive. - has no special meaning. Allowed: letters/digits, ., _, -, spaces, grouping with (), list commas inside in(...), operators &&/and, ||/or, !/not, wildcard *, preset @, metadata @@, and comparisons =, ==, !=, >, >=, <, <=.
The topic filter matches topic names and selected topic metadata such as partitions, cleanup policy, and built-in ignore predicates. The old topic-filter rules based on space/comma for OR and + for AND should be considered retired for topic filtering.
How to think about the language
Build filters in layers: start with a term, combine terms with and / or, remove unwanted matches with not, and add metadata checks with @@... when needed.
Think of it like plain-English logic: orders and not retry, (crm or cust) and @@partitions>=3, @billing and not dlq.
Operator precedence is not first, then and, then or. When a filter mixes AND and OR, use parentheses so the intention is obvious.
Saving
Open Topic filter builder… from the Saved filters menu. The current topic-list filter is loaded into the editor automatically, where you can refine and save it as a preset.
Topic Filter Builder tabs
AI topic filter assistant
Describe the topics you want in plain language. The assistant uses the visible topics, filter syntax, metadata rules, and saved presets to generate a filter in the editor.
Filter samples
Browse grouped, ready-to-adapt examples for wildcards, AND/OR/NOT, nesting, in(...), saved presets, metadata aliases, and complete recipes. Each expression is shown beside an explanation of what it does.
Using presets in the filter
You can reference a saved preset by name using @presetName. Type @ to open autocomplete, then press Enter or Tab to insert.
Presets can be combined with other terms and operators (e.g. @orders and pri) and can be negated (e.g. not @billing or !@billing).
Good preset examples: @core_finance and @@cleanupPolicy=compact, @payments and not in(dlq,retry,error), (@orders or @billing) and @@partitions>2.
Internal topics
The simplest way to hide Kafka internal topics is @@ignoreInternals. It keeps normal topics and filters out internal ones such as __consumer_offsets and __transaction_state.
Name-based filters still work too. Kafka internal topics commonly start with __, so not __* or !__* can also hide them.
To show only internal topics, use __*. Example: __* and not __transaction_state keeps only internal topics but removes one noisy internal topic.
For common Kafka Connect and MirrorMaker system topics, add @@ignoreConnectors. This is intentionally conservative and targets known connector system-topic naming patterns rather than guessing arbitrary business topics produced by connectors.
Basic matching
A plain term matches anywhere in the topic name. orders matches pri-ord-orders, team-orders-live, and orders-dlq.
Terms are not quoted strings and do not need regex syntax. Just type the meaningful fragments that appear in your topic names.
orders
pub-fin
crm
__*
Boolean building blocks
OR
billing or payments or billing || payments matches either alternative.
AND
pri and orders or pri && orders matches only when both terms are present.
NOT
not invoices or !invoices excludes matching topics.
Grouping with parentheses
Use parentheses whenever a filter mixes AND and OR. Example: (crm or cust) and not retry.
Operator precedence
The parser evaluates not first, then and, then or. Even though that works, explicit parentheses are still easier to read and safer to maintain.
Examples
orders and not retry
billing or payments
(crm or cust) and not (dlq or retry)
((crm and cust) or (pub-fin* and payments)) and @@partitions=3
Important: standalone and, or, and not are operators.
Wildcards
* matches anything.
Wildcards work inside terms, so pre*, *acct*, and pub-fin* all work.
Use prefix* for prefix-style matching, *middle* for contains-style matching, and *suffix for suffix-style matching.
If you want to match the literal words and, or, or not in topic names, use wildcard terms such as *and*, *or*, or *not*.
orders**dlq**retry*or* and *and*
Compact lists with in(...)
in(a,b,c) is shorthand for (a or b or c). It is especially useful when excluding several similar topic fragments.
Wildcards are allowed inside the list, so in(ledger,cashflow,*acct*) is valid.
in(crm,cust,payments)not in(dlq,retry,error)-fin- and not in(ledger,cashflow,*acct*)(inv or payments) and not in(calc,line,test)
Metadata filters
Use metadata variables when you want to constrain by topic properties, not just the name.
Comparison metadata variables are @@partitions and @@cleanupPolicy. Standalone boolean predicates are @@ignoreInternals and @@ignoreConnectors.
Short aliases are @@prt for @@partitions, @@cp for @@cleanupPolicy, @@ii for @@ignoreInternals, and @@ic for @@ignoreConnectors.
Supported comparisons are =, ==, !=, >, >=, <, and <=.
Cleanup policy values are delete for delete-only topics, compact for compact-only topics, and both for topics configured with both compact and delete cleanup. @@cleanupPolicy=both matches the Kafka values compact,delete and delete,compact.
@@partitions=3@@prt > 1@@cleanupPolicy==compact@@cleanupPolicy=both@@cp=both@@ignoreInternals@@ignoreInternals and @@ignoreConnectors@@ii and @@icorders and @@cleanupPolicy=delete@billing and @@ignoreInternals and @@partitions>=3
Common mistakes to avoid
Do not use legacy + for AND. Write and or && instead.
Do not use square brackets [] for grouping. Use parentheses ().
Do not use commas as general OR separators outside in(...).
Do not use quotes, regex syntax, SQL fragments, or CEL-style expressions. Keep filters as simple topic-filter expressions.
Use this: orders and not retryNot this: orders + !retryUse this: (crm or cust) and not dlqNot this: [crm cust] !dlq
Filter samples
Learn the filter language from complete examples. Expressions are on the left; their effect is described on the right.
Topic names and wildcards
orders
Matches any topic name containing “orders”. Matching is case-insensitive.
orders*
Matches topic names that start with “orders”.
*orders
Matches topic names that end with “orders”.
*orders*
Same effect as “orders”: matches any topic name containing “orders”, but makes the contains wildcard explicit.
*
Matches every topic.
AND, OR, and NOT
orders and prod
Matches topics whose names contain both “orders” and “prod”.
orders && prod
Symbol form of AND; equivalent to “orders and prod”.
orders or payments
Matches topics containing either “orders” or “payments”.
orders || payments
Symbol form of OR; equivalent to “orders or payments”.
orders AND (payments OR billing) AND NOT retry
AND, OR, and NOT are case-insensitive. Capitalizing them is optional, but makes the filter logic easier to scan.
not retry
Excludes topic names containing “retry”.
!retry
Symbol form of NOT; equivalent to “not retry”.
orders and not (retry or dlq)
Keeps order topics while excluding retry and dead-letter queue topics.
Grouping and nesting
(orders or payments) and prod
Requires “prod” and at least one of “orders” or “payments”.
(prod or stage) and not (retry or dlq)
Selects production or staging topics, then removes retry and DLQ topics.
((orders or payments) and (prod or stage)) and not (retry or dlq)
A nested filter combining two inclusion groups with a grouped exclusion.
Compact lists with in(...)
in(orders,payments,billing)
Shorthand for “orders or payments or billing”.
in(ord*,pay*,*invoice*)
An OR list whose entries can each use wildcards.
prod and in(orders,payments) and not dlq
Requires “prod”, accepts either business area, and excludes DLQ topics.
Saved preset references
@billing
Uses the saved topic-filter preset named “billing”.
@billing and not retry
Reuses a saved preset and adds a local exclusion.
(@billing or @orders) and @@ignoreInternals
Combines saved presets and removes Kafka internal topics.
Topic metadata
@@partitions=3
Matches topics with exactly three partitions.
@@partitions>=3
Matches topics with three or more partitions.
@@cleanupPolicy=compact
Matches topics whose cleanup policy is compact only.
@@cleanupPolicy=both
Matches topics configured with both compact and delete cleanup policies.
@@cp=both
Short alias form of “@@cleanupPolicy=both”.
@@cleanupPolicy!=delete
Excludes topics whose cleanup policy is delete only.
@@ignoreInternals
Keeps non-internal topics and removes Kafka internal topics.
@@ignoreConnectors
Removes recognized Kafka Connect and MirrorMaker system topics.
@@ii and @@ic
Short alias form of “@@ignoreInternals and @@ignoreConnectors”.
@@prt>=3
Short alias form of “@@partitions>=3”.
@@ignoreInternals and @@ignoreConnectors and @@partitions>=3
Keeps larger user topics while excluding internal and recognized connector topics.
Literal operator words
*or*
Matches the literal text “or” inside a topic name instead of using OR.
*and*
Matches the literal text “and” inside a topic name instead of using AND.
*not*
Matches the literal text “not” inside a topic name instead of using NOT.
*or* and *and*
Requires both literal words to appear in the topic name.
Combined recipes
(prod and in(orders,payments)) and not (retry or dlq)
Production order or payment topics without retry or DLQ variants.
(@billing or finance) and @@cleanupPolicy=compact
Billing-preset or finance topics that use compact cleanup.
@@ignoreInternals and @@ignoreConnectors and (auth or user)
User-facing auth topics without Kafka internal or recognized connector topics.
Topicstopics.contextMenu
Topic list context menu
topic listcontext menuright clickempty topicdelete topiccopy topic datapaste topic dataplaybook builderwizard
Right-click a topic row (or the empty area below the list) to open the context menu.
Common actions
Refresh topics
Reload the topic list for the current connection.
Create topic…
Create a new topic (opens the Create Topic modal).
Create based on…
When invoked on a topic row, opens the create dialog using that topic as a template.
Playbook builder wizard…
Open the wizard to generate a playbook based on selected topic(s) and common operations.
Create Topic modal notes
Create topic… starts from defaults; Create based on… pre-fills partitions, replication factor, and configs from the source topic.
Replication factor is validated against the current broker count (if available) and is capped/validated to 1..brokerCount.
Topic configs can be edited either as rows or as JSON. In JSON mode, values are normalized to strings where possible and *.msvalues accept convenience durations.
cleanup.policy is normalized and must be one of delete, compact, or compact,delete.
Live feed action
Stop live feed appears only for topics that are currently in Live mode.
Copy / paste topic data
Copy topic data stores a source reference (connection + topic).
Paste topic data copies from the last copied topic into the selected destination topic. It is disabled if nothing has been copied yet, or if you try to paste into the same topic.
If you right-click the empty area (not a specific topic), you can use Paste topic data… to paste into a new destination topic name.
Danger zone
Empty topic
Clear topic data (confirmation required). This may be disabled by settings.
Delete topic
Delete the topic (confirmation required). This may be disabled by settings.
Copy helpers
Copy topic name
Copy the selected topic name.
Copy topic name & config (JSON)
Copy a JSON object containing the topic name and known configs.
Copy all visible topic names
Copy the currently visible topic names (after topic list filtering), one per line.
Copy all visible topic names & config (JSON)
Copy a JSON array for all currently visible topics.
Topicstopics.acls
Kafka ACLs (Access Control Lists)
aclaclssecurityauthorizationprincipalresourceoperationallowdenyprefixedliteral
Kafka ACLs are server-side authorization rules. They determine whether a principal (user/service) is allowed to perform an operation on a resource (topic, group, cluster, etc.).
How to read an ACL row
Principal
The identity (often User:alice or a service account) that the rule applies to.
Resource
What is being protected (Topic / Group / Cluster / TransactionalId). Topics and groups are the most common.
Name + Pattern
Resource name with a pattern type. Literal matches exactly; Prefixed matches any name that starts with the prefix.
Host
Client host/IP constraint (* means any host).
Operation
The action (Read / Write / Describe / Create / Alter / Delete / All, etc.).
Permission
Allow or Deny. If any matching Deny applies, the request is denied. If no matching Allow exists, the request is denied.
Using the ACLs pane
The ACLs pane lists ACLs for the active connection and supports filtering by principal/resource/operation.
+ Create ACL opens a form to add an ACL entry. Creating ACLs usually requires admin-level privileges on the cluster.
Topicstopics.kafkaConnectors
Kafka Connectors (Kafka Connect REST)
kafka connectconnectorsconnectorrestpauseresumerestartdeletecreatetemplatevalidate
The Kafka Connectors pane is a UI for listing and managing Kafka Connect connectors via the Kafka Connect REST API.
Prerequisite: Connect URL
Configure the Connect REST base URL in Connection settings (Kafka Connect tab). Typically this is port 8083.
If your Connect endpoint is protected, set the optional API key/secret (HTTP Basic auth).
Reading the table
Status is shown for the connector and its tasks (running/paused/failed). Failed tasks are the most common indicator that the connector is unhealthy.
Use the search box to filter by name, type, plugin class, or status.
Common actions
Pause / Resume
Stop/start work without deleting the connector.
Stop
Stop a connector (if supported by the Connect API / configuration).
Restart
Restart connector and tasks (useful after fixing a config or external dependency).
Delete
Remove the connector from Kafka Connect.
Creating connectors (Create modal)
The Create modal lets you pick a connector plugin and provide Config JSON. The selected plugin supplies connector.class automatically.
Use load template to populate a starting config (when available), and validate to run server-side validation before creating.
Validation may report errors but still allow creation; Kafka Connect may still reject the final request depending on plugin requirements.
Common errors
401/403: Connect REST authentication is wrong or missing.
Connection refused / timeout: Connect URL is wrong, unreachable, or blocked by network rules.
Validation errors: required fields for the selected plugin are missing or have invalid values.
Topicstopics.topicSettings
Topic Settings (per-topic overrides)
topic settingsguardrailspermissionsallow deleteallow emptyallow sendsamplespersistconfirm before sendpartitioningformatschema
Topic Settings lets you override connection defaults for a single topic. This is useful for applying stricter guardrails or different performance tuning on one topic without changing the entire connection.
Guardrails (destructive actions)
Settings like Delete topic, Empty topic, and Send messages can be overridden per topic.
Most toggles support inherit so the topic follows the connection defaults.
Consumer profiles & tuning
Consumer tuning values (poll time, max poll records, and timeouts) can be overridden per topic.
Larger profiles improve throughput but can reduce UI responsiveness and increase memory use during Live feeds.
Partitioning & formats
Topic Settings can control partition calculation/partitioner behavior, plus Key/Message formats. This affects how messages are displayed and how produced records choose partitions.
If you use schema formats, set the Key/Message schema ids for the topic.
Topicstopics.partitionInfo
Partition Information (offsets, counts, and disk usage)
partition infopartitionoffsetsearliestlatestcountdisk usagecompact
Partition Information is a per-topic breakdown showing per-partition offsets and (where supported) record counts and disk usage.
Offsets (earliest vs latest)
Earliest is the lowest available offset; Latest is the end offset (the next offset that would be written).
For non-compacted topics, an approximate message count per partition is often latest - earliest.
Compacted topics
On compacted topics, offset deltas do not represent current record count reliably (compaction removes older keys).
In that case, the UI may use scan count to compute a more accurate count.
Disk usage
Disk usage is loaded per topic and partition when supported by the backend. Some connection types (for example Confluent) may not provide disk usage.
Topicstopics.information
Topic Information (audit topics at a glance)
topic informationtopicspartitionsreplication factorimbalanceskewoffsetsrecordsdiskconfigscolumnssortingfilterhot partitions
Topic Information is a quick “audit table” for your cluster: a sortable, filterable list of topics with partitions, offsets, record estimates, disk usage (if available), and selected topic config values.
Common uses
- Check what topics are populated (records/offsets) and how many partitions they have
- Spot unbalanced partitions (Imbal / hot partitions highlight)
- Quick eyeballing of key parameters (partitions, RF, cleanup policy, retention, min ISR, etc.)
- Filter topics by selected config values and derived metrics using the column filter
- Sort to find inconsistencies across configs, partitions, replication factor, records, or disk usage
- Estimate average message size (Avg msg) to spot unusually large payload topics
Filtering
Filter topics narrows the topic list by name. It supports the same preset-style filtering as the topic list (type @ to autocomplete saved presets).
Filter columns matches against the currently visible columns (offsets/records/disk and your selected config columns). This is useful for finding outliers by value.
Sorting & columns
Click any column header to sort (click again to toggle asc/desc). Use Columns to add/remove topic config columns (for example retention.ms or min.insync.replicas).
Partition breakdown
Expand a topic row to see a per-partition breakdown (begin/end offsets, record deltas, and disk usage where available). Hot partitions are highlighted when they exceed the average load.
Imbal is a quick skew factor:maxPartitionDelta / avgPartitionDelta, wheredelta = latestOffset - earliestOffset per partition. 1.0x is perfectly balanced; higher values indicate a few partitions hold much more data.
Avg msg is computed as totalBytes / recordCount when both inputs are available.
For compact topics, recordCount is not reliably represented by offset deltas (because compaction can remove older records). In those cases, you may see scan count, which runs a scan to compute a more accurate current record-count estimate for the topic.
For compact topics, Imbal is computed from disk-by-partition skew (because offsets are not a reliable proxy for current records after compaction).
Topicstopics.allConsumers
All Consumers (consumer groups across topics)
consumersconsumer groupsoffsetslagcommittedendclienthostnavigatetopic consumers
All Consumers shows consumer groups across all topics in the selected connection. It’s a quick way to spot lagging groups and see where they are consuming.
Navigation to Topic Consumers
Click a Topic value to open that topic and jump directly to its Consumers tab.
Filtering & grouping
Use the filter to match group id, topic, partition, state, client id, and host. Filtering is case-insensitive.
Use Group to group rows by consumer group and reduce repeated group ids.
Offsets and lag
Committed is the last committed offset for the group; End is the current partition end offset; Lag is the difference.
Deleting a consumer group is only possible when it is empty (no members attached).
Topicstopics.topicConsumers
Topic Consumers (per-topic offsets and lag)
topic consumersconsumersconsumer groupsoffsetslagcommittedendearliestlatestlockedclient attached
Topic Consumers shows consumer group offsets for a single topic, broken down by partition.
Why offsets can be locked
An offset can only be changed when there is no client attached to the consumer group (i.e. there is no active application consuming).
When a client is attached, the row is shown as locked and the Apply action is disabled.
Changing offsets (earliest / latest / manual)
Use earliest / latest to jump quickly, or type a specific offset and press Apply.
Use the + / − buttons to nudge the draft offset and to load an example record into the Inspect message panel.
Messagesmessages.details
Message panels (details, editing, samples, and sending)
messagedetailssendtesttombstonesamplesfavoriteshistorycopypasteformatautosession varssession variablesvariablesparent samplerecalc@@s
The Message panel exists in two layouts (right dock and bottom dock), but both support the same actions for inspecting a selected message and using it as a base for producing new messages.
Why this panel is useful
The Message panel is where investigation turns into action: you can inspect what happened, copy the exact payload shape, adjust it, and replay or produce a new message.
It is especially useful for recreating a bug, testing a downstream fix, or building a reusable library of sample flows for your team.
Common examples include tweaking one field in a real message, saving a good sample as a favorite, or applying a template repeatedly with different session variables.
Selection history
UseBack/Forwardto navigate recently selected messages.
UseRecentto open a menu of recent selections (typically the last ~10). This is handy for reusing a message as a sendable sample.
Formats (Key/Message type)
The panel shows the currently selected Key/Message format. Click the   if the format is wrong, it will try to detect the correct format.
Editing, copy/paste, and templates
The panel has tabs for Message, Key, and Headers. You can copy and paste per-tab.
Template resolution happens in a couple of layers. One layer is @@s.* session vars (described below), and another layer is the general template engine for tokens like @@guid(...), @@rnd(...), etc. Use Test to preview the fully-resolved payload.
Session vars (sample variables) and @@s.someVar
Session vars are stored on a data sample (in the Sample Editor under variables). When you pick an “Apply sample” in the Message panel, those session vars become available to the editor.
Use @@s.someVar anywhere in:
Headers
Header values can include @@s.*.
Key
The key editor can include @@s.*.
Message
The message editor can include @@s.*.
{
  "orderId": "@@s.orderId",
  "customerId": "@@s.customerId"
}
If a session var is missing, Streamyard replaces it with ??@@s.someVar?? and shows a warning listing the missing tokens.
Parent samples (inherit variables)
Samples can have a Parent Sample. Session vars are resolved across the chain:
Order
Parent vars are applied first, then the child sample overrides any keys with the same name.
Scope
This inheritance is for variables. Key/value/headers do not automatically merge from parent samples.
Circular parent chains are blocked by the Sample Editor.
Recalc (stable vs. regenerated values)
Each session var can be marked Recalc in the Sample Editor.
Recalc off
The computed value is cached (per renderer session). This is useful for IDs that should stay consistent while you test/send repeatedly.
Recalc on
The value is recomputed every time session vars are resolved. This is useful for values that should change each send (e.g. timestamps or random numbers).
Session var values can also reference other session vars, so you can compose values like @@s.traceId inside@@s.correlationId.
When are session vars applied?
In both layouts (right dock and bottom dock), the flow is the same:
1) Apply session vars
Replace any @@s.* tokens in headers/key/value using the selected “Apply sample” (including its parent chain).
2) Resolve remaining templates
The template engine resolves other @@ tokens.
3) Use the resolved output
Test shows the resolved output. Send produces the resolved output. Calc partition (when available) also uses the resolved key/headers so the partition calculation matches what will actually be sent.
Samples & favorites
You can load a saved sample to overwrite the current Key/Message/Headers, save the current content as a sample, and mark samples as favorites for quick access.
Favorite sample
Favorited samples are scoped by connection/topic context (tags) and are ideal for maintaining a reusable “send library”.
In both the favorites row and the sample dropdown menu, there is a small <> (code) action next to each sample. Clicking it loads the raw sample text into the editor (no resolution of @@ tokens, and no substitution of @@s.* session vars). This is useful when you want to edit templates in their original form.
Test / Send / Tombstone
Use Test to resolve templates and optionally validate JSON before sending.
SendTombstone
If your message uses template tokens like @@, use Test to preview the fully resolved payload before sending.
Test
Use Send to produce to the current topic. If the topic is compacted, you can also send a tombstone (null value) for a key.
You can also pick a destination partition. Default lets the producer choose the partition (typically key-based when a key is present, otherwise the producer may distribute records). Manual lets you force an explicit partition number (0..N-1).
Copy full message / reuse elsewhere
Use Copy all to copy the full message details as JSON (meta + headers + key/value). This is useful for creating a data sample, or pasting into other tools.
Copy all
You can also copy individual tabs, and paste into another connection/topic workflow using the “paste into new topic” tools.
Messagesmessages.partitioning
Partitioning (producing messages)
partitionpartitioningpartitionersendproducedefaultpartitionerroundrobinrandommodulocrc32fnvjavahash
Partitioning controls which topic partition a produced record is written to. In Streamyard, there are two ways a partition can be chosen:
Manual: you explicitly pick a partition number.
Automatic: the producer/partitioner decides based on the selected strategy.
Manual partition (always deterministic)
If you pick Partition: N, Streamyard sets the record partition explicitly.
This overrides any partitioner strategy.
Automatic strategies (when you do not pick a partition)
DefaultPartitioner
Kafka default partitioner. When a key is present, it is key-based. When the key is missing, Kafka may distribute records in a way that is not strictly predictable (producer-dependent).
RoundRobinPartitioner
Uses org.apache.kafka.clients.producer.RoundRobinPartitioner. Records are spread across partitions over time; the exact partition for a given key is not deterministic.
Random
Chooses a random partition. This is indeterministic and cannot be calculated ahead of time.
Numeric Modulo
Interprets the key as an integer (up to a long/big int) and uses key % partitionCount (supports negative keys). If the key is not numeric, Streamyard cannot calculate a partition.
JavaHash
Uses Java string hashCode() of the key, then maps it to a partition.
CRC32
Uses CRC32 of the key bytes, then maps it to a partition.
FNV-1a
Uses FNV-1a of the key bytes, then maps it to a partition.
Current Key Calc
For calculatable strategies, Streamyard can compute the expected destination partition for the current key.
Clicking Current Key Calc will calculate the partition and switch the UI to a manual partition.
Strategies like RoundRobinPartitioner and Random are indeterministic and cannot be calculated.
Data toolsdata.sampleLoaderPartitioning
Sample Loader help (wait time, quantity, partitioning)
sample loaderpartitioningpartitionermodulusnumeric modulomanualtemplates@@m@@keywaitdelayburstquantity-1until stoppedmssh
Sample Loader produces messages as a background task. Keys and values may contain templates (for example @@rnd(...) or @@m.id) which are resolved by the backend per produced record.
Why teams use Sample Loader
Sample Loader is useful when you want realistic traffic without hand-producing every record one by one.
It can simulate steady background activity, bursty workloads, or endless test data streams while you validate consumers, dashboards, alerts, and retry behaviour.
Common uses include load-testing a dev workflow, generating many variants of a business object, or keeping a topic active while a tester validates end-to-end behaviour.
Wait time + quantity
Waits are optional. Leave the wait checkbox off when you want Sample Loader to send as fast as possible.
Burst from and Burst to control how many records are sent together before the next optional wait. The default is 1 to 1.
Wait from and Wait to accept ms, s, m, and h.
Use one value for a fixed cadence (for example 1000ms, 1s, or 5m), or set a range such as 10s to 15s for a random wait between produced messages.
Set burst mode to values like 10 and 20 to send a random batch of 10..20 messages together each time.
Enter -1 in Quantity to load to keep producing until you stop/delete the task from the Tasks window.
Numeric Modulo
Numeric Modulo requires the resolved key to be a valid signed 64-bit integer (Java long).
If your key is templated (example: @@m.id), it is validated after template resolution. If the resolved value is not numeric, the task will fail and the error will be shown in the Tasks window.
# Example
value: {"id": @@rnd(10000000,99999999)}
key: @@m.id  # OK (id resolves to digits)
Manual
Manual partitioning requires an explicit partition number (0..N-1). Sample Loader will validate the partition exists on the target topic.
Default / Round-robin / Random
These strategies do not require numeric keys. Random and Round-robin are indeterministic and cannot be predicted ahead of time.
Data toolsdata.variableSubstitution
Variable Substitution Help
sampledatasubstitutionrandomguidheaderskeyvaluepaymentshipmentproductorganizationaccountsubscription
Overview
You can use @@ template tokens inside headers, key, and value. They are resolved by the backend endpoint POST /api/message-template/resolve.
The goal is to quickly generate synthetic test data and to reuse values across fields (e.g. use the same generated GUID in the key and in the JSON value).
Why this is useful
Variable substitution is useful when you need realistic but disposable business data for testing, demos, or replay scenarios.
It helps you keep related fields in sync, such as using the same generated id in the key, headers, and message body.
Typical examples include generating people, orders, invoices, payments, shipments, and products without maintaining a giant folder of hand-written JSON files.
Resolution order (important)
Resolution happens in passes so you can reference other fields:
Pass 1: Generators (randoms / timestamps / composite generators like @@person.*, @@organization.*, @@account.*, @@subscription.*, @@payment.*, and @@shipment.*).
Pass 2: Context references (read values from headers / key / message JSON): @@h.*, @@k.*, @@m.*, @@keyRaw.
Conditional pass: @@if(...) is resolved and then the chosen branch is re-resolved (generators + references again). This is repeated a few times to catch nested expressions.
If a reference cannot be resolved (unknown token / missing header / invalid JSON), the token is usually left as-is (so you can see what didn’t resolve).
Quick examples (copy/paste)
Generate an id once, reuse everywhere
key: order-@@guid(1)
value: {"id":"@@guid(1)","createdAt":"@@now.toIsoDate()"}
Pick from a list
value: {"env":"@@rndText('dev','stage','prod')"}
Counter for sample playback
key: order-@@count(1000,1)
value: {"seq":@@count(1,0.0001)}
Reference a header
headers: {"x-correlation-id":"@@guid()"}
value: {"correlationId":"@@h.x-correlation-id"}
Conditional based on message JSON
value: {"kind":"@@if(m.type,'invoice','invoice','order')","ref":"@@if(m.type,'invoice', rnd(10), rnd(8))"}
Generators
@@guid()
Random UUID string
@@guid(1)
Stable slot UUID (same index produces the same UUID within one resolve call)
@@guid().toLower()
Format UUID casing (also supports .toUpper())
@@rnd(6)
Random digits (0 .. 10^n-1)
@@rnd(100,999)
Random inclusive numeric range (min/max order doesn’t matter)
@@rnd(6).toPadded(6)
Pad with zeros (useful for fixed-width ids)
@@now
Current time as ISO instant string
@@now.toMillis()
Unix epoch milliseconds (also toSeconds() and toIsoDate())
@@rndText('a','b','c')
Pick one value at random (also accepted as alias: @@rnd('a','b'))
@@count(start,step)
Stateful counter for sample playback. Starts at start and increments by step per produced message (stable within one message).
@@rndChars(12).toMixed()
Random token-like string (also supports toUpper() / toLower())
@@rndAlphas(8)
Random alphanumeric string (no hyphen/underscore)
Random data generators
Random data generators create a shared context the first time they are used in a resolve call, so related fields stay consistent. All generated values are synthetic and random, and any resemblance to real people or companies is purely coincidental.
These JSON examples can be pasted into a key, message, or a header value. You can also use individual fields directly as tokens such as @@person.fullName or @@shipment.status.
Person
{
  "name": "@@person.fullName",
  "email": "@@person.email",
  "country": "@@person.country",
  "phone": "@@person.phone"
}
Order / Invoice
{
  "orderNo": "@@order.number",
  "lines": @@order.lines,
  "invoiceNo": "@@invoice.number"
}
Payment
{
  "id": "@@payment.id",
  "amount": "@@payment.amount",
  "currency": "@@payment.currency",
  "payer": "@@payment.payerEmail"
}
Shipment
{
  "tracking": "@@shipment.trackingNumber",
  "status": "@@shipment.status",
  "carrier": "@@shipment.carrier",
  "events": @@shipment.events
}
Product
{
  "sku": "@@product.sku",
  "name": "@@product.name",
  "price": "@@product.price",
  "tags": @@product.tags
}
Device
{
  "type": "@@device.type",
  "hostname": "@@device.hostname",
  "ip": "@@device.ip",
  "mac": "@@device.mac",
  "os": "@@device.os",
  "related": @@device.related,
  "interfaces": @@device.interfaces
}
Support ticket
{
  "id": "@@ticket.id",
  "status": "@@ticket.status",
  "priority": "@@ticket.priority",
  "reporter": "@@ticket.reporterName",
  "tags": @@ticket.tags
}
Organization
{
  "id": "@@organization.id",
  "name": "@@organization.name",
  "industry": "@@organization.industry",
  "hq": @@organization.headquarters
}
Account
{
  "id": "@@account.id",
  "number": "@@account.number",
  "owner": "@@account.ownerEmail",
  "tags": @@account.tags
}
Subscription
{
  "id": "@@subscription.id",
  "plan": "@@subscription.planName",
  "amount": "@@subscription.amount",
  "features": @@subscription.features
}
Composite generator fields (full reference)
Note: @@gen.<field> is accepted as a backwards-compatible alias of @@person.<field>.
@@person.*
@@person.title
@@person.forename
@@person.firstname
@@person.first
@@person.surname
@@person.lastname
@@person.last
@@person.name
@@person.fullName
@@person.hobby
@@person.hobbies
@@person.profession
@@person.employmentStatus
@@person.employmentHistory
@@person.employment_history
@@person.workHistory
@@person.work_history
@@person.careerHistory
@@person.career_history
@@person.address1
@@person.address_line1
@@person.line1
@@person.street
@@person.address2
@@person.address_line2
@@person.line2
@@person.city
@@person.town
@@person.county
@@person.region
@@person.state
@@person.province
@@person.postcode
@@person.postal
@@person.zip
@@person.country
@@person.phone
@@person.phone_number
@@person.email
@@person.username
@@person.user
@@person.login
@@person.lat
@@person.latitude
@@person.long
@@person.lng
@@person.longitude
@@person.countryCode
@@person.country_code
@@person.dob
@@person.date_of_birth
@@person.dateOfBirth
@@person.age
@@person.age_years
@@person.ageYears
@@person.gender
@@person.ssn
@@person.national_id
@@person.nationalId
@@person.driver_license
@@person.driverLicense
@@person.driving_license
@@person.drivingLicense
@@person.passport_number
@@person.passportNumber
@@person.passport
@@person.vat
@@person.vat_number
@@person.vatNumber
@@person.tax
@@person.tax_id
@@person.tax_number
@@person.taxNumber
@@person.employmentHistory resolves to a JSON array string with entries like [{"type":"employment","status":"employed","fromDate":"2022-03-01","toDate":"2024-11-30","title":"Software Engineer","organization":"Northstar Systems","employmentType":"full-time","salaryPeriod":"annual","salary":84250}].
@@organization.*
@@organization.id
@@organization.organizationId
@@organization.organization_id
@@organization.orgId
@@organization.name
@@organization.organizationName
@@organization.organization_name
@@organization.orgName
@@organization.legalName
@@organization.legal_name
@@organization.industry
@@organization.sector
@@organization.segment
@@organization.domain
@@organization.website
@@organization.url
@@organization.country
@@organization.countryCode
@@organization.country_code
@@organization.phone
@@organization.phone_number
@@organization.registrationNumber
@@organization.registration_number
@@organization.companyNumber
@@organization.company_number
@@organization.vatNumber
@@organization.vat_number
@@organization.vat
@@organization.taxNumber
@@organization.tax_number
@@organization.taxId
@@organization.currency
@@organization.employeeCount
@@organization.employee_count
@@organization.employees
@@organization.annualRevenue
@@organization.annual_revenue
@@organization.revenue
@@organization.tags
@@organization.tagsJson
@@organization.tags_json
@@organization.headquarters
@@organization.hq
@@organization.tags resolves to a JSON array string, and @@organization.headquarters resolves to a JSON object string.
@@account.*
@@account.id
@@account.accountId
@@account.account_id
@@account.number
@@account.accountNumber
@@account.account_number
@@account.externalId
@@account.external_id
@@account.crmId
@@account.status
@@account.state
@@account.type
@@account.accountType
@@account.account_type
@@account.segment
@@account.region
@@account.lifecycleStage
@@account.lifecycle_stage
@@account.stage
@@account.openedAt
@@account.opened_at
@@account.createdAt
@@account.created_at
@@account.renewalAt
@@account.renewal_at
@@account.nextBillingAt
@@account.next_billing_at
@@account.currency
@@account.balance
@@account.outstandingBalance
@@account.outstanding_balance
@@account.creditLimit
@@account.credit_limit
@@account.paymentTermsDays
@@account.payment_terms_days
@@account.paymentTerms
@@account.riskScore
@@account.risk_score
@@account.ownerName
@@account.owner_name
@@account.primaryContactName
@@account.primary_contact_name
@@account.ownerEmail
@@account.owner_email
@@account.primaryContactEmail
@@account.primary_contact_email
@@account.organizationName
@@account.organization_name
@@account.orgName
@@account.organizationId
@@account.organization_id
@@account.orgId
@@account.tags
@@account.tagsJson
@@account.tags_json
@@account.tags resolves to a JSON array string.
@@subscription.*
@@subscription.id
@@subscription.subscriptionId
@@subscription.subscription_id
@@subscription.number
@@subscription.accountNumber
@@subscription.account_number
@@subscription.account
@@subscription.organizationName
@@subscription.organization_name
@@subscription.orgName
@@subscription.planCode
@@subscription.plan_code
@@subscription.planName
@@subscription.plan_name
@@subscription.plan
@@subscription.status
@@subscription.state
@@subscription.billingPeriod
@@subscription.billing_period
@@subscription.term
@@subscription.channel
@@subscription.source
@@subscription.quantity
@@subscription.seats
@@subscription.autoRenew
@@subscription.auto_renew
@@subscription.startedAt
@@subscription.started_at
@@subscription.startDate
@@subscription.start_date
@@subscription.trialEndsAt
@@subscription.trial_ends_at
@@subscription.trialEnd
@@subscription.trial_end
@@subscription.renewalAt
@@subscription.renewal_at
@@subscription.nextBillingAt
@@subscription.next_billing_at
@@subscription.currency
@@subscription.amount
@@subscription.grossAmount
@@subscription.gross_amount
@@subscription.value
@@subscription.mrr
@@subscription.monthlyRecurringRevenue
@@subscription.monthly_recurring_revenue
@@subscription.features
@@subscription.featuresJson
@@subscription.features_json
@@subscription.features resolves to a JSON array string.
@@order.*
@@order.number
@@order.date
@@order.country
@@order.countryCode
@@order.currency
@@order.vatRate
@@order.subtotalNet
@@order.totalVat
@@order.totalGross
@@order.lines
@@order.lines resolves to a JSON array string.
@@invoice.*
@@invoice.number
@@invoice.date
@@invoice.dueDate
@@invoice.country
@@invoice.countryCode
@@invoice.currency
@@invoice.vatRate
@@invoice.subtotalNet
@@invoice.totalVat
@@invoice.totalGross
@@invoice.lines
@@invoice.lines resolves to a JSON array string.
@@payment.*
@@payment.id
@@payment.paymentId
@@payment.payment_id
@@payment.number
@@payment.createdAt
@@payment.created_at
@@payment.date
@@payment.processedAt
@@payment.processed_at
@@payment.settledAt
@@payment.settled_at
@@payment.capturedAt
@@payment.status
@@payment.state
@@payment.method
@@payment.paymentMethod
@@payment.provider
@@payment.gateway
@@payment.currency
@@payment.amount
@@payment.grossAmount
@@payment.feeAmount
@@payment.fee
@@payment.netAmount
@@payment.reference
@@payment.paymentReference
@@payment.authCode
@@payment.authorizationCode
@@payment.authorisationCode
@@payment.cardLast4
@@payment.last4
@@payment.maskedPan
@@payment.masked_pan
@@payment.payerName
@@payment.payer_name
@@payment.customerName
@@payment.payerEmail
@@payment.payer_email
@@payment.customerEmail
@@shipment.*
@@shipment.id
@@shipment.shipmentId
@@shipment.shipment_id
@@shipment.number
@@shipment.tracking
@@shipment.trackingNumber
@@shipment.tracking_number
@@shipment.carrier
@@shipment.service
@@shipment.shippingService
@@shipment.status
@@shipment.state
@@shipment.shippedAt
@@shipment.shipped_at
@@shipment.dispatchAt
@@shipment.estimatedDeliveryAt
@@shipment.estimated_delivery_at
@@shipment.eta
@@shipment.destinationCountry
@@shipment.destination_country
@@shipment.destinationCountryCode
@@shipment.destination_country_code
@@shipment.country
@@shipment.countryCode
@@shipment.warehouse
@@shipment.warehouseCode
@@shipment.warehouse_code
@@shipment.packageCount
@@shipment.package_count
@@shipment.packages
@@shipment.weightKg
@@shipment.weight_kg
@@shipment.weight
@@shipment.shippingCost
@@shipment.shipping_cost
@@shipment.cost
@@shipment.currency
@@shipment.events
@@shipment.trackingEvents
@@shipment.tracking_events
@@shipment.events resolves to a JSON array string.
@@product.*
@@product.sku
@@product.id
@@product.productId
@@product.product_id
@@product.name
@@product.title
@@product.category
@@product.subcategory
@@product.sub_category
@@product.type
@@product.brand
@@product.vendor
@@product.manufacturer
@@product.status
@@product.availability
@@product.currency
@@product.price
@@product.unitPrice
@@product.unit_price
@@product.listPrice
@@product.list_price
@@product.taxRate
@@product.tax_rate
@@product.vatRate
@@product.inventoryLevel
@@product.inventory_level
@@product.stock
@@product.onHand
@@product.reorderLevel
@@product.reorder_level
@@product.reorderPoint
@@product.warehouseBin
@@product.warehouse_bin
@@product.bin
@@product.location
@@product.barcode
@@product.ean
@@product.gtin
@@product.tags
@@product.tagsJson
@@product.tags_json
@@product.tags resolves to a JSON array string.
@@device.*
@@device.type
@@device.deviceType
@@device.device_type
@@device.category
@@device.vendor
@@device.manufacturer
@@device.make
@@device.brand
@@device.model
@@device.os
@@device.osName
@@device.os_name
@@device.osVersion
@@device.os_version
@@device.hostname
@@device.host
@@device.name
@@device.domain
@@device.fqdn
@@device.mac
@@device.macAddress
@@device.mac_address
@@device.ip
@@device.ipv4
@@device.ipv4Private
@@device.privateIp
@@device.private_ip
@@device.lanIp
@@device.lan_ip
@@device.ipv4Public
@@device.publicIp
@@device.public_ip
@@device.wanIp
@@device.wan_ip
@@device.ipv6
@@device.subnet
@@device.subnetCidr
@@device.subnet_cidr
@@device.cidr
@@device.gateway
@@device.gatewayIp
@@device.gateway_ip
@@device.protocol
@@device.port
@@device.serial
@@device.serialNumber
@@device.serial_number
@@device.assetTag
@@device.asset_tag
@@device.asset
@@device.imei
@@device.cpu
@@device.cpuModel
@@device.cpu_model
@@device.cores
@@device.cpuCores
@@device.cpu_cores
@@device.memoryGb
@@device.memory_gb
@@device.ramGb
@@device.ram_gb
@@device.storageGb
@@device.storage_gb
@@device.diskGb
@@device.disk_gb
@@device.rack
@@device.datacenter
@@device.dc
@@device.related
@@device.interfaces
@@device.related resolves to a JSON object string, and @@device.interfaces resolves to a JSON array string.
@@ticket.*
@@ticket.id
@@ticket.ticketId
@@ticket.ticket_id
@@ticket.number
@@ticket.createdAt
@@ticket.created_at
@@ticket.created
@@ticket.openedAt
@@ticket.opened_at
@@ticket.updatedAt
@@ticket.updated_at
@@ticket.updated
@@ticket.lastUpdated
@@ticket.last_updated
@@ticket.dueAt
@@ticket.due_at
@@ticket.due
@@ticket.status
@@ticket.state
@@ticket.priority
@@ticket.severity
@@ticket.category
@@ticket.type
@@ticket.source
@@ticket.channel
@@ticket.queue
@@ticket.team
@@ticket.tags
@@ticket.tagsJson
@@ticket.tags_json
@@ticket.subject
@@ticket.title
@@ticket.description
@@ticket.body
@@ticket.reporterName
@@ticket.reporter_name
@@ticket.requester
@@ticket.requesterName
@@ticket.reporterEmail
@@ticket.reporter_email
@@ticket.requesterEmail
@@ticket.assigneeName
@@ticket.assignee_name
@@ticket.agent
@@ticket.agentName
@@ticket.assigneeEmail
@@ticket.assignee_email
@@ticket.agentEmail
@@ticket.sla
@@ticket.slaMinutes
@@ticket.sla_minutes
@@ticket.tags resolves to a JSON array string.
References (read from key/value/headers)
Use these to reuse already-generated values (or values you typed in yourself).
@@keyRaw
Raw key string
@@h.x-foo
Header reference (recursive: if the header value contains tokens, it is resolved too)
@@k.id
Read a field from key JSON (supports nested paths: @@k.customer.id)
@@m.type
Read a field from message JSON (supports nested paths: @@m.person.identity.age)
Note: JSON references require the key/value to be valid JSON objects. If parsing fails, the token is left unchanged.
Conditionals
@@if(ref, expected, thenExpr, elseExpr?)
ref must be a reference like m.<path>, k.<path>, or h.<header>.
expected can be an exact match, or comparisons like >=65, !=foo. Date comparisons also work for ISO dates (YYYY-MM-DD).
Branch expressions can contain generators; inside branches you can use short form like rnd(8) (it is treated as @@rnd(8)).
value: {"age":"@@person.age","tier":"@@if(m.age,'>=65','pension',@@if(m.age,'>=18','adult','minor'))"}
Automationautomation.playbooks
Playbook help
playbooktaskscreate-topicdelete-topicscopy-dataimport-datandjsonjsonburstdelay
Overview
A playbook is a line-based script for running bulk operations (create/copy/delete/etc.) against Kafka topics.
Each non-empty line is a single command. Validation happens before running.
You can also use the Playbook builder wizard to generate scripts. It is available from the topic list context menu and from the playbook editor (New with Wizard).
Use Dry run in the playbook editor to preview the expanded script and validation output before submitting a task.
Playbooks can now also load message envelopes from disk using import-data, reusing the same import/export envelope shape you see in the topic import modal.
Important: playbooks are not designed to set up a production environment. Production provisioning should be handled via CI/CD and IaC (Infrastructure as Code). Playbooks are intended for controlling your own environment (or a slice of an environment) and for quick refreshes to prepare for manual testing with Kafka Streamyard. For example, if you have a Docker environment or a development environment, playbooks help you quickly get it back into a known state that is ready for testing again.
Script basics
Comments: lines starting with # are ignored.
One command per line: playbooks are line-based, so import-group, each import-data, and end-group must all be on separate lines.
Targets: most commands take <connectionName>:<topicPattern>.
Repeat blocks: you can repeat an indented block using repeat <count>. The block is defined by indentation (spaces/tabs).
For-each blocks: you can expand a topic pattern into concrete topic names and run an indented block for each topic using for-each-topic <connectionName>:<topicPattern> as <var>. Inside the block, @@<var> is replaced with the matched topic name.
Import groups: wrap multiple import-data commands in import-group / end-group to run them in parallel.
Import group rules: only import-data commands are allowed inside a group, and nested groups are rejected during validation.
# Send the same operation 3 times
repeat 3
  echo "Round"
  copy-data dev:orders* stage
for-each-topic <connectionName>:<topicPattern> as <var>
  <indented commands using @@<var>>
# Clear matching topics one by one
for-each-topic dev:orders* as t
  echo Clearing @@t
  clear-topics dev:@@t
Topic pattern matching
Patterns are evaluated as:
AND: split by +. Every + part must match.
Exclude: within each + part, you can exclude with *-* (example: *foo*-*bar* means “include foo, exclude bar”).
Wildcards: * is a glob wildcard. If there is no *, the token behaves like a “contains” match.
orders**dlq*orders*+prod*-*.tmp**orders*-*test*
Command reference
echo
Log a message to the task output (quotes are optional).
create-topic
Create one topic on one connection (requires partitions + replication-factor).
create-topics
Create topics on destination based on the source topic specs (optional overrides).
clear-topics
Empty matching topics (does not delete/recreate; clears messages).
delete-topics
Delete matching topics (waits for disappearance).
update-configs
Copy configs from source to destination, or apply fixed configs to a target.
copy-data
Copy records from source topics to destination topics (creates destination topic if missing).
slow-copy
Like copy-data, but sleeps up to a max delay between produced records (maxDelay like 5s or 250ms).
import-data
Load exported message envelopes from a file and produce them to a topic with optional delay and burst controls.
import-group
Run multiple import-data commands in parallel until each file is exhausted.
end-group
Closes an import-group block.
Import-data details
Syntax: import-data "<filePath>" -topic <connection:topic> [-delay <duration | min-max>] [-burst <count | min-max>]
-delay accepts ms, s, m, and hvalues, for example 250ms, 10s, 2m, 1h, or a random range like 1s-5s.
-burst controls how many records are sent each time before the next delay; it accepts a fixed count like 10 or a range like 5-10.
import-group is also line-based: do not place multiple import-data commands on the same line as import-group or end-group.
-topic must always use the explicit connection:topic form, just like the other playbook commands.
The backend checks that the file exists before sending anything. If the file is missing, unreadable, or invalid, the task logs the error and stops.
Accepted file formats
Files can be a single JSON object, a JSON array, or NDJSON. The preferred envelope shape is: { headers, key, messageValue, partition }
headers can be an array from export/import modal format or a simple object map; headers and key may be omitted.
messageValue can be a JSON string payload or plain text. payload is also accepted.
If a row is just a plain JSON object without envelope fields, it is sent as the message body with no key, no headers, and automatic partitioning.
{"headers":[{"key":"dataType","valueUtf8":"hr-candidate"}],"messageValue":"{\"id\":87831168}","key":"87831168","partition":0}
Examples (copy/paste)
Echo
echo Starting migration
echo "Copying topics..."
Create a single topic
create-topic dev:orders.v1 -partitions=12 -replication-factor=3 -configs cleanup.policy=compact,retention.ms=7d
Notes: -configs is a comma-separated k=v list. For *.ms keys you can use durations like 7d, 1h, 30s.
cleanup.policy must be one of: delete, compact, compact,delete. The UI normalizes delete,compact to compact,delete, but playbook -configs should use the canonical compact,delete explicitly.
Create topics on destination (copy topic specs)
create-topics dev:orders* stage
Optional overrides: -partitions=<n>, -replication-factor=<n>
create-topics dev:orders* stage -partitions=6 -replication-factor=2
Clear topics (empty messages)
clear-topics dev:*dlq*
Delete topics
delete-topics dev:*tmp*
Update configs (copy from source to destination)
update-configs dev:orders* stage
Update configs (apply fixed config values)
update-configs dev:orders* -configs cleanup.policy=compact,retention.ms=7d,replication.factor=3
Tip: replication.factor is handled specially and may be adjusted to the destination broker count.
Tip: cleanup.policy should be provided as compact,delete (not delete,compact).
Copy data
copy-data dev:orders* stage
Slow copy data
slow-copy dev:orders* stage 5s
maxDelay supports ms and s only (example: 250ms, 5s).
Import exported messages from a file
import-data "c:/data/orders.ndjson" -topic stage:pri-ord-orders -delay 1s-5s
-topic must include the connection name, for example stage:pri-ord-orders.
Files can be a single JSON object, a JSON array, or NDJSON in the same envelope format used by the import/export modal.
Import bursts with a fixed batch size
import-data "c:/data/warehouse.json" -topic stage:pri-whr-warehouse -delay 1s -burst 10
Run multiple imports in parallel
import-group
  import-data "c:/data/people.ndjson" -topic pri-hr-employees -delay 10s
  import-data "c:/data/orders.json" -topic pri-ord-orders -delay 1s-5s
  import-data "c:/data/warehouse.json" -topic pri-whr-warehouse -delay 1s-2s -burst 5-10
end-group
One full “migration” playbook (create + configs + data)
# 1) Create destination topics from source specs
create-topics dev:orders* stage

# 2) Ensure configs match (or re-apply)
update-configs dev:orders* stage

# 3) Copy records (will create missing topics too)
copy-data dev:orders* stage
Automationautomation.playbookWizard
Playbook builder wizard
playbookwizardbuildergeneratecreate topicscopy datadelete topics
The Playbook builder wizard helps generate a playbook script from a guided UI, so you don’t have to remember command syntax.
Where to open it
Open it from the Topic list context menu (Playbook builder wizard…) or from the playbook editor (New with Wizard).
What it generates
Depending on the chosen action, the wizard produces commands like create-topics, copy-data, clear-topics, and delete-topics, targeting the topics you selected.
You can save the generated script as a playbook and then run it like any other playbook.