Log Term Search Tokenization

This tokenization applies to Kloudfuse 4.3.0 and later.

Term search breaks each log line into tokens and matches your query against those tokens. Kloudfuse now uses a simpler tokenizer for log lines.

What changed

  • Every non-alphanumeric character is a delimiter. Dots, underscores, colons, slashes, hyphens, and equals signs all split tokens. Previously, dots and underscores kept a value together as one token.

  • Stop words are no longer removed. Common English words such as is, not, in, and the are now indexed and searchable.

  • Tokens are still lowercased, so term search remains case-insensitive.

The result is that sub-parts of dotted names, paths, IP addresses, and underscore-joined identifiers are now searchable on their own.

Examples

Log line text Tokens before Tokens now

com.example.service.QuartzJob

com.example.service.quartzjob

com, example, service, quartzjob

connection_timeout

connection_timeout

connection, timeout

192.168.1.10

192.168.1.10

192, 168, 1, 10

user.name=admin

user.name, admin

user, name, admin

record is not valid

record, valid

record, is, not, valid

Searches that previously returned nothing now match. For the log line com.example.scheduler.Job failed: connection_timeout:

Search Why it matches now

scheduler

A dot is a boundary, so scheduler is a token of its own rather than part of com.example.scheduler.job.

timeout

An underscore is a boundary, so connection_timeout indexes connection and timeout separately.

failed: connection

Punctuation is dropped from the query too — it looks for the tokens failed and connection.

What to expect

  • Term search matches more broadly. A query such as 192.168 matches any line containing both 192 and 168, even if they are not adjacent. Use grep search when you need an exact, adjacent character sequence.

  • Grep searches for punctuated values such as com.example.service or an IP address are substantially faster, because the index can now narrow the scan for them.

  • Tokenization is fixed when a log is indexed. Logs ingested before the upgrade keep the previous tokenization until they age out of your retention window, so results can differ across the transition period.

For a full comparison of the two search modes, see Logs, Term vs Grep search.