Parse Operators

Parse operators enable you to extract specific fields from log messages during query execution. They provide a flexible ad hoc method for processing data and refining queries. When using parse operators, you can extract valuable information dynamically, and tailor queries to specific use cases.

FuseQL supports the following parse operators:

regex: Parses variable patterns using Regex.
anchor: Parses variable patterns using an anchor.
json: Parses structured data from JSON-enabled logs.
split: Parses delimited log entries, splits strings into substrings, and extracts specific fields from a string.

regex

Parses variable patterns using Regex.

If you are familiar with regular expressions, use the Parse Regex operator to extract structured fields from unstructured data. This is especially useful for extracting nested or complex fields in log messages.

Syntax

| parse regex "<start_expression>(?P<field_name><field_expression>)<stop_expression>"

| parse regex "<start_expression>(?P<field_name><field_expression>)<stop_expression>" [noDrop]

| parse regex [field=<field_name>] "<start_expression>(?P<field_name><field_expression>)<stop_expression>"

Examples

| parse regex "user_id=(?P<user_id>[a-zA-Z0-9]+)"

| parse regex "error_code=(?P<error_code>\d+)" nodrop

anchor

Parses variable patterns using an anchor.

The Parse operator enables you to extract predictable patterns using specified start and stop anchors.

Syntax

| parse "<start_expression>*<stop_expression>" as <field>

| parse "<start_expression>*<stop_expression>" as <field> [noDrop]

| parse [field=<field_name>] "<start_expression>*<stop_expression>" as <field>

Examples

| parse regex "user_id=*;" as user

| parse regex "[*]" as my_field nodrop

json

Parses structured data from JSON-enabled logs.

The JSON Operator enables efficient extraction of values from JSON-formatted logs, using JSONPath expressions. This operator is ideal for parsing structured JSON data, enabling users to extract single fields, multiple fields, or nested keys for further analysis. It simplifies working with JSON data, and makes queries more precise and effective.

Additionally, we support extraction of arrays.

Syntax

| json "<name_or_key>"[, "<name_or_key>", ...] [as <field> ...]

| json "<name_or_key>"[, "<name_or_key>", ...] [as <field>] [nodrop]

| json [field=<field_name>] "<name_or_key>"[, "<name_or_key>", ...] [as <field> ...]

| json "<name_or_key>".[*]."<name_or_key>" multi type=["string" or "int" or "double"]

Examples

| json "user_id" (1)

| json "user_id", "session_id" as user, session (2)

| json "user.details.age" as age (3)

| json field=log_message "transaction_id", "status" as txn_id, txn_status (4)

| json "users.[*].score" multi type="int" | sum(score) (5)

| where source="app" | json "users.[*].score" type="int" multi as user_scores | timeslice 1m | sum (user_scores)  by (_timeslice) (6)

1	Extracts the `user_id` column from data.
2	Extracts the `user_id` column from data and displays it under the user heading, and the `session_id` column under the session heading.
3	Extracts the `age` field from the `user.details.age` data, and displays it under the age heading.
4	Extracts `transaction_id` and `status` from the log message, and displays them under the txn_id and txn_status headings, respectively.
5	Extracts an array of integer `score` values from each `user` instance, and then computes an aggregation (sum) of all scores.
6	Whenever the source is "app", extracts an array of integer `score` values from each `user` instance, displays them under the user_scores heading, buckets them into 1-minute intervals, and then computes an aggregation (sum) of all user scores for each bucket.

split

Parses delimited log entries, splits strings into substrings, and extracts specific fields from a string.

The Split operator enables you to split strings into multiple substrings. You can use it to parse delimited log entries, like space-delimited or comma-separated value (CSV) formats. You can also use it to extract specific fields from a string, and use them for further analysis and processing.

Syntax

The split operator offers several flexible approaches for extract fields; match them to your business needs.

Examples

Extract fields using index

This method uses numerical indexes to identify the fields you want to extract. The first field after the split is index 0, the second is index 1, and so on. You assign aliases (<A>, <B>, <E>, <F>) to the extracted fields for easier referencing after the split.

split <field> / (1)
  extract 0 as <A>, 1 as <B>, 4 as <E>, 5 as <F> (2)

1	`field`: the original string
2	A, B, E, F: the aliases of the named split sections

Extract fields using position

This method relies on the position of the field within the split string. You list the aliases for the fields you want to extract in the desired order.

Use an underscore _ to skip a position if you don’t need a particular field.

split <field> extract <A>, <B>, _, _, <E>, <F>

Mix positional and index-based extraction:

You can also combine positional and index-based extraction within the same split operation. This provides flexibility when dealing with strings where some fields are easily identified by position and others by index.

split <field> extract <A>, <B>, 4 as <E>, <F>

Specify delimiter, escape, and quote characters

This form enables you to define custom delimiter, escape, and quote characters.

split <field> escape='\', delim=':', quote='''' extract <A>, <B>, _, _, <E>, <F>

Default Characters

By default, the split operator uses the following characters:

Delimiter: Comma (,)
Escape: Backslash (\)
Quote: Double quote (")

Custom Character Restrictions

When you define your own escape, delimiter, or quote characters, they must adhere to these rules:

All three characters must be distinct.
Each character must be a single character.
Each character must match the Java regular expression [\s\S]; it can be any single character.

Required Field

You must always have field from which to extract the split.

To extract from your original message, use the log_line field.

This example splits the log_line field and extracts the first three fields, aliasing them as timestamp, level, and message respectively.

Split, extract, and alias

split log_line extract 1 as timestamp, 2 as level, 3 as message

Output

source="json" | parse "http://www.*.*/" as domain, ext | timeslice 60s | concat("https://kloudfuse.com/search/create?query=",ext) as url | json "method" | json "bytes" | where method = "POST" | split url delim="=" extract 1 as ext2