FuseQL Parse Operators

Parse operators enable you to extract specific fields from log messages during query execution. They provide a flexible ad hoc method for processing data and refining queries. When using parse operators, you can extract valuable information dynamically, and tailor queries to specific use cases.

Parse Variable Patterns Using Regex

If you are familiar with regular expressions, use the Parse Regex operator to extract structured fields from unstructured data. This is especially useful for extracting nested or complex fields in log messages.

Syntax

| parse regex "<start_expression>(?P<field_name><field_expression>)<stop_expression>"

| parse regex "<start_expression>(?P<field_name><field_expression>)<stop_expression>" [noDrop]

| parse regex [field=<field_name>] "<start_expression>(?P<field_name><field_expression>)<stop_expression>"

Examples

| parse regex "user_id=(?P<user_id>[a-zA-Z0-9]+)"

| parse regex "error_code=(?P<error_code>\d+)" nodrop

Parse Variable Patterns Using an Anchor

The Parse Operator enables you to extract predictable patterns using specified start and stop anchors.

Syntax

| parse "<start_expression>*<stop_expression>" as <field>

| parse "<start_expression>*<stop_expression>" as <field> [noDrop]

| parse [field=<field_name>] "<start_expression>*<stop_expression>" as <field>

Examples

| parse regex "user_id=*;" as user

| parse regex "[*]" as my_field nodrop

JSON Operator

The JSON Operator enables efficient extraction of values from JSON-formatted logs, using JSONPath expressions. This operator is ideal for parsing structured JSON data, enabling users to extract single fields, multiple fields, or nested keys for further analysis. It simplifies working with JSON data, and makes queries more precise and effective.

Syntax

| json "<name_or_key>"[, "<name_or_key>", ...] [as <field> ...]

| json "<name_or_key>"[, "<name_or_key>", ...] [as <field>] [nodrop]

| json [field=<field_name>] "<name_or_key>"[, "<name_or_key>", ...] [as <field> ...]

Example

| json "user_id"

| json "user_id", "session_id" as user, session

| json "user.details.age" as age

| json field=log_message "transaction_id", "status" as txn_id, txn_status

Split Operator

The split operator enables you to split strings into multiple substrings. You can use it to parse delimited log entries, like space-delimited or comma-separated value (CSV) formats. You can also use it to extract specific fields from a string, and use them for further analysis and processing.

Syntax

The split operator offers several flexible approaches for extract fields; match them to your business needs.

Examples

Extract fields using index

This method uses numerical indexes to identify the fields you want to extract. The first field after the split is index 0, the second is index 1, and so on. You assign aliases (<A>, <B>, <E>, <F>) to the extracted fields for easier referencing after the split.

split <field> / (1)
  extract 0 as <A>, 1 as <B>, 4 as <E>, 5 as <F> (2)
1 field: the original string
2 A, B, E, F: the aliases of the named split sections
Extract fields using position

This method relies on the position of the field within the split string. You list the aliases for the fields you want to extract in the desired order.

Use an underscore _ to skip a position if you don’t need a particular field.

split <field> extract <A>, <B>, _, _, <E>, <F>
Mix positional and index-based extraction:

You can also combine positional and index-based extraction within the same split operation. This provides flexibility when dealing with strings where some fields are easily identified by position and others by index.

split <field> extract <A>, <B>, 4 as <E>, <F>
Specify delimiter, escape, and quote characters

This form enables you to define custom delimiter, escape, and quote characters.

split <field> escape='\', delim=':', quote='''' extract <A>, <B>, _, _, <E>, <F>

Default Characters

By default, the split operator uses the following characters:

Delimiter

Comma (,)

Escape

Backslash (\)

Quote

Double quote (")

Custom Character Restrictions

When you define your own escape, delimiter, or quote characters, they must adhere to these rules:

  • All three characters must be distinct.

  • Each character must be a single character.

  • Each character must match the Java regular expression [\s\S]; it can be any single character.

Required Field

You must always have field from which to extract the split.

To extract from your original message, use the log_line field.

This example splits the log_line field and extracts the first three fields, aliasing them as timestamp, level, and message respectively.

split log_line extract 1 as timestamp, 2 as level, 3 as message
Output
source="json" | parse "http://www.*.*/" as domain, ext | timeslice 60s | concat("https://kloudfuse.com/search/create?query=",ext) as url | json "method" | json "bytes" | where method = "POST" | split url delim="=" extract 1 as ext2