Rate Control for Logs stream ingestion

You can customize an ingestion rate control mechanism that enables you to limit the amount of ingested logs data.

  1. Follow the steps in Navigate to Rate Control Policy Management.

  2. In the list of streams, hover over the Logs stream, and click the Edit (Edit) icon.

    Edit the rate control policy for Logs

Configure Logs Rate Control

To add or change the ingestion rate control for logs, configure the rate control in the LOGS - Rate Control interface:

Stream Type

The default stream type is Logs.

You can choose another stream type: Metrics, Events, Traces, or RUM.

Ingestion Rate

Specify the number of log lines per second.

The default is 100 log lines per second.

This represents the maximum number of items that Kloudfuse can ingest for the stream, per second.

Burst

The maximum number of log lines that Kloudfuse can ingest from a single burst.

The default is 6000 log lines.

To enable bursting while maintaining average rates, we recommend that you modify the burst size to match your business needs:

burst=rate

Set the burst equal to the rate to ensure that Kloudfuse ingests up to 1 second’s worth of traffic at the same time.

burst=rate * N

Set the burst equal to a multiple of the rate to ensure that Kloudfuse ingests N seconds' worth of traffic at the same time.

Default Rate

The absolute rate (in lines/sec) set for the default class of the Logs stream.

You can distribute the total ingestion rate across the default class and custom classes; see Split the stream allocation. Be sure to ensure that the sum of all class rates does not exceed the ingestion rate.

The initial value of this default class rate should be set based on your logs ingestion requirements.

Click Save Rate Control (Save Rate Control) to finalize the rate control assignment to the stream.

After configuring the rates, the logs stream definition may look like this. The Logs Drop Percentage (1hr) changes over time.

Defined rate control

Split the stream allocation

At a more granular level, you can distribute the ingestion rate of your logs stream across distinct classes.

  1. Complete the steps in Add Classes.

  2. Complete the steps in Manage Class Rules.

Click Save Rate Control (Save Rate Control) to finalize the rate control assignment to the stream.

Add Classes

To assign rates to classes, click Add Class (Add Class):

Add class to metrics rate control

  1. In the Class field, enter the name of the Class, such as Class 1, Class 2, and so on.

  2. Assign a rate (in the appropriate units for the stream type).

  3. Continue adding as many classes as required.

    The total assigned rate across all classes (default and custom) cannot exceed the ingestion rate.

    The Default Rate is a configurable rate that applies to streams that don’t match any of the defined custom classes. For example, if your ingestion rate is 1000 samples/sec, you might configure the default class to 300 samples/sec and custom classes to 400 samples/sec and 200 samples/sec respectively, totaling 900 samples/sec.

After configuring your classes and rates, click Save Rate Control (Save Rate Control) to save your changes.

Ensure that the sum of the default rate and all custom class rates does not exceed the configured ingestion rate. If the total exceeds the ingestion rate, you will receive an error message.

Manage Class Rules

To use classes, you must first define them. Click Manage Class Rules:

rate control manage classes logs

In the Rate Control Manage Class Rules page, proceed to define the classes.

  1. Ensure that you select the correct Stream Type. Each stream type has its own classes.

  2. Click Add class rule (Add class rule).

  3. Name the class. For example, Class 1.

  4. Specify at least one filter for the class:

    Key

    The label name that can be extracted from, or assigned to, the stream. For example, availability-zone.

    Value

    The value of the key. For example, us-west-2c.

    You can use a regex pattern to define a value. This results in multiple possible value matches.

  5. [Optional]

    To add another filter to the class definition, click Add filter (Add filter).

    Order: Kloudfuse evaluates the rules in order, top-down.

    Combination: When a class definition has multiple key:value filter pairs, Kloudfuse combines them using AND logic.

    Multiple rules for the same class: You can add multiple rules referring to the same class. These rules are treated as OR logic, meaning data matching any of the rules will be assigned to that class.

  6. To delete a filter from a class, click the Delete (Delete) icon next to the filter definition.

  7. To add more class definitions, click Add class rule (Add class rule).

  8. To delete a class, click the Delete (Delete) icon next to the class name.

  9. Click save classes to finalize class definition for rate control on that data stream.

Delete Logs Rate Control

  1. On the Rate Control Policy Management page, hover over the Logs stream, where you plan to delete the rate control policy.

  2. Click the Delete (Delete) option.

    Delete logs rate control policy

  3. In the Delete Rate Control confirmation dialog, click Delete.

    Confirm deleting logs rate control policy