Rate Control for Logs stream ingestion

You can customize an ingestion rate control mechanism that enables you to limit the amount of ingested logs data.

  1. Follow the steps in Navigate to Rate Control Policy Management.

  2. In the list of streams, hover over the Logs stream, and click the Edit (Edit) icon.

    Edit the rate control policy for Logs

Configure Logs Rate Control

To add or change the ingestion rate control for logs, configure the rate control in the LOGS - Rate Control interface:

Stream Type

The default stream type is Logs.

You can choose another stream type: Metrics, Events, Traces, or RUM.

Ingestion Rate

Specify the number of log lines per second.

The default is 100 log lines per second.

This represents the maximum number of items that Kloudfuse can ingest for the stream, per second.

Burst

The maximum number of log lines that Kloudfuse can ingest from a single burst.

The default is 6000 log lines.

To enable bursting while maintaining average rates, we recommend that you modify the burst size to match your business needs:

burst=rate

Set the burst equal to the rate to ensure that Kloudfuse ingests up to 1 second’s worth of traffic at the same time.

burst=rate * N

Set the burst equal to a multiple of the rate to ensure that Kloudfuse ingests N seconds' worth of traffic at the same time.

Default Percentage

The percentage of quota set for the Logs stream, as a whole.

You can split this into defined classes; see Split the stream allocation. Be sure to adjust this value to compensate for the allocations to custom classes.

The initial value of this default class is 100%.

Click Save Rate Control (Save Rate Control) to finalize the rate control assignment to the stream.

After splitting the percentage, the logs stream definition may look like this. The Logs Drop Percentage (1hr) changes over time.

Defined rate control

Split the stream allocation

At a more granular level, you can split the percentage quota of your logs stream into distinct classes.

  1. Complete the steps in Add Classes.

  2. Complete the steps in Manage Class Rules.

Click Save Rate Control (Save Rate Control) to finalize the rate control assignment to the stream.

Add Classes

To assign percentages to classes, click Add Class (Add Class):

Add class to metrics rate control

  1. In the Class field, enter the name of the Class, such as Class 1, Class 2, and so on.

  2. Assign a percentage.

  3. Continue adding as many classes as required.

    The total assigned percentage cannot exceed 100%.

    The Default Percentage applies to the streams that don’t match the defined classes and selected classes. So if the total for the named classes adds up to 90%, then the Default Percentage can have a maximum value of 10%.

Manage Class Rules

To use classes, you must first define them. Click Manage Class Rules:

rate control manage classes logs

In the Rate Control Manage Class Rules page, proceed to define the classes.

  1. Ensure that you select the correct Stream Type. Each stream type has its own classes.

  2. Click Add class rule (Add class rule).

  3. Name the class. For example, Class 1.

  4. Specify at least one filter for the class:

    Key

    The label name that can be extracted from, or assigned to, the stream. For example, availability-zone.

    Value

    The value of the key. For example, us-west-2c.

    You can use a regex pattern to define a value. This results in multiple possible value matches.

  5. [Optional]

    To add another filter to the class definition, click Add filter (Add filter).

    Order: Kloudfuse evaluates the rules in order, top-down.

    Combination: When a class definition has multiple key:value filter pairs, Kloudfuse combines them using AND logic.

  6. To delete a filter from a class, click the Delete (Delete) icon next to the filter definition.

  7. To add more class definitions, click Add class rule (Add class rule).

  8. To delete a class, click the Delete (Delete) icon next to the class name.

  9. Click save classes to finalize class definition for rate control on that data stream.

Delete Logs Rate Control

  1. On the Rate Control Policy Management page, hover over the Logs stream, where you plan to delete the rate control policy.

  2. Click the Delete (Delete) option.

    Delete logs rate control policy

  3. In the Delete Rate Control confirmation dialog, click Delete.

    Confirm deleting logs rate control policy