RTLD may automatically deliver compressed log data to an Azure Blob Storage container by submitting HTTPS PUT requests to it. Each request creates a block blob within the container. This block blob contains a compressed JSON or CSV document that uniquely identifies a set of log data and describes one or more log entries.
Learn more: RTLD CDN | RTLD Rate Limiting
RTLD applies gzip compression to log data. Azure Blob Storage stores compressed log data as an object with a gz file extension.
Learn more.
Setting up log delivery to Azure Blob Storage requires:
An existing Azure Blob Storage account.
A base URL that points to your container.
Blob Container URL:
Sample Blob Container URL:
Either a SAS token or an access key through which our service will authorize requests to upload content to your Azure Blob Storage account.
If you plan on providing a SAS token, make sure that the token has permission to write to the blob/container. Additionally, it should start with sv= and it should not include a ?.
Sample SAS token:
In addition to the above requirements, you may specify an optional prefix that defines the location where log data will be uploaded within your container. Content will be uploaded to the location defined by this prefix as indicated by the following URL.
Key information:
A trailing forward slash means that the specified value only defines a virtual directory path within your container where logs will be stored. If the specified value ends in a character other than a forward slash, then the characters specified after the last forward slash will be prepended to the file name for each log file uploaded to Azure Blob Storage.
Sample Scenario
This scenario demonstrates how a prefix that does not end with a forward slash will affect the naming convention for log files.
Sample prefix:
The above prefix will store log files in the following virtual directory:
The file name for each log file uploaded to Azure Blob Storage will start with "siteA_."
Sample log file name:
To create a log delivery profile
Create or identify an Azure storage account and a container to which log data will be posted.
View Microsoft Azure documentation on how to create a storage account.
Navigate to the Real-Time Log Delivery CDN | Rate Limiting
Set the Blob Container URL option to a URL that points to the container to which log data will be posted.
Optional. Set the Prefix option to a value that defines a virtual log file storage location and/or a prefix that will be added to each log file added to your container.
From the Access Type option, select whether log data uploads will be authorized via a SAS token or an access key and then paste it in the field below it.
If you plan on providing a SAS token, make sure that the token has permission to write to the blob/container. Additionally, it should start with sv= and it should not include a ?.
From the Log Format option, select whether to format log data using our standard JSON format, as a JSON array, as JSON lines, or as a CSV (RTLD CDN only).
Learn more: RTLD CDN | RTLD Rate Limiting
From the Downsample the Logs option, determine whether all or downsampledReduces the amount of log data that will be delivered. For example, you may choose to only deliver 1% of your log data. log data will be delivered.
Downsampled Log Data: Downsample logs to 0.1%, 1%, 25%, 50%, or 75% of total log data by enabling the Downsample the Logs option and then selecting the desired rate from the Downsampling Rate option.
Use this capability to reduce the amount of data that needs to be processed or stored by Azure Blob Storage.
RTLD CDN Only: Downsampling log data also reduces usage charges for this service.
Log delivery setup varies according to whether you are delivering log data for CDN traffic
From the Filter by Edge CNAME section, perform one of the following steps:
Filter log data by one or more edge CNAME(s)
Select one or more edge CNAMEs from the option directly to the right of the above option.
Filter the list by typing the entire or partial hostname. For example, typing co will filter the list to include all hostnames that contain co (e.g., cdn.example.com and corp.example.org).
Upload all log data regardless of edge CNAME
Verify that an edge CNAME has not been defined within this section.
From the Filter by User Agent option, perform one of the following steps:
Filter log data by user agent
Type a RE2-compatible regular expressionThis type of value is interpreted as a regular expression that defines a pattern of characters. pattern that identifies the set of user agents by which log data will be filtered.
Upload all log data regardless of user agent
Set it to a blank value.
From the Filter by Status Code section, perform one of the following steps:
Filter log data by status code
Mark each status code class (e.g., 2xx or 3xx) for which log data will be delivered. Clear all other status code classes.
Upload all log data regardless of status code
Clear all status code classes (e.g., 2xx and 3xx).
From the Log file contains the following fields section, perform the following steps:
Add the request headers, response headers, and cookies that will be logged for each request from the Custom Request Headers, Custom Response Headers, and Custom Cookies options.
You may either select or type the name of the desired headers and/or cookies. Click on the list to add additional headers or cookies. Remove a header or cookie by clicking on its x.
Although other settings take effect quickly, it may take up to 90 minutes before data for custom request/response headers and cookies is logged.
Add or clear all of the fields associated with a category by marking or clearing the category's header.
From the Filter by Edge CNAME section, perform one of the following steps:
Filter log data by one or more edge CNAME(s)
Select one or more edge CNAMEs from the option directly to the right of the above option.
Filter the list by typing the entire or partial hostname. For example, typing co will filter the list to include all hostnames that contain co (e.g., cdn.example.com and corp.example.org).
Upload all log data regardless of edge CNAME
Verify that an edge CNAME has not been defined within this section.
From the Filter by Country section, perform one of the following steps:
Filter log data by one or more countries:
Select one or more countries from the option directly to the right of the above option.
Filter the list by typing the entire or partial country name. For example, typing un will filter the list to include all countries that contain un (e.g., United States and United Kingdom).
Upload all log data regardless of country of origin:
Verify that a country has not been defined within this section.
From the Filter by User Agent option, perform one of the following steps:
Filter log data by user agent
Type a RE2-compatible regular expressionThis type of value is interpreted as a regular expression that defines a pattern of characters. pattern that identifies the set of user agents by which log data will be filtered.
Upload all log data regardless of user agent
Set it to a blank value.
From the Filter by Client IP option, perform one of the following steps:
Filter log data by one or more IP addresses:
Upload all log data regardless of IP address:
Verify that an IP address has not been defined within this section.
From the Filter By Action Type option, perform one of the following steps:
Filter log data by one or more enforcement action(s):
Select or type the name for one or more enforcement action(s).
Upload all log data regardless of enforcement action:
Verify that an enforcement action has not been defined within this section.
From the Filter By Request Method option, perform one of the following steps:
Filter log data by one or more request method(s):
Upload all log data regardless of request method:
Verify that a request method has not been defined within this section.
From the Filter By Scope Name option, perform one of the following steps:
Filter log data by one or more security application manager(s):
Select or type the name for one or more security application manager(s).
Upload all log data regardless of security application manager:
Verify that a security application manager(s) has not been defined within this section.
From the Filter By Action Limit ID option, perform one of the following steps:
Filter log data by one or more rate rule(s):
Type the name for one or more rate rule(s).
Upload all log data regardless of rate rule:
Verify that a rate rule has not been defined within this section.
From the Filter By URL Regex option, perform one of the following steps:
Filter log data by URL
Type a RE2-compatible regular expressionThis type of value is interpreted as a regular expression that defines a pattern of characters. pattern that identifies the set of URLs by which log data will be filtered.
Upload all log data regardless of URL
Set it to a blank value.
From the Log file contains the following fields section, perform the following steps:
From the Filter by Edge CNAME section, perform one of the following steps:
Filter log data by one or more edge CNAME(s)
Select one or more edge CNAMEs from the option directly to the right of the above option.
Filter the list by typing the entire or partial hostname. For example, typing co will filter the list to include all hostnames that contain co (e.g., cdn.example.com and corp.example.org).
Upload all log data regardless of edge CNAME
Verify that an edge CNAME has not been defined within this section.
From the Filter by Country section, perform one of the following steps:
Filter log data by one or more countries:
Select one or more countries from the option directly to the right of the above option.
Filter the list by typing the entire or partial country name. For example, typing un will filter the list to include all countries that contain un (e.g., United States and United Kingdom).
Upload all log data regardless of country of origin:
Verify that a country has not been defined within this section.
From the Filter by User Agent option, perform one of the following steps:
Filter log data by user agent
Type a RE2-compatible regular expressionThis type of value is interpreted as a regular expression that defines a pattern of characters. pattern that identifies the set of user agents by which log data will be filtered.
Upload all log data regardless of user agent
Set it to a blank value.
From the Filter By Security Application Manager option, perform one of the following steps:
Filter log data by one or more security application manager(s):
Select or type the name for one or more security application manager(s).
Upload all log data regardless of security application manager:
Verify that a security application manager(s) has not been defined within this section.
From the Filter By Access Rule option, perform one of the following steps:
Filter log data by one or more access rule(s):
Select or type the name for one or more access rule(s).
Upload all log data regardless of access rule:
Verify that an access rule has not been defined within this section.
From the Filter By Custom Rule option, perform one of the following steps:
Filter log data by one or more custom rule(s):
Select or type the name for one or more custom rule(s).
Upload all log data regardless of custom rule:
Verify that a custom rule has not been defined within this section.
From the Filter By Managed Rule option, perform one of the following steps:
Filter log data by one or more managed rule(s):
Select or type the name for one or more managed rule(s).
Upload all log data regardless of managed rule:
Verify that a managed rule has not been defined within this section.
From the Log file contains the following fields section, perform the following steps:
Set the Log Delivery Enabled option to the "on" position.
RTLD CDN: On 12/8/2019, the log file naming convention was updated to include the profile ID for your Real-Time Log Delivery configuration.
The log data stored within an object is compressed using gzip. Each object follows this naming convention:
The JSON document contained within an object follows this naming convention:
Sample file name (RTLD CDN - JSON log format):
Sample file name (RTLD Rate Limiting - JSON log format):
Sample file name (RTLD WAF - JSON log format):
Each of the above file naming variables are described below.
Variable | Description |
---|---|
Represents the type of log data.
|
|
Represents your CDN account number (e.g., 0001). This account number may be viewed from the upper-right hand corner of the MCC. |
|
Represents the system-defined ID for your Real-Time Log Delivery configuration. You cannot currently view the system-defined ID assigned to your Real-Time Log Delivery configuration from within the MCC. |
|
Represents the date on which the log file was generated. Syntax: YYYYMMDD
Example: 20210110
|
|
Represents a unique ID that identifies the Real-Time Log Delivery software agent that generated the log file. |
|
Represents a sequential number that identifies the order in which the log file was generated by the software agent identified above. Each software agent assigns a sequential number to the log files that it generates. A gap between log files generated on the same day by the same software agent indicates missing log data. Key information:
|
|
Represents the file extension for the log file. This file extension varies by log format.
|
RTLD may deliver log data to one or more of the following destinations:
Log fields vary by RTLD module.
Learn more: RTLD CDN | RTLD Rate Limiting