Setting up AWS S3 Log Delivery

RTLD may automatically deliver compressed log data to an AWS S3 bucket by submitting HTTPS PUT requests to it. Each request adds an object to the bucket. This object contains a compressed JSON or CSV document that uniquely identifies a set of log data and describes one or more log entries.

Learn more: RTLD CDN | RTLD Rate Limiting | RTLD WAF

RTLD applies gzip compression to log data. AWS S3 stores compressed log data as an object with a gz file extension.
Learn more.

AWS S3 may automatically decompress files downloaded via the S3 Management Console into JSON or CSV files. No additional decompression is required to process this data.

To create a log delivery profile

  1. Create or identify an AWS S3 bucket to which log data will be posted.

    View AWS documentation on how to create a bucket.

  2. Apply the following bucket policy to the AWS S3 bucket identified in step 1. This bucket policy authorizes our service to upload content to your bucket.

    View AWS documentation on how to add a bucket policy.

    {
    	"Version": "2012-10-17",
    	"Statement": [{
    			"Sid": "CDNRealTimeLogDelivery",
    			"Effect": "Allow",
    			"Principal": {
    				"AWS": "arn:aws:iam::638349102478:user/real-time-log-delivery"
    			},
    			"Action": [
    				"s3:PutObject",
    				"s3:GetBucketLocation",
    				"s3:PutObjectTagging",
    				"s3:PutObjectACL"
    			],
    			"Resource": [
    				"arn:aws:s3:::Bucket-Name",
    				"arn:aws:s3:::Bucket-Name/*"
    			]
    		}
    	]
    }

    Replace the term "Bucket-Name" in lines 16 and 17 with the name of the AWS S3 bucket to which this policy is being applied.

  3. Optional. Set up AWS to process the log data that will be posted to it.

    Example:

    Leverage AWS Lambda to mine specific data from log entries.

  4. Navigate to the Real-Time Log Delivery CDN | Rate Limiting | WAF page. From the main menu, navigate to More and then find Real-Time Log Delivery under Analytics. Select either CDN, WAF, or RL.

  5. Click Add Profile.
  6. From the Log Delivery Method option, select AWS S3.
  7. Set the Bucket option to the name of the AWS S3 bucket to which log data will be posted.
  8. Optional. Set the Prefix option to the desired prefix that defines a virtual log file storage location and/or a prefix that will be added to each object added to your bucket.

  9. From the AWS Region option, select the region assigned to the AWS S3 bucket.
  10. From the Log Format option, select whether to format log data using our standard JSON format, as a JSON array, as JSON lines, or as a CSV (RTLD CDN only).

    Learn more: RTLD CDN | RTLD Rate Limiting | RTLD WAF

  11. From the Downsample the Logs option, determine whether all or downsampledReduces the amount of log data that will be delivered. For example, you may choose to only deliver 1% of your log data. log data will be delivered.

    • All Log Data: Verify that the Downsample the Logs option is disabled.
    • Downsampled Log Data: Downsample logs to 0.1%, 1%, 25%, 50%, or 75% of total log data by enabling the Downsample the Logs option and then selecting the desired rate from the Downsampling Rate option.

      Use this capability to reduce the amount of data that needs to be processed or stored by AWS S3.
      RTLD CDN Only: Downsampling log data also reduces usage charges for this service.

  12. Log delivery setup varies according to whether you are delivering log data for CDN traffic, threats identified by WAF, or rate limited requests.

  13. Set the Log Delivery Enabled option to the "on" position.

  14. Click Save.

Log File Naming Convention

RTLD CDN: On 12/8/2019, the log file naming convention was updated to include the profile ID for your Real-Time Log Delivery configuration.

The log data stored within an object is compressed using gzip. Each object follows this naming convention:

The JSON document contained within an object follows this naming convention:

Sample file name (RTLD CDN - JSON log format):

adn_0001_123_20210111_50550000F98AB95B_1.json

Sample file name (RTLD Rate Limiting - JSON log format):

rl_0001_123_20210111_50550000F98AB95B_1.json

Sample file name (RTLD WAF - JSON log format):

waf_0001_123_20210111_50550000F98AB95B_1.json

Each of the above file naming variables are described below.

Variable Description

Log Type

Represents the type of log data.

  • RTLD CDN: Identifies the delivery platform for which CDN activity was logged. Valid values are:

    • wpc: HTTP Large platform
    • wac: HTTP Small platform
    • adn: Application Delivery Network platform
  • RTLD Rate Limiting: This variable is always set to rl.
  • RTLD WAF: This variable is always set to waf.

AN

Represents your CDN account number (e.g., 0001). This account number may be viewed from the upper-right hand corner of the MCC.

Profile ID

Represents the system-defined ID for your Real-Time Log Delivery configuration.

You cannot currently view the system-defined ID assigned to your Real-Time Log Delivery configuration from within the MCC.

Date Stamp

Represents the date on which the log file was generated.

Syntax:

YYYYMMDD

Example:

20210110

Agent ID

Represents a unique ID that identifies the Real-Time Log Delivery software agent that generated the log file.

Sequence Number

Represents a sequential number that identifies the order in which the log file was generated by the software agent identified above.

Each software agent assigns a sequential number to the log files that it generates. A gap between log files generated on the same day by the same software agent indicates missing log data.
Learn more.

Key information:

  • This number starts at 0.
  • This number resets to 0 at the start of a new day (UTC).

File Extension

Represents the file extension for the log file. This file extension varies by log format.

  • JSON Log Format: json
  • JSON Array Log Format: json_array
  • JSON Lines Log Format: json_lines
More Information