LeoCybSec

Our Pricing

Billed Monthly

Billed Yearly (save
15%)

Request
Demo

$0

Description of the tier list will go here, copy should be concise and impactful.

    Smart

    $25

    / user / month

    All the security software you need

    Protection Level

    Advanced

    $50

    / user / month

    Everything in Smart Plus…

    Protection Level

    criblblog

    How Our Customers Recovered 50% Of Their Splunk Cost(Part 2 AWS-CloudTrail)

    Leo CybSec Team

    Cribl

    Introduction

    Following our last LinkedIn poll, the most voted option was AWS CloudTrail and as promised, we will continue our blog series  to demonstrate how we help our customers reduce their Splunk cost using Cribl LogStream.

    AWS CloudTrail

    AWS CloudTrail services enable you to manage governance, compliance, operational and risk auditing requirements of AWS accounts as CloudTrail events cover API and non-API calls made through the AWS Management Console, AWS CLIs, AWS SDKs, and other AWS services. These logs offer enhanced visibility and better insights into user and resource activities across AWS infrastructure and serve as a valuable intelligence source for security investigations.

    CloudTrail records three types of events:

    • Management events / Control plane operations: Provides details about management activities performed on an AWS account’s resources; logged by CloudTrail by default.
    • Data events / Data plane operations: Logs resource operations conducted on or within a resource.
    • CloudTrail Insights events: Records unusual API call rates or error rate activities.

    Best practice requires every organisation to enable CloudTrail Logs to be effectively queried either proactively as a countermeasure or in response to an incident. But given the breadth and depth of the logs, inspecting CloudTrail events can become challenging. This is why it is common to integrate SIEM tools for real-time monitoring and analysis to help respond proactively to security incidents. However, as time progresses, CloudTrail logs will be laden with noisy, non-security related events, and as a result, licence charges of these analytical platforms could soar, which dissuade organisations from taking full advantage of CloudTrail’s potentials. 

    This blog will demonstrate how your organisation can use Cribl LogStream to formulate your CloudTrail for more effective use of Splunk, helping reduce your licence, infrastructure and storage cost whilst ensuring an improved security posture.

    Why Cribl

    Cribl enables observability by giving you the power to make choices that best serve your business, without the negative tradeoffs. As your goals evolve, you have the flexibility to make new choices including new tools and destinations.

    At Leo CybSec, we believe in the value Cribl can bring to our customers, which is why we deliver here Splunk-based evidence on Cribl capability to reduce Splunk licensing while maintaining logs fidelity for a variety of noisy log sources that cause headache and sleepless nights to Splunk admins.

    Ingesting AWS CloudTrail Logs

    Our lab consists of 3 tiers: AWS, Cribl and Splunk.

    AWS:

    • Configure SQS-based S3 input for CloudTrail delivery as per AWS documentations
      • Set up a dedicated S3 Bucket which will host the CloudTrail log files
      • Set up an SQS with sufficient permissions to be used for S3 notifications 
      • Configure S3 notifications to SQS  

    Our lab is hosted on AWS and consists of the below systems:

    Server roleIP
    Cribl Worker 1172.31.70.181
    Cribl Worker 1172.31.66.108
    Splunk Indexer 1172.31.66.108
    Splunk Indexer 2172.31.72.157
    Splunk Indexer 3172.31.72.93
    Splunk Indexer 4172.31.73.235
    Splunk Cluster Master17​​2.31.69.131
    Heavy Forwarder172.31.64.212

    The Splunk Architecture

    A multi-site Splunk deployment with 4 Indexers and Index Discovery enabled

    The Splunk Architecture

    Cribl LogStream  Architecture

    Cribl LogStream  Architecture

    Enabled Source For S3 Using SQS

    One of the many sources that Cribl LogStream supports is S3 over SQS which requires 3 LogStream UI steps to be enabled (account number and API key have been changed):

    1. Configure the input parameters
    1. Configure the input parameters

    2. Enter the credentials (API key and secret key)

    2. Enter the credentials (API key and secret key)

    3. Enter the event breaker settings (LogStream has predefined rules for reading AWS logs)

    3. Enter the event breaker settings (LogStream has predefined rules for reading AWS logs)

    Based on the above, CloudTrail logs re getting to LogStream as shown in the image below;

    CloudTrail logs

    Create A Route In LogStream To Deal With CloudTrail Logs

    The below route consists of a LogStream Pipeline to apply the processing on the CloudTrail Logs

    consists of a LogStream Pipeline 1
    consists of a LogStream Pipeline 2

    Create A Load Balanced Logstream Destination To The Splunk Cluster

    Create A Load Balanced Logstream Destination To The Splunk Cluster

    The Log Sources

    The Log Sources
    • Upload the sample to S3 ⇒ SQS ⇒ Splunk Heavy Forwarder ⇒ Splunk indexers 
    • Upload the sample to S3 ⇒ SQS ⇒ LogStream worker ⇒ Splunk indexers

    We used 4 samples from the publicly available CloudTrail logs: 

    • flaws_cloudtrail01.json.gz
    • flaws_cloudtrail05.json.gz
    • flaws_cloudtrail19.json.gz
    • flaws_cloudtrail11.json.gz

    Findings

    Using Cribl LogStream

    We used a mix of 9 functions divided into 4 groups as shown below to achieve reduction via: 

    • Simplifying the logs from a nested multilayer json into a key-value pair
    • Renaming long names to short ones (abbreviated) and re-apply the mapping at Splunk
    • Removing duplicates and fields that can be calculated 
    • Dropping non-security related events

    In this pipeline we will demonstrate 2 reduction scenarios: 

    • Dropping all not security related events and applying event-level reduction
    • Keeping all events and applying event-level reduction
    Using Cribl LogStream

    Dropping Non-Security Events

    One efficient way of trimming CloudTrail log size is by removing irrelevant events that do not contribute value to Security Assessments. In our scenario, “Describe*” and “List*” events were identified as unnecessary elements; therefore, we used Cribl to drop all events containing Describe or List as eventName.

    After successfully dropping those events, we are able to demonstrate the achieved reduction by using LogStream’s Basic Statistics for the selected sample files:

    LogStream’s Basic Statistics 1
    LogStream’s Basic Statistics 2
    LogStream’s Basic Statistics 3
    LogStream’s Basic Statistics 4

    As Shown Above, We Are Able To Achieve A Minimum 36.49% Reduction In Logs Volume!

    Keeping Non-Security Events

    In this case we keep the Describe and List events by disabling the drop function.

    Keeping Non-Security Events 1
    Keeping Non-Security Events 2
    Keeping Non-Security Events 3
    Keeping Non-Security Events 4

    Using Splunk

    To demonstrate the above findings using Splunk we push now the 4 samples into Splunk via 3 stages:

    1. On Splunk HF we enable SQS-based S3 input for CloudTrail logs
    On Splunk HF we enable SQS-based S3 input for CloudTrail logs

    2. On LogStream we disable the SQS-based S3 input to ensure logs will only flow to Splunk HF

    2. On LogStream we disable the SQS-based S3 input to ensure logs will only flow to Splunk HF

    3. We upload the 4 sample files into the destination S3

    3. We upload the 4 sample files into the destination S3

    4. Verify the logs are arriving at Splunk under the AWS index

    4. Verify the logs are arriving at Splunk under the AWS index

    5. Disable the Splunk HF SQS-based S3 input and enable the LogStream source for SQS based S3

    Enable The Drop Function 

    • At this stage we enable the drop function to remove all the Describe and List functions  
    • Delete all the files from the S3 and upload them again 
    • Validate that the logs are getting to LogStream
    Enable The Drop Function 

    Validate that the logs are heading to the destination (Splunk)

    Validate that the logs are heading to the destination (Splunk)

    Validate that the logs are reaching Splunk and are searchable 

    Validate that the logs are reaching Splunk and are searchable 
    Validate Splunk are searchable 

    License Reduction

    Now let’s compare the consumed license in Splunk from the 4 sample log files;

    License Reduction
    License Reduction

    Disable The Drop Function

    Same comparison but this time without the Drop Function enabled;

    Disable The Drop Function
    Disable The Drop Function

    Conclusion

    • Not all CloudTrail events are essential for security, and Cribl’s LogStream can help filter them either by dropping or by directing them to another destination 
    • Utilizing LogStream’s inherent capabilities, we are able to drop redundant fields and fields that can be calculated at Splunk side using lookups e.g. (we only send eventName and using lookups we calculate eventSource and eventType) these are static values of CloudTrail events. For example, in the table below we only send the 1st column values and use lookup to fill 2nd and 3rd ones.
    eventNameeventSourceeventType
    AssumeRolests.amazonaws.comAwsApiCall
    AttachRolePolicyiam.amazonaws.comAwsApiCall
    AttachVolumeec2.amazonaws.comAwsApiCall
    ConsoleLoginsignin.amazonaws.comAwsConsoleSignIn
    • We managed to achieve a license reduction between 35 and 40% following an effortless but powerful use of LogStream.

    What other log sources do you want to see reduced with LogStream? 

    What other destinations do you want to reduce the logs ingestion volume to?

    Please visit our website https://leocybsec.com/ or drop us an email at info@leocybsec.com  and will be more than happy to help and answer any questions.

    Contact us

    Learn more about our cyber security solutions and address any questions you may have.

    Get in touch with us today, we are here for you.

    London Office

    167-169
    Great Portland Street,
    London, England,
    W1W 5PF.
    London: +44 7463239665

    Dubai Office

    Business Bay,
    Ontario Tower Building,
    SR-G-01-042, Dubai,
    United Arab Emirates
    Dubai: +971 501716764