Splunk Search Query – Linux Systems Auditing

The auditing of the linux systems is achieved by using the auditd service that is provided by installing audit package. All the system audit log is generated and dumped to /var/log/audit/audit.log.

All these audit.log is forwarded to Splunk indexer for indexing this data and then in turn leverage this data to audit the linux systems by using the Splunk search query. We will consider couple of audit cases as below.

  • List of failed login attempts by users
  • File Audit – Who modified the critical file and when

Audit#1 : List of failed login attempts by users

Log Source: /var/log/audit/audit.log

Splunk Query

index=<indexname> host=<hostname> sourcetype=linux_audit type=USER_LOGIN res=failed | stats count by acct| sort - count

How to read this Query
We are monitoring linux audit logs for host “ip-172-31-3-123.ap-southeast-2.compute.internal” and looking for the audit source type “linux_audit” and event type “USER_LOGIN” (that is triggered when a user logs in) with result as failed. Then we are performing a statistically search of count variable on account name “acct” and report the results by sorting the count in descending order.

Of course you should adjust query accordingly if the index has been customized.

NOTE: In Splunk, the audit.log data input of the linux systems is indexed with source type “linux_audit” while using Automatic Source type option.

Output of the events returned by running the query.

The query yields the below Statistics about the failed login counts by the Users.



The corresponding Bar Chart of the User’s failed login attempt counts is displayed as follows.

NOTE: Replace the res=success in the query to get the successful login attempts by users.

File Audit – Who modified the critical file and when

We are assuming the audit rule is in place for watching the file for any modification to it on the host as follows

[root@ip-172-31-3-123 ~]# auditctl -w /etc/rsyslog.conf -p wa -k critfile_watch

[root@ip-172-31-3-123 ~]# auditctl -l | grep -i rsyslog
-w /etc/rsyslog.conf -p wa -k critfile_watch

Where,

-w path : Insert  a watch for the file system object at path
-p permissions: permissions that are logged. This value can be one or a combination of r(read), w(write), x(execute), and a(attribute change)
-k keyname: keyname is an optional string that helps you identify which rule(s) generated a particular log entry

Log Source: /var/log/audit/audit.log

Splunk Query

index=<indexname> host=<hostname> sourcetype=linux_audit key=<keyname> path=<file_to_monitor> type=config_change op="updated rules" | dedup _time | table _time, auid, path

How to read this Query
In our case, we are running the query to find the list of users WHO modified the critical file (/etc/rsyslog.conf) and WHEN on the host “ip-172-31-3-123.ap-southeast-2.compute.internal”. We are using the audit source type “linux_audit” and keyname “critfile_watch” with audit event type “config_change” & operation “updated rules”. Any duplicates has been removed with respect to time as there can be other non interest audit lines and lastly the resulting output is presented in a table with time, used id and file modified.

Output of the events returned by running the query.



Below is the Statistical data of the results in Table with columns _time (when was the file modified) and auid(who modified the file) and path(file being modified)

NOTE: This auditing data can be used to save as Report or Dashboard Panel.

In the coming article, I would be showing the real-time monitoring and alerting (email) for any critical file modification using Splunk.

Leave a Reply

Your email address will not be published.