Code block - name 'int' is not defined

Hello,

we are trying to use the code block to return the computer name of a log entry in the data stream OTHER. Executing the commands in a python shell (with inward_array filled with dummy data) works. We however get an exception in DNIF (name ‘int’ is not defined). (we cast to int, since we otherwise receive an error ‘string indices must be integers’ for the slice function (computer_name[start_index,end_index]))

def transform(inward_array):
for log in inward_array:
computer_name = log[’$ComputerName’]
start_index = int(computer_name.find(’“computer_name”:"’) + len(’“computer_name”:"’))
end_index = int(computer_name.find(’.DOMAIN.NAME’, start_index) + len(’.DOMAIN.NAME’))
log[’$ComputerName’] = computer_name[start_index,end_index]
return inward_array

Is there a way we can get this to work in DNIF?

Kind Regards
Jonas

Hi Jonas,
Thanks a lot for trying DNIF and posting your query on the Forum.

Can you provide sample values of $ComputerName and the expected outcome for those ?

Regards,
Bodhi

Hi Bodhi,

thank you for the quick reply! Sorry, I initially posted the wrong querry. This was the query:


def transform(inward_array):
for log in inward_array:
computer_name = log[’$ComputerName’]
start_index = computer_name.find(’“computer_name”:"’) + len(’“computer_name”:"’)
end_index = computer_name.find(’.auenland.lkt’, start_index) + len(’.auenland.lkt’)
log[’$ComputerName’] = computer_name[start_index:end_index]
return inward_array

We used the following simplified data & code to test this in a python shell:

dummy_data_computer_name = ‘thisisadummydataset"computer_name":"test-computername.DOMAIN.NAME"thisisadummydataset’
start_index = dummy_data_computer_name.find(’“computer_name”:"’) + len(’“computer_name”:"’)
end_index = dummy_data_computer_name.find(’.DOMAIN.NAME’, start_index) + len(’.DOMAIN.NAME’)
print(dummy_data_computer_name[start_index:end_index])

Expected result is: test-computername.DOMAIN.NAME

Which works in the shell.


But when we execute the full query (the new one from this post) we get the error name ‘slice’ is not defined.

Regards,
Jonas

Hi Jonas,
Thanks for the details. Now that we know what is expected, I would like to suggest following approach.

SQL block: SELECT regexp_extract($LogEvent, ‘“computer_name”:"([^"]+)’) AS $ComputerName, $LogEvent FROM OTHER WHERE $Duration=1d LIMIT 10

If Log event is in JSON format, you can make use of json_parse in the code block.

However, more effective solution would be to ‘Parse’ the required field using the Extractors. Please see this Extractor.

Kindly also refer to our Extractor support policy here.

Let us know if above helps.

Regards,
Bodhi

1 Like

Hi Bodhi,

awesome - thank you! I did not know regexp_extract was supported. We had to make a small adjustment (escape the character ") for the regex to work.

SELECT REGEXP_EXTRACT($LogEvent, ‘“computer_name”:"([^"]+)’) AS $ComputerName, $LogEvent FROM OTHER LIMIT 10

Quick question regarding the extractors: Can we define custom event channels (additionally to already existing like FIREWALL, SYSMON-FILE, etc.) to parse the logs to. I did not find anything regarding this in the documentation - sorry if I might have missed it. We have an application sending logs to our DNIF PoC that doesn’t fit in existing event channels context-wise.

Regards,
Jonas

1 Like

Hi Jason,

Glad it worked for you. SQL block allows one to use all standard functions and operations of SQL. SQL Block

Yes, it is very much possible to define your own streams ( your Custom Event Chanel is DNIF’s steam :slight_smile: )
Stream is a domain-specific collection of data from different sources that contributes to a unique dataset.
It denotes log event/log description of the particular category and Stream forms an integral part of Detection rules.

You can also refer to the DNIF Data Model to have a better understanding of the streams and fields incorporated in it. We encourage you to edit the extractor YAML to incorporate the stream that you wish to see and a video tutorial for the same will be published very soon.

Hi,

thank you! Just tried parsing into a new custom stream an it works without any issues :smiley:

One last question: The stream shows up in the stream overview, data is being parsed and I can search it with SQL. But I am unable to select the stream in the search-block. Is this not supported yet or does it just take some time for the stream to appear there?

Regards,
Jonas

Hi Jonas,
Amazing to see how quickly you are progressing !

The dropdown of Stream from Search block gets updated instantly and we expect no delay once the yaml file gets updated.

Can you please share the SQL query that you are running, for us to gain more context around this observation.

Regards,
Bodhi

Hi Bodhi,

the stream shows up in the stream overview:

I can search for data in it. But the Search-Block for quick queries does not allow me to select the created test stream “WINLOGBEAT-GENERIC”.

Regards,
Jonas

Hi Jonas,
Apologies for late reply.
So, we replicated this problem in our labs. I will keep you posted on the progress.

Regards,
Bodhi

Hi Bodhi,

no problem & thank you for the update!

Regards,
Jonas

Hi Jonas,
Is it possible for you to share the .yaml file that you modified in order to create the custom stream ?

Regards,
Bodhi

Hi Bodhi,

of course! For testing purposes I just copied the default ms-windows-winlogbeat extractor, changed the channel OTHER to WINLOGBEAT-GENERIC and added parsing for event id 4722 - user account enabled. (I then manually changed the extractor to it under collection status).

I can’t upload .yaml files here - I hope it is ok to just add it to my comment as a code bock

Regards,
Jonas

event-details:
- decoder: json
  event-key-format: '{winlog[event_id]}'
  event-key-mapping:
    '1':
      annotate:
        Action: PROCESS_ADDED
        Status: PASSED
        Stream: SYSMON-PROCESS
      translate:
        winlog[event_data][CommandLine]: CommandLine
        winlog[event_data][Company]: Company
        winlog[event_data][Description]: Description
        winlog[event_data][Image]: Image
        winlog[event_data][ParentCommandLine]: ParentCommandLine
        winlog[event_data][ParentImage]: ParentImage
        winlog[event_data][Product]: Product
        winlog[event_data][User]: User
    '10':
      annotate:
        Action: PROCESS_ACCESSED
        Status: PASSED
        Stream: SYSMON-PROCESS
      translate:
        winlog[event_data][CallTrace]: CallTrace
        winlog[event_data][Description]: Description
        winlog[event_data][GrantedAccess]: GrantedAccess
        winlog[event_data][TargetImage]: TargetImage
    '11':
      annotate:
        Action: FILE_CREATED
        Status: PASSED
        Stream: SYSMON-FILE
      translate:
        winlog[event_data][CreationUtcTime]: CreationUtcTime
        winlog[event_data][Image]: Image
        winlog[event_data][TargetFilename]: TargetFilename
    '12':
      annotate:
        Status: PASSED
        Stream: SYSMON-REGISTRY
      translate:
        winlog[event_data][Description]: Description
        winlog[event_data][EventType]: Action
        winlog[event_data][Image]: Image
        winlog[event_data][TargetObject]: TargetObject
    '13':
      annotate:
        Status: PASSED
        Stream: SYSMON-REGISTRY
      translate:
        winlog[event_data][Description]: Description
        winlog[event_data][Details]: Details
        winlog[event_data][EventType]: Action
        winlog[event_data][Image]: Image
        winlog[event_data][TargetObject]: TargetObject
    '15':
      annotate:
        Action: FILE_CREATED_STREAM_HASH
        Status: PASSED
        Stream: SYSMON-FILE
      translate:
        winlog[event_data][CreationUtcTime]: CreationUtcTime
        winlog[event_data][Image]: Image
        winlog[event_data][TargetFilename]: TargetFilename
    '16':
      annotate:
        Action: STATE_CHANGED
        Status: PASSED
        Stream: SYSMON-CONFIG
      translate:
        winlog[event_data][ConfigurationFileHash]: ConfigurationFileHash
        winlog[event_data][Configuration]: Configuration
    '17':
      annotate:
        Action: PIPE_CREATED
        Status: PASSED
        Stream: SYSMON-PIPE
      translate:
        winlog[event_data][Image]: Image
        winlog[event_data][PipeName]: PipeName
    '18':
      annotate:
        Action: PIPE_CONNECTED
        Status: PASSED
        Stream: SYSMON-PIPE
      translate:
        winlog[event_data][Image]: Image
        winlog[event_data][PipeName]: PipeName
    '19':
      annotate:
        Action: FILTER_ACTIVITY
        Status: DETECTED
        Stream: SYSMON-WMI
      translate:
        winlog[event_data][EventType]: Action
        winlog[event_data][Name]: Name
        winlog[event_data][Operation]: Operation
        winlog[event_data][User]: User
    '2':
      annotate:
        Action: PROCESS_FILE_TIME_CHANGED
        Status: PASSED
        Stream: SYSMON-PROCESS
      translate:
        winlog[event_data][Image]: Image
    '20':
      annotate:
        Action: CONSUMER_ACTIVITY
        Status: DETECTED
        Stream: SYSMON-WMI
      translate:
        winlog[event_data][Destination]: Destination
        winlog[event_data][Name]: Name
        winlog[event_data][Type]: Type
        winlog[event_data][User]: User
    '21':
      annotate:
        Action: CONSUMER_TO_FILTER_ACTIVITY
        Status: DETECTED
        Stream: SYSMON-WMI
      translate:
        winlog[event_data][Consumer]: Consumer
        winlog[event_data][Filter]: Filter
        winlog[event_data][Operation]: Operation
        winlog[event_data][User]: User
    '22':
      annotate:
        Action: DNS_QUERY
        Status: PASSED
        Stream: SYSMON-DNS
      translate:
        winlog[event_data][Image]: Image
        winlog[event_data][QueryName]: QueryName
        winlog[event_data][QueryResults]: QueryResults
        winlog[event_data][QueryStatus]: QueryStatus
    '23':
      annotate:
        Action: FILE_DELETED
        Status: DETECTED
        Stream: SYSMON-FILE
      translate:
        winlog[event_data][Image]: Image
        winlog[event_data][TargetFilename]: TargetFilename
        winlog[event_data][User]: User
    '3':
      annotate:
        Action: NETWORK_CONNECTION
        Status: PASSED
        Stream: SYSMON-NETWORK
      translate:
        winlog[event_data][Description]: Description
        winlog[event_data][DestinationHostname]: DstHost
        winlog[event_data][DestinationIp]: DstIP
        winlog[event_data][DestinationPort]: DstPort
        winlog[event_data][Image]: Image
        winlog[event_data][SourceHostname]: SrcHost
        winlog[event_data][SourceIp]: SrcIP
        winlog[event_data][SourcePort]: SrcPort
    '4':
      annotate:
        Action: SERVICE_STARTED
        Status: PASSED
        Stream: SYSMON-SERVICE
      translate:
        winlog[event_data][SchemaVersion]: SchemaVersion
        winlog[event_data][State]: State
    '5':
      annotate:
        Action: PROCESS_REMOVED
        Status: PASSED
        Stream: SYSMON-PROCESS
      translate:
        winlog[event_data][CommandLine]: CommandLine
        winlog[event_data][Company]: Company
        winlog[event_data][Description]: Description
        winlog[event_data][Image]: Image
        winlog[event_data][ParentCommandLine]: ParentCommandLine
        winlog[event_data][ParentImage]: ParentImage
        winlog[event_data][Product]: Product
        winlog[event_data][User]: User
    '6':
      annotate:
        Action: DRIVER_LOADED
        Status: PASSED
        Stream: SYSMON-DRIVER-LOAD
      translate:
        winlog[event_data][User]: User
    '7':
      annotate:
        Action: PROCESS_ACCESSED
        Status: PASSED
        Stream: SYSMON-IMAGE-LOAD
      translate:
        winlog[event_data][Company]: Company
        winlog[event_data][Description]: Description
        winlog[event_data][Product]: Product
        winlog[event_data][User]: User
    '8':
      annotate:
        Action: CREATE_REMOTE_THREAD
        Status: PASSED
        Stream: SYSMON-PROCESS
      translate:
        winlog[event_data][Description]: Description
        winlog[event_data][TargetImage]: TargetImage
  fallback:
    annotate:
      EventName: Generic WinlogBeat Event
      Stream: WINLOGBEAT-GENERIC
  first-match: \"Microsoft\-Windows\-Sysmon/Operational\"
  globals:
    translate:
      winlog[computer_name]: System
      winlog[event_data][UtcTime]: SystemTstamp
  subs:
    Action:
      CONSUMER_ACTIVITY: CONSUMER_ACTIVITY
      CONSUMER_TO_FILTER_ACTIVITY: CONSUMER_TO_FILTER_ACTIVITY
      CREATE_REMOTE_THREAD: CREATE_REMOTE_THREAD
      CreateKey: OBJECT_CREATED
      DNS_QUERY: DNS_QUERY
      DRIVER_LOADED: DRIVER_LOADED
      DeleteKey: OBJECT_DELETED
      DeleteValue: OBJECT_MODIFIED
      FILE_CREATED: FILE_CREATED
      FILE_CREATED_STREAM_HASH: FILE_CREATED_STREAM_HASH
      FILE_DELETED: FILE_DELETED
      NETWORK_CONNECTION: NETWORK_CONNECTION
      PIPE_CONNECTED: PIPE_CONNECTED
      PIPE_CREATED: PIPE_CREATED
      PROCESS_ACCESSED: PROCESS_ACCESSED
      PROCESS_ADDED: PROCESS_ADDED
      PROCESS_FILE_TIME_CHANGED: PROCESS_FILE_TIME_CHANGED
      PROCESS_REMOVED: PROCESS_REMOVED
      SERVICE_STARTED: SERVICE_STARTED
      STATE_CHANGED: STATE_CHANGED
      SetValue: OBJECT_MODIFIED
- decoder: json
  event-key-format: '{winlog[event_id]}'
  event-key-mapping:
    '1102':
      annotate:
        Action: AUDIT_LOG_CLEARED
        Status: PASSED
        Stream: IAM
      translate:
        winlog[user_data][SubjectUserName]: User
  fallback:
    annotate:
      EventName: Generic WinlogBeat Event
      Stream: WINLOGBEAT-GENERIC
  first-match: \"Microsoft\-Windows\-Eventlog\"
  globals:
    translate:
      '@timestamp': SystemTstamp
      winlog[computer_name]: System
- decoder: json
  event-key-format: '{winlog[event_id]}'
  event-key-mapping:
    '4611':
      annotate:
        Action: LOGIN
        Status: PASSED
        Stream: AUTHENTICATION
      translate:
        winlog[event_data][SubjectUserName]: User
    '4624':
      annotate:
        Action: LOGIN
        Status: PASSED
        Stream: AUTHENTICATION
      translate:
        winlog[event_data][IpAddress]: SrcIP
        winlog[event_data][SubjectUserName]: User
    '4625':
      annotate:
        Action: LOGIN
        Status: FAILED
        Stream: AUTHENTICATION
      translate:
        winlog[event_data][IpAddress]: SrcIP
        winlog[event_data][SubjectUserName]: User
    '4634':
      annotate:
        Action: LOGOUT
        Status: PASSED
        Stream: AUTHENTICATION
      translate:
        winlog[event_data][TargetUserName]: User
    '4648':
      annotate:
        Action: LOGIN
        Status: PASSED
        Stream: AUTHENTICATION
      translate:
        winlog[event_data][IpAddress]: SrcIP
        winlog[event_data][SubjectUserName]: User
    '4662':
      annotate:
        Action: AUDIT_LOG_CLEARED
        Status: PASSED
        Stream: IAM
      translate:
        winlog[event_data][SubjectUserName]: User
    '4663':
      annotate:
        Action: OBJECT_ACCESSED
        Status: PASSED
        Stream: IAM
      translate:
        winlog[event_data][SubjectUserName]: User
    '4672':
      annotate:
        Action: PRIVILEGE_SET
        Status: PASSED
        Stream: IAM
      translate:
        winlog[event_data][SubjectUserName]: User
    '4720':
      annotate:
        Action: USER_ACCOUNT_CREATED
        Status: PASSED
        Stream: IAM
      translate:
        winlog[event_data][SubjectUserName]: User
        winlog[event_data][TargetUserName]: TargetUser
    '4722':
      annotate:
        Action: USER_ACCOUNT_ENABLED
        Status: PASSED
        Stream: IAM
      translate:
        winlog[event_data][SubjectUserName]: User
        winlog[event_data][TargetUserName]: TargetUser
    '4724':
      annotate:
        Action: ACCOUNT_PASSWORD_RESET
        Status: PASSED
        Stream: IAM
      translate:
        winlog[event_data][SubjectUserName]: User
        winlog[event_data][TargetUserName]: TargetUser
    '4732':
      annotate:
        Action: SECURITY_LGRP_MEMBER_ADDED
        Status: PASSED
        Stream: IAM
      translate:
        winlog[event_data][SubjectUserName]: User
        winlog[event_data][TargetUserName]: TargetUser
    '4738':
      annotate:
        Action: USER_ACCOUNT_CHANGED
        Status: PASSED
        Stream: IAM
      translate:
        winlog[event_data][SubjectUserName]: User
        winlog[event_data][TargetUserName]: TargetUser
    '4742':
      annotate:
        Action: SYSTEM_USER_ACCOUNT_CHANGED
        Status: PASSED
        Stream: IAM
      translate:
        winlog[event_data][SubjectUserName]: User
        winlog[event_data][TargetUserName]: TargetUser
    '4768':
      annotate:
        Action: KRBTGT_REQ_ISSUED
        Status: PASSED
        Stream: AUTHENTICATION
      translate:
        winlog[event_data][TargetUserName]: User
    '4771':
      annotate:
        Action: KRB_PREAUTH_ERR
        Status: FAILED
        Stream: AUTHENTICATION
      translate:
        winlog[event_data][TargetUserName]: User
    '4776':
      annotate:
        Action: LOGIN
        Stream: AUTHENTICATION
      translate:
        winlog[event_data][TargetUserName]: User
    '5136':
      annotate:
        Action: OBJECT_MODIFIED
        Status: PASSED
        Stream: IAM
      translate:
        winlog[event_data][SubjectUserName]: User
    '5140':
      annotate:
        Action: OBJECT_ACCESSED
        Status: PASSED
        Stream: IAM
      translate:
        winlog[event_data][SubjectUserName]: User
    '5142':
      annotate:
        Action: OBJECT_ADDED
        Status: PASSED
        Stream: IAM
      translate:
        winlog[event_data][SubjectUserName]: User
    '5145':
      annotate:
        Action: NET_SHR_ACCESS_CHK
        Status: PASSED
        Stream: IAM
      translate:
        winlog[event_data][SubjectUserName]: User
  fallback:
    annotate:
      EventName: Generic WinlogBeat Event
      Stream: WINLOGBEAT-GENERIC
  first-match: \"Microsoft\-Windows\-Security\-Auditing\"
  globals:
    translate:
      '@timestamp': SystemTstamp
      winlog[computer_name]: System
- decoder: json
  event-key-format: '{winlog[event_id]}'
  event-key-mapping:
    '18454':
      annotate:
        Action: LOGIN
        Status: PASSED
        Stream: AUTHENTICATION
      translate:
        winlog[host][name]: User
    '18456':
      annotate:
        Action: LOGIN
        Status: FAILED
        Stream: AUTHENTICATION
      translate:
        winlog[host][name]: User
  fallback:
    annotate:
      EventName: Generic WinlogBeat Event
      Stream: WINLOGBEAT-GENERIC
  first-match: \"MSSQLSERVER\"
  globals:
    translate:
      '@timestamp': SystemTstamp
      winlog[computer_name]: System
extractor-id: 710
integration: Winlogbeat
master-filters:
- \"beat\"\:\"winlogbeat\"
product-name: Windows
provides-streams:
- SYSMON-PROCESS
- SYSMON-NETWORK
- SYSMON-FILE
- SYSMON-IMAGE-LOAD
- SYSMON-REGISTRY
- SYSMON-WMI
- SYSMON-PIPE
- SYSMON-SERVICE
- SYSMON-DNS
- IAM
- AUTHENTICATION
- WINLOGBEAT-GENERIC
schema-version: 1.0
source-description: Extractor for Windows Winlogbeat Events
source-name: WINDOWS
source-type: OS
vendor-name: Microsoft

2 Likes

Hi Jonas,

The ‘Fallback’ section where the Generic events are classed as ‘Other’ is unexpectedly working with a fixed value ‘Other’ only. Thank you for bringing this bug to our attention. We have recorded it, and this will be fixed in DNIF’s next release.

Regards,
Bodhi

Hi Bodhi,

sorry for the delayed response - I was out of office. Thank you for the update!

Regards,
Jonas

1 Like

Hi Jonas,
This bug is fixed on Version 9.1.0

Regards,
Shweta