File pattern analysis: tool to count frequency of string of bytes

Question

I'm looking for a tool to help me analyze a binary file. There are tools which provide you with rank, the character that appears most of the time, (such as the character that appeared the most times is c with frequency f). But I want one which is able to do that not just for one character but for a string of bytes, (rather than just a string of chars).

For example, looking at the paragraph above, the string of hex bytes '7468', ("th"), appeared 11 times.

It would be a plus if the tool is available for Linux.

Answer 1

I also don't understand what you want exactly, but here is something off the top of my head:

One such command line tool to extract characters from binary files is strings.

With perl you can analyse the output.

This command will count the number of vowels in strings extracted from a linux kernel binary:

sudo strings /boot/vmlinuz-4.15.0-55-generic | perl -000 -nlE "my \$s = tr/[aeiou]//; say \$s;" -

33627

If you want to count the number of fields (words) containing the string "Canonical" or "Douglas" in the Ubuntu Linux kernel, try this:

sudo strings /boot/vmlinuz-4.15.0-55-generic | perl -alne 'map { /Canonical|Douglas/i && $t++ } @F; END { print $t }'

8

Answer 2

Here is a simple python code implementation. The code reads as a binary [rb] the file [file_path], then for the given set of characters [user_set], it counts how many each set occurred in the file and makes a dictionary [stat] with 'character set in hexadecimal' as key, and 'count of that character set' as value. Finally, prints character set in hex with their corresponding count, ordered by count.

import operator
file_path = 'example.bin'
with open(file_path, 'rb') as f:
    content = f.read()

length = content.__len__()
user_set = 3

stat = {}

for i in range(length - user_set):
    select = content[i:i + user_set]
    count = 0
    for j in range(length - user_set):
        if select == content[j:j + user_set]:
            count += 1
            i += user_set - 1
    if count>1:               # if count=1, then its not included
        stat[select] = count


sorted_stat = sorted(stat.items(), key=operator.itemgetter(1), reverse=True)

for x in range(1,len(sorted_stat)):
    print(sorted_stat[x][0].hex()," :appeared -->",sorted_stat[x][1],"  times")

an example result

f7d483  :appeared --> 5   times
0191ed  :appeared --> 2   times
019075  :appeared --> 2   times

Answer 3

Strings or characters?

1. If it's strings...

   Using strings to show the Top 10 strings in the kernel:

   sudo strings /vmlinuz | tr -s '[-[:space:]_]' '\n' | sort | 
        uniq -c | sort -rn | head
   

   Output:

       521 SPRQ
       443 3021
       418 rq
       328 OHDLBJFNAIEMCKG
       323 g`dbf
       278 f
       270 z
       260 g
       260 /
       254 L
   

   To narrow this down, just add a grep in there, with whatever set is of interest, (let's say it was "aeiouy"):

   sudo strings /vmlinuz | tr -s '[-[:space:]_]' '\n' | grep -i '[aeiouy]' | 
        sort | uniq -c | sort -rn | head
   

   Output:

   328 OHDLBJFNAIEMCKG
   215 U
   211 o
   209 O
   206 u
   197 y
   195 i
   193 Y
   193 E
   175 I
   

2. If it's characters...

   sudo hexdump -v  -e '/1 "%.2x"'  -e '/1 " _%_u\_\n"' /vmlinuz | 
        sort | uniq -c | sort -rn | head
   

   Output shows Top 10 chars:

    204518 00 _nul_
     44349 ff _ff_
     38958 fe _fe_
     38507 7f _del_
     37624 bf _bf_
     37101 fd _fd_
     37066 fc _fc_
     36655 3f _?_
     36560 df _df_
     36509 fb _fb_
   

   To narrow this down, just add a grep in there, with whatever set is of interest, (let's say it was "aeiouy"):

   sudo hexdump -v  -e '/1 "%.2x"'  -e '/1 " _%_u\_\n"' /vmlinuz | 
        grep -i '_[aeiouy]_'  | 
        sort | uniq -c | sort -rn | head
   

   Output:

     34369 6f _o_
     33810 4f _O_
     33809 79 _y_
     32162 61 _a_
     31528 45 _E_
     31453 49 _I_
     30951 59 _Y_
     30358 75 _u_
     30204 65 _e_
     30190 69 _i_
   

   See also: "How to gather byte occurrence statistics in binary file?"