1

I'm looking for a tool to help me analyze a binary file. There are tools which provide you with rank, the character that appears most of the time, (such as the character that appeared the most times is c with frequency f). But I want one which is able to do that not just for one character but for a string of bytes, (rather than just a string of chars).

For example, looking at the paragraph above, the string of hex bytes '7468', ("th"), appeared 11 times.

It would be a plus if the tool is available for Linux.

2
  • 1
    Practically all your text is about characters, yet you ask for a binary file. Please edit and remove the ambiguity. Also: what is the limit to the set of characters - any logical one or just the number of characters. What are you trying to accomplish? Price? – user416 Aug 1 '19 at 7:56
  • 1
    What's up with "characters". Of course, a binary file has ASCII character, maybe unprintable but are still called characters. And as for "text". binary file can contain readable text. – aven desta Aug 1 '19 at 9:11
0

I also don't understand what you want exactly, but here is something off the top of my head:

One such command line tool to extract characters from binary files is strings.

With perl you can analyse the output.

This command will count the number of vowels in strings extracted from a linux kernel binary:

sudo strings /boot/vmlinuz-4.15.0-55-generic | perl -000 -nlE "my \$s = tr/[aeiou]//; say \$s;" -

33627

If you want to count the number of fields (words) containing the string "Canonical" or "Douglas" in the Ubuntu Linux kernel, try this:

sudo strings /boot/vmlinuz-4.15.0-55-generic | perl -alne 'map { /Canonical|Douglas/i && $t++ } @F; END { print $t }'

8

3
  • Links to documentation on the web for strings and perl would make this so much more useful an answer. – Eric S Aug 1 '19 at 16:31
  • 1
    If you don't understand the question, you can ask in the comment. And your answer is not what I was looking for if you read my question. But I appreciate the effort. P.S. I have already found a way to solve my problem. – aven desta Aug 2 '19 at 5:31
  • 2
    @avendesta If you have solved your problem, you should post that solution as an answer. It is perfectly fine for you to do so and encouraged. You can even accept your own answer. – Eric S Aug 3 '19 at 16:46
0

Here is a simple python code implementation. The code reads as a binary [rb] the file [file_path], then for the given set of characters [user_set], it counts how many each set occurred in the file and makes a dictionary [stat] with 'character set in hexadecimal' as key, and 'count of that character set' as value. Finally, prints character set in hex with their corresponding count, ordered by count.

import operator
file_path = 'example.bin'
with open(file_path, 'rb') as f:
    content = f.read()

length = content.__len__()
user_set = 3

stat = {}

for i in range(length - user_set):
    select = content[i:i + user_set]
    count = 0
    for j in range(length - user_set):
        if select == content[j:j + user_set]:
            count += 1
            i += user_set - 1
    if count>1:               # if count=1, then its not included
        stat[select] = count


sorted_stat = sorted(stat.items(), key=operator.itemgetter(1), reverse=True)

for x in range(1,len(sorted_stat)):
    print(sorted_stat[x][0].hex()," :appeared -->",sorted_stat[x][1],"  times")

an example result

f7d483  :appeared --> 5   times
0191ed  :appeared --> 2   times
019075  :appeared --> 2   times
2
  • Please clarify which of the following hex strings would, (and would not), be counted amount the 5 appearances of "f7d483" if they were converted to binary: 1) "f7df83" 2) "df83f7" 3) "f78383" 4) "f783" 5) "f700d40183". – agc Sep 4 '19 at 14:11
  • 1
    @agc only the first one will be counted as f7d483. the code is looking for a three character string, that appears more than once. I probably did not understand your confusion. – aven desta Sep 4 '19 at 21:16
0

Strings or characters?

  1. If it's strings...

    Using strings to show the Top 10 strings in the kernel:

    sudo strings /vmlinuz | tr -s '[-[:space:]_]' '\n' | sort | 
         uniq -c | sort -rn | head
    

    Output:

        521 SPRQ
        443 3021
        418 rq
        328 OHDLBJFNAIEMCKG
        323 g`dbf
        278 f
        270 z
        260 g
        260 /
        254 L
    

    To narrow this down, just add a grep in there, with whatever set is of interest, (let's say it was "aeiouy"):

    sudo strings /vmlinuz | tr -s '[-[:space:]_]' '\n' | grep -i '[aeiouy]' | 
         sort | uniq -c | sort -rn | head
    

    Output:

    328 OHDLBJFNAIEMCKG
    215 U
    211 o
    209 O
    206 u
    197 y
    195 i
    193 Y
    193 E
    175 I
    
  2. If it's characters...

    sudo hexdump -v  -e '/1 "%.2x"'  -e '/1 " _%_u\_\n"' /vmlinuz | 
         sort | uniq -c | sort -rn | head
    

    Output shows Top 10 chars:

     204518 00 _nul_
      44349 ff _ff_
      38958 fe _fe_
      38507 7f _del_
      37624 bf _bf_
      37101 fd _fd_
      37066 fc _fc_
      36655 3f _?_
      36560 df _df_
      36509 fb _fb_
    

    To narrow this down, just add a grep in there, with whatever set is of interest, (let's say it was "aeiouy"):

    sudo hexdump -v  -e '/1 "%.2x"'  -e '/1 " _%_u\_\n"' /vmlinuz | 
         grep -i '_[aeiouy]_'  | 
         sort | uniq -c | sort -rn | head
    

    Output:

      34369 6f _o_
      33810 4f _O_
      33809 79 _y_
      32162 61 _a_
      31528 45 _E_
      31453 49 _I_
      30951 59 _Y_
      30358 75 _u_
      30204 65 _e_
      30190 69 _i_
    

    See also: "How to gather byte occurrence statistics in binary file?"

1
  • Thank you bro. @agc – aven desta Sep 4 '19 at 10:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.