2

This is a abstract question for me, hope someone can direct me to one lane.

I want to analyze content of file and want to know info like below:

Is the dedup ratio high or low before I run dedup on files? I'm not sure the if dedup ratio can pre-examine. (dedup means de-duplication. it's used to remove the duplication part of files and achieve compress-like function)

I want to know if any tools(like python, C or else) can achieve analyze-like function?

Improve this question
7
  • Also, please explain what you mean by "dedup". Do you mean "Is the file suitable to be compressed?" – Nicolas Raoul♦ Oct 3 '18 at 8:40
  • Sounds like "de-duplicate" to me. Maybe an existing tool? – Mawg says reinstate Monica Oct 3 '18 at 8:49
  • 2
    @NicolasRaoul , I updated. Dedup means one tool to de-duplication and make the duplication part apprear once. It's like compression. – code_worker Oct 3 '18 at 8:50
  • 1
    @Mawg . It's tool right. I wonder if I can pre-check the dedup ratio before I start to dedup? or something like that? – code_worker Oct 3 '18 at 8:52
  • 1
    @Mawg , maybe I can analyze if the data of file is messy? If messy, I can predict it's not efficient after dedup – code_worker Oct 4 '18 at 1:35
1

Use a fast compressor on the file data; if the compressed size is:

  • much smaller than the raw data, then the file would probably benefit from deduplication.

  • the same size as the raw data, then the file would probably not benefit from deduplication.

Example code using fast compressor lzop:

lzop < raw.dat | wc -c raw.dat - | head -n -1
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.