0

Please recommend a gratis command-line (CLI) tool for Windows that will convert all HTML entities as well as common Unicode character numeric code representations within ANSI/UTF-7 encoded text files to their UTF-8 equivalents.

It can either directly output the contents to a UTF-8 encoded text file, or simply send the output to stdout, and I'll programmatically redirect the output to a file.

The data is all pages that have been scraped from the internet using HTTPS GET & POST requests.

I'm currently using a sed script I wrote to do this, and it has gotten very lengthy (and thus somewhat slow).

For example, &, &, and & all need to be converted to &.

Thus, when a file with this content is used for input:

This & that.
Bells & whistles.
Dogs & cats.

The output needs to be:

This & that.
Bells & whistles.
Dogs & cats.

Ideally, I would like the tool to also handle converting:

ç
ê
ñ

to:

ç
ê
ñ

Open source code is preferred, but not required. Portable (no installer and no registry modifications) is required. Cross-platform compatibility with Linux is a big plus, but not required.

1 Answer 1

1

There is a unix command called recode that can do this. You can find a Windows port as part of UnixUtils and there is a Chocolatey package which would install and add it to your PATH.

Example output:

C:\>type temp.txt
This & that.
Bells & whistles.
Dogs & cats.
ç
ê
ñ
C:\>recode html..utf8 temp.txt

C:\>type temp.txt 
This & that.
Bells & whistles.
Dogs & cats.
ç
ê
ñ
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.