1

I have text files 100MB+ in size and they have a lot of special chars. I can't import them into my DBMS because PostgreSql doesn't recognize these chars. How can I remove them? Which software does this?

I am currently using EmEditor but I have to find special chars manually and replace all. There's tons of them. Over 10k. I'm on Windows.

  • Do you have access to a Linux machine? It would be a single command line on Linux, if you can boot it for example with a Live CD – Jonas Stein Jun 13 '17 at 17:35
1

A very simple python script, or even from a terminal/command line interactive session, could read from an input file and write to an output file while changing the encoding to ASCII - you would have a choice of what to do about the nonconforming characters of:

  • ignore skip none ascii characters
  • replace with ?
  • xmlcharrefreplace output in a format like ꀀ
  • backslashreplace output in a format like \\ua000
  • namereplace output in a format like \\N{YI SYLLABLE IT} where names are available.

The code, in Python 3 would be something like, untested:

with open('somefilename') as infile: # You may need to add 'rb' to the open command
   with open('outname', 'wt') as outfile:  # You may need to use 'wb' here
      outfile.write(infile.read().encode('ascii', 'backslashreplace'))

The above code will read the input file transcode it and write it out - you could also, for large file specify a maximum buffer size in the read method.

See https://docs.python.org/3/howto/unicode.html for more details.

Python is free, gratis & open source, and available for just about every platform, it is pre-installed on most non-Windows platforms.

| improve this answer | |
1

Install CudaText editor.

  • Call menuitem "Plugins - Make Plugin"
  • Enter plugin name "MyRemove", and enter new plugin text, below.
  • Save, restart CudaText, new plugin appears "Plugins - MyRemove".
  • Open your file with non-ASCII
  • Run plugin
  • Save file as...

Plugin text:

from cudatext import *

class Command:
    def run(self):
        s=ed.get_text_all()
        for i in range(32):
            if not i in [10,13,9]:
                s=s.replace(chr(i), '')
        ed.set_text_all(s)
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.