2

This is a non-trivial task, e.g.

  • 'Á' -> 'A' in Brazilian
  • 'ö' -> 'oe' in German
  • 'ẞ' -> 'Ss' in German normal text, -> 'SS' in German ALLCAPS*
  • 'Ø' -> 'Oe' in Danish normal text, -> 'OE' in Danish ALLCAPS*
  • 'ǿ' -> 'ø' -> 'oe' in Danish normal text, -> depending on previous letter 'Oe' or 'oE' in Danish aLtErNaTiNg CaSe*

etc. so the API would have to take as inputs

  • source text
  • language
  • meta-information (likely more than I am aware from the examples above)

and output transliterated text.

As many languages have these constructs, I'd like to not list them all myself and wonder whether there is not some library out there that has done the job for me.

Partial answers encouraged - if e.g. an open-source library supported this for Brazilian but not for German, I'd still rather contribute the German mappings myself than write my own library from scratch.

Encoding is guaranteed to be UTF-16, conversion to other Unicode variants would not pain me either.

Note that pure unicode normalisation not only does not catch all forms of umlaut - e.g. not 'ß', 'ł', 'ø', only partially 'ǿ' - and it also cannot transliterate them language-appropriately. All it does is strip characters.

(I do not think there can be a purely algorithmic solution.)

* - Changing the case of some text is well enough encapsulated that doing this separately, outside the library, would be fine. That is (please correct me, if this is an i18n "popular-but-wrong" misconception of mine) one could always first transliterate, then apply case in two, strictly segregated steps.
I am only using this as an obvious example for how the correct transliteration may depend on other letters in the source.

Addendum: It occurs to me that writing any particular language binding ought to be mostly trivial, if only the dictionary existed in whatever sort of structured language. So, I'll also accept answers pointing to e.g. a Word Net or similar machine-processible collection, as long as it lists specifically these umlauts and the relevant rules for their transformation.
To that end I opened another question at Linguistics.

Improve this question
5
  • "Ss" in German text? "ss", I suppose.
    – Thomas Weller
    Commented Nov 19 at 16:33
  • Do you expect aLtErNaTiNg CaSe support out of the box? It looks to me as if you would do the replacement first and then do the casing afterwards. Otherwise it won't work anyway. Consider SøM ØxAmPlE, which would become SoeM OExAmPlE (all replacements in the same case, no longer alternating) or SoEm OeXaMpLe (expected by me, each character alternating, but where the casing completely flips around).
    – Thomas Weller
    Commented Nov 19 at 16:38
  • @ThomasWeller: No, I mentioned that casing is just my example how replacement may be context sensitive. Casing is a bad real-world example, as doing it as post-processing will always be easier - but I am not linguist enough to think up a better one, nor am I confident that none exist. I pretty much only know German umlaut transliteration rules and that's it, everything else is already looked up. (E.g. I had no idea about 'Ø' before writing this question, all I knew from the linked answer was that 'Ø' is a letter, which unicode normalisation does not replace correctly.)
    – Zsar
    Commented Nov 19 at 17:42
  • It is sometimes hard to find proper examples, when one has little experience with the subject, but I would like to avoid traps like 'Falsehoods programmers believe about names' - without first having to do the miniscule task of learning every single alphabet there is .
    – Zsar
    Commented Nov 19 at 17:45
  • That's a great link. Thanks for sharing. I had to deal with at least 5 of those misconceptions already.
    – Thomas Weller
    Commented Nov 19 at 18:21

1 Answer 1

Reset to default
1

I suppose that ICU4NET (I've used PyICU, but I suppose they're similar) can help and you can always override the Transliterator to create language specific rules

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.