I am looking for a program, or library that can "untokenize" a list of tokens.

For example, given the list of tokens (each element of the list is separated by a space) Forty-four patients ( 88 shoulders) completed the study ., the output should be Forty-four patients (88 shoulders) completed the study.

Any operating system, price or license is fine. I am mostly interested in the English language.

  • Apart from the couple of removed spaces the output seems to be identical to the input! – Steve Barnes May 7 '17 at 18:53
  • @SteveBarnes correct, the issue being which spaces to remove :-) – Franck Dernoncourt May 7 '17 at 19:07
  • I feel like I would need to read your mind to understand what you're after here. How about some input-output pair examples? – aparente001 May 13 '17 at 5:27
  • @aparente001 just use the typical English language rule regarding spacing and punctuation. – Franck Dernoncourt May 13 '17 at 13:24
  • Are you saying that the input has some extraneous spaces inserted in strange places, and the output should have the extraneous spaces removed? For the example you provided ("the study ."), my first thought would be to make a macro in MS Word that does a bunch of replacements. – aparente001 May 14 '17 at 5:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.