0

I'm looking for a program for library that can take as input the Wall Street Journal (WSJ) section of the Penn TreeBank (PTB), and output the corresponding Probabilistic Context-Free Grammars (PCFGs).

If possible:

  • option to get a lexicalized PCFG
  • Python/Java/C++
  • runs on Windows 7 SP1 x64 Ultimate or Ubuntu

The Penn TreeBank consists of thousand of annotated sentences. Below are 3 of them:

( (S 
    (NP-SBJ 
      (NP (NNP Pierre) (NNP Vinken) )
      (, ,) 
      (ADJP 
        (NP (CD 61) (NNS years) )
        (JJ old) )
      (, ,) )
    (VP (MD will) 
      (VP (VB join) 
        (NP (DT the) (NN board) )
        (PP-CLR (IN as) 
          (NP (DT a) (JJ nonexecutive) (NN director) ))
        (NP-TMP (NNP Nov.) (CD 29) )))
    (. .) ))
( (S 
    (NP-SBJ (NNP Mr.) (NNP Vinken) )
    (VP (VBZ is) 
      (NP-PRD 
        (NP (NN chairman) )
        (PP (IN of) 
          (NP 
            (NP (NNP Elsevier) (NNP N.V.) )
            (, ,) 
            (NP (DT the) (NNP Dutch) (VBG publishing) (NN group) )))))
    (. .) ))

( (S 
    (NP-SBJ-1 
      (NP (NNP Rudolph) (NNP Agnew) )
      (, ,) 
      (UCP 
        (ADJP 
          (NP (CD 55) (NNS years) )
          (JJ old) )
        (CC and) 
        (NP 
          (NP (JJ former) (NN chairman) )
          (PP (IN of) 
            (NP (NNP Consolidated) (NNP Gold) (NNP Fields) (NNP PLC) ))))
      (, ,) )
    (VP (VBD was) 
      (VP (VBN named) 
        (S 
          (NP-SBJ (-NONE- *-1) )
          (NP-PRD 
            (NP (DT a) (JJ nonexecutive) (NN director) )
            (PP (IN of) 
              (NP (DT this) (JJ British) (JJ industrial) (NN conglomerate) ))))))
    (. .) ))
1

This sounds like a job for pythons SciKit Learn and the Natural Language Toolkit (NLKT) as it has its own CFG and treebank modules as demonstrated here.

  • Free
  • Python
  • Looks like a lot of academic work heading in the same direction using these tools.
  • Multi-platform, (even Windows)!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.