3

I have a large collection of PDFs that are highly structured, and I want to turn them into some other format that can preserve the structure (either .doc with special characters, xml, whatever, pretty much any standard, easy to parse format).

I need to be able to specify the follow things:

Sections are demarcated by enter image description here, section titles are all caps following the image, and end when encountering non-cap letters.

I need to preserve the bold/italic nature of the text if possible.

Requirements:

  • Can be paid or free (prefer free of course)
  • Must run on windows (8.1)
  • Scriptable (or have some sort of bulk process capabilities)
  • Be able to output something reasonable (doc/docx/xml/html/json)
  • Can be given images to recognize the section markers
  • If there is no GUI must support calling from .NET

I am comfortable writing regular expressions if need be. I would also take something that just preserved the structure (all section markers became qqqqqqqqqq, all line breaks preserved, etc)

I have already tried iTextSharp, PDFBox, and AbbyyFineReader 12 (professional), and none of them seem to be able to do this (all have issues with preserving line breaks in some cases).

Update: sample: http://www.gpo.gov/fdsys/pkg/CREC-2014-09-19/pdf/CREC-2014-09-19.pdf Ideas?

  • soandos, did you manage to succeed with this? I'm also trying to parse current issues of Congressional Record and if this has already been done, it would be extremely beneficial. – Daniel Janus Mar 24 '17 at 14:38
  • I did, but I did it by scraping the html pages, not the pdfs. Let me know if you want more details. I still have the code somewhere if you want it – soandos Mar 24 '17 at 14:40
1

PDF files are display oriented and not content oriented. Usually the page content has no internal structure (it is just a bunch of drawing instructions), this is why it is so hard to extract structured content.
No matter what PDF library you choose you will have to write some code to process your PDF files. If you can post a sample file I can take a look at it and offer some suggestions.

UPDATE: I looked at the file and the solution below shows a possible implementation using XFINIUM.PDF library (Disclaimer: I work for the company that develops XFINIUM.PDF).

As I said before the PDF file content has no structure, it is just a long list of drawing instructions (no matter it is showText or strokePath). This requires to extract the text content along with its position and perform the analysis of the text fragments based on their position, the analysis which will be specific to your files.

In your text the section separators are not images, the symbol is actually the letter 'f' printed with a symbol font so it is easy to detect when a section starts.
The text that follows is analysed based on its position and each text fragment is either added to current line (if Y baseline is roughly the same) or to a new line. A line of text consists of one or more text fragments, depending on how the PDF generator decides to draw it.
After the section content (a list of strings) has been retrieved, it is processed to detect the title. The first n lines that use only upper-case are considered to be the title.
The code below shows how to implement this logic, but it is just the basics to get you started.

The Section class:

public class Section
{
    public string Title { get; set; }

    private List<string> content = new List<string>();
    public List<string> Content
    {
        get { return content; }
    }
}

Load the document, extract the text fragments from the first page and transform them into sections.

PdfFixedDocument document = new PdfFixedDocument(CREC-2014-09-19.pdf");
PdfContentExtractor ce = new PdfContentExtractor(document.Pages[0]);
PdfTextFragmentCollection textFragments = ce.ExtractTextFragments();
List<Section> sections = ProcessSections(textFragments);
UpdateSectionTitle(sections);

Transform text fragments into sections:

public List<Section> ProcessSections(PdfTextFragmentCollection textFragments)
{
    List<Section> sections = new List<Section>();

    Section currentSection = null;
    string currentLine = null;
    for (int i = 0; i < textFragments.Count; i++)
    {
        PdfTextFragment tf = textFragments[i];

        if ((tf.Text == "f ") && (tf.PostScriptFontName == "Gpospec5"))
        {
            // Store the last line of the section that just ended.
            if ((currentSection != null) && (currentLine != null))
            {
                currentSection.Content.Add(currentLine);
            }

            currentSection = new Section();
            sections.Add(currentSection);
            currentLine = null;
            continue;
        }

        if (currentSection != null)
        {
            // Current text fragment has a different baseline from the previous fragment, create a new line.
            // Let's consider that if the difference is greater than 1/3 of font size we have a new line.
            if (Math.Abs(tf.FragmentCorners[0].Y - textFragments[i - 1].FragmentCorners[0].Y) > tf.TransformedFontSize / 3)
            {
                if (currentLine != null)
                {
                    // Store the current line.
                    currentSection.Content.Add(currentLine);
                }
                // Begin a new line.
                currentLine = "";
            }

            if (currentLine != "")
            {
                // If the X position of current text fragment is larger with a third of font size than end X position of previous fragment 
                // then we have a graphic space.
                if ((tf.FragmentCorners[0].X - textFragments[i - 1].FragmentCorners[1].X) > tf.TransformedFontSize / 3)
                {
                    currentLine = currentLine + " ";
                }
                currentLine = currentLine + tf.Text;
            }
            else
            {
                currentLine = tf.Text;
            }
        }
    }

    return sections;
}

Retrieve section title from content:

private static void UpdateSectionTitle(List<Section> sections)
{
    for (int i = 0; i < sections.Count; i++)
    {
        sections[i].Title = "";

        while (sections[i].Content.Count > 0)
        {
            // If the text is uppercase move it from content to title.
            if (IsStringUppercase(sections[i].Content[0]))
            {
                sections[i].Title += sections[i].Content[0] + " ";
                sections[i].Content.RemoveAt(0);
            }
            else
            {
                break;
            }
        }
    }
}
  • The text is actually encoded in this case as text. My issue is the linebreaks. I'll post a sample in a bit – soandos Oct 6 '14 at 19:34
  • Going to try it out at some point tomorrow and let you know how it goes. – soandos Oct 8 '14 at 1:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.