4

To start, I have used Splunk for log data analysis. One of the great features about Splunk is that it can automatically detect the date from a log event, no matter what format you give it (it is 99% accurate from my experience)

Is there an existing library that is capable of this, being able to parse the timestamp into a date, regardless of the format and where it lies in the string? (Preferably in Java).

Here are some examples (these come from actual logs):

(Actual Date in format yyyy-MM-dd hh:mm:ss.SSS ZZZ): Log Entry
2015-11-19 13:19:24.000 -0500: 172.24.133.22 23958 online.acme.com - - [19/Nov/2015:13:19:24 -0500] "POST /app/services/jsevents/loginfo?request HTTP/1.1" 204 - - 1898041 "https://online.acme.com/app/web/index/home" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"
2015-11-19 13:21:11.000 (no Time Zone): [Thu Nov 19 13:21:11 2015] [error] [client 172.24.133.27] File does not exist: /server/appportal/docroot/app/ui/public/res/css/img, referer: https://online.acme.com/app/web/documents/viewDocuments?req_type=INSDOC&di=151118AH01CT31447828851329810A53&oc_id=ep0109&lob=AUTOHOME&
2015-11-19 13;23:59.912 -0500: [11/19/15 13:23:59:912 EST] 0000009a PmiRmArmWrapp I   PMRM0003I:  parent:ver=1,ip=172.24.237.31,time=1447880900823,pid=28380,reqid=1245500,event=1 - current:ver=1,ip=172.24.237.31,time=1447880900823,pid=28380,reqid=1245505,event=1 type=Web Services Requestor detail=wsrequestor:DocServicesBeanPort.findDoc?transport=https&parameters=xref elapsed=167
2015-11-19 13:29:36.603 (no Time Zone): 2015-11-19 13:29:36,603 [WebContainer : 26] WARN  172.24.133.26 - - - - Default c.a.p.w.c.user.FindUserProfileBean - Invalid User Type Argument. Deferring to default.
2015-11-19 07:00:40.000 (no Time Zone): 19-Nov-2015.07:00:40: [INFO ] com.acme.app.legacy.LegacyConnector  - Succesful CHANGE Event.
1

And the best answer I found so far is a Java library called Natty (with a little bit of help from Regex and Joda Time). Natty is a natural language parser capable of parsing all sorts of dates in many different formats, but it is not always terrific with times. For me, times are the easy part, because in every log event I listed above, the time is a fairly standard format (hh:mm:ss, with an occasional SSS (for milliseconds) separated by a dot (.) comma (,) or colon (:). I am actually very impressed with this library's ability to parse a date from natural language in so many different forms.

If I can get Natty to tell me where the date is in a string (and refine it), then the time is usually pretty close to it, so I can use a regular expression to get the time.

For anyone interested, I have posted a sample usage of this library, with some augmentation by regular expressions and Joda time:

package org.joestelmach.natty;

import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import org.joda.time.DateTime;
import org.joda.time.MutableDateTime;

import com.joestelmach.natty.DateGroup;
import com.joestelmach.natty.Parser;

public class ParserTest {

    private static Parser parser = new Parser();

    public static void main(String[] args) {
        String [] lines = {
                "172.24.133.22 23958 online.acme.com - - [19/Nov/2015:13:19:24 -0500] \"POST /app/services/jsevents/loginfo?request HTTP/1.1\" 204 - - 1898041 \"https://online.acme.com/app/web/index/home\" \"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)\"",
                "[Thu Nov 19 13:21:11 2015] [error] [client 172.24.133.27] File does not exist: /server/appportal/docroot/app/ui/public/res/css/img, referer: https://online.acme.com/app/web/documents/viewDocuments?req_type=INSDOC&di=151118AH01CT31447828851329810A53&oc_id=ep0109&lob=AUTOHOME&",
                "[11/19/15 13:23:59:912 EST] 0000009a PmiRmArmWrapp I PMRM0003I: parent:ver=1,ip=172.24.237.31,time=1447880900823,pid=28380,reqid=1245500,event=1 - current:ver=1,ip=172.24.237.31,time=1447880900823,pid=28380,reqid=1245505,event=1 type=Web Services Requestor detail=wsrequestor:DocServicesBeanPort.findDoc?transport=https&parameters=xref elapsed=167",
                "2015-11-19 13:29:36,603 [WebContainer : 26] WARN 172.24.133.26 - - - - Default c.a.p.w.c.user.FindUserProfileBean - Invalid User Type Argument. Deferring to default.",
                "19-Nov-2015.07:00:40: [INFO ] com.acme.app.legacy.LegacyConnector - Succesful CHANGE Event.",
                "DEBUG|2015-11-19-01:14:17.628|WebContainer : 0|          TRACE:BEGIN (876.411s) ContractDAO.findByContractNumberAndSuffix",
        };

        SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS ZZZ");

        Pattern timePattern = Pattern.compile("(\\d{2}):(\\d{2}):(\\d{2})(?:[\\.,:](\\d{3}))?");

        final int DATE_TEXT_SPAN_TOLERANCE = 15;

        for (String line : lines) {
            List<DateGroup> dateGroups = parser.parse(line);
            // Refine the list, since it's possible the parser will return multiple matches
            // (though usually only one of them is an actual match)
            DateGroup group = refineDateGroupList(dateGroups);
            List<Date> dateList = group.getDates();
            Date firstDate = dateList.get(0);
            int column = group.getPosition();
            int length = group.getText().length();
            String dateText = group.getText();
            int hour = 0, minute = 0, second = 0, milli = 0;
            boolean matched = false;
            Matcher timeMatcher;
            // First let's check if the time is in the matched date
            // We always check the last match in the regex because of the assumption that the time
            // is usually after the date in the string
            if ((timeMatcher = findLastMatch(dateText, timePattern)) != null) {
                matched = true;
                hour = new Integer(timeMatcher.group(1));
                minute = new Integer(timeMatcher.group(2));
                second = new Integer(timeMatcher.group(3));
                if (timeMatcher.group(4) != null) {
                    milli = new Integer(timeMatcher.group(4));
                }
            }
            // If not, go X characters forward and backward and see if we can find a time
            if (!matched) {
                int timeSearchStart = Math.max(column - DATE_TEXT_SPAN_TOLERANCE, 0);
                int timeSearchEnd = Math.min(column + length + DATE_TEXT_SPAN_TOLERANCE, line.length());
                String timeSearchSubstring = line.substring(timeSearchStart, timeSearchEnd);
                timeMatcher = timePattern.matcher(timeSearchSubstring);
                if ((timeMatcher = findLastMatch(timeSearchSubstring, timePattern)) != null) {
                    hour = new Integer(timeMatcher.group(1));
                    minute = new Integer(timeMatcher.group(2));
                    second = new Integer(timeMatcher.group(3));
                    if (timeMatcher.group(4) != null) {
                        milli = new Integer(timeMatcher.group(4));
                    }
                }
            }
            MutableDateTime jodaTime = new MutableDateTime(firstDate.getTime());
            jodaTime.setHourOfDay(hour);
            jodaTime.setMinuteOfHour(minute);
            jodaTime.setSecondOfMinute(second);
            jodaTime.setMillisOfSecond(milli);
            firstDate = jodaTime.toDate();
            System.out.printf("DATE: %s [%d] (from matched text \"%s\")\n%s\n====\n", dateFormat.format(firstDate), firstDate.getTime(), dateText, line);
        }

    }

    // Refines the date groups returned from the Natty parser by making sure the date
    // retrieved from the entire line is the same as the date retrieved from the matched
    // text
    private static DateGroup refineDateGroupList(List<DateGroup> dateGroups) {
        if (dateGroups.size() == 1) {
            return dateGroups.get(0);
        }
        if (dateGroups.size() == 0) {
            return null;
        }
        for (DateGroup group : dateGroups) {
            List<DateGroup> subDateGroups = parser.parse(group.getText());
            DateGroup subDateGroup = refineDateGroupList(subDateGroups);
            if (subDateGroup == null) {
                return null;
            }
            List<Date> dateList = group.getDates();
            if (dateList.size() == 0) {
                // This shouldn't actually happen
                return null;
            }
            // Choose the first date
            Date expectedDate = dateList.get(0);
            List<Date> subDateList = subDateGroup.getDates();
            if (subDateList.size() == 0) {
                // Again, this shouldn't happen
                return null;
            }
            // Again, choose the first date
            Date actualDate = subDateList.get(0);
            if (isSameDate(expectedDate, actualDate)) {
                return group;
            }
        }
        // If none of them match, the first one wins
        return dateGroups.get(0);
    }

    // Makes sure that the yyyy, MM, and dd are the same between two dates
    private static boolean isSameDate(Date expectedDate, Date actualDate) {
        DateTime expectedDateTime = new DateTime(expectedDate.getTime());
        DateTime actualDateTime = new DateTime(actualDate.getTime());
        return expectedDateTime.year().equals(actualDateTime.year()) &&
                expectedDateTime.monthOfYear().equals(actualDateTime.monthOfYear()) &&
                expectedDateTime.dayOfMonth().equals(actualDateTime.dayOfMonth());
    }

    private static Matcher findLastMatch(String text, Pattern pattern) {
        int length = text.length();
        if (text.length() == 0) return null;
        for (int start = length - 1; start > 0; start --) {
            String subText = text.substring(start, length);
            Matcher matcher = pattern.matcher(subText);
            if (matcher.find()) {
                return matcher;
            }
        }
        return null;
    }

}

And sure enough, the output to this is:

DATE: 2015-11-19 13:19:24.000 -0500 [1447957164000] (from matched text "19/Nov/2015")
172.24.133.22 23958 online.acme.com - - [19/Nov/2015:13:19:24 -0500] "POST /app/services/jsevents/loginfo?request HTTP/1.1" 204 - - 1898041 "https://online.acme.com/app/web/index/home" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"
====
DATE: 2015-11-19 13:21:11.000 -0500 [1447957271000] (from matched text "Thu Nov 19 13:21:11")
[Thu Nov 19 13:21:11 2015] [error] [client 172.24.133.27] File does not exist: /server/appportal/docroot/app/ui/public/res/css/img, referer: https://online.acme.com/app/web/documents/viewDocuments?req_type=INSDOC&di=151118AH01CT31447828851329810A53&oc_id=ep0109&lob=AUTOHOME&
====
DATE: 2015-11-19 13:23:59.000 -0500 [1447957439000] (from matched text "11/19/15 13:23:59")
[11/19/15 13:23:59:912 EST] 0000009a PmiRmArmWrapp I PMRM0003I: parent:ver=1,ip=172.24.237.31,time=1447880900823,pid=28380,reqid=1245500,event=1 - current:ver=1,ip=172.24.237.31,time=1447880900823,pid=28380,reqid=1245505,event=1 type=Web Services Requestor detail=wsrequestor:DocServicesBeanPort.findDoc?transport=https&parameters=xref elapsed=167
====
DATE: 2015-11-19 13:29:36.000 -0500 [1447957776000] (from matched text "2015-11-19 13:29:36")
2015-11-19 13:29:36,603 [WebContainer : 26] WARN 172.24.133.26 - - - - Default c.a.p.w.c.user.FindUserProfileBean - Invalid User Type Argument. Deferring to default.
====
DATE: 2015-11-19 07:00:40.000 -0500 [1447934440000] (from matched text "19-Nov-2015")
19-Nov-2015.07:00:40: [INFO ] com.acme.app.legacy.LegacyConnector - Succesful CHANGE Event.
====
DATE: 2015-11-19 01:14:17.628 -0500 [1447913657628] (from matched text "2015-11-19")
DEBUG|2015-11-19-01:14:17.628|WebContainer : 0|          TRACE:BEGIN (876.411s) ContractDAO.findByContractNumberAndSuffix
====

It's not bullet-proof yet, but it sets the stage for exactly what I need to do.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.