Searching Summary

In the previous few chapters, we covered a number of the regex searching patterns to find strings in text, and in this chapter, we will provide a few examples of how to use those options to solve some text processing problems.

Help System Parser

In the beginning of the book, we described the help system we wanted to process messages from. We can now use our various regex patterns to create an object to process those help messages. The purpose of the object is to take a text message and fill the following properties if they can be extracted:

• AssignedTo: Help desk rep who took the message.
• UserDate: Date the user indicated they called.
• UserTime: Time of the call.
• EmailAddr: Email address, if one is found.
• PhoneNumber: Phone number, if found.
• ComplainURL: Website the user complained about.

To create the parse class, we first need to create the class and declare a few variables.

We are keeping the class very simple, using public variables rather than properties. In our constructor, we are going to load our regex expression list. It is often a good idea to keep the regular expressions loaded in one spot, in this case in a dictionary, to make it easier to update them as testing progresses or to enhance the expressions.

And finally, we include a method to take the message string and extract the pieces it can get from the message. There is not a lot of code in the method, and the power of the regex engine is on display here.

While there are improvements we can make (and will in later chapters), this simple class illustrates how powerful regular expressions can be when dealing with a larger amount of unstructured text.

Named Entity Parsing

Natural Language Processing (NLP) is a part of artificial intelligence that deals with having a computer appear to understand native language text, such as English. The complexities of the English language make this an extremely difficult task for computers to handle. One component of NLP is trying to extract “named entities” from text; named entities could be phone numbers, currency amounts, city and state names, people, etc. This is done with lists, grammar rules, and often with some regular expressions.

For example, if the following ad were found online, the highlighted text might be the named entities we want to extract: the year, the type of car, and the price.

2011 Honda Civic excellent condition color BLUE 56K miles for questions ask for George 215 555 1212 from 8AM TO 5PM THANK YOU... Firm $10,300.

To find the type of car, you would most likely have a list of car makes and models and do a simple string search for these. However, the year and price could easily be handled using regex values.

By combining the list lookup and the regex patterns, writing a program to “read” online classified ads and report on year, type of car, and price would not be very difficult.

An example CarSearch class is illustrated below:

The class provides a method called ReadAd() which takes the ad content as a parameter. It will search for one of several models, and if the model is found, will attempt to return the price and year (as long as it is greater than 2000) of the car it found.

Note: The ? after the INT and DOUBLE type declarations allow us to place a NULL value in those fields, which is more appropriate if we don’t find a match in the message text. It is a shortcut for Nullable<int> or Nullable<double>.

Validation

One caveat with regular expressions is that they will sometimes find patterns that look alright, but might not be accurate. While regex searching by itself is a very useful tool, you should also look further to see if the returned string is valid, not just that it matches a pattern. Let’s look at some examples.

Credit Cards

The following regex will identify most credit cards:

^((4\d{3})|(5[1-5]\d{2}))(-?|\040?)(\d{4}(-?|\040?)){3}|^(3[4,7]\d{2})(-?|\040?)\d{6}(-?|\040?)\d{5}

However, even values that match the pattern might not be a valid credit card number. But credit card numbers have a checksum digit built (using the Luhn algorithm) into the number that ensures not only the pattern looks OK, but that it is also a valid combination of numbers. The following C# function performs the checksum function and returns a Boolean value indicating whether or not the credit card number is valid.

If you need to validate credit card numbers, use the regex in combination with the CC_Check() function to ensure you’ve got a valid card number.

Note: It is still possible after both checks to have a valid, but unissued credit card number, but such a function would require some sort of API communication with the credit card issuer.

EAN 8 number

The EAN 8 number is a simple 8 digit number typically used with a barcode for smaller products. The regex is valid and the EAN-8 number is simply 8 digits, \d{8}. This is a primary example where a regular expression would very likely find more matches than only EAN-8 bar codes. Fortunately, the last digit is a checksum; the code below is a C# function to validate that checksum.

Email

Email is often validated using regular expressions; a quick search for regex email validation will yield dozens of email validation regex patterns. However, most patterns are either too broad (i.e. (\w[-._\w]*\w@\w[-._\w]*\w\.\w{2,3}) ) or will prevent some valid email patterns from getting through. For example, many expressions don’t accept a + sign in the email address, even though it is a valid character. Email addresses are based on RFC (Request for Comment) 5321 (and others), so a true regex pattern to validate email should handle that standard. You can view such a pattern at the website http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html. However, the pattern is over 6,000 bytes long.  

Note: The example email regex pattern above does not work in .NET, only Perl, but illustrates how complex a pattern to validate email can become.

If you are creating an application that requires email validation, you should visit Dominic Sayer’s website http://www.isemail.info/ which provides a great email validator that handles email addresses that many regex patterns don’t. Of course, it would probably be better to accept an email address from a potential customer rather than display an error message for an invalid email address and lose that customer.

Summary

In the first half of this book, we focused on how to search with regular expressions and covered a fair amount of what can be done with the regex literal characters and metacharacters. It is important to keep in mind that while regex is a powerful tool for searching text, you should also consider additional validations, such as checksum, API look-ups, etc. You can use the regex to determine whether or not a string is worth running a checksum against or sending to an API, but don’t rely on it for strict validation that the data is OK.

In the next section of the book, we are going to explore some additional regex features, and while we continue to focus on searching, we will also begin to explore what can be done with the resulting string or group(s) that a regex finds. For example, we might to want some code to not only extract phone numbers, but to also slice the phone number into area code, main number, and possible extension. So if you feel comfortable with using regex patterns for searching, read on to see what else can be done.