Using a third-party tool—parse

While manually parsing data, as seen in the previous recipe, works very well for small strings, it can be very laborious to tweak the exact formula to work with a variety of inputs. What if the input has an extra dash sometimes? Or it has a variable length header depending on the size of one of the fields?

A more advanced option is to use regular expressions, as we'll see in the next recipe. But there's a great module in Python called parse (https://github.com/r1chardj0n3s/parse), which allows us to reverse format strings. It is a fantastic tool that's powerful, easy to use, and greatly improves the readability of code.

Getting ready

Add the parse module to the requirements.txt file in our virtual environment and reinstall the dependencies, as shown in the Creating a virtual environment recipe.

The requirements.txt file should look like this:

delorean==1.0.0
requests==2.22.0
parse==1.14.0

Then, reinstall the modules in the virtual environment:

$ pip install -r requirements.txt
...
Collecting parse==1.14.0
  Downloading https://files.pythonhosted.org/packages/4a/ea/9a16ff916752241aa80f1a5ec56dc6c6defc5d0e70af2d16904a9573367f/parse-1.14.0.tar.gz
...
Installing collected packages: parse
  Running setup.py install for parse ... done
Successfully installed parse-1.14.0

How to do it…

1. Import the parse function:

   >>> from parse import parse
   

2. Define the log to parse, in the same format as in the Extracting data from structured strings recipe:

   >>> LOG = '[2018-05-06T12:58:00.714611] - SALE - PRODUCT: 1345 - PRICE: $09.99'
   

3. Analyze it and describe it as you would do when trying to print it, like this:

   >>> FORMAT = '[{date}] - SALE - PRODUCT: {product} - PRICE: ${price}'
   

4. Run parse and check the results:

   >>> result = parse(FORMAT, LOG)
   >>> result
   <Result () {'date': '2018-05-06T12:58:00.714611', 'product': '1345', 'price': '09.99'}>
   >>> result['date']
   '2018-05-06T12:58:00.714611'
   >>> result['product']
   '1345'
   >>> result['price']
   '09.99'
   

5. Note the results are all strings. Define the types to be parsed:

   >>> FORMAT = '[{date:ti}] - SALE - PRODUCT: {product:d} - PRICE: ${price:05.2f}'
   

6. Parse once again:

   >>> result = parse(FORMAT, LOG)
   >>> result
   <Result () {'date': datetime.datetime(2018, 5, 6, 12, 58, 0, 714611), 'product': 1345, 'price': 9.99}>
   >>> result['date']
   datetime.datetime(2018, 5, 6, 12, 58, 0, 714611)
   >>> result['product']
   1345
   >>> result['price']
   9.99
   

7. Define a custom type for the price to avoid issues with the float type:

   >>> from decimal import Decimal
   >>> def price(string):
   ...   return Decimal(string)
   ...
   >>> FORMAT = '[{date:ti}] - SALE - PRODUCT: {product:d} - PRICE: ${price:price}'
   >>> parse(FORMAT, LOG, {'price': price})
   <Result () {'date': datetime.datetime(2018, 5, 6, 12, 58, 0, 714611), 'product': 1345, 'price': Decimal('9.99')}>
   

How it works…

The parse module allows us to define a format, as a string, that reverses the format method when parsing values. A lot of the concepts that we discussed when creating strings apply here—put values in brackets, define the type after a colon, and so on.

By default, as seen in step 4, the values are parsed as strings. This is a good starting point when analyzing text. The values can be parsed into more useful native types, as shown in steps 5 and 6 in the How to do it section. Please note that while most of the parsing types are the same as the ones in the Python Format Specification mini-language, there are some others available, such as ti for timestamps in ISO format.

Though we are using timestamp in this book in a more liberal way as a replacement for "Date and time," in the strictest sense, it should only be used for numeric formats, such as Unix timestamp or epoch, defined as the number of seconds since a particular time.

The usage of a timestamp that includes other formats is common anyway as it's a clear and understandable concept, but be sure to agree to formats when sharing information with others.

If native types are not enough, our own parsing can be defined, as demonstrated in step 7 of the How to do it section. Note that the definition of the price function gets a string and returns the proper format, in this case, a Decimal type.

All the issues about floats and price information described in the There's more section of the Extracting data from structured strings recipe apply here as well.

There's more…

The timestamp can also be translated into a delorean object for consistency. Also, delorean objects carry over time zone information. Adding the same structure as in the previous recipe gives the following object, which is capable of parsing logs:

import parse
from decimal import Decimal
import delorean
class PriceLog(object):
  def __init__(self, timestamp, product_id, price):
    self.timestamp = timestamp
    self.product_id = product_id
    self.price = price
  def __repr__(self):
    return '<PriceLog ({}, {}, {})>'.format(self.timestamp,
                                            self.product_id,
                                            self.price)
  @classmethod
  def parse(cls, text_log):
    '''
    Parse from a text log with the format
    [<Timestamp>] - SALE - PRODUCT: <product id> - PRICE: $<price>     to a PriceLog object
   '''
    def price(string):
      return Decimal(string)
    def isodate(string):
      return delorean.parse(string)
      FORMAT = ('[{timestamp:isodate}] - SALE - PRODUCT: {product:d} - '
              'PRICE: ${price:price}')
      formats = {'price': price, 'isodate': isodate}
      result = parse.parse(FORMAT, text_log, formats)
      return cls(timestamp=result['timestamp'],
               product_id=result['product'],
               price=result['price'])

So, parsing it returns similar results:

>>> log = '[2018-05-06T14:58:59.051545] - SALE - PRODUCT: 827 - PRICE: $22.25'
>>> PriceLog.parse(log)
<PriceLog (Delorean(datetime=datetime.datetime(2018, 6, 5, 14, 58, 59, 51545), timezone='UTC'), 827, 22.25)>

This code is contained in the GitHub file, https://github.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/blob/master/Chapter01/price_log.py

All supported parse types can be found in the documentation at https://github.com/r1chardj0n3s/parse#format-specification.

See also

• The Extracting data from structured strings recipe, covered earlier in this chapter, to learn how to use simple processes to get information from text.
• The Introducing regular expressions recipe, covered later in this chapter, to learn how to detect and extract patterns from text.
• The Going deeper into regular expressions recipe, covered later in this chapter, to further your knowledge of regular expressions.