What’s With Python Decorators?

Nearly four years ago, I started learning a new language, Python, because I felt that as a newer language it would make sense to use it as the implementation language for a water endpoint/meter configuration and reads store and forward system. That reason extended to someone needing to maintain the system after I was gone. Using Python alone did not make the application’s modules easier to maintain, but I give some credit to the language for it.

Of all things in Python that are visually difficult for me to comprehend are decorators. I have never used one or did not know I needed to. If I were using Python on a more regular basis, like when I was designing and implementing the system, I would just decide to conquer this subject once and for all. However, one thing I have learned over the years is to be suspicious of overly complicated things that perhaps should not be, and I leave decorators there. I could have missed it, but I was not able to find a description of decorators in Mark Pilgrim’s Dive Into Python.

Perhaps I will find that post out there that just puts it all together for me, but so far, I just see confusion where decorators are concerned.

Epilogue:

I try never to give up, so this tutorial is a good start.


Is Learning Functional Programming a Good Thing?

Those folks who follow this blog or http://octopusgrabbus.wordpress.com might come to the conclusion that I’ve abandoned Python, but that is not the case. I started learning Python in 2009 by writing some applets to perform database housekeeping, and then eventually implemented an application in 2010 for our water department. This application, which includes a Django-powered web site was written almost exclusively in Python.

I have been interested in functional programming for a few years now, and started looking at Haskell a couple of years ago. As the deadline to start designing our water department application loomed closer,  I did not know Haskell well enough to use it. Also, as a language, Haskell did not appear to have the broad library support enjoyed by Python. So, I chose Python as the implementation language for this project.

Earlier this year I came across a description of The Joy of Clojure at my favorite book sellers — http://softpro.com — and started learning more about Clojure. Having never programmed in a functional programming language or a Lisp dialect, I am finding Clojure a little daunting. However, a project has presented itself, which involves a faster way of processing our billing data through address verification.

A project is like a carrot and stick; that is it provides cause to use extraordinary effort to learn something new quickly or solve a particularly difficult problem. The project will require more than a Clojure program. Some 4GL and bash scripts will be used. However, the Clojure program will be substantial enough to get me familiar with the language and will have required enough effort to start writing code.

One of the things that has attracted me to functional programming (FP) languages is the same thing that attracted me to object oriented languages when I first learned C++ in the 1990s. Learning C++ improved my overall programming skills, whether I was writing object-oriented software or not. With learning C++, my analogy was it was like rotating your mind 90 degrees. With Clojure it is that and more.

Just one of the ways that learning Clojure has made me think differently is using the various sequence functions. Most of what I want to do with billing data fits well into Clojure’s strengths. That is you can apply the same functions to all the data and sort out data easily that does not fit your needs. At the moment, I do not see a use for using Clojure in a multi-process/multi-threaded environment.

I am also appreciating providing for data at the beginning, binding in Clojure, declaring variables in imperative languages like C. Python’s strength in the the ability to declare an object anywhere is also in my opinion a weakness that can lead to sloppy style, and I know this from direct experience.

So my conclusion is this. If I never write another line of Clojure code after my current project, my time will not have been wasted. Clojure requires a different way of thinking. It never hurts to take a different point of view, especially with problem solving. However, I think more Clojure projects will be forthcoming.


Tolerant Technical Communities

I don’t know what it is about the Clojure community, but it is very accepting of beginners and questions that sometimes ask  for feedback stating the obvious.  I have been programming over thirty years and paid to do so for over twenty-five years. I have used a lot of languages, but Lisp syntax has always thrown me for a loop.

The Clojure users group on Google http://groups.google.com/group/clojure?hl=en, Getting Clojure http://www.gettingclojure.com/, and resources on this blog’s Clojure Resource Page https://drknucklehead.wordpress.com/clojure-resources/ are good places to ask questions and get help.

It would be unfair of me not to include the Python community in this tribute as well. The core Python community is always encouraging and willing to answer questions. They are also a great group of people.


Programming Languages Used and Using

These are the computer programming languages I have used over the past few years.

1985 – 1987

PL/I Subset G

1987 – 1990

C, Bliss, Assember (DEC VAX)

1990 – 2000

C, C++

2000 – 2002

VBScript/VBA

2002-2003

Java, C, C++, Perl

2004 – current

Informix 4GL, C, VB, Perl, Python, JavaScript, Clojure


Book Review: Pro Python Marty Alchin

Pro Python is one of those books whose detail and clear writing took some time to appreciate. I purchased it while still learning Python, and other than taking a cursor look at the chapters, I did not have time to read it. I purchased the book during the implementation of an Automated Meter Reading (AMR) application that handles meter configuration and reads.

Eventually, I had time to read the book in greater detail. First, it contains the first, full, good description of Python decorators I have ever read. I had searched the web pretty thoroughly for an explanation before finding in Pro Python.  The book breaks down advanced topics into basics, functions, classes, meta programming, and includes distribution, not always covered by other texts.

This book does more to explain the Python mindset needed for clear, concise code, than some of the explanations I have read that seem to favor dictating a “Python Way” to the reader. Therefore, once you have learned Python basics, this book is a must for your library.

Bon Apetit!


Parsing An Industrial XML File

Attached is code to parse an XML files that we will soon receive from our Automated Meter Reading (AMR) vendor. We bill our water accounts from daily reads, which we receive seven days a week. Hourly reads will assist our water department with helping customers figure out water usage. In other words, hourly reads serve as diagnostic information to see when water was used during a twenty-four hour period.

Each of the search tags in the Python program, which I believe are XML leaves, are the actual parts of the XML file. Adding these to the dictionary this way ties the XML file to the dictionary, which in this case I believe is a good thing, if someone looks at the XML file all on its own.

This example uses lxml, which from what I can interpret out in the Python community is the XML parsing library of choice.

"""
Pulls a name space away from an element. This is for easier searching.
"""

def parseHrNs(qname):
 try:
 namespace, element_name = re.search('^{(.+)}(.+)$', qname).groups()

 except:
 namespace = None
 element_name = qname

 return namespace, element_name

def parseHrXmlDoc(fnam):

 rc = 0
 se_read = []

 current_ch = 0
 current_endpoint = 0

 ert_ch_text_key = ' '

 hrDict = {}

 try:
 context = etree.iterparse(fnam)

 except:
 context = None
 rc = -1

 if 0 == rc:
 nspace = None

 """
 These searches are roughly in order of appearance in the XML document.

 Channel number found after ID.

 ID is the place to write the last data before assigning new ID.

 Check on se_read[] provides initialization step, so we don't assign rubbish at the
 beginning.
 """

 for action, elem in context:
 nspace, search_tag = parseHrNs(elem.tag)

 #print("%s: %s" % (search_tag, elem.text))

 if 'exportGuid' == search_tag:
 hrDict[search_tag] = elem.text

 elif 'exportDateTime' == search_tag:
 hrDict[search_tag] = elem.text

 elif 'collectionSystemType' == search_tag:
 hrDict[search_tag] = elem.text

 elif 'collectionSystemID' == search_tag:
 hrDict[search_tag] = elem.text

 elif 'startTime' == search_tag:
 hrDict[search_tag] = elem.text

 elif 'endTime' == search_tag:
 hrDict[search_tag] = elem.text

 elif 'intervalLengthInSeconds' == search_tag:
 hrDict[search_tag] = elem.text

 elif 'ID' == search_tag:

 """
 There really are null elements, so perform a continue.
 """

 if not elem.text:
 continue
 else:
 """
 This is an initialization step.
 se_read is an empty list on initialization, so
 just assign the current endpoint if se_read is an empty list. That
 should only happen once.

 After initialization, basically, this is already visted logic.
 We've already seen and cached the ID in current_endpoint and
 the channel in current_ch. Before assigning the new ID,
 assign what we have in the dictionary. The fact we're here
 means we've seen a new ID, so the prior data must be put in dictionary.
 """

 if se_read:

 """
 Endpoint ids are not unique for the dictionary if they are dual-port, so
 we need to create a unique dual key based on the channel number.

 current_ch was already saved, and look below. We have not taken
 elem.text's value yet. We're still going on already cached data, which
 is our intent. See explanation above.
 """

 ert_ch_text_key = str(current_endpoint) + '-' + str(current_ch)

 hrDict[ert_ch_text_key] = se_read

 se_read = []

 current_endpoint = elem.text

 elif 'channelID' == search_tag:
 """
 Channel must be preserved, even though we've already seen ID.
 current_ch will be used before next ID assigned.
 """

 current_ch = elem.text

 elif 'value' == search_tag:
 se_read.append(elem.text)

 return hrDict


Sometimes Simple Is Better

Python is a great language and can solve a variety of problems. But sometimes, a plain shell script will do just fine.

#!/bin/bash
#
# drop_csm_users.sh -- grabs anyone running fglgo csm and kills their
# process. This must run under root's user, no matter who calls it.
# This has to run and allow sqlexecd to continue running. It's a little
# bit different than drop_ics_users.pl
#
# Change History:
#

cat /dev/null > /tmp/csm_users

ps -ef | grep -v 'grep' | grep 'fglgo csm' > /tmp/csm_processes

RC1=$?

if [ $RC1 -eq 0 ]; then
    while read line
    do
        echo -e "$line\n"
        ccmd=`echo $line  | cut -b 8-14`
        kpl="kill -9 $ccmd"
        echo "Forking this kill command $kpl"
        $kpl
    done < /tmp/csm_processes
fi