Tag: software engineering


JSP is just a name

June 28th, 2011 — 10:30am

At EuroPython this year Erik Groeneveld gave a talk on Python generators and he introduced it by referencing Jackson Structured Programming. As it happens, I’m quite a fan of JSP. For many tasks it solves the ‘blank page’ problem; it tells me what to do when faced with a new task. And, generally, I can get the program structure right first time. Most useful programs have both input and output streams, and Jackson recognised that there are a number of ways in which these two data structures might be mismatched. He also recognised that programs might have multiple input streams and multiple output streams and that these inputs and outputs needed to be mapped to each other (Jackson called this multi-threading, but it need not be asynchronous). So what happened? Does it matter that only 3 people out of a conference room-full had heard of JSP?

Simplifying somewhat, Jackson’s overarching solution to all of these problems was program inversion. (He also used look ahead and backtracking.) By ‘inversion’ he meant turning the program inside out so that each new call to the inverted program stepped through the structure. In effect, he produced co-routines. Of course, creating co-routines in a language (such as Cobol or Fortran) that does not support such things is hard work. You have to maintain a state, and make sure that every new call gets to the right place in the code. All of this is error prone and requires additional mechanical effort beyond what seems necessary to solve the problem. I suspect that, in the early days, this was one of the reasons for JSP being seen as ‘difficult’.

Whatever the difficulty, just because something is hard work does not mean that the problems it is designed to solve just go away. We still have to write programs where the inputs and outputs don’t quite match up, where a single input produces multiple outputs, or whatever. So what has happened to program inversion? Well, I think there are three answers to this.

  • Buffering

    This is the main change since those far off days. Most structure clashes can be resolved by posting data into a random access buffer. Jackson, of course, did not generally have access to such techniques. Now, we can have large data structures in memory, or we can use databases of many styles and capabilities. For many of us, this approach is natural and obvious. I’ve met one university course that explicitly teaches students to read in an entire file before processing it. Frankly, I’m nervous of going to that extent as a standard approach. I still like to process a file one line at a time, as it were. But, even so, I cannot deny the huge advantages of Python’s dictionary structures, and I confess to writing programs that spend a good deal of effort in transposing data from one structure to another. Perhaps that make-work of structure transposing is a clue to why the other options I give here are still valuable.

  • Explicit Inversion

    Yes, it still happens, particularly on user interfaces. From IBM’s CICS to the Python framework Twisted we have to write inverted programs to solve the interleaving clash between a user input stream for many users on the one hand, and the user interface logic for a single user on the other. Even an ordinary web page is an inverted program, with, perhaps, a database behind it to maintain the state. We seem so used to the pattern that we do not notice.

    We are not limited to interleaving clashes, either. The Python SAX parser offers an incremental parser that takes data a chunk at a time. And any facility that uses callback handlers is also using a form of inversion. I don’t know if the designers of the SAX API were explicitly using JSP, but the use of inversion to solve a structure clash is clear.

  • Implicit Inversion

    For some, perhaps many, of the problems with structure clashes, it is so much easier if the language supports some mechanism that supports the effect of inversion, without requiring that the poor programmer explicitly manages state and re-entry. Each part of the overall program can be written to explicitly demonstrate the structure it is intended to process, and the final program comes down to some reasonably obvious mechanism for connecting the parts together. As Erik Groeneveld pointed out, Python generators do just this. In fact, any Python programmer who has used, or written, a generator (or an iterator, which can be thought of as a special case), has been using implicit inversion. In some cases, of course, the generator may be quite trivial, but it is useful to remember that we can use program inversion to link data structures even when there is no structure clash. Indeed, Erik advocates writing all programs as interlinked generators, supported, perhaps, by his weightless library. I’m very tempted.

I see that JSP is still discussed in the academic world from time to time, but, as an explicit approach, it has not caught the imagination of people working in the real world. On the one hand, this seems to be a pity, because JSP is very much about using what we know of the real world (the data structures) to design programs. On the other, modern tools and techniques make the explicit identification of JSP almost redundant. For myself, I like the discipline of enumerating a data structure, but I’m not going to loose sleep over the demise of JSP as a recognised name. And for the future, Erik’s talk has given me a lot to think about, and I’m grateful for that.

Comments Off on JSP is just a name | Development

In celebration of filepath

June 26th, 2010 — 4:44pm

I was reminded the other day of some of the problems that we (the software industry, and the human race in general) have in thinking of software as some sort of engineering. The term Software Engineering has a nice ring to it, but the reality is disappointing. As Jeff Atwood so nicely pointed out, the advantage of, er, real world engineering is that it has immutable laws of physics. In software the programmer is free to invent his or her own laws as the project progresses. Of course, it is possible to relate physical laws to the software world, but, not, I suspect in the way that I, or Jeff Attwood, have in mind.

The particular issue I’m looking at here is software re-use. This is where project A and project B can both use some component, even though the two projects are very different. The mutable laws of physics thing comes in when the component in question has an API that is tuned to project A and therefore doesn’t cut it for project B. This is a known problem, with a set of causes, discussed by Douglas C. Schmidt in 1999, and one key aspect of software engineering is the disciplined approach that tries to address these causes. Interestingly, Schmidt picks up on two problems that are relevant here. One is that re-usable software has to be deliberately written that way. This means that the author has to understand a wide range of use cases, and have the funding or flexibility to be allowed to do it that way. The other is that re-usable components have to be attractive. Once the component has been written others must be attracted to it. For this, Schmidt uses the concept of a “re-use magnet” and he thought that the open source development process is a good and effective way of creating re-use magnets.

There’s a whole can of worms that opens up when software engineering is discussed, but that is not why I’m writing this. What reminded me of all of this is the recent release of filepath. This is a package that makes file handling that bit easier by hiding the details of the Python os.path module. In that way alone it is a potential re-use magnet. Why I like the fact of this release, however, is that filepath used to be part of Twisted. Now, Twisted is a good and useful web frame work, but I don’t happen to use it, and I would not want to install it just for the sake of using filepath. After all, I already have lots of code that uses os.path, so I’m not going to go out of my way to dig a new component out of a bunch of software that I’m not going to use. The advantages are not enough. On the other hand, with filepath as a down-loadable component on its own the re-use magnet is no longer shielded and I can feel the attraction. What is more, now that it is out there it can grow (or shrink) to fit the wider world (as a result of more use cases and more discussion).

So, thank you to the guys at Twisted. The software re-use paradigm lives on and Chaos and Old Night have been kept a bay in one small corner of the virtual world.

Comments Off on In celebration of filepath | Development

Back to top