Idioms and Patterns as Architectural Literature

IEEE Software Special Issue on Objects, Patterns, and Architectures, January, 1997

by
James O. Coplien, Bell Laboratories
 ILL650 1G341
1000 East Warrenville Road
Naperville, IL 60566 USA
(630) 713-5384; FAX (630) 713-4982
cope@bell-labs.com



©1996 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.

This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Abstract

Some people assume that software patterns are primarily about objects or component interconnection schemes, and thereby fail to reap their benefits. To understand the potential of patterns and how to apply them, it is helpful to understand the history of patterns, their relationship to objects and language-specific idioms of object-oriented programming, and to the broader stage of software architecture. Patterns not only support object-oriented design and architecture, they are reaching out into areas far beyond the structure of software, such as the software development organization, markets, and software usability. But a true understanding of software architecture must include these facets as well, so patterns truly support the whole of sotware architecture.

Keywords: idiom, systems engineering, multi-paradigm design, anthropology, process, business process re-engineering, structure

Introduction

Patterns have achieved the status as a must-have or must-do both in object-oriented circles and among software architects. Though a few foresighted software folks have had Alexander on their bookshelf for 15 years or more, it is only recently that I've found a table full of his books at a software conference. Patterns are here with a vengeance, and most programmers will tell you that we have objects to thank for it.

There are strong parallels between Alexander's patterns of building architecture and our own object patterns. Alexander strove to use software for semi-automated urban design in his early work. He built tools to lower the cognitive load of design by exploring large design spaces on behalf of the architect.[Notes] Patterns were in part a reaction to the shortcomings of this approach.

Software design has traversed a similar history. We, too, went through an era when we were fascinated with CASE tools and the prospects of automating the design process. With or without CASE tools, early adopters of the object paradigm focused on the "find-the-objects" exercise, deferring or forever losing the system perspective of interactions between classes or objects.

Though much of the value of patterns comes from their contribution to a sound foundation of design literature, a good deal of it lies in their non-object-oriented nature. More specifically, patterns are constrained by the rules of no single paradigm, which leaves them free to describe the incredibly rich structures of increasingly complex software systems. This, to me, is what most distinguishes patterns from most other contemporary software design. Alexander found that patterns helped him express design in terms of the relationships between the parts of a house, and the rules to transform those relationships. That is a much different focus than a method that builds houses from pre-designed modules.

Unveiling structures

So it is with software patterns: they are not just another way to capture legacy knowledge, but they capture legacy knowledge that is operative at the system level, above the component level. Patterns have given us a vocabulary to talk about structures larger than modules, procedures, or objects, structures that outstrip the vocabularies of the sound object design methods that have served us for the past decade. Many of these structures aren't new; a few people have known them for decades. Patterns bring these techniques, and a vocabulary to talk about them, to the everyday programmer.

This perspective is crucial to any operative definition of (software) architecture. Traditional software architecture focused on parts of a system in isolation, with each one defined in terms of its internal structure and external interfaces. While these interface definitions capture the formal behavior of system parts, the behavior of the system as a whole is richer than the sum of its parts. Architecture captures and articulates the larger picture, the relationships as well as the parts.

Patterns--at least, as Alexander posed them, and as many are practicing them in software--focus directly on this issue. While I don't believe the pattern discipline is or should follow Alexander slavishly, analogies to many of his building architecture principles strike home. Consider this excerpt from Timeless Way of Building:

Design is often thought of as a process of synthesis, a process of putting together things, a process of combination.

According to this view, a whole is created by putting together parts. The parts come first: and the form of the whole comes second.

But it is impossible to form anything which has the character of nature by adding preformed parts.

When parts are modular and made before the whole, by definition then, they are identical, and it is impossible for every part to be unique, according to its position in the whole.[Timeles1]

Misused and misunderstood

Yet people have difficulty letting go of old ways. Pattern papers submitted to (and occasionally, published by) software conferences and journals too often are CASE or object sheep in pattern clothing. The literature describes tools that automatically "instantiate patterns," i.e., by assembling preformed parts. "Why will patterns fail," "Patterns: Cult to Culture" and similar panels are making the popular software conference scene. As with any new technique, patterns have tremendous potential for abuse and misinterpretation. In fact, this problem is not unique to patterns, but is symptomatic of much contemporary software literature: design, architecture, process and programming are understood only as disconnected concerns. Today's programmer can better appreciate patterns through the eyes of history, and by studying the relationship between patterns, objects, and architecture. Such a perspective can take us beyond objects and beyond paradigm to attack the central problems of contemporary software development. This article offers my perspective on that history, and a vision of where patterns could have their greatest value in the future.

Patterns: beyond objects

Their most important contemporary value is that they complement existing design methods, solving problems that are often beyond the reach of those methods. They help a large body of programmers to gain competence, and sometimes excellence, in object-oriented programming and design, using micro-architectures such as those published in Design Patterns[GOF]. Patterns are also reaching to broader abstractions at the framework level, where the designer must often mix the object paradigm with other paradigms: Patterns take us beyond objects.

Patterns: beyond architeture

Patterns are just starting to make inroads into areas beyond software design. Teaching, organization and process, and other areas have growing bodies of pattern literature. Structures, skills, and patterns of behavior from these domains are as important to the success of a software product as are any of the software patterns: Patterns take us beyond architecture.

Patterns: back to a holistic approach

If we really take seriously the term "system" in "software system," we must consider facets of software development that range from technological to humanistic. It isn't that we need to forge new relationships between these perspectives, but to recognize and manage the relationships that have been there, beneath the surface, all along. These considerations combined form a structure, an architecture, much as Alexander's urban planning patterns have both structural and human components. Alexander consciously brought all these components into his pattern languages. At this degree of integration, we can aspire to the ideals of classic architecture, as we find in Vitruvius's durability, utility, and aesthetics.[Vitruv] The software community will also approach this threshold of integration, which truly puts objects, architecture, and patterns in their proper perspective: patterns bring us back to a more holistic view of architecture.

Beyond Objects

Alexander's A Pattern Language has been around for almost 20 years: why has its influence taken root in the object paradigm? Part of it has to do with complexity: software problems, and the solutions to address them, are becoming exponentially more complex. While individual programmers could wrap their minds around most software problems 20 years ago, most of today's problems are larger than one person. Today's problems call for something bigger than objects, something that cuts across objects and ties them together into larger structures. Objects started people thinking abstractly; patterns can take us further than objects could.

Limited Abstraction

Why are objects high on the complexity scale? It's because of their abstracting power. Under the procedural paradigm, we chunked hierarchies of procedures under some top-level procedure, using structured design techniques. The object paradigm supports hierarchies too (class hierarchies) but now we chunk several procedures at each level of the hierarchy. Also, the abstractions of the procedural paradigm were fairly concrete: we could look at a procedure call at compile time and tell where it would branch to at run time. In the object paradigm, even the procedure names are just abstractions-they might be bound to one of any number of functions at run time. These are the properties that distinguish the object paradigm from its siblings. These complexities show up as relationships: Complexity shows up in these relationships and patterns thrive on capturing and articulating them.

Relationship Focus

Before Alexander pioneered his pattern-based design approach, he used tools to arrange the "objects" of his domain--rooms, windows, walls--to fit a specification.[Notes] One can speculate that one reason Alexander moved away from this approach and toward patterns is that the solution focused on the "objects" in the final solution, rather than the structure of the solution itself: the relationships between the objects. To study relationships is to study systems, and we find patterns in the structure of system relationships. Good and bad relationships are respectively the strength and downfall of all systems: architectural, social, economic and software.

The importance of focusing on relationships was recognized, but not widely subscribed, even in the early days of the popularization of the object paradigm. Advanced C++ Programming Styles and Idioms[Coplien] captured object relationships that are important to accomplished C++ programmers. The book's idioms captured the practices of expert C++ programmers of the time (that is, of the early creators and adopters of the language). These practices solved problems encountered by most inexpert C++ programmers as they struggled to master the language. By pointing out the idiomatic nuances of C++, Advanced C++ supplemented the basic literature of the time with the techniques necessary to exhibit competence, if not excellence. One can master a language only after its idioms become second nature. Most of these idioms captured interactions between objects, not just with finding the right objects.

Jim Waldo credits Advanced C++ as the foundational work behind software patterns in his 1995 UNIX Review.[Waldo] That's certainly a C++-centric view, but idioms figure prominently as one foundation of patterns in the object community. Most of the idioms in Advanced C++ were design rules that helped programmers use C++ as an even-more-object-oriented-programming language. Each object-oriented programming language lacks features available in others, and many hold some of these features to be fundamental to object-oriented programming. Multiple dispatch is one example, and one early idiom shows programmers the "right way" to fake multiple dispatch in C++. The idioms aren't in pattern form, but they provide a "solution to a problem in a context" (Alexander's straightforward definition of a pattern).

If one views idioms as proto-patterns then, yes, early patterns were very tightly tied to object-oriented design. But even then, the patterns addressed object and class relationships for specific design problems. These problems were the ones left unaddressed by object-oriented design methods and by fundamentals of the paradigm itself: multiple dispatch, reference counting and other memory management schemes, brokering, type promotion, and dozens of others. Such concerns sometimes reached beyond the object paradigm because they are low-level design considerations (true of many idioms), such as reference counting. But others, like multiple dispatch, needed patterns to map from the problem to the solution because they outstripped available methods. Most methods don't really explain the relationships between classes or between objects. In other words, the meaning of many relationships is left implicit, except for subtyping and containment, which obey well-defined conventions. (Even today, few design methods deal coherently with multiple dispatch.) Patterns underscore the meaning, the rationale, behind such relationships.

This body of knowledge generalized and broadened over time as the object paradigm and its idioms-and patterns-matured and became better understood. The Design Patterns[GOF] book acknowledges these early idioms as one influence on its authors (p. 357), though Design Patterns is organized around patterns while Advanced C++ is organized around language features. Designers can apply the Design Patterns material to most programming languages, while Advanced C++ focuses on one language. The literature explores the mapping between these two models in more detail.[Cope2] Mastery of the patterns of Design Patterns is perhaps the best measure of competency in the object paradigm, yet even the best methods rarely produce the structure captured by the design patterns. This is particularly true for patterns of dynamic behavior.

Beyond idioms

Idioms--and later, design patterns--armed early C++ adopters with training that took them beyond curious inquiry to serious applications. But time and again, technological soundness alone did not win the day; process, organizational structure, politics, culture, personalities, and a host of other "outlying" factors figured heavily in the success of new techniques and tools. I decided to focus on these areas as I moved from development into Research, rising above the "details" of technology to address the fundamental problems of process.

Our studies of the organizational and process side of software development uncovered patterns there as well. Not only do sound software architectures exhibit the same, recurring patterns, but accomplished organizations likewise share patterns of organization and process. These patterns of software development process and organization cannot be separated from the patterns of architecture: an OO project can succeed only if supported with the right patterns from the human side.

I wrote my first process pattern (Buffalo Mountain[PLoPa] ) at the first Hillside workshop in August of 1993. Many authors have contributed to the body of organizational patterns since then, including Norm Kerth[Kerth] and Bruce Whitenack.[Whnack] At least one author has achieved the ultimate integration: in Steve Berczuk's pattern language,[Bercz] process and architecture patterns work together.

There is of course an abundance of earlier literature of this genre that did not enjoy the pattern label. Cultural anthropology has been studying patterns of relationship in organizations for the better part of a century, as have cultural observers dating back to Lao Tzu and beyond. Patterns provide an outlet for these time-honored world-views to once again be fashionable, and objects have been of the vehicles to usher patterns into the modern world.

Patterns of Preservation

Patterns' biggest payoff may lie in capturing the great truths that are about to be lost to history, instead of focusing on pattern support for the best-funded buzzwords. At Bell Laboratories, we are mining the patterns of classic embedded systems to capture the core competencies of our businesses. Telecommunication system architectures have evolved over the past 40 years to the point where they are among the most reliable systems in the world. Why? We can trace availability and fault tolerance to patterns, and we have extracted those patterns from the minds of long-standing experts. These patterns are a fundamental component of emerging systems that face equally stringent requirements. Of the 150 or so patterns we've gathered, none are really object-oriented.[Becketal] [Mining] [PLoP96]

We have progressed a bit beyond our initial flirtations with patterns of object-oriented programming and design, into other design disciplines, and even into domains such as training and human enterprise organization. A wide spectrum of patterns lies beneath the code and practices of our recurring successes, waiting to be mined and woven into the emerging literature of good software practice. Patterns take us beyond objects into grander structures: perhaps, they take us beyond software architecture itself.

Beyond Software Architecture

Alexander's patterns emphasize the human element of environmental design; in fact, this is the prime focus of his patterns. Buildings don't "care" about their architectures; people do. Alexander saw his patterns as a way to free us from all method, and to surface the good practices we all draw from deep cultural roots. Patterns are mostly about people-and much less about houses, software, or design methods.

The human focus has two major components: utilitarian and aesthetic. On one hand, Alexander's patterns eschew aesthetics for its own sake, deferring to basic human comforts. This is an eminently practical point of view. But, on the other hand, many human needs transcend physical world models and touch on issues of sociology, psychology, and other soft sciences far from the comfortable realm of tangible, physical architecture. Alexander takes us squarely into these fields with patterns such as Wings of Light[PatLanga] and Structure Follows Social Spaces.[PatLangb]

Humanizing Software

We face the same dilemma in software. In the age of "software factories" and repeatable software processes, modern software tries to distance itself from unreliable human beings. We try to capture (and sometimes formalize) software architecture. Most architecture documents still capture just APIs and data structures, missing the important nuances of relationships. And most architecture documents still pretend that architecture can be divorced from downstream design and implementation concerns, which runs counter to predominate empirical practice, particularly in successful development organizations.[PLoPa] And few architecture documents draw in issues of paradigm, process, expertise, organization, aesthetics, or even the obvious concern of long-term maintainability. Software is of, by, and for the people too, and few software development problems or solutions lack human overtones; to omit the human element is to miss the thrust of most interesting problems. Consider the following pattern from Richard Gabriel:[Gab]

Pattern: Simply Understood Code

...at the lowest levels of a program are chunks of code. These are the places that need to be understood to confidently make changes to a program, and ultimately understanding a program thoroughly requires understanding these chunks.

In many pieces of code the problem of disorientation is acute. People have no idea what each component of the code is for and they experience considerable mental stress as a result.

Suppose you are writing a chunk of code that is not so complex that it requires extensive documentation or else it is not central enough that the bother of writing such documentation is worth the effort, especially if the code is clear enough on its own. How should you approach writing this code?

People need to stare at code in order to understand it well enough to feel secure making changes to it. Spending time switching from window to window or scrolling up and down to see all the relevant portions of a code fragment takes attention away from understanding the code and gaining confidence to modify it.

People can more readily understand things that they can read in their natural text reading order; for Western culture this is generally left to right, top to bottom.

If code cannot be confidently understood, it will be accidentally broken.

Therefore, Arrange the important parts of the code so it fits on one page. Make that code understandable to a person reading it from top to bottom. Do not require the code to be repeatedly scanned in order to understand how data is used and how control moves about.

This pattern can be achieved by using the following patterns: Local Variables Defined and Used on One Page, which tries to keep local variables on one page; Assign Variables Once, which tries to minimize code scanning by having variables changed just once; Local Variables Re-assigned Above their Uses, which tries to make a variable's value apparent before its value is used while scanning from top to bottom; Make Loops Apparent, which helps people understand parts of a program that are non-linear while retaining the ability to scan them linearly; and Use Functions for Loops, which packages complex loop structure involving several state variables into chunks, each of which can be easily understood.

This pattern may portend more for long-range quality than many prima facie "architectural" principles, yet it is not object-oriented, and is hardly architectural. It is an intensely human pattern, woven together with words like "confidently," "stress," "secure," "culture," and "understand." It is intuitive because it speaks to our common human sense, but it is worth committing to literature because its precepts are so often ignored.

Right-brain patterns

Some of our biggest pattern successes have come not from patterns of software architecture and design, but from patterns of organization and process. Such patterns help organizations see beyond themselves and develop a vision of what is possible. The organizational patterns in Chapter 13 of the PLoP[PLoPa] book helped the engineering organization of an established software company revitalize itself.[People]

The human aspect of patterns displaces most software engineers from their comfort zones. Few software engineers are tooled in experimental psychology or sociometric science; to them, these areas are mysterious, nameless black holes of nonsense. These are disciplines of the right brain, of emotion, and of the soul. They defy the left brain's attempts at quantification: for most of us, such Qualities thrive unnamed. Bean counters can (and do) ascribe numbers to these properties: cyclomatic complexity and function points for: "this feels complicated," for example. But such numbers are only shadows of the gut feelings of the intuitions of accomplished architects, and most metric systems don't know how to measure "good guts." We are finding that patterns can and do help us, both in software architecture[Becketal] and in (sometimes remarkable) organization changes.[People]

Architecture Restored

We have followed patterns beyond building architecture into objects, beyond software architecture into organizations. Are patterns intrinsically linked to "architecture"-this notion of the parts of a system and the relationships between them? If so, what are the architectural principles of an organization?

I believe that the answer is "yes," at least for an insightful definition of architecture that goes beyond enumeration of parts or interfaces. Even organizations have architecture; the phrase "process architecture" crept into the literature about a decade ago. In fact, "organizational architecture" is a refreshingly different perspective than the flowchart-like "development process."

Peter Sengé speaks of the patterns of structure of organizations. These patterns can be characterized by a small number of system archetypes, each one of which has a structure with opposing forces. It's not a purely Alexandrian formulation, but the similarity in patterns, structure, and forces is striking.[Sengé]

Even Gabriel's pattern on Simply Understood Code is curiously structural. It talks about system granularity from the human cognitive perspective-a crucial aspect of the abstraction structure of the system. If that isn't architecture, what is?

A system needn't be something huge or something containing computer hardware. And we don't need to limit ourselves to the most abstract meaningful view, either. Systems comprise smaller systems, each of which obeys its own set of patterns. Handle/body pairs are a system; model-view-controller is a system; so are client-server or a pair of half-objects with a protocol between them (each of these has been the topic of at least one noteworthy pattern). In his seminal book on systems engineering, Arthur Hall relates this about systems (in 1962, two years before Alexander's Notes on Synthesis of Form hit the bookshelves):

Of all the possible ways of defining the systems engineering function, the most significant and explicit is an operational one, which gives a description of the general pattern of work from formulation of a program of projects to completion of a specific project. For it is the pattern, more than anything else, which gives the function its essential structure and characteristics. Furthermore, there is more agreement about the pattern, and the detailed steps within it, than about any other aspect. Finally, and most important, experience shows that this mode of definition is the most useful to those who want to learn how to do systems engineering better.[Hall62]
Hall understood the scope of system engineering to be very broad. His book contains chapters on communication problems, objectives of the value system, the psychological theory of value and the psychological aspects of synthesis. He recognized the limitations of an objective-based definition of system engineering because of the force of personal value judgments.[HallB] If patterns are about systems, they are about more than software, and certainly are about more than objects.

A Conclusion

This article started with the early roots of patterns in Alexander's architecture, following their progress through the idioms of early object-oriented programming into design, software architecture, and at last into a wide variety of domains related to software development. Though early patterns focused on objects, they focused on those aspects of object-oriented structure that could not easily be regularized with a design method. Patterns of organization, training, and process went even further afield from method in the knowledge they captured. Yet, patterns at all these levels seem to apply to many phases of analysis, object-oriented design, programming, verification, and validation.

If patterns are not about objects, and they reach beyond software architecture, then what is a pattern? Most of this article has worked to explain the relationships that might exist between objects, patterns, and architecture. But since objects and architecture have fueled the hype many ascribe to patterns, patterns might best be served by a definition distanced from these influences. Such a perspective should be important both to the educator planting seeds in the designers' mind, and to the practitioner seeking more pragmatic ends.

To me, patterns are literature, but more than just documentation. They capture an important structure, a central idea, a key technique long known to expert practitioners. It can be an architectural structure, a process practice, or a marketing strategy. What ties this body of literature together is that all patterns solve problems. We can take inspiration from a primordial source:

These patterns in our minds are, more or less, mental images of the patterns in the world: they are abstract representations of the very morphological rules which define the patterns in the world.

However, in one respect they are very different. The patterns in the world merely exist. But the same patterns in our minds are dynamic. They have force. They are generative. They tell us what to do; they tell us how we shall, or may, generate them; and they tell us too, that under certain circumstances, we must create them.

Each pattern is a rule which describes what you have to do to generate the entity which it defines.[Timeles2]

System concerns and software architecture have too long received the short shrift. And too many people who have taken the 16-hour design method course feel they understand objects. These are both problems that will seriously hurt our craft, left unaddressed. If patterns produce literature that attacks the problems in these areas, then the alliances between patterns, objects and architecture will have served us well. We shouldn't lose sight of the goals, however: to serve human needs, to restore dignity to programming, and to add that nameless right-brain Quality to our outlook on software development and project management.

Bibliography

[Notes] Alexander, Christopher. Notes on Synthesis of Form. Cambridge, MA: Harvard University Press, ©1964.

[Timeles1] Alexander, Christopher. The Timeless Way of Building. New York: Oxford University Press, ©1979, p. 368.

[Vitruv] Vitruvius. The Ten Books of Architecture, translated by Morris Morgan. New York: Dover Publications, 1960.

[Coplien] Coplien, James. Advanced C++ Programming Styles and Idioms. Reading, MA: Addison-Wesley, ©1992.

[Waldo] Waldo, Jim. Minor Patterns. UNIX Review 13(4), 1995, pp. 79-82.

[GOF] Gamma, Erich, et al. Design Patterns: Elements of Reusable Object-Oriented Software. Reading, MA: Addison-Wesley, ©1995.

[Cope2] Coplien, James O. The Column Without a Name: Patterns and idioms in circles, complex ellipses, and real bridges. C++ Report 7(4), May, 1995.

[PLoP1] Colien, James. A Development Process Generative Pattern Language. In Pattern Languages of Program Design, edited by J. O. Coplien and D. C. Schmidt, pp. 218-220.

[Kerth] Kerth, Norm. Caterpillar's Fate. In Pattern Languages of Program Design, op. cit.

[Berc] Berczuk, Steve. A Pattern Language for Ground Processing of Science Satellite Telemetry. Excerpted in "The column without a name: the human side of patterns," C++ Report 8(1), January 1996.

[Mining] Coplien, James. The column without a name: Pattern Mining. C++ Report 7(8),October 1995.

[PLoP96] Vlisside, John, et al., eds. Pattern Languages of Program Design -- 2. Reading, MA: Addison-Wesley, 1996.

[PatLanga] Alexander, Christopher, et al. A Pattern Language. New York: Oxford University Press, 1977, p. 529.

[PatLangb] Ibid, p. 940.

[Gab] Gabriel, Richard. Personal correspondence with the author; also appears in "The Column Without a Name: Software Development as Science, Art and Engineering," C++ Report 7(6), July/August 1995.

[PLoPa] Coplien, James. A Development Process Generative Pattern Language," op. cit., pp. 183-238.

[People] Coplien, James. The column without a name: the human side of patterns. C++ Report 8(1), January, 1996.

[Becketal] Beck, Kent, et al. Industrial experience with design patterns. Proceedings of ICSE-18 -- 1996 Los Alamitos, California: IEEE Computer Society Press,pp. 103-114.

[Senge] Senge, Peter. The fifth discipline. New York: Doubleday, ©1990, Chapter 6.

[Hall] Hall, Arthur D. A Methodology for System Engineering. Princeton, NJ: Van Nostrand, ©1962, Section 1.4.6, p. 11.

[HallB] Hall, op. cit., Section 1.5, p. 11.

[Timeles2] Alexander, Christopher. The Timeless Way of Building, op. cit., pp. 181-182.