Software architecture, software engineering, and Renaissance Jazz

Grady is an IBM Fellow who has served as architect and architectural mentor for numerous complex software-intensive systems around the world in just about every domain imaginable. Grady is the author of six best-selling books and has published several hundred articles on software engineering, including papers published in the early '80s that originated the term and practice of object-oriented design. At random times, the laws of physics do not apply to him. He is not dead yet.

Microsoft and Domain Specific Languages posted by Grady Booch on 2004 Dec. 03 09:33 PM Permalink The past couple of months, Microsoft has unleashed a torrent of words detailing their marketecture for software factories. Alan Brown and Simon Johnston of IBM Rational have previously and very ably commented on this work, but Steve Cook of Microsoft has drawn me into the fray in his blog, so I feel compelled to reply: Steve wrote "I hope Grady Booch reads this"; well, I did :-). I know many of the folks involved in Microsoft's software factory effort, and I very much respect what they are trying to do. We have differences of opinion, as you'll see in this blog, but it's good to watch Microsoft putting some energy into improving the development experience. As I've said many times here and elsewhere, software development has been, is, and will remain fundamentally hard, and whatever can be done to improve the profession of developing software is a Good Thing. That being said, I'm disappointed that Microsoft choose the term "software factory," because it's an emotionally-laden phrase that harkens to extremely mature manufacturing methods that focus on stamping out endless copies of the same thing, although perhaps with slight variants therein. There's no doubt that reuse at the level of design patterns or, even better, vertically-oriented architectural patterns is a Good Thing, but what Microsoft is proposing to do is not exactly like the manufacturing metaphor, and so their use of the term is a bit misleading (although Steve has curiously used the image of a conveyor belt when describing the Microsoft factory process). Tom Demarco in his book Why Does Software Cost So Much? sets aside a chapter on software factories in which he notes - and I agree with him - that "I think factory methods for software are dead wrong, witless, and counter-effective. Organizations that build good software know that software is an R&D activity, not a production activity. Organizations that try to make it into a production activity produce bad software (though potentially lots of it)." At OOPSLA, Rick Rishad of Microsoft publically spoke of their strategy (in a somewhat controversial way, as reported by Spencer F. Katt, which was a bit surprising given the typically-frictionless Microsoft marketing machine). That strategy was subsequently reviewed in ADT Magazine. While I agree with much of that article, they too fell into the pit of taking the software factory term a bit too literally. Perhaps the best source of Microsoft's deep thinking in this space, in addition to their site, is the book by Jack Greenfield and others, titled Software Factories: Assembling Applications with Patterns, Models, and Tools. Therein you'll see Microsoft's emphasis upon resuable assets and tooling to support them. To that end, there's considerable common ground between IBM and Microsoft's approaches to the problem: we both agree that resuable components, as manifest both in code as well as in patterns, are the right next stage in cutting the Gordian knot of software. Indeed, IBM's been in the pattern space for sometime, starting with many of the authors of the seminal book Design Patterns to the current work led by Grant Larsen and as manifest in the open standard we pioneered through the Object Management Group, the Reusable Asset Specification. However, we do disagree with Microsoft's rejection of the UML in favor of proprietary domain-specific languages, as noted not only in Jack's book but also in Alan Will's blog. To be clear, as Jim Rumbaugh has commented back to me, our observation - and that of our customers - is that the UML has proven itself useful much of the time, yet there are a few purposes for which it may be less appropriate. In many cases, the semantics of the UML are pretty close to what you need, although they are deeper than necessary; in such cases, a suitable UML profile is sufficient to focus the language, which allows you to leverage standard UML tools and training and yet eliminate the bloat. In those cases where the business concepts are more naturally expressed in a specialized syntax, then inventing a suitable DSL is reasonable. At the extreme, this is essentially the path that Charles Simonyi has been trodding for some years, a path that requires a very very deep and integrated underlying semantic model. Indeed, as I've pointed out in one of my earlier blogs, the root problem is not simply making one set of stakeholders more expressive, but rather weaving their work into that of all the other stakeholders. This requires common semantics for common tooling and training, so even if you start with a set of pure DSLs, you'll most often end up covering the same semantic ground as the UML. Will's blog had a number of errors of fact, which Bran Selic has pointed out to me and so which I'll paraphrase here. Alan wrote "So here's why we don't want to limit ourselves to the UML as a basis for our users' domain-specific language" and then went on to say: "A careful look at the specialization mechanisms for UML reveals their limitations. Stereotypes and tagged values allow you to change icons etc, although even simple alterations like decorating a box to show the state of some property isn't within range. You can't change the semantic constraints, or invent new sorts of diagram or new categories of element." This is incorrect: a stereotype allows you to define a set of associated constraints (in OCL, for example) that can capture the characteristics of your domain-specific context. While it is true that you cannot violate the semantics of the metaclass that you have stereotyped, this is actually an advantage of the stereotypeing mechanism. A stereotype is a type-compatible specialization of an existing UML concept. Consequently, you can reuse standard UML tools and expertise even though you are using a domain-specific language. Of course, if you want a language that is incompatible with UML, that is OK as well (specifically, you can define it using MOF), but you will be losing some of those benefits. "You can't take stuff away, so your users are always distracted by other options and elements and diagrams that aren't relevant to your language. Tools that use a UML editor as a front-end have to have a (very annoying) validation step before they generate their DB schema or whatever it is." Also incorrect: as Jim noted above, a UML profile can remove any metaclasses it chooses. "UML only includes certain types of graphical format. If you want your language to include tree-structured diagrams, or tables, or math formulae, or block-structured text, or prose, or if you want hyperlinks - well, I think you'd have a hard time. While our initial offering won't include all those styles, we'd certainly like to support them at some stage in the future." Actually, the current UML spec really does not restrict graphical formats in any way -- it simply provides a standard set of notations, but not at the exclusion of other notations. In other words, there really is no "illegal" UML graphical syntax. The formal definition of a UML graphical syntax is an outstanding item before the OMG. While this is not good, it also means that Alan's criticisms about its graphical restrictions are misguided - and Microsoft too acknowledges that these different graphical representations are a future desire for them, not a present reality. "An important aspect of the definition of a modern language includes how you interact with it: how you navigate and elide or elaborate and edit etc - think of your favorite GUI composing tool, in which the GUI-defining language is scarcely separable from its editor. You can't do anything about these aspects in UML." We agree - but this has nothing to do with the UML but rather is all about the tool environment. IBM's tooling approach is to build upon the open standard of Eclipse, not a proprietary IDE. "What you get out the back of one of our tools says things like CheckinDesk and ConveyorBelt. What you get out the back of a UML tool says things like Class and Stereotype - much more difficult to read. (Yes, of course you could write translators, but it's an extra layer of hassle.)" This is also incorrect: using the UML, at the model level you get back stereotypes called CheckinDesk and ConveyorBelt. At any rate, why would anyone want to look at things at the XMI or metamodel level? XML is not really for human consumption, and ultimately, MDD is all about raising the level of abstraction for the developer. "You have to understand all of UML (stereotypes etc) before you can create your language on top of it." Not true: you just need to know the subset that you are specializing (mostly it has to do with things such as classes and associations which is what most profiles specialize). Of course, if you want to specialize state machines, you need to know the metamodel. But, if you don't care about state machines, you can ignore them safely. "Your users have to understand an editor that's intended for something far more complex than they probably have in hand - or at least whose complexities are in a different direction." Again, this is a issue that confuses tooling with language definition. I hope that Steve and Alan read this :-)

  Comments

Mr. Booch, one of your points is: I'm disappointed that Microsoft choose the term "software factory," because it's an emotionally-laden phrase that harkens to extremely mature manufacturing methods that focus on stamping out endless copies of the same thing, although perhaps with slight variants therein.... Tom Demarco in his book Why Does Software Cost So Much? sets aside a chapter on software factories in which he notes - and I agree with him - that "I think factory methods for software are dead wrong, witless, and counter-effective. Organizations that build good software know that software is an R&D activity, not a production activity. Organizations that try to make it into a production activity produce bad software (though potentially lots of it). This argument contends that all apects of building systems is creative to the point that software factories cannot accomodate the level of variance between systems. This argument is fundamentally flawed, as evidenced by the prevalence of established design patterns. I would argue that companies building good software actually want to take creativity out of the development process when it comes to components for which mature design patterns exist. In other words, the building blocks should come from software factories and be assembled in a creative way to solve the problem at hand. Only the truly unique components should be hand-crafted. At MAKE Technologies we currently implement this methodology -- we call it Standards Based Automation (SBA) -- presenting DSL models graphically in UML where appropriate or using other visual representations. From the models we can then generate code that conforms to industry-standard design patterns. The key success behind DSLs is the ease with which people can create them evolve them and implement solutions based on DSL modeling. To my knowledge, successful heavyweight MDA extensions through MOF are non-existant. I can think of dozens of examples of successful DSL implementations. The problem here is that MOF is too complicated. posted by dgreen on 2004 Dec. 15 11:27 AM To produce a software a factory-like stile we need two things which made contemporary engineering posssible: 1) a draft-like language which presents software in a way that makes it easy to validate it; 2) a system of measurement or validation which allows QA person to say immediately that this module passed or failed. These are ABCs of mass production be it a mechanical device or electonic one. Unfortunately software either did not come to this phase of maturity or, whatis more probable, it is a totally different animal. As to UML, though I have more than 10 years of experience, I am not deeply impressed. Programmers mostly do not need it because there are simpler way to excahnge ideas. As to high-level folks most whom I met are not very technical and thus totally confused by UML. So the task as I formulated it is a definite challenge. We, designers and programmers should either come up with a language similar to drafts in engineering and simple way of module validation against "software draft" or to obtain deeper understanding why it does not work for software problems and figure out what works. posted by Leon on 2004 Dec. 13 05:53 PM Dear Blogger, I totally agree with Grady, but only if he talks about UML 2 (the new flavor of UML). UML version 2 is offering great more than before and especially a modular and semantically well defined core. I do not see any inconvenient of using UML 2.0 as a strong basis for building DSL. In my opinion, the main problem is that people continue to think that UML is just a digramming language (I call it the "Visio" view). You can design a UML profile to describe enterprise architecture with UML 2.0 and it is very easy and staighforward (if you have a clear idea of what you want!). You can also use pattern and transformation at the model and metamodel level. So you have all the feature needed for customizing UML in the way you want. Tools are now ready to support UML 2.0, so please before making yourself an idea, try UML 2.0, build your profile, create some patterns and see the result. So why so much noise around DSL? For building DSL, you will have to design your model and then use it (run time phase). Microsoft will offer a complete suite for building DSL (you can try the beta now!), and will also offer some pre-defined dialects for usage in its tools. I see Microsoft DSL vision more as a "configuration language" building set (see the Microsoft Dynamic System Intitative at http://www.microsoft.com/windowsserversystem/dsi/default.mspx). Finally, as you see, both camps propose the model driven based approach, but with a clear difference on the process and tools/language to use. You can adopt it using a strategic unified approach (UML language and may be MDA) or a tactical approach (proprietary DSL, proprietary DSL with UML documentation, DSL generated from UML). posted by billyBoy on 2004 Dec. 13 04:52 AM From my experience, using uml to document high level software architecture may not be always optimal. The business users who look at the component diagrams/package diagrams find it difficult to identify the logical flows. Using basic block diagrams with arrows for flows seems to be a better way to provide high level architecture to non technical business users. UML Profiles will further bloat UML. For instance, there are many profiles for modelling web applications. which profile to use is confusing. posted by evijay on 2004 Dec. 11 12:04 AM "XML is not really for human consumption, and ultimately, MDD is all about raising the level of abstraction for the developer" Looking at the above quote by Grady and points made by Bob Wilmes, the difference in opinion appears to boil down to "economic relativity". What I mean is that if you know UML and you are a developer, then using UML represents an economy of effort for the developer. If you are a Vertical Industry "Subject Matter Expert", or just an "average Joe" who has spent 10-20 years in a business, then UML is a foreign and confusing language. In this case, having a DSL is key to providing "economy of effort". If proven, patterned, well-architected code can be generated from the DSL -what practical value does UML then provide? posted by mgunter on 2004 Dec. 10 05:30 PM As a user of UML, I feel that, just as with any other language or tool, UML has its range of applicability. UML is well suited for identifying and visualizing the structure and the collaborations of the components in a system. It is also an invaluable aid in the iterative process of discovering and refining requirements. On the other hand, a UML tool may not be the best suited for modeling UI components, or for creating business rules or even writing math formulae. I don't feel the need to defend the fact that UML does not do all these things well. It is always conceivable that, given enough specialization, a particular tool or mechanism will perform better than others in a given context. Software is about patterns, and if something doesn't look like a pattern to you, just broaden the context. UML captures patterns. A good software development tool or language (let's not exclude DSLs), like UML, should aid in the discovery, refinement and validation of requirements ('cause if requirements were always complete, accurate, unambiguous and the same for everyone, then maybe we could use software factories). In addition, it should also capture patterns and, in my opinion, those patterns should be expressed as semantically validated models. It would then be possible to automatically transform those models into a particular target technology in a coherent way within the context of our architecture or platform of choice. Only at this point in the software development process can we talk about factory-like productivity gains. It would help of course if all your favorite development tools were to share a common meta-model, even more if they were to share the same technology. The Eclipse Modeling Framework (EMF) is coming very close to making this vision a reality, a reality where there is no exclusion of UML, or anything that may help you develop your application, even DSLs. posted by jdf on 2004 Dec. 10 03:02 PM In partial defense of Microsoft, I think the work that they are doing with visual architecture tools, such as the "Whitehorse" technology in the Visual Studio 2005 code is very forward looking. Software systems design and methodologies are inherently addressing economic value derived from information systems. What Microsoft is doing with Domain Specific Languages serves two purposes: First, it addresses a broader spectrum of need than current generation UML tools, by starting a a conceptual architecture level, then moving down through a logical and physical deployment, based on reference architectures. It's the reference architectures which today have higher economic value to an enterprise. Second, DSL's create economic opportunities for ISV's to add value by providing standardized visual components which cannot be seperated from implementation details. This ecosystem drives has the potential to drive more choice and fit for purpose when integrating commercial of the shelf software with customized integration, which is exactly the way most enterprises operate today. Finally, Microsoft is not alone in realizing that development of software related systems is inherently an economic value judgement for most enterprises. Sun Microsystems certainly promotes a competitive view - see the recent book "Software by Numbers", which expresses this using incremental development using agile techniques. What the UML tools lack today are the abstraction capabilities. I've heard this described as "the police skecth artist" view of a system architecture. Ideally the tool would allow visual representation of a system to be described in abstractions, while at the same time driving the abstraction down through the logical and physical reference architecture layers, which themselves incorporate standard reference patterns. posted by bobwilmes on 2004 Dec. 10 11:19 AM