Session:Evaluation--Discussions

From WICSA Conference Wiki

Jump to: navigation, search

Please use this page to capture ideas, thoughts, reactions, opinions and so on about the content of this working session.

Contents

Interests of participants

David Garlan: What people are doing, generally. Understand better what are the really hard problems that we cannot do.

Tahar Khamashi. Academia. Presenting a paper. Interested in learning more about various analyses.

Jorma Myyrylainen: Industry. Get a better understandind of what we have achieved, and we have not achieved. How do we evaluate our systems, and our customer's systems. How we can make our customers convinced that they get value out of architecting with concrete evidence.

[Neno Medvidovic]. USC. Research touches upon evaluation and analysis. Asked by asked by authors of paper who could not be here to summarize their papers.

[Aline Vasconcelos]Univ. of Rio. Architectural recovery. Recover comprehensive models of legacy systems. Arch. evaluations based on scenarios on other methods.

[Nour Ali] Ph.D. Student. Automatic generaiton of distributed applications and mobile system from ADLs. Interested in analyzing the architecture.

Jonathan Aldrich. CMU. Research interests in how architecture relates to code.

SM. Infosys India. What are the enterprise architecture issues that you should take into account when designing a software. Software is often not in a silo mode. It has to integrate and interoperate with many other systems.

Anthony Tan. Industry/Research. Evaluation and validation is somewhat ad-hoc through mostly peer reviews. Want to learn more about some techniques.

Jeff Tyrene. Industry. Work on large projects. Interested in new ideas that might be injected into that process.

Bradley Schmerl. 1) What kinds of analyses are possible. 2) What kinds of tools can we provide? 3) How we can facilitate design decisions.

Johan Muskens. Interested in software for dependable systems.

[Name] Industry. Tried scenario-based analyses techniques. Interested in model-based architecture. Bosch insti. ATAM.

[Eelco Rommes] Research in software architectures for medical systems. Merging of different systems into a product line.

[Jens Knodel]. Fraunhofer. Research in product lines architecture department.

Jason Smith. Detection of design patterns from source code. Learn more about reverse engineering at the architectural level.

Nasim [Name]. Measure properties in a quantitative way. Interested also in automated tools that take little effort to build models, and use that to do causal analysis of the problems detected during the evaluation of the architecture.

[Alan Coleman] Working with adaptive frameworks. Come up with high-level abstractions at the organzational level for software.

Jennifer Pérez : Integration of Software Architectures and Aspect-Oriented Software Development. Research work: The PRISMA Approach

[Name] Research

[Name] Research work relates to interesting aspects in software architectures.

Introduction by David Garlan

Software Architecture: Evaluation and Analysis

  • Many informal representation; hard to check for conformance
  • But the reality we like is that we want this to be an engineering discipline
  • Reuse patterns and styles are becoming more widely adopted
  • Still big gap between worst of practice and what we would like to do
  • Today, companies recognize architects as a particular role
  • Investment in product lines and frameworks
  • Material available: courses, textbooks, certificates, ...
  • Why aren't we there?
    • Things tend to be informal; no checking or formal analysis
    • No practical tools that scale

Measuring Architecting Effort

SlidesPresented by Eelco Rommes

  • How much (upfront) architecting is enough?
    • Data collected by Boehm and Turner, 2003
    • On a 10,000KLOC, spending only 5% of time in architecting, may mean that 90% of time spent in rework
    • Question: is this related to failure? Project termination? Delivered system undelivered? Cost overruns?
  • So how much architecting is going on within Philips Medical systems
    • System Daisy: 5 M LOC and 5 M LOC of test code
      • Timekeeping database made it easy to collect the data
      • "General" category does not capture spent time doing architecture
      • More fine-grained categories not added until later
    • System Daffodil:
      • No time-keeping
      • Many different architecture roles: "System architect", "software architect", "senior architect"
  • The Sweet spot depends on more than just system size
    • Requirements volatility
      • With stable requirements, more upfront architecting possible
      • If requirements change a lot, do less initially
    • Efficiency is not the main driver
      • Time-to-market may lead to less architecting
    • Product line approach may require more time architecting, for a bigger payoff later
    • "Reuse of effort" by people on the same project
      • "Technical debt"
    • Quality of work
      • Focus on getting a few critical decisions really right, instead of having many dubious architectural decisions.
  • Open questions:
    • How do define "architecting" in a measurable why?
    • What factors influence the sweet spot
    • Do we need this many architects?

Discussion

      • Question: How do you identify those goals? Does hindsight help?
      • If you do the right decisions, no one notices; but everyone notices the bad decisions
    • The important ones are often the controversial ones, e.g., the ones that involve integration.
      • This may be related to when multiple quality attributes are affected, i.e., there are many stakeholders, and each one has a different view of things.
    • the numbers seem high
    • 35% of the time spent on architecting seems excessive
    • Are we putting requirements with architecture then?
    • What about reusing existing code? 10M LOC probably involve reuse existing components. What are you counting?
    • also, notion of what a line of code means varies quite a bit in that context.
  • War story: performance was 10% of the desired one. [Sorry, couldn't fill in the details]
  • We don't document reasons for failures. Will the Grady Booch handbook have explanations of the horror stories.
  • Rainer made a dubious architectural decision: pointed out a flaw in the architecture, and ended up not getting the consulting assignment.
  • Software engineers do not like to have their mistakes pointed out.
  • How do architects work? It's really the "tribal memory". It's not capture in the diagram.
  • same idea with design patterns: the diagrams is just the formal description. It does not have the how and the why.
  • The real issue is how can we make these low-effort activities: we should be able to pick things from a library. Documenting patterns alone is not enough. We need to understand why these things work, rationale, etc. We often design from scrach instead of pulling in different components.
  • Cannot often justify to management why need 30+ architects on a large system
  • Organizational issues: if the architects don't all report to a chief architect
  • When dividing roles in 'fellow architects' and 'senior architects', are there role conflicts?
  • Architects often align themselves with their area of expertise
  • Design is not a multi-level thing: everyone is doing the same basic process of design, at different levels of abstractions. Developers are making small decision decisions (methods, classes), but the same basic processes are going in their heads, and are being represented in various design artifacts.
    • Counter-point: not necessarily. Developers may not be aware of the big picture.

Predicting Architctural Styles from Component Specifications

Slides Presented by Neno Medvidovic

  • Key idea: component-based development deals with grabbing off-the-shelf-code; whereas architectural styles are more high-level rules... These two ideas do not necessarily converge.
  • How can you get the benefits of arch. styles with those of building systems from components?
  • If you take a no. of components off the shelf, what style(s) of application structure will you get, and what properties will you expect from the system?
  • Shaw and Clements work on feature-based classification of architectural-styles
    • Constituent parts
    • Control issues
    • Data issues
  • E.g., In a P2P architecture: looking for deadlocks might make sense.
  • Authors provide an algorithm for predicting an arch. style
    • Identify a use case
    • For each service in use case, look at repository, and look for best fit candidate
      • What if the component is a partial fit? "good enough"?
      • For each component, you look at its "Component Type Attribute"
    • ...
    • Finally, you figure out which style(s) are the best fit using the taxonomy by Shaw & Clements
  • Discussion
    • Not that interested in primitive styles anymore; we typically buy large coarse-grained components? These styles do not show in integration of products.
    • Not so. See Garlan architectural mismatch paper. The "glue code" (the magic to make them work together) can be trivial or incredibly complex
    • The idea of styles is more of a vocabulary of building blocks, and not to enforce uniformity in the whole system. Typically, systems use multiple styles at once. But there is a difference between "call return", and accessing "shared data". Need to get beyond the buzzwords, and into the details. But knowing the vocabulary is important. The end result is not to pick one of the styles; the end result is to pick a good architecture.
    • When you put together a system based on external components, they all fit if they have been designed: e.g., Linux, webserver, etc...The components have been designed for such combination in mind. But in an engine controller, you cannot plug-in anything into it.
    • If you mandate to use off-the-shelf components that make different assumptions, then you end up with serious architectural problems.
    • Do we need new vocabulary to talk about styles? We matured past pipe-and-filter being the vocabulary.
    • But we really haven't matured. There's still a big gap between implementation and abstractions.
    • We need to characterize the styles better. Need to talk about limitations, strengths associated with the styles.
  • Sydney Opera House. Brilliant architecture. Poor execution. Whereas Sydney bridge is working flawlessly. Brought the original architect back onto the project to help them redo. This is a basic principle that form follows function. But, it's not clear what the function is.

TODO: Add picture from Neno.

  • There are many beautifully designed disasters. Examples:
  • Discussion: this approach does not predict any of the quality attributes that would result from putting together all these components.

Static Evaluation of Software Architectures

Slides Presented by Jens Knodel


  • PuLSE-DSSA
  • PuLSE: product-line software engineering
    • Turn various goals into scenarios
    • Define architectural views/viewpoints
    • Then, go through process loop
      • Planning:
      • Realization:
      • Documentation:
      • Assessment:
  • Late Architecture evaluation (source code present)
    • how to compare the intended architecture against the implemented architecture
    • look for divergences and convergences


  • Basic principle of static evaluation
    • High-level model
    • Low-level model
    • Domain expert provides you with some mapping to bridge the gap between the two
    • Various tools available (e.g., reflexion models)
  • Purpose or needs to evaluate an architecure
    • Planning:
      • Product line potential: Analysis whether several existing systems comply to a single product line architecture although the existing systems are separately developed products that not yet constitute a real product line.
      • Product alignment: Evaluation of the product line architecture against a product to detect variabilities and to estimate the degree of required modifications in order to align it to the product line architecture.
    • Realization:
      • Reuse potential: Analysis of the reuse potential of an existing architecture, a component or an architectural part in order to answer whether or not to reuse it.
      • Component adequacy: Assessment of the adequacy of a component for the product line architecture and the component’s internal quality.
      • Comprehension: Assessment of program comprehension on a higher level of abstraction (i.e., an architectural level or a component level).
    • Documentation:
      • Consistency: Assessment of the consistency of documentation (e.g., architectural views, component engineering models) against the implementation.
      • Completeness: Detection of elements that form part of an implementation, but are not yet documented.
      • Re-documentation: Static decomposition (on a low level) to create high level architectural descriptions.
    • Assessment:
      • Evolution control: Monitoring the evolution of system or a product line once released.
      • Structure: Assessment of the structural decomposition of a system or a product line.
  • The distinct purposes were derived from practical experiences in industrial and academic case studies.


  • Discussion:
    • does this scale?
      • It was used on a system and product lines of several hundreds of KLOC, currently systems implemented in C/C++, Java, and Delphi can be evaluated
    • Often start by documenting existing architecture.
    • Question: how do you abstract the implemented architecture from the code?
    • use domain expert; also use name guessing. clustering techniques were not effective.
    • How do you represent that architecture? what annotations?
      • We use UML.
    • How do you find the connectors? Components are usually easier. Do you have good ideas on how to recover connectors?
      • We're extracting call dependencies, import dependencies, and use dependencies.
      • Currently, only working statically; in the future work, we might be interested in doing runtime traces, and lifting them up to the architectural level.
    • How much effort was spent typically in case studies?
      • Parsing was a matter of a few hours
      • Getting the mapping between the arch and the code is the hardest part
      • Comparing was a matter of a few minutes
      • Visualizing was a matter seconds
      • Interpretation of results guides further architectural development

Introducing Ambient Calculus in Mobile Aspect-Oriented Software Architectures

Presented by Nour Ali

  • Many software systems are distributed
  • Need to consider many non-functional requirements
  • Usage of middleware technologies
  • Objective: offer developes of distributed applications automatic code generation frameworks
  • Proposing PRISMA and Ambient Calculus.
  • Integrates component-based software development and aspect-based software development
  • Each side of the "prism" can be seen as an aspect modeling a specific concern
  • Weaving mechanism weaves together the different aspects
    • Translate a graphical notation to an XML.
    • XML compiled to generate source code on .NET framework
    • Functionality required by Prisma not provided by .NET framework; PrismaNet layer on top of .NET provides that functionality. .NET does not allow reconfiguration and evolution mechanisms.
  • Prisma middleware can execute on multiple nodes of a distributed architecture
  • Ambient calculus (process algebra) can express mobility in a precise way
    • Ambient = bounded place where a computation happens.
      • Each ambient has a name, and local processes executing on it.
      • Sub-ambients (tree hierarchy)
    • Introduces mobility primitives to the Pi-calculus and communication primitives
    • Graphical representation
  • Detailed example
  • For more infomration [1]


  • Discussion:
    • Does mobility introduce new requirements for thinking about security, etc, which we had been working with for a while?
    • Making "mobility" an aspect
  • Have you made any specific modeling of the "context" in which the computation takes place?
    • This is really an architectural description language. In the ADL, you can describe the properties for each ambient, and add predicates to represent the context.
  • What about taking into account reconfiguration based on different "availabilities": moving from laptop to a pc is relatively comparable. Having to deal with reduced bandwidth or smaller physical display introduces additional complexity.
    • These advanced reconfiguration can be added to the Prismanet middleware tier, which currently implements some reconfiguration primitives. We're interested in finding out more about the various "adaptation" patterns or requirements.
  • Many formalists seem to be gravitating towards the Pi-calculus. Once you modelled things in Pi-calculus, how hard is it to analyze these models?
    • The formal models help us generate correct code. The idea is not for the architect to use the Pi-calculus. The architect will use the graphical representation and not see the calculus. the graphical notation is very intuitive.
  • General question: What really is evaluation and analysis? Is this something you do on a complete architecture? This is a framework for creating an architecture. Is it really an analysis technique?
    • We're doing a case study involving robots moving around to do a cleaning task. The operator is controlling the system from a distance. We're modeling the robots; and we will use automatic code generation.

Towards a set of application-independent clustering criteria within an architecture recovery approach

Slides Presented by Aline Vasconcelos

  • Proposing a set of application-independent clustering criteria to support architectural recovery
  • Extraction phase
    • Static and dynamic reverse engineering
  • Abstraction phase
  • Evaluation
  • Odyssey environment implements tool support for the approach
  • Static reverse engineering
    • Recover a class model from the Java source code
  • Dynamic reverse engineering
    • Uses AspectJ to collect trace executions
  • Clustering criteria
    • Based on the dynamic analysis of the system
    • Applying data mining techniques to detect common functionality
  • Criterion 1: interaction patterns
    • Sequences of methods frequently executed
    • If these occur more than a threshold, the classes are candidates to be clustered
  • Question: Can the method calls be interleaved? Yes.
  • Criterion 2: using apriori rule for mining the associating rules
  • Applied in two case studies involving software tools.
  • Recovering the architectura at the 4+1 model
    • Logical View
    • Process View
    • Scenario View:
      • Sequence diagrams associated to use-case scenarios

Discussion:

  • Is this the right criterion for understanding what the architecture is? If a client and a server talk a lot, you will tend to put them together into the same "thing". The logical distinction might be quite orthogonal to the level of interaction.
    • We can find the decomposition in a micro-kernel style.
    • What you find is highly-interactive clusters? But this is not one of the 4+1 views
  • Is the architecture use-case based? Is this really the way to come up with an architecture?
  • Using dynamic tracing does not provide a complete coverage. e.g., if a large portion of the code-based is for exception-handling purposes.
    • Currently, not giving developer support to
  • If using a rule-based system, the rules are represented in a database. Can you capture that?
  • What about parameter-driven systems? Mnay of our systems use late-binding.
  • Can I use this information for refactoring or re-engineering purposes?
  • How big were the case studies?
    • First case study had around 4 scenarios, 400 classes
    • Second case study had 9 scenarios, and 60 classes; almost all the classes
  • Did you get feedback from the original developers and architects? Did they evaluate the recovered architectural views?
  • Typically, in a recovery problem, you often have some initial knowledge about what the architecture looks like. But you may not have complete knowledge. The approaches that seem to work best are the ones that have a good mix of using "top-down" and "bottom-up"
  • In a fairly well documented architecture, you still learn a lot form using the traces. This can uncover divergences between the implemented and the recovered architecture.
  • You can use some additional clues: e.g., if the developer uses the following library in our organization, this means that they're using some kind of connector. But the trick is finding ways to express those things.
  • Is this how rule-based systems? Are there "rule architects"? A tool cannot hope to recover the architecture of a rule-based system, if we cannot figure out how a human would do it.

Extending SPQR to Architectural Analysis Using Semi-Automated Training

Slides Presented by Jason Smith.

  • Detects instances of instances of design patterns from source code
  • Based on analysis of concepts, not constructs
  • Defining "Elemental Design Patterns" (EDP)
  • Rho calculus (denotational semantic). Extension of Sigma calculus
    • Added relations: objects, methods, fields and types
  • Some of these EDP constructs may not be directly supported by most programming languages
    • Abstract Interface, RedirectInFamily, ExtendMethod
  • AbstractInterface: promise of a fullfillment of a method at a later day.
  • Objectifier pattern (Zimmer 1995)
  • Merging AbstractInterface and RedirectInFamily produces "ObjectRecursion" (Wolf 1998)
  • Adding a 3rd one, ExtendMethod, produces the Decorator pattern (Gamma et al. 1995)
  • The goal is to define GoF patterns
  • Surprised that we can get from denotational semantics to Decorator patterns in 3 steps
  • Pattern/Object Markup Language
    • XML version of rho calculus
    • Encapsulates notion of pattern directly
    • XSLTs can expand power tremendously
  • Allows practioners to do is to create canonical examples in code
    • Semi-automatic training
    • Helps refine the canonical example and get it down to essentials
  • One of the goals is to make it look like a compiler
  • SPQR sits at the gap between the design documentation and the source code
  • SPQR sits between the existing syste, and the architecture and systemdesign. The design patterns sit in the gap analysis between reverse engineering and requirements engineering, because they deal with expressing concepts.
  • SPRQs can be used for describe design decisions
  • In UAM

elements of arch -- elements of OOP functionality -- intent how they relate -- similarity principle how they interact -- reliance operators

  • Questions:
    • Where can the formalisms and approaches of SPQR fit into the architectural analysis realm?
    • Is there an undiscovered body of Elemental Architectural Patterns (EAPs)?
    • Where can the architetural analysis work fit into SPQR?

Discussion

  • Will this be more abstract? Or more complex?
    • Referring to the "Decorator" pattern introduces a shared vocabulary. Now, we can define the high-levels of abstraction using simple concepts.
  • All the GoF patterns can be defined in terms of Deleate or Recursion pattern. Are EDPs more elemental?
    • The EDPs are written as design patterns. We found them useful for teaching design patterns to students.
  • Design patterns are very centered around the quality attribute of maintainbility. At the architectural level, the driving attribute may no longer be that; they could be more about achieving reliability.
    • E.g., replication is an arch. technique to achieve reliability.
    • Fault-detection techniques
    • Very specific building blocks
    • There may be no pure patterns, but more hybrids.
  • For more information: [2]
  • Do you detect aspects at all?
    • Interested in viewing AOP programs as observer pattern at runtime.

Discussion questions

  • What is the future of convergence of research ideas/work on evaluation & analysis of architecture on industry practice? It is good to hear that some formal ideas have been abstracted, such as Ambient Calculus, from users, however, what are some thoughts/techniques for ensuring that industry architects can use the work of researchers?
  • There seems to still be a gap between research areas. For example, several presentations mentioned the notion of aspects, but, it did not seem to leverage a common body of experience and knowledge from this topic. Thoughts?
  • There seems to be a notion or an assumption about what architects actually do. When a researcher uses phrases such as "... an architect prescribes this using the ADL ...", there is an assumption about who the architect is and what tools are available. It would be useful to describe the roles and expectations up front so that a common understanding can be had be all. Thoughts?
  • It seems that the majority of the work in the field of architecture research in this area is on "niche" areas, versus the mainstream. Why is this the case?

Pre-session Discussions

Slides provided by the participants

Session Minutes

Post-session Summary

Architecture Evaluation and Analysis Session Lead: David Garlan

1. Introductory discussion and elicitation of focus areas.

We began by considering the broad vision for analysis and evaluation of architectures: to provide an engineering discipline that allows architects to make principled architectural decisions, evaluate the impact of those decisions, determine the conformance between architectures and other artifacts (code, requirements, etc.), and extract architectural representations from implementations.

Participants in the room represented both industry and academia in roughly equal numbers. A survey of interests revealed specific interest in the following topics.

  • Conformance (arch-impl) and reverse eng
  • Relation between Enterprise Arch & SW Arch
  • Improving arch practices in industry.
  • Disciplined processes for arch development
  • What is hard to do? What is state of art?
  • Tradeoff analysis
  • Analysis for dependability
  • Use of scenarios and other eval techniques
  • Product lines and mergers
  • Patterns (artificial and natural)
  • Causal Analysis

2. Presentations and Insights

The paper presentations covered many of the topics above.

Architecture reconstruction: About half of the papers dealt with the problem of extracting architectural information from code. Approaches included: using component specifications to determine architectural styles, finding patterns from primitive building blocks, looking for code clusters based on degree of interaction.

Organizational issues: With emergence of formal architect roles in organizations important practical issues include (a) how much architecting is enough, and what factors does that answer depend on; (b) how many architects and architectural roles are needed to cover the needs of architecting. Discussion of the first revealed that categorization of sweet spots for architectural ROI based on lines of code is too naïve. Important additional factors include issues of the routine-ness of a system, the level of reuse, the complexity of the domain, and others. Discussion of the second issue revealed that organizations now have many architect roles and many architects working on the same system. Additionally there are different sociologies of architects in different organizations, including hierarchical (master architect, sub-architects, etc.); domain-aligned (performance architects, security architects); temporal (architect that worries about long-term evolution of a system and product lines, architect that worries about immediate needs).

Making architects more effective: We know intuitively that some architectural decisions are fundamental – if you get them wrong the system will not succeed. Others are important, but secondary. How do you identify the critical architectural decisions that need to be made? One possible avenue is to focus on “architectural failures”. Specifically we should be including such examples in our architect’s handbooks. Other factors that may point one in the right direction are places where quality attributes interact and can’t be fully realized without compromise. (I can either give you high security or high performance, but not both.).

Emerging architectural techniques: We had several talks that described new formal approaches, including the use of Ambient calculus for mobile architectures, Rho calculus for semantics of patterns, use of Aspects for dynamic monitoring and for code construction. It was noted that many of the artifacts that we have in practice are not being addressed by current architecture research. Examples: How do you represent the architecture of a rule base? How do you deal with legacy systems written in non-object oriented languages such as COBOL.

3. Results

The primary result of the working session was a deepening of our understanding about issues of evaluation and analysis. We see that the subfield is maturing, but remains wide open for additional research and application of research results to practical settings. Issues that are most pressing include those of architectural conformance, making good architectural decisions on the right problems, understanding organizational and coordination issues of architectural expertise, and providing ways of representing and analyzing a larger varieties software related artifacts.

Post-mortem

Retrieved from "http://wwwp.dnsalias.org/wiki/Session:Evaluation--Discussions"
Personal tools