WICSA 2009 WS3 -- Analysis
From WICSA Conference Wiki
Session Chairs: Eoin Woods and Ali Babar
Tuesday, 15th September, 13:30 - 15:30
Contents |
Participants (Tentative)
- Please sign your name here if you are thinking of attending this working session. (Click the signature button in the editor). Tell us something about your background. Add a few sentences about the working session topic such as your position, questions you would like to see discussed, etc.
- Eoin Woods, Barclays Global Investors. Co-leading the session with Ali Babar. Interested in how analysis can be made directly valuable on iterative industrial projects, that don't have long architectural design phases.
- Raghvinder Sangwan, Pennsylvania State University. Presenting paper on incidental and essential complexity. Interested in determining the proportion of complexity in a software-intensive system that is an incidental byproduct of the development methodology.
- Bettina Biel, Leipzig University. Presenting paper on finding architecture-level causes of usability problems. Interested in integrating analysis into iterative software development, in this case into architectural design and usability engineering.
- Ali Babar, Lero, University of Limerick. Co-leading the session with Eoin Woods. Interested in several aspects of architecture level analysis methods, techniques, and tools. One particular interest is how the outcomes of research on analysis can be transferred to industry by demonstrating the quantifiable benefits of allocating resources to analysis activities.
Papers
- "Characterizing Essential and Incidental Complexity in Software Architectures", by Raghvinder S. Sangwan, and Colin J. Neill
- This paper reports results from an experimental case study that seeks to characterize essential and incidental complexity in the design of a complex software product using design structure matrices (DSMs). A DSM captures structural dependencies among the modules and can be used to identify parts of a system that lack cohesion and/ or are highly coupled. We consider such parts as excessively complex. In the case study, we capture the complexity of an Internetbased collaborative system as it was originally designed and after it was redesigned using an architecture-centric development methodology. We find significant reduction in excessive complexity of the redesigned system suggesting that excessive complexity can be an incidental byproduct of a development methodology that lacks focus on the systemic properties of a system that strongly influence its architecture.
- "From Retrospect to Prospect: Assessing Modularity and Stability from Software Architecture", by Kanwarpreet Sethi, Yuanfang Cai, Sunny Wong, Alessandro Garcia and Claudio Sant'Anna
- Architecture-level decisions, directly influenced by environmental factors, are crucial to preserve modularity and stability throughout software development life-cycle. Tradeoffs of modularization alternatives, such as aspect-oriented vs. object-oriented decompositions, thus need to be assessed from architecture models instead of source code. In this paper, we present a suite of architecture-level metrics, taking external factors that drive software changes into consideration and measuring how well an architecture produces independently substitutable modules. We formalize these metrics using logical models to automate quantitative stability and modularity assessment. We evaluate the metrics using eight aspectoriented and object-oriented releases of a software product-line architecture, driven by a series of heterogeneous changes. By contrasting with an implementation-level analysis, we observe that these metrics can effectively reveal which modularization alternative generates more stable, modular design from highlevel models.
- "Towards a Method for Analyzing Architectural Support Levels of Usability", by Bettina Biel and Volker Gruhn
- The user acceptance of a software product especially depends on its usability. Therefore, it is a matter that needs to be addressed early in the software development process. In existing scenario-based software architecture analysis methods that focus on usability, the usage context is not employed to select scenarios used for analysis, although it is known that understanding a specific usage context is important to carefully design for usability. In order to address this, we propose in this position paper the Method for Analyzing Architectural Support Levels of Usability, which incorporates a usage context analysis and a simple standards inspection of a user interface prototype. It furthermore provides a knowledge base of standardized analysis scenarios and analysis templates that are derived from general usability requirements and design patterns. By defining Architectural Support Levels (ASL), which are defined based on analysis scenario gradings, the proposed procedure model shall guide evaluators to accomplish a repeatable analysis resulting in a rating of the current design for usability in a traceable way.
Working Session Organisation
Welcome to the working session on Architectural Analysis. This working session will comprise short presentations of three relevant research papers and then small group discussion of some of the topics that emerge from these presentations, with the groups each reporting their conclusions at the end of the working group for further dissemination.
Timetable
| 13:30 - 13:35 | Introductions |
| 13:35 - 14:05 | Paper Presentations (10 minutes each) |
| 14:05 - 14:15 | Identification of discussion topics and formation of groups |
| 14:15 - 15:10 | Discussion Groups |
| 15:10 - 15:30 | Presentation of results |
Results
After the papers, we solicited discussion topics from the group and from a list of 6 or 8, chose two for further discussion in small groups:
- The effect of the level of abstraction on analysis
- Transferring DSM based analysis methods into practice
The discussion in each group and the conclusions drawn are summarised in the sections below. Thank you to the paper authors and all who took part in the discussions for such a productive session.
Effect of the Level of Abstraction on Analysis
The discussion in this group remained quite diversed and at an abstract level becuase the topic appeared to be relevant to many aspects of software architecture process and artefacts. The participants discussed the role of different factors, such as domain, involved stakeholders, and quality attributes, on the required level of abstraction. For example, the level of abstraction for architectural description required for analysing design-time quality attributes, such as modifiability, is likely to be different to the level of abstraction that may be required for analysing run-time quality attributes, such as performance and security. The participants also discussed the role of domain in the requirements for a certain level of abstraction as they were of the opinion that the level of abstraction for software architecture required for analysing quality attributes in an embedded system domain is likely to be different to the abstraction level required in the Enterprize systems' domain. An example provided was analysis for required memory as code structure is needed for such kind of analysis.
The participants also discussed the requirements of stakeholders for different levels of abstraction of software architecture. That means it is important to know that who will be using the software architecture, business stakeholders, hardware designers, software engineers, or maintenance people.
Different analysis techniques may also require different levels of abstractions. For example, scenario-based analysis and prototyping and simulatons would have different requirements for abstraction levels required for software architecture description. One particular quality attribute discussed in the group was usability as the participants thought that usability analysis related to software architecture can be performed using scenarios and patterns reported to support architectural level usability.
Accuracy versus inacuracy - organize the system will allow cancel command, security or performance...tradeoff analysis early on - abstract architecture level to know how the quality attributes critical systems for formal approaches - right design decisions, we try to have designs early as early possible - models help design the systems. importance of model is to able to analysis the systems
Self-adaptive systems - techniques are different - verification may run many day - run the model check for several days - time-constraints for self-adaptive system - we do analysis by looking into indication properties which may be going outside of the required properties - observe runtime components - and then report what is going on those components - the way we make analysis needs to change - we need to bring some constraints. we are telling system how to adapt - balancing different quality attributes within a certain threshold - specify the system's requirements -
Conclusions
Rather than definite conclusions, this group found it more productive to highlight questions that need to be answered when developing and assessing analysis approaches:
- Which quality attributes are sensitive to the level of abstraction? (i.e. which can be analysed at a high level, which only at a detailed level?)
- How sensitive is the accuracy of analysis to the stage of development at which it is performed?
- Which levels of abstraction is a particular analysis method applicable to?
- How do we bring enough detail up the abstraction levels to allow analysis? (e.g. memory usage is estimated at a low level, how can this be allowed to ripple up through more abstract descriptions to allow high level analysis?)
- Can tools be created that allow the same analysis approach to be used at different levels?
- How can you perform analysis of concerns like security at an architectural level?
Transferring DSM Analysis into Practice
- Current shortcomings of DSMs in practice include:
- need realization of architecture
- use DSMs at design level of UML (although Magic Draw creates DSM directly from UML)
- using a DSM probably restricts us to (semi-)formal notation?
- Can DSMs be used for reverse engineering approaches for identifying model?
- There are some scalability limitations of DSMs as used today as people often use a SAAM like manual walkthough of code to identify components and it's difficult to scale such manual processes (there is a hypothesis that there are few number of architecturally significant requirements so perhaps this isn't a problem as we focus on those?)
- Integration into design method can ease identification of feature mapping as developers/designers initially build features
- DSMs give insight into visibility and structures
- but people need to learn to read DSMs for adoption
- clustering of modules in DSM helps scale
- We seem to need more semantic information to relate components to requirements
- challenge to integrate environmental parameters into component diagrams to automatically generate DSMs
- people are willing to add custom annotations to UML diagrams for a payoff
- easier if integrated into development process
- may help to estimate costs for changing requirements
- Question as to how to calculate metrics from DSMs? only by adding rows and columns
- metrics can be further extended
- inclusion of probability of change
- inclusion of cost model
- metrics can be further extended
- key problem is how to get enough information into DSMs
- tools and techniques can handle it once data is there
- need people to put data in and keep it up to date
- techniques like UML 2, code annotations, can help provide data
- annotations that can be automatically analyzed by tool chain for definite payback will make it more likely to be adopted
- many existing tools only identify when something goes wrong
- Focus today seems to be retrospective analysis - can we predict that things are going to go wrong?
- Structure 101 can generate metric values on structure
- can specify "design rules" for allowed usage to identify violations
- can identify conformance of code to design
- allows custom plugins to populate your own dependency structure
- code dependencies can miss architectural dependencies
- Possible integration to use Magic Draw to get DSM from UML and other tools to create code DSM for conformance checking
- DSMs do not consider temporal dimension for evolution of design, for this you need many DSMs
- how to visual with time dimension? 3D matrices?
Conclusions
- Our experiences in research and practice seem to suggest that DSMs are very widely applicable to architectural analysis problems as they're flexible and provide a general framework that different sorts of analysis can be performed within and they can be used at levels from architecture to code.
- Initial readability of DSMs is difficult for people (particularly spotting patterns) but they're very scalable once learned.
- Getting the data into the DSM is the difficult part, particularly when annotations (such as measures) are needed beyond just simple structure. Possible solutions include
- code annotations where available (Java, .NET, Doxygen)
- manually during the software development process (or from the build tool chain?)
- via design tools (e.g. UML2 tools that can generate DSMs which could be exported)
- manual updates to the DSMs where people are motivated to do it (example being a manager who might be prepared to add cost model information to a DSM because of the usefulness of the resulting DSM for planning and estimating)
- Most DSM based tools spot problems after the event. Can we start to use them for predictive analysis because this would be further motivation to use them.
- Visualisation of data across multiple DSMs is difficult and most of the tools don't really use it. This would be very valuable to spot differences and trends over time
