Credit: Anton Croos
There are numerous ways a software project can fail: projects can be over budget, they can ship late, they can fail to be useful, or they can simply not be useful enough. Evidence clearly shows that success is highly contextual and stakeholder-dependent: success might be financial, social, physical and even emotional, suggesting that software engineering success is a multifaceted variable that cannot explained simply by user satisfaction, profitability or meeting requirements, budgets and schedules (Ralph & Kelly 2014).
One of the central reasons for this is that there are many distinct software qualities that software can have and depending on the stakeholders, each of these qualities might have more or less importance. For example, a safety critical system such as flight automation software should be reliable and defect-free, but it's okay if it's not particularly learnable—that's what training is for. A video game, however, should probably be fun and learnable, but it's fine if it ships with a few defects, as long as they don't interfere with fun (Murphy-Hill et al. 2014).
There are a surprisingly large number of software qualities (Boehm 1976):
| Correctness | The extent to which a program behaves according to its specification. If your specifications are ambiguous, correctness is ambiguous. |
| Reliability | The extent to which a program behaves the same way over time. If your online banking app crashes sometimes, it's not reliable. (It's probably also not correct, unless it's specification said that it should crash randomly). |
| Robustness | The extent to which a program behaves similarly in different operating environments. A touch screen is less robust because it stops working consistently in the rain. Web forms that reject my last name as invalid are not robust. |
| Performance | The extent to which a program uses computing resources economically. Synonymous with "fast" and "zippy". Performance is directly determined by how many instructions a program has to execute to accomplish it's operations, but it is difficult to measure because operations, inputs, and the operating environment can vary widely. |
| Learnability | The ease with which a person can learn to operate a program. Learnability is multi-dimensional and can be difficult to measure (Grossman et al. 2009) |
| User efficiency | The speed with which a person can perform tasks with a program. For example, think about how many taps and keystrokes it takes you to log in to an app on your phone compared to using a fingerprint sensor like Apple's TouchID. |
| Accessibility | The diversity of physical or cognitive abilities that can successfully operate software. For example, something that can only be used with a mouse is less accessible than something that can be used with a mouse, keyboard, or speech. Software can be designed for all abilities, and even automatically adapted for individual abilities (Wobbrock et al. 2011). |
| Usefulness | The extent to which software solves a problem. Utility is often the most important quality because it subsumes all of the other lower-level qualities software can have (e.g., part of what makes a messaging app useful is that it's performant, user efficient, and reliable). That also makes it less useful as a concept, because it can be so difficult to measure for most problems. That said, usefulness is not always the most important quality. For example, if you can sell a product to a customer and get a one time payment of their money, it might not matter that the product has low usefulness. |
| Verifiability | The effort required to verify that software does what it is intended to do. For example, it is hard to verify a safety critical system without either proving it correct or testing it in a safety-critical context (which isn't safe). Take driverless cars, for example: for Google to test their software, they've had to set up thousands of paid drivers to monitor and report problems on the road. In contrast, verifying that a simple static HTML web page works correctly is as simple as opening it in a browser. |
| Maintainability | The extent to which software can be corrected, adapted, or perfected. This depends mostly on how comprehensible the implementation of a program is. |
| Reusability | The extent to which a program's components can be used for unintended purposes. APIs are quite reusable, whereas black box embedded software (like the software built into your car's traction systems) is not. |
| Portability | The extent to which an implementation can run on different platforms and environments |
| Interoperability | The extent to which a system uses standard interfaces. |
| Security | The extent to which a system prevents access to information that is restricted to a certain population |
Although the list above is not complete, you might have already noticed some tradeoffs between different qualities. A secure system is necessarily going to be less learnable, because there will be more to learn to operate it. A robust system will likely be less maintainable because it it will likely have more code to account for its diverse operating environments. Because one cannot achieve all software qualities, and achieving each quality takes significant time, it is necessary to prioritize qualities for each project.
These external notions of quality are not the only qualities that matter. For example, developers often view projects as successful if they offer intrinsically rewarding work (Procaccino et al. 2005). That may sound selfish, but if developers aren't enjoying their work, they're probably not going to achieve any of the qualities very well. Moreover, there are many organizational factors that can inhibit developers' ability to obtain these rewards. Project complexity, internal and external dependencies that are out of a developers control, process barriers, budget limitations, deadlines, poor HR planning, and pressure to ship can all interfere with project success (Lavallee & Robillard 2015).
As I've noted before, the person most responsible for isolating developers from these organizational problems, and most responsible for prioritizing software qualities is a product manager. Check out the podcast below for one product manager's perspectives on the challenges of balancing these different priorities.
Boehm, B.W. 1976. Software Engineering, IEEE Transactions on Computers, 25(12), 1226-1241.
Grossman, T., Fitzmaurice, G., & Attar, R. (2009, April). A survey of software learnability: metrics, methodologies and guidelines. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 649-658).
Emerson Murphy-Hill, Thomas Zimmermann, and Nachiappan Nagappan. 2014. Cowboys, ankle sprains, and keepers of quality: how is video game development different from software development? In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). ACM, New York, NY, USA, 1-11.
Procaccino, J. D., Verner, J. M., Shelfer, K. M., & Gefen, D. (2005). What do software practitioners really think about project success: an exploratory study. Journal of Systems and Software, 78(2), 194-203.
Paul Ralph and Paul Kelly. 2014. The dimensions of software engineering success. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). ACM, New York, NY, USA, 24-35.
Mathieu Lavallee and Pierre N. Robillard. 2015. Why good developers write bad code: an observational case study of the impacts of organizational factors on software quality. In Proceedings of the 37th International Conference on Software Engineering - Volume 1 (ICSE '15), Vol. 1. IEEE Press, Piscataway, NJ, USA, 677-687.
Wobbrock, J. O., Kane, S. K., Gajos, K. Z., Harada, S., & Froehlich, J. (2011). Ability-based design: Concept, principles and examples. ACM Transactions on Accessible Computing (TACCESS), 3(3), 9.
Software Engineering Daily, Product Management with Suzie Prince