One of the issues I see in many open source software packages, particularly those developed by academics, is that they are designed very poorly. This is not to say they are not engineered well, and that is the point of this post. Every package/program I have used does what it says it will do, but often times it is quite difficult to figure out how to make it do what it says it does. So today, I wanted to highlight some software I think is well designed, some I think are poorly designed, and talk a bit about how to design open source software with users in mind.

UX is undertaught

Most of these issues with software design come into play at the user interface level. User experience or UX is not a factor of software design commonly taught in non-computer science fields (in my field, quant psych, there was absolutely no training in it). The fundamental difference between a well designed software package and a poorly designed software package, I believe, comes down to how much effort a user needs to make to use the package.

Before we get into the weeds with some examples, I want to highlight an excellent book about product design, “The Design of Everyday Things“ by Donald A. Norman. This is a fantastic discussion about the psychology of product design (in fact it was originally titled “The Psychology of Everyday Things“), and I am planning on making it required reading for any software design course I teach. In Chapter 1, the author describes the design of a thermometer control in a refrigerator/freezer. There are two dials, one to control the cooling unit, and one to control the valve that partitions the cold air between the fridge and the freezer. This setup is absolutely reasonable from a engineering standpoint, as you have two distinct mechanisms to control. However, from a user standpoint it is quite difficult. The author shows the user guide for the fridge, with combinations such as A on the first dial and 5 on the second for normal settings, C-7 for colder freezer, etc… To make matters worse, the dials are labeled “Freezer“ and “Fresh Food“. Again from an engineering perspective, this is a fine choice, but from a design perspective it is counter-intuitive and difficult to use. This tension between fidelity to the actual mechanisms and user experience is a constant in software design, and it requires careful thought and planning to create a package that performs complex tasks while being easy to use and understand. Let’s go through a couple of examples.

Simple vs. Comprehensive

For readers who are in psychology, or indeed, any social science, you probably have heard of structural equation modeling or SEM. SEM is a powerful tool for modeling relations between sets of variables, as well as latent constructs, but it requires the use of specialized programs. Two of these programs really highlight two approaches to user experience that in my opinion are equally valid, which I will term the simple approach and the comprehensive approach.

Lavaan, simple syntax for complex problems.

Lavaan is an open source R package (found here) for structural equation modeling that is extraordinarily usable, particularly for actual data analysis tasks (as opposed to simulation studies). The user experience follows the simplification approach, in that while you can customize any aspect of the model you want, the package is fairly intelligent in choosing correct defaults. That means you can quickly specify models and get to interpreting results. Below is the code for running a two factor model, where the two latent factors, f1 and f2 are correlated, and have three indicators each.

model <- "
  f1 =~ x1+x2+x3
  f2 =~ y1+y2+y3
  f1~~f2
"
model.fit <- sem(model = model, data = data)

To me, this is quite intuitive. Latent variables are defined as being “equal“ to a combination of indicators, covariances are specified using “~~“. All observed variables correspond to variables in the dataset “data“. Of course, if you are not steeped in SEM for years, this might be less intuitive. But you also have to think about who the users are going to be. Is lavaan going to be used by undergraduate lab assistants taking their first statistics course? I hope not. But a graduate student/advanced undergraduate who has seen SEM before or a faculty member who took SEM many years ago and now has a need to use it? This design will likely work very well for them.

The key to lavaan’s simplicity is sensible defaults, a term that I will likely use over and over again in this blog. Here, the sensible default is to use the first indicator of the latent factors as the scaling indicator. Lavaan automatically sets the loading of x1 and y1 to 1, which then pins the variance of the latent factors to be the same as the variance of x1 and y1 respectively. Note, the user did not need to specify this at all, rather the package has this as a default. Lavaan is also very flexible. For example, this is the argument list for the sem function:

sem(model = NULL, data = NULL, ordered = NULL, sampling.weights = NULL,
    sample.cov = NULL, sample.mean = NULL, sample.th = NULL,
    sample.nobs = NULL, group = NULL, cluster = NULL,
    constraints = "", WLS.V = NULL, NACOV = NULL,
    ...)

There are a number of different options, all of which will completely change how the model is estimated. In this way, Lavaan simplifies the life of the user by only asking for directions if the user explicitly needs more complex options. It provides an intuitive syntax for model specification that allows for quite a bit of flexibility, while being easy to use for more simple models. Finally, the capabilities of lavaan are quite extensive. Overall, lavaan is a very well designed and engineered package and is a joy to use. Now, lets take a look at a program that embodies a comprehensive approach to design, LISREL.

LISREL, the brussel sprouts of SEM.

I have to admit, I love both LISREL and brussel sprouts. My SEM training was all done in LISREL, and I credit that in developing a deeper understanding of the modeling process behind SEM. However, while it might be good for you to learn LISREL (or eat brussel sprouts), you might not necessarily enjoy it. This is because LISREL takes a comprehensive approach to user interface. Newer versions of LISREL have GUIs and a simplified syntax called SIMPLIS, but here I am going to focus on the classic LISREL syntax. LISREL is proprietary software, which you can find here.

I am not going to go into the full specification of a model, which involves file input and a couple of other things. Instead, here is the necessary syntax for specifying the previous two factor model (at least, I think, I don’t use LISREL anymore, due to it being proprietary, so if you see errors, reach out and I am happy to correct!):

LK
f1 f2
VA 1.0 LX(1,1) LX(4,2)
FR LX(2,1) LX(3,1) LX(5,2) LX(6,2) PS(1,2)

Let’s break this down. LISREL using matrix notation for SEM, which means the commands above are explicitly freeing or fixing parameters that correspond elements in the design matrices. The “LK“ command is simply labeling our two exogenous latent variables, “f1“ and “f2“. “VA“ is fixing our scaling indicators loadings to 1.0 in the lamba_X matrix, which is a 6x2 matrix of factor loadings. So, LX(1,1) corresponds to x1 loading onto f1 with a value of 1. “FR“ stands for free parameters. This is how we specify which indicators correspond to which factor. the PS(1,2) indicates freeing the covariance between factors “f1“ and “f2“.

As you might be thinking, this type of user interface is not the easiest to use for the uninitiated. But it is comprehensive. Unlike Lavaan, which simplifies the model specification away from the specific model matrices, LISREL requires you to completely specify the model in its mathematical form. This corresponds much more closely to the actual “engineering“ of SEM. I still consider this to be good design, because 1) it is comprehensive, allowing the user to access every possible part of the model, and 2) makes no claim to simplicity. The worst design occurs when you try and fail to simplify a complex process. Lavaan succeeds in simplification, while LISREL doesn’t attempt to simplify.

In determining if the design of a piece of software is good or not, we also have to consider the intended audience. I believe that both Lavaan and LISREL succeed with different audiences. Lavaan is accessible by new researchers, while LISREL is built for people who know the matrix algebra like the back of their hand. While somebody might not pick up LISREL as quickly as lavaan, both are excellent examples of consistent user designs that provide a map from mechanism to interface, albeit for different audiences.

Now lets get into a couple of objectively bad design choices.

Bad defaults and inconsistent arguments

One thing guaranteed to result in user difficulties is when your default options are in some sense “bad.” This is best illustrated with an example. In my own research, I use an R package called igraph quite a bit. This package can do everything and anything with networks and is well engineered as well, in that it runs quickly, and I trust the results that come out of it. I use it, I will continue to use it, and I suggest its use to colleagues. It does have some design issues,and the one I want to highlight is “diag = FALSE“.

When you create a network in igraph, you are often converting an adjacency matrix or edge list into an igraph graph object. For adjacency matrices, the command looks like this:

net <- graph.adjacency(your_matrix, mode = "directed", diag = FALSE)

What I want to focus on is the “diag” argument. This argument, when true, considers elements on the diagonal of an adjacency matrix (which refer to self loops) as valid. The issue here is that with a vast number of network applications, self loops are nonsensical. For example, in a social network, a self loop of a friendship nomination is nonsense, while in a correlation matrix, 1’s on the diagonal are not meaningful. The key issue here is that “diag“ is set to TRUE by default, and this has massive impacts on the results of algorithms down the line. The reason I bring this up as a prime example of poor design is that this is a default option, where the default is not common and the choice of TRUE vs FALSE has large consequences to subsequent analyses. This is not only annoying to work with, it is quite dangerous. I once had to redo several months worth of simulations because of this very option.

Now, a caveat. We have to consider the user base. igraph is a general R package for graph theoretic analysis, which is used in both social and hard sciences. It might be the case that in physics and biology, self loops are the default (I am fairly certain they are not, but let us give the benefit of the doubt here). If that were the case, then, okay, I can see an argument for having it be the default. However, a much much better default would be to not have a default at all. Force the user to explicitly decide to use the diagonal or not. Don’t ever make large decisions for your user without asking them first.

The previous example was of what I consider dangerously poor design, while the next is what I consider annoyingly poor design. Consider the two following functions for plotting a network:

plot_network(network, vertex.size, edge.size,
             vertex.color, edge.color, vertex.label, edge.labels)

plot(network, radius, labels, color, e.width, e.color, e.labs)

Which one is easier to work with? I hope you will agree with me and say it is the first one. The issue with the second one is not that it won’t work, but that the arguments are entirely inconsistent and vague. What does “radius“ refer to? Likely to the radius of individual nodes, but it could refer to the radius of the whole network. “labels“ and “e.labs“ likely refer to node and edge labels, but it is a) unclear and b) inconsistent. You will likely have to look at the documentation for the second function every time you use it, and that could have been avoided by having a consistent labeling scheme consistent with user expectations.

Good design is about knowing the user

So what is my point? Whenever I use software, I make the good faith assumption that it does what it says on the box. The thing that personally drives me away/to a given package, is ease of use. Now, I personally am a bit of a specialized user, in that I live in software and scientific programming, and my interface needs are very different than most. But that highlights my last point: When you are designing software, the target user is never you. As you build your program/package, make sure that you are thinking of how other people will use this, not how to make it intuitive for you to use. Let me wrap up with some summary suggestions:

Know your user - Is this software going to be used by other quanty people or is it going to be used by applied researchers? What can you assume about the skill level of your user base? How complex can your user handle?
Make it simple or make it comprehensive - Either spend the time to make the default options and workflow simple, or explicitly tell the user they need to specify everything. Don’t try to split the difference, and make parts of the package easy, and the other parts require extensive user input.
Sensible defaults, but no dangerous defaults - It’s alright to put in sensible default options, but make sure that they are clearly labeled, and that the consequences of changing them are laid out. Don’t have default options that violate norms/expectations, and that have major consequences in other sections of the program.
Be consistent - Decide on how your user will interact with your software, and stick with a single general schema. Don’t change gears mid workflow. Keep in mind the cognitive effort a user has to make to learn new interfaces. They should struggle with their problems, not with your software.

UX is an underappreciated aspect of open source software development, and I hope this post gave some reasonable suggestions! As always, get in touch if I made any errors or if you want to guest post about scientific software development!