I have blogged on this before, but I just read an article in the Washington Post that made me want to revisit the issue. As with emotional decisions, moral decisions seem to draw on different parts of the brain than intellectual decisions. So rather than integrating moral and intellectual attributes into an overall cost-benefit kind of decision, two different responses get activated (if the moral and intellectual answers diverge) and we are forced to decide which one to accept.
This is one of the consequences of the way our brains evolved. Since only one response is possible, it would be nice if we could integrate the two sets of criteria and come up with one overall best response. And we try to do this in reality (I have a whole course dedicated to doing it for public policy analysis). But it is so hard because of this brain structure.
System designers have to be careful when their designs have a moral or emotional component. If users will have moral or emotional responses, the decision they make when using our systems may not be what we expect. It is important for more of us to learn what we can about emotion and moral cognition.
