Benjamin Sago / ogham / cairnrefinery / etc…
Saturday, 13th June, 2020

Optional Type Misconceptions

More and more programming languages are being updated to gain a dedicated “optional value” type, rather than relying on null. Java gained Optional in Java 8, C++ gained std::optional in C++17, and newly-designed languages such as Kotlin or Swift have special syntax for optional values with the ? operator on a type.

I wholeheartedly welcome this change! I find that it makes my code easier to read, easier to write, and easier to modify.

Why is this so? The problems with null are well-documented by now, but here’s a summary:

  1. There’s no way of telling which functions return null and which return an actual value, so it’s harder to tell where you need to write error checks (and where you don’t need to).
  2. null values have a habit of propagating themselves far away from where they were initially created. If your program fails because a value is unexpectedly null, you have to go back and trace where this null originally came from.

I was lucky enough to have started programming in a language that had no concept of null from the beginning — Haskell — so I got to experience its benefits from the start. This reversed the question about null: I didn’t have to think “should we find some way to solve the null problem in a future version?”, but instead “should we add null in a future version?”. To which the answer was, of course, no.

However, this means that when I use an entrenched lanugage such as Java, which cannot remove null for the sake of backwards compatibility, I know that the experience of using a type like java.util.Optional isn’t as good as it could be. With an enormous amount of existing code, as well as the standard libraries, best practices, and developer habits of working with it, there are cases that using an Optional type makes the language a little more awkward to use. It helps, but it also hurts.

This post is a list of responses to the most common complaints about the Optional type I’ve seen on the internet, all of which I disagree with.

Misconception #1:
Optional doesn’t help because you can still ignore the error

The main benefit of using an Optional type is not just that you can tell which functions can return null, but that you can tell which functions can’t. So paradoxically, I find myself writing a lot more sloppy code using unchecked get or unwrap calls, purely because it’s much easier to come back and fix it later.

This is best appreciated when doing exploratory design work. For example, here’s some fragile, buggy code I wrote recently, to read values from a database and return them:

final User user = usersTable.findById(1234).get();  // (1)

final String name = user.name().formattedName();
final String address = user.getDefaultAddress().get().formattedAddress();  // (2)

final String phone = user.getPrimaryPhone();
final PhoneCall lastPhoneCall = phoneCallsTable.findLastCallTo(phone).get();  // (3)
final String lastCalledDate = dateFormatter.format(lastPhoneCall.callDate());

return String.format("%s (last phoned at %s)%n%s", name, lastCalledDate, address);

Look at all those calls to .get()! This code is very fragile:

  • (1): The user with ID 1234 might not exist in the database.
  • (2): They might exist, but have no default address.
  • (3): There may be no last phone call associated with their phone number.

Any of these three situations would throw an exception when the get call fails, crashing our program.

However, when you’re done with the happy path and it’s time to add error handling, these calls to get will stick out like a sore thumb. I can see where get is being used inappropriately, and replace it with better, sturdier code.

Here’s the same code written without using the Optional type. All the functions now just return null, and it now looks like this:

final User user = usersTable.findById(1234);

final String name = user.name().formattedName();
final String address = user.getDefaultAddress().formattedAddress();

final String phone = user.getPrimaryPhone();
final PhoneCall lastPhoneCall = phoneCallsTable.findLastCallTo(phone);
final String lastCalledDate = dateFormatter.format(lastPhoneCall.callDate());

return String.format("%s (last phoned at %s)%n%s", name, lastCalledDate, address);

Now, which function calls do we need to add null checks to? You could probably guess that the two that return entities from the database could return null, but what about the others? A user might not have a default address, but could they have no primary phone? Could formattedAddress fail if their address doesn’t fit the standard format? Could a phone call have no date?

Without get explicitly marking places in the code that fail to handle their errors, it becomes a lot harder to deal with missing values where appropriate. I’m forced to think about both the success path and the error path when I just want to be doing exploratory design work.

The usual way I’ve seen developers solve this problem is to add null checks everywhere, including the places where it’s not needed. And now your function is twice as long.

Misconception #2:
You now have two failure cases to deal with

Another reason why developers may not trust the Optional type is that in languages where null is already a thing, such as Java, you now have two failure cases to deal with: an empty Optional<T>, and null itself, which is still part of the language.

While this is a problem in theory, it does not turn out to be a problem in practice. There’s a simple rule that cuts one of these failure cases out:

If your Optional value is null, then you have a bug in your code and your program should crash, 100% of the time.

One of the problems with null is that you can’t tell the diffence between a null caused by something being missing, such as a user not being present in the database, and a null caused by a bug in the program, such as an initialisation function not being run. Using Optional types will separate these two cases; using straight nulls will not.

Here’s an example to demonstrate:

static Optional<String> overrideAppName = null;  // gets loaded later

static void loadGlobalConfig() {
    final Properties props = new Properties();
    props.load(new FileInputStream("application.properties"));

    overrideAppName = Optional.ofNullable(props.getProperty("app-name"));
}

void welcomeUser() {
    if (overrideAppName.isPresent()) {
        System.out.println("Welcome, user, in " + overrideAppName.get());
    } else {
        System.out.println("Welcome, user");
    }
}

We want our application to have a configurable name, and the name gets loaded from a config file in loadGlobalConfig. Then, later, we greet our user in one of two different ways, depending on whether the application name has been set.

However, this whole code is predicated upon the loadGlobalConfig function actually being called. If it gets skipped, or if it stops halfway through for whatever reason, then the overrideAppName will remain null.

This is a bug! If we changed the code to cater for this case, it would look like this:

if (overrideAppName != null && overrideAppName.isPresent()) {
    System.out.println("Welcome, user, in " + overrideAppName.get());
} else {
    System.out.println("Welcome, user");
}

And sure, this particular function will no longer throw an exception. But not only is it highly likely that some other function further down the program will just throw an exception instead, but you’ve covered up a bug.

So an empty Optional and a null serve two different purposes: one is intentional, and you should handle that case; the other is unintentional, and you should treat as a bug, and crash.

Misconception #3:
We don’t need Optional when we already have null

I can’t argue against this statement, as from a technical perspective, it’s entirely correct. There are no programs that can be written using Optionals that can’t be written using null:

  • value.isPresent() is the same concept as value != null.
  • value.orElse(other) has the same motive as value != null ? other : null.
  • Chains of map and filter methods can be re-written as… well, as code, like we’re used to writing.

The difference with null is instead a social one.

Java needs a special mention here, because before the Optional type came along, it tried to solve the problem using @Nullable and @NotNull annotations on types. If your method returned a string that could be null, then it returned a @Nullable String, not just a String. Or if it took a parameter that should never be null, then it could take a @NotNull Object, rather than just an Object. And then you’d have a compile-time processor, such as as the Nullness Checker in the Checker Framework, that would analyse your code and point out anywhere you aren’t checking for null when you should, or are checking for null when you don’t need to.

This is essentially the same safety guarantees you get with null, just done in a different way. At least it would have been, had this approach taken off; but it hadn’t, which is why Optional needed to be added to the language instead. For one thing, it’s opt-in: while your IDE might use these annotations to show warnings, your build server won’t, unless you’ve gone out of your way to set it up so it checks your code as it builds it. For another, you end up having more non-nullable types than nullable ones, and having to annotate the places that are not null gets annoying fast.

So while it’s true that you can get away with just using annotations, it’s more error-prone. And if humans were better at not making errors, we wouldn’t have issues around null to begin with.

The secret about absence

Those are the three main complaints about Optional types that I’ve seen, and I hope this clears up some of the details around their use. But there’s one more conceptual reason that I favour this type.

Earlier I mentioned languages that have special syntactic support for null, such as Swift or Kotlin, that were designed after the industry realised the mistakes about sticking with it. Instead of having to define an Optional string as literally an Optional<String>, you can just put a ? after the type, as String?. These languages go further and have special syntax for dealing with missing values: the safe navigation operator, ?. in Kotlin or ?? in Swift, make it easy to cut through null-heavy procedures without making the code really long.

But languages descended from ML have resisted adding special syntax and operators around missing values. In Haskell or Idris, for example, the Maybe type is a simple sum type with two possibilities, Just and Nothing. And instead of syntax like ?? or ?:, you have functions like map or orElse.

What’s the benefit of avoiding dedicated syntax for this? It’s precisely because it’s dedicated syntax that I don’t like. Having Optional be just another type lets you know its deep, dark secret:

Optional types are not special!

Haskell’s Maybe is just another data structure, one that you could implement yourself if you wanted. It’s like a collection, except instead of containing zero or more elements, it contains zero or one element — and has operations like map or filter that work in the same way.

I recently worked with some code that had to cache the results of running some functions. When the functions took a string argument, we had to store one result per string, which was put in a Map:

Map<String, InvocationResult> invocations;

When the functions did something different the second, third, or fourth time they were called, we stored the results in a List:

List<InvocationResult> invocations;

And when the functions did the same thing each time, the only thing we needed to know was whether we had cached data for that function or not, so we stored the result in an Optional:

Optional<InvocationResult> invocation;

It’s being treated like any other container, because that’s what it is: a container of values, just zero or one of them.

These are the reasons I prefer to use Optional types when they are available, instead of falling back to using null.∎

All posts