The values of Emacs, the Neovim revolution, and the VSCode gorilla

In 2018 Bryan Cantrill gave a brilliant talk where he shared his recent experiences with the Rust programming language. More profoundly, he explored a facet of software that is oftentimes overlooked: the values of the software we use. To paraphrase him slightly:

Values are defined as expressions of relative importance. Two things that we're comparing could both be good attributes. The real question is, when you have to make a choice between two of them, what do you choose? That choice that you make, reflects your core values.

He goes ahead to contrast the core values of some programming languages with the core values we demand from systems software, like operating system kernels, file systems, microprocessors, and so on. It is a really good talk and you should watch it.

It is important to think about values because they are core to the decisions that we make.

Unlike systems software, the values demanded from text editors or IDEs vary greatly depending on who you ask. These are much more personal tools and make room for a diverse set of desires.

The following listing enumerates values that could be attributed to development tools.

In my view, Emacs has the following core values:

Emacs

• Extensibility
• Freedom
• Introspectability
• Keyboard centrism
• Stability
• Text centrism

We can feel the clasp of stability in the following—rather poetic—exchange in the Emacs development mailing list, which also provides useful historical perspectives.

Emacs is older than the operating systems people use today. (It is almost as old as the first Unix, which barely resembled the Unix of later decades.) It is much older than Linux, the kernel.

The oldest design elements were not designed for the uses we make of them today. And since we wrote those, people have developed other areas of software which don't fit Emacs very well. So there are good reasons to redesign some of them.

However, people actually use Emacs, so a greatly incompatible change in Emacs is as unthinkable as a greatly incompatible change in the New York City subway.

We have to build new lines through the maze of underground pipes and cables.

— Richard Stallman in "Re: Discoverability (was: Changes for 28)" (2020)

The following exchange reifies freedom and stability while demonstrating a disinclination to progressiveness. Which is neither good nor bad; it's just what it is.

If Emacs was to become a "modern" app tomorrow, an editor extended in Lisp still only has appeal for a minority of programmers, much like the Lisp language itself. Most programmers looking for easy and modern experiences will likely stick with Atom and Sublime.

Most of the push for a "modern look" comes from the desire for Emacs to play more nicely with proprietary platforms. Rather, the goal of Emacs is to support platforms like GNU/Linux. Platforms that respect your freedom, and also do not push a corporate UI/UX vision of "modernity".

(Perhaps if we do move forward with modernization, we should think of modernization in the context of something like GNOME rather than MacOS or Windows. Surely Emacs could be a better citizen of GNOME.)

Given that many of the people complaining about "how Emacs looks" are not submitting patches to fix the problem themselves, resources would be diverted from actual functionality to "modernity".

By the time we do major code refactoring "modernizing" Emacs on the major proprietary platforms, what is "modern" has now once again changed, and our resources were put towards a project with a poor return on investment.

Basically, I don't see a "modernizing" project playing out well. We will spend extensive time and energy on a moving target, and even if we succeed, our Lisp-based vision still has limited appeal. Additionally, I don't think "modernizing" Emacs advances the cause of free software, given that there are other more popular casual libre tools for text editing that individuals can use.

— Ahmed Khanzada in "Re: Why is emacs so square?" (2020)

Core values are self-reinforcing. They attract like-minded people, who will then defend them.

I'm an Emacs user, and reading the Emacs mailing lists serves to remind me that my values are very different from the values held by maintainers and core contributors. I don't value freedom or stability nearly as strongly and have an inner affinity for progressiveness and velocity.

* * *

Now let's talk about Vim. I see Vim as intersecting with a few of Emacs' values, but ultimately diverging radically with its narrow focus on providing really efficient editing capabilities.

Vim

• Doing one thing well
• Editing efficiency
• Keyboard centrism
• Stability
• Text centrism

One might notice that extensibility is not in the list. That's intentional. Vim is certainly extensible to a degree, but it just does not compare to Emacs. Vim has a "plugin system", while Emacs is the system. Your code becomes part of it the moment it's evaluated. Since I'm sticking to yes/no indicators for values I'm giving it a no.

Stability emanates from communications with the primary maintainer.

Vim development is slow, it's quite stable and still there are plenty of bugs to fix. Adding a new feature always means new bugs, thus hardly any new features are going to be added now. I did add a few for Vim 7.3, and that did introduce quite a few new problems. Even though several people said the patch worked fine.

— Bram Moolenar in "Re: Scrolling screen lines, I knew, it's impossible." (2011)

And of course in this famous exchange in a QA session.

How can the community ensure that the Vim project succeeds for the foreseeable future?

Keep me alive.

— Bram Moolenaar in "10 Questions with Vim's creator" (2014)

At the end of 2013, a few folks were trying to get new concurrency primitives merged into Vim. This would empower plugin authors to create entirely new types of functionality and by extension, make Vim better.

This is what one of them had to say about the process:

The author of Neovim (Thiago de Arruda) tried to add support for multi-threaded plugins to Vim and has been stymied.

I'm not sure how to get a patch merged into Vim. Bram Moolenar is the only person with commit access, and he's not a fan of most changes beyond bug fixes. My co-founder and I tried to add setTimeout & setInterval to vimscript. Even six weeks of full-time effort and bending over backwards wasn't enough. Eventually we were just ignored.

I've contributed to a lot of open source projects, and the Vim community has been the most difficult to work with. I've been writing C for almost two decades, and the Vim codebase is the worst C I've ever seen. The project is definitely showing its age, and I'd love for something new to replace it.

— Geoff Greer in "Neovim (HN)" (2014)

While they understood that some of their values were ultimately incompatible with the values of the Vim maintainers—who prioritized stability—they still tried to push for a change, because they treasured the idea of Vim, embodied by some of its values.

It didn't happen, so a Vim fork came to life: Neovim.

The vision was grand, and is summarized in a statement of its values:

Neovim is a Vim-based text editor engineered for extensibility and usability, to encourage new applications and contributions.

— neovim.io/charter

Some of their concrete plans included

• improving testing, tooling, and CI to simplify maintenance, make aggressive refactorings possible, and greatly reduce contributor friction
• decoupling the core from the UI, making it possible to embed the Vim core into browsers or IDEs (or any computer program really), also making way for more powerful and diverse GUIs
• embedding a Lua runtime and providing concurrency primitives to open the doors for smoother, more efficient, and powerful plugins
• extensive refactoring: bringing C code to modern standards (C99, leveraging new compiler features), replacing platform-specific IO code with libuv, removing support for legacy systems and compilers, including automatic formatting, and fixing static analysis warnings and errors
• creating a scriptable terminal emulator

And they delivered it.

In a very short amount of time they were able to, and I don't use this word lightly, revolutionize Vim. The impact can be seen in Vim development, which picked up considerably as Neovim gained ground, with features and processes ending up being reimplemented in Vim.

Neovim builds upon Vim, and the way I see it, holds the following core values:

Neovim

• Approachability
• Editing efficiency
• Extensibility
• Keyboard centrism
• Progressiveness
• Text centrism
• Velocity

As I see it, it also currently has a better story than Emacs on:

• development process: non-mailing-list-driven-development, extensive automated builds, a public roadmap, recurrent funding, frequent stable, automated releases
• user interface: all UI-related code for every single platform was removed from the core, replaced by a consistent API (both TUIs and GUIs use it) that made a diversity of decoupled display implementations possible (UIs are literally plugins!)
• out of the box experience: better defaults, built-in LSP client

These items are the outcome of massive change that came about through consistent hard work from a few individuals who shared a vision and a set of values. Crucially, it included aggressively improving the human side of software: raising money to support development, lowering contribution friction, unblocking contributors, reconciling and combining efforts, documenting processes. In other words, the type of invaluable work non-software engineers do in technology companies. To our detriment, in open source these tasks are often neglected.

Code is the easy part of building software.

It's hard to contest that Neovim's achievement happened because of its approachable development process focused on maintainability and velocity, while in contrast, it could be argued that current progress in Emacs happens despite its development process.

For example, because Emacs highly values freedom, contributing to Emacs core (or to packages in the official repository) requires assigning copyright to the FSF. To incorporate packages into the main repository, everyone who committed to the project needs to have gone through that procedure. Even in the case of a very willing, actual core Emacs maintainer, of an uncontroversially valuable package used by virtually everyone, this process can take years.

It is also not hard to find criticism coming from folks who have already and continue to give so much to the community and ecosystem.

It all comes down to core values.

* * *

Let's now address the 800-pound gorilla in the room: VSCode.

VSCode was released just five years ago, and in this short amount of time it was able to capture half of the world's software developers.

It provides a powerful, refined, cohesive out of the box experience with great performance.

It has immense leverage by building on top of Electron, NodeJS, and Chromium, projects that receive contributions in the millions of person-hours of work, from both the open source community and heavily invested corporations.

Here's how I see its values.

VSCode

• Approachability
• Integration
• Maintainability
• Progressiveness
• Velocity

We can now put it all together in this very uncontroversial table.

Irrespective of values, VSCode is looking more and more as an acceptable Emacs replacement.

• It is somewhat extensible and very configurable
• It can be mostly driven from a keyboard
• It has a great extension language, TypeScript (which is in my opinion superior to Emacs Lisp in terms of maintainability for non-trivial projects)
• It even has a libre variant

It also shines in areas where Emacs doesn't: if you're a programmer working on typical contemporary projects, mostly just wanting to get stuff done, things usually... just work. You install VSCode, open a source code file, get asked to install the extension for that particular language, and that's it. You get smart completion, static analysis, linting, advanced debugging, refactoring tools, deep integration with git, and on top of that, great performance and a cohesive user experience.

This type of experience is the selling point of Doom and Spacemacs, two initiatives driven by relentless maintainers. These projects bring approachability and integration to Emacs, and are in my view, along with LSP, Magit, and Org, the biggest reasons drawing people to Emacs nowadays. It is however clear from looking at their issue trackers just how difficult it is to provide this cohesive experience by combining parts from the ecosystem.

With Neovim also shaping up as a worthwhile up-and-comer, this is probably the first time Emacs has actual competition in its own turf.

In "Emacs is my "favourite Emacs package"" Protesilaos Stavrou talks about the importance of Emacs, the platform. While Emacs packages can be valuable in isolation, combined, they amplify the platform that made them possible. The whole becomes greater than the sum of its parts.

It wouldn't be a stretch to say that Org represents 10% of my cognitive function. Magit really is "Git at the speed of thought", and I have yet to see a more integrated and rich interactive shell than Eshell.

And yet, even being a very enthusiastic Emacs user, I have a hard time recommending it to folks who mostly just want to get stuff done. Some will argue that those who aren't willing to build their computing environment from scratch shouldn't be using a "power tool" like Emacs anyway. I don't see a fundamental reason for that to be the case, and believe that not having young folks trying out, using, and contributing to Emacs, is not a good thing for Emacs.

This existential threat seems to be acknowledged by maintainers.

One of the gravest problems I see for the future of Emacs development is that we slowly but steadily lose old-timers who know a lot about the Emacs internals and have lots of experience hacking them, whereas the (welcome) newcomers mostly prefer working on application-level code in Lisp. If this tendency continues, we will soon lose the ability to make deep infrastructure changes, i.e. will be unable to add new features that need non-trivial changes on the C level.

— Eli Zaretskii in Re: [PATCH] Add prettify symbols to python-mode (2015)

For better or worse, Emacs overfits to the needs and priorities of its maintainers, and contributors who overcome its barriers to entry. Being a decentralized, volunteer-based project, people will commonly scratch their own itches or work on whatever they find interesting. Which is only fair: they could be doing literally anything else, and yet they choose to sacrifice their time and do their best to advance Emacs according to their values. They owe no one anything and deserve gratitude.

* * *

In a 2015 keynote, while laying out an argument for why the Go programming language is open source at all, Russ Cox portrayed an active and intentional effort to lower barriers to entry and deliberately improve the human side of Go, so that as many people as possible used and contributed to it.

Go was created to make Google's developers more productive and give the company a competitive advantage by being able to build products faster and maintain them more easily. Why share it with the world?

Russ argues that the business justification for it is that it is the only way that Go can succeed.

A language needs large, broad communities.

A language needs lots of people writing lots of software, so that when you need a particular tool or library, there's a good chance it has already been written, by someone who knows the topic better than you, and who spent more time than you have to make it great.

A language needs lots of people reporting bugs, so that problems are identified and fixed quickly. Because of the much larger user base, the Go compilers are much more robust and spec-compliant than the Plan 9 C compilers they're loosely based on ever were.

A language needs lots of people using it for lots of different purposes, so that the language doesn't overfit to one use case and end up useless when the technology landscape changes.

A language needs lots of people who want to learn it, so that there is a market for people to write books or teach courses, or run conferences like this one.

None of this could have happened if Go had stayed within Google. Go would have suffocated inside Google, or inside any single company or closed environment.

Fundamentally, Go must be open, and Go needs you. Go can't succeed without all of you, without all the people using Go for all different kinds of projects all over the world.

— Russ Cox in the GopherCon 2015 keynote

The parallel to Emacs isn't direct, but it's clear.

Emacs evolved greatly since its inception in 1976 as a collection of macros for the programmable TECO text editor, which is itself from 1962. Take a look at this example TECO program, from Wikipedia:

0uz
<j 0aua l
<0aub
qa-qb"g xa k -l ga -1uz '
qbua
l .-z;>
qz;>

It was a different world then. Updating the display text in real-time as users typed into the keyboard was a recent innovation.

Since then, Emacs Lisp was created, Emacs forks came and went, a graphical UI was added, lexical scope was implemented, rudimentary networking and concurrency primitives were introduced, and more.

It can be argued that stability and incremental evolution are the reasons why Emacs survived and is still thriving. Stability though is necessarily antithetical to progressiveness. The very thing that likely made it succeed is what slows it down.

It doesn't, however, affect progressiveness as much as freedom. Because of freedom, when faced with a question of using technology that is

1. Non-free, but objectively better
2. Free, but objectively worse

the latter will always be picked. Given that most technological progress happens through the mechanisms of capitalism, "free" alternatives commonly lag behind to a large degree.

Freedom has a price.

It can be seen clearly and succinctly in this exchange:

[...] I think freedom is more important than technical progress. Proprietary software offers plenty of technical "progress", but since I won't surrender my freedom to use it, as far as I'm concerned it is no progress at all.

If I had valued technical advances over freedom in 1984, instead of developing GNU Emacs and GCC and GDB I would have gone to work for AT&T and improved its nonfree software. What a big head start I could have got!

— Richard Stallman in "Re: New maintainer" (2015)

Agree with him or not, RMS has a point. The ability to inspect and change software running on our computing machines gives us control.

Apple for example seems to be growing increasingly antagonistic to the privacy of its users (and bullish to its developers). As I write this, it's been discovered that newer versions of macOS include anti-malware functionality that transmits tracking information almost every time any program is run. This information is sent unencrypted via a third-party CDN, so not only this could be seen as a privacy violation but also a dangerous data breach: anyone listening on the network can roughly know which applications you use, how often you use them, when do you use them, and from where.

There are measures that can still be taken to ameliorate this situation and others, but it's ultimately outside of our control. macOS is a proprietary operating system and can easily prevent users from taking these steps in the future.

I choose to pay the price of compromising my freedom by tolerating invasions of my privacy so that I can have a computer that mostly just works and allows me to be productive towards achieving my life goals. We have to pick our battles, and the hills we die on depend strongly on our values.

We are increasingly finding ourselves in a world where we have to choose between extremes: do you want a computing machine that respects your privacy, or a modern, powerful one?

* * *

Back to Emacs.

Maintainers and core contributors likely use it in very different ways than the majority of users and casual contributors and therefore have very different priorities. For example, until it got stolen in 2012, RMS used a 9-inch netbook because "it could run with free software at the BIOS level" (these days he uses an 11-year-old T400s). The recent Emacs User Survey 2020 might help identify prevailing usage patterns, and hopefully have an impact on the direction of Emacs.

Emacs doesn't need a Neoemacs as much as Vim needed Neovim. Unlike Vim, Emacs always had a rich ecosystem of active contributors, and maintainers who—to some degree—listen to user feedback. There is still tension, rooted in ideology and values, which throughout Emacs history materialized as forks: Lucid/XEmacs, Guile Emacs, Aquamacs, Mac port, Remacs.

Forking is incredibly difficult to pull off. A successful one requires not only an initial momentum and enthusiasm, but also unrelenting, sustained hard work from a group of individuals, not to mention buy-in from a critical mass of users. As someone said to me, you have to be a "special kind of crazy" to start an Emacs fork.

I hope that Emacs doesn't find itself becoming "perfectly suited for a world that no longer exists". Still, I understand and appreciate the difficulty of the situation. People have diverging values and are highly fallible. Everything requires so much effort. So is our condition.

Ultimately, building software is a complex and deeply human activity. Everything is contextual and there are rarely easy answers. Most meaningful progress happens through consensus, compromise, luck, and lots of hard work.

In the end, a lot can be understood through the lens of values.

What are yours?

This article is part of How to open a file in Emacs: A short story about Lisp, technology, and human progress, published in January 03, 2021.