I started using Emacs almost exactly four years ago, after almost a decade of Vim. I made the switch cold turkey. I vividly remember being extremely frustrated by unbearable slowness while editing a Clojure file at work. With no sane way of debugging it, just moving the cursor up and down would result in so much lag that I had to step away from the computer to breathe for a while. When I came back I quit Vim (I knew how at that point), opened Emacs, and started building my configuration.
More recently, I've been very put off by the performance and stability (or lack thereof) of building large scale software via Tramp. This has been sufficient to have me looking out again. On a whim, I installed VSCode for the first time and tried its "remote development" capabilities and holy smokes are they good. Getting up and running was trivial and the performance was great. Saving files was snappy and LSP worked out of the box. What a different experience from my carefully-put-together, half-working, slow Emacs setup.
My common denominator for rage-quitting software seems to be consistent: bad performance.
There has recently been more discussion than usual regarding "modernizing" Emacs, by making keybindings more consistent with other applications and using more attractive color schemes and visuals, with the end goal of attracting more users and by extension more contributors.
In my view improving these aspects of user experience wouldn't hurt. The way I see it, though, is that for Emacs to attract more users it needs to be objectively better than the alternatives. And the way to do it is for Emacs to become even more like Emacs.
I see Emacs as being fundamentally two things: a programmable runtime, and a beacon for free software. I'm talking more about the former.
It needs to be a more robust, more efficient, and more integrated platform with a more powerful extension language, to empower its users to build their own environment.
Getting LSP integrated pervasively in Emacs in a way that it reliably just works and performs well out of the box, would go a long way towards making Emacs more attractive not just to new users, but to existing ones too. Imagine an experience similar to VSCode's:
- Open Emacs for the first time
- Open a source code file
- Emacs asks if you want it to configure itself for the programming language of that source file
- Saying "yes" automatically sets up Emacs to have a modern programming environment for that programming language with smart code completion, navigation, and refactoring, rich hover information, highlighting, automatic formatting, snippets, etc. Maybe even open a side window with a buffer with a short "getting started" tutorial showing the available keybindings.
Providing good out of the box support for LSPis one of the current priorities in the Neovim project.
Given enough users, opinionated community-built Emacs "distributions" like Spacemacs, Doom, and Prelude will do the job of making it easier for newcomers to get started with typical contemporary tasks: building software with popular programming languages, writing documents, managing machines, etc.
Building and maintaining these "distributions" also becomes much easier given a more robust, more efficient, and more integrated platform with a more powerful extension language.
Having a wizard showing up in new Emacs installations might be a great low-hanging fruit way of making Emacs more accessible. Assuming buy-in from core maintainers, the wizard could even directly reference popular Emacs "distributions" like the ones mentioned above, so that new users can kickstart their lives in Emacs.
The way to attract contributors can also be stated simply: directly improve the contribution process.
Easier said than done.
Many have created their Emacs wishlists. This is mine.
Table of Contents
Let's get into it.
1. Improved single-core efficiency
There are two dimensions to this:
- garbage collection efficiency
- code execution efficiency
For the past one and a half years, Andrea Corallo, a compiler engineer, has been working on adding native compilation capabilities to the Emacs Lisp interpreter. His work is available in a branch in the official Emacs repository. Folks have been trying it out, and according to the reports I'm hearing, the results are staggeringly positive. I am very excited about Andrea's work, which seems to bring enough improvement to the "code execution speed" side of the equation to make it a non-issue for now.
Andrea's work will also allow for more of Emacs to be implemented in Emacs Lisp itself (instead of C), which is what most contributors are used to. This is a great win for maintainability and extensibility: incrementally having more and more of Emacs be implemented in the language with which it's extended.
The garbage collector is still in much need of improvement. Many resort to hacks to ameliorate frequent and sometimes long pauses that seem to be unavoidable while working on large git repositories, fast-scrolling font-locked Eshell buffers, displaying dynamically updating child frames, navigating big Org files, and many other tasks.
(setq garbage-collection-messages t)2. Improved display efficiency and rendering engine
The display implementation in Emacs core is... less than ideal.
GNU Emacs is an old-school C program emulating a 1980s Symbolics Lisp Machine emulating an old-fashioned Motif-style Xt toolkit emulating a 1970s text terminal emulating a 1960s teletype. Compiling Emacs is a challenge. Adding modern rendering features to the redisplay engine is a miracle.
It would be great if Emacs did like Neovim and decoupled the editor runtime from the display engine. This would make it possible for the community to build powerful GUIs without having to change Emacs core, possibly using technology not fully sanctioned by core maintainers.
Take a look at the screenshots of these Neovim GUIs:
They're powerful, look great, perform well, and more importantly, are based on industry standard, cross-platform graphics APIs (Vulkan and WebGL respectively) that get lots of personpower contributions from companies and individuals alike.
The Onivim and Xi text editors could also be sources of inspiration:
- Separating the core runtime from the user interface
- Ropes for faster incremental changes and parallelization of text operations
- Game-like drawing pipelines
Check out this talk by Raph Levien: Xi: an editor for the next 20 years.
3. Leveraging preemptive parallelism
Emacs does not support parallel code execution via multi-core processing. Code execution happening on any buffer will freeze the whole program, preventing not only user interaction but other cooperative threads of execution from making progress as well.
Adding parallelism to Emacs in a way that automatically makes existing code run in parallel is about as close to impossible as it can get. What would be more feasible is including new primitives for parallel execution that new code could leverage, to build more powerful extensions to Emacs.
Emacs-ng is a recent effort that implements just that: an additive layer over Emacs that brings not only parallelism, but also asynchronous I/O capabilities via an embedded Deno runtime, and GPU-based rendering via WebRender. I am super excited about the very fast progress from the folks working on emacs-ng, and I think the project holds great promise for the future of Emacs itself.
Join the emacs-ng Gitter chat room to get involved!
There also seems to be advances in the area of immutable data structures that could be leveraged by the Emacs core, as seen in "Persistence for the Masses: RRB-Vectors in a Systems Language". Persistent data structures would make building thread-safe parallel code much easier.
4. Enhanced stability
It is very easy to either freeze Emacs or cause it to run very slowly. Multiple
times a day I have to hit C-g incessantly to bring it back from being frozen.
When that fails, I am sometimes able to get it back with pkill -SIGUSR2 Emacs.
At least once per week I have to pkill -9 Emacs because it turned completely
unresponsive. I suspect doing more work outside of the main thread might help
with this?
There are many hacks to ameliorate issues caused by long lines, but they're still fundamentally there. Advancements in the "display efficiency and rendering engine" effort would help with this too.
I recently tried a package that displays pretty icons on completion prompts, and noticed that it made scrolling through candidates really slow. Profiling showed that the package was creating thousands of timers, which were somehow causing the issue. There are lots of cases like this, where folks attempt to create something nice, but inevitably have to resort to hacks to either achieve acceptable performance, or to be able to implement the thing at all. Having a more robust/efficient/integrated core with a more powerful extension language would help here.
Impressive efforts from folks like Lars Ingebrigtsen who routinely comes in and obliterates 10% of all reported Emacs bugs also have a sizable impact. We users should follow the lead and do a better job not only creating good bug reports but also dipping in our toes and helping out: fixing bugs, writing tests, and documentation.
Yuan Fu recently wrote a nice guide for contributing to Emacs.
5. Emacs Lisp improvements
Emacs Lisp is a much better language than Vimscript. Unfortunately, that's not saying much. It's not a particularly good Lisp and has lots of room for improvement.
For example, if you want to use a map, you have three choices: you can use alists, plists or hash maps. There are no namespaces in Emacs Lisp, so for each of the three data types you get a bunch of functions with weird names. For alists get is
assocand set isadd-to-list, for hash maps get isgethashand set isputhash, for plists get isplist-getand set isplist-put.For each of those types it is easy to find basic use cases that are not covered by the standard library, plus it is easy to run into performance pitfalls, so you end up rewriting everything several times to get something working. The experience is the same across the board, when working with files, working with strings, running external processes etc. There are 3rd party libraries for all those things now because using the builtins is so painful.
Emacs Lisp APIs evolved incrementally while maintaining backwards compatibility over a long period of time. This is good: code written more than a decade ago still runs. These increments came about via decentralized volunteer efforts, and it shows: there are many inconsistencies and conflicts between and within libraries, which feel like having evolved without an overarching design.
Programmers used to languages that did go through careful, deliberate design brought some of it to Emacs Lisp:
| Package | For working with |
|---|---|
| a.el | alists, hash tables, and vectors |
| dash.el | lists |
| f.el | files |
| ht.el | hash tables |
| map.el | alists, hash tables, and arrays |
| s.el | strings |
| seq.el | sequences |
It would be great to have more and more of these influencing and being incorporated to the Emacs Lisp standard library and made to be very performant. map.el and seq.el seem to already be in thanks to Nicolas Petton!
Assuming a multi-core future for Emacs, it will also be critical to have good ergonomics for writing concurrent code. It should be easy to do the right thing (writing thread-safe code), and hard to do the wrong thing. I believe Clojure can also be a source of inspiration.
Other than that, a great module system, possibly one that allows different versions of libraries to coexist, would also be a great addition. Andrea Corallo seems to be trying out some new ideas in this space (discussion).
6. Dealing with non-text
It is currently possible to browse the web in Emacs in an embedded fully-featured WebKit widget. We need to go further—I want to have the same experience of the likes of Nyxt and vimperator, integrated to Emacs:
- switch between tabs with fuzzy completion (ivy, helm, etc.)
- navigate via link hinting
- Isearch web pages
- easily copy text content from web pages, paste it elsewhere
- create macros to repeat actions on web pages
I can kinda do some of these things right now with the existing WebKit widget along with some clever hacks. There are other more adventurous hacks which work around Emacs to create a full graphical interface. It would be great if this type of functionality was deeply integrated to Emacs. It's a difficult thing to do because of the existing display engine implementation and Emacs Lisp limitations.
I believe doing like Neovim (and others) and separating the core from display would help here. But, it would likely bring its own problems. There are unfortunately no silver bullets.
Less importantly but still desirable: email. Even though I write most of my email messages in Emacs, I read them mostly outside of it. I prefer to exchange plain text email, but sometimes I receive HTML email. When I do, I'd prefer to visualize it as the author intended. This is currently technically possible, but suffers from the same challenges as web browsing.
7. Improved contribution and development process
Contributing to Emacs core and packages in the official repository requires assigning copyright to the FSF. Employed software developers need to get paperwork signed by their employers' legal departments, a process that takes many days. Copyright assignment is likely not going away—can it be made more convenient?
It would be great to move to a forge style of contribution. It is honestly incredible to me how people keep track of patches flying around in email threads. Unless something like sourcehut or Patchwork is being used there's no automated CI making sure individual patches and overall contributions are in a good state. Hopefully the Emacs GitLab instance starts being more actively used and becomes the official way to contribute.
Copyright assignment and mailing-list driven development are definitely off-putting to folks who just want to contribute, and aren't looking forward to having to sign paperwork or learn a special way to contribute to every project they work with. The GitHub generation of open source developers are used to a standardized, powerful and convenient platform—anything other than that just feels not worth it.
Emacs will likely always have a niche of users, but it could grow to not have developers. Having large parts of the core implementation be in C makes it not very approachable to anyone outside the handful of contributors who do feel confident to change it.
It probably makes sense to continuously look for functionality implemented in the C core that could be replaced with focused libraries, like Neovim did by replacing almost all of their hacky, platform-specific code with libuv.
Also, would it make sense to start an Emacs Open Collective to fund work on Emacs?
Last "small" thing
Improve Tramp performance to match the experience of using terminal Emacs via SSH, or VSCode's Remote Development.
* * *
Talking is easy. Accomplishing any of these would require lots of work. It may not seem like it but text editors are a hard problem. And people, an even harder one.
I wonder if Emacs will stick around long enough and grow the necessary
functionality for us to someday run M-x neuralink-mode and evaluate Lisp in
the brain?
This article is part of How to open a file in Emacs: A short story about Lisp, technology, and human progress, published in January 03, 2021.