"Creeping Featurism is a Strength" if it is done right.

(Reading notes for Chapter 11 of "Beautiful Architecture" GNU Emacs: Creeping Featurism Is a Strength)

This chapter describes the software architecture of Emacs and discusses its creeping featurism "problem". Creeping featurism means that people tend to put more and more features into a system and eventually make it too complicated to manage. Many people think creeping featurism leads to no good, but if it can certainly become a strength if it is done right. Software becomes very complicated these days and people expect more and more from software. So a closed system is very hard to be very successful. We can see this tendency from desktop applications, like Firefox, iPhoto and so on, to enterprise software, like SplunkBase, to web-based applications, like Facebook.

Extension is especially important for open source software which usually has a lot of uncentralized developers and users.  I will talk about a few thoughts on how to make a software system open to extensions but in the meantime prevent creeping featurism.

Firstly, a system should have a fairly simple data model. The book chapter compares the data model of Emacs and the data model of Microsoft Word. I think this is a very good example. The simplicity of Emacs data model, strings of characters, makes it much easier to extend then MS Word, I'm not criticizing MS Word, after all, Emacs is an editor, not a word processor.

Secondly, the interface between the core part of a system and its other rich features should be clear and well defined. Emacs' MVC model is a good example again. Views are automatically updated once a relevant model change is detected. The core controller is written in C, but the controller code of most command we use is written in Emacs Lisp, which is also used as the major language to develop extensions. This design lets extensions integrated into the whole system naturally. Most extension code can be modified without recompiling the core Emacs, even without restarting Emacs.

Thirdly, a system should have a good default set of features that fit common needs. Most Emacs users only use a small subset of its features, but a complete installation of Emacs is already pretty big, it takes time to start and occupies quite some system resource when running. This is actually one of the major attacks from Vi advocators. To find the balance between efficiency and reasonable set of features is hard. This article presents one solution of having different sets of default features, like expert mode, normal mode, hardcore mode and such.

Fourthly, to make it very easy to install an extension. If it takes hours to install an extension, I don't believe people will do it at all. Emacs extensions are not trivial to install. For most of them, you need to modify your ".emacs" configuration file and copy package files to the right place. For some of them, it even requires compiling and/or installing new stuff. Software like Eclipse and Firefox do a better job on this.

Finally, to have a good platform for people to share and look for extensions is important. If a system is designed to be extend, extensions should be managed efficiently and provide good communication between people who write extensions and people to require new features.

Making enterprise software extendable and control its feature scale is quite different from to do so to open source software. I will probably write another post about that.

Domain specific distributed programming platform

(Reading notes for Chapter 3 of "Beautiful Architecture" - Architecting for Scale)


This chapter talks about Project Darkstar, a Sun-led open source distributed programming platform for online games and virtual worlds. One major thing I learned from this Chapter is to analyze a particular domain thoroughly and design a distributed programming platform for it, instead of trying to build a platform that works everywhere. Nowadays, distributing computing is an unavoidable task in front of almost every programmer. However, to write robust and efficient distributed programs is not trivial at all. It requires deep knowledge of how computer systems work and very careful design. Distributed programs are also much more difficult to test and debug. Does it mean that we need to train very programmer for it? No! Programmers should not be distracted from their own domain specific tasks. In this case,  to have a platform helps programmers focus on what they are good at and quickly scale their programs up. Unfortunately, distributed computing is very complicated and it often involves many design trade-offs. There is no one general  design that fit in everywhere. Project Darkstar is designed for the uniqueness of massively multiplayer online games (MMOs) and virtual world.

The target domain of Project Darkstar has a similar programming model with many web based services. It has a server and a huge amount of clients (players). Players send Tasks to server to query or update states. In the meantime, it has a few differences which lead to the design trade-offs of this system. I am especially interested in the design of its data model. I also found this technique report cover many details about this. It is very interesting to read.

First, latency is more critical than throughput for MMOs and virtual worlds. It requires very low latency to make sure that people can play games smoothly. This become an especially big problem when we talk about data storage, because data accessing is slow due to disk performance. Achieving low latency could be hard even in a non-distributed environment. Project Darkstar uses a distributed caching system with callbacks to avoid frequent permanent storage accesses and network communication.

Second, data access pattern is different. MMOs and virtual worlds have thick clients, which interactive with users by fancy graphic interfaces. Servers usually just keep states. So servers process a lot of small data access and about half of them are modifications. On the other hands, for most other web-based applications, clients are significantly simpler and servers handle most of the work. This character also mainly affects the design of the storage layer.

Third, users of MMOs and virtual worlds are more open to tolerate temporary data unavailability or even loss. So the design of Project Darkstar trade offs on this.

Fourth, a centralized and globally shared data management is especially important for MMOs and virtual worlds. The reason is two folds. It does not require game designer to divide games into playing areas. Every player is able to interact with every other player. It also prevents cheating more effectively.

Besides data storage, Project Darkstar also put a lot of efforts in other aspects of scaling an application, such as load balancing (demo), dynamic scaling, transaction management and so on.