Sergey Mikhanov  

The world’s most complicated software (May 19, 2015)

A typical software developer in a company possessing some level of technical sofistication routinely switches between abstraction levels during a working day. He or she may go from reasoning about product structure on a web page level to the intricacies of file allocation in their database system. Those dealing with some sort of message processing can switch from a byte-level layout of the protocol messages to a more general view of interconnected queues within the system — you get the drift. From the very early days of being in the profession, programmers are told that abstraction is the key to fighting complexity natural to all software. It’s only after spending few years in the profession, some may discover that few domains are surprisingly resistant to abstraction alone. Without the only tool to fight complexity, developers are left to accept the difficulty of the field as a given. I’ll give an example of a truly difficult problem.

Meet the calendar, the world’s most complicated software.

I’m talking about a product like Microsoft Outlook (actually, its calendaring part and the server). On a first sight, there’s nothing special about it, but if you try to think about it, complexity starts manifesting itself from a very basic level. The interaction protocol between participants trying to agree on a meeting is surprisigly hard to get right. For example:

  • When someone receives a meeting invitation, should it be shown to him if a meeting has already passed? How do we detect this (note that we should take time zones in consideration, including the cases when participant is not in his default time zone)?
  • When someone proposes a new time for a meeting, but it can’t be sent because the participant was offline should we try re-sending it when participant goes online? What about if the meeting time has already passed? What if not, but his proposed time did?
  • What about the time zone changes? If the participant changed a zone, should we reschedule his events? What if all participants changed the time zone? What would be the time zone of the newly proposed events? Of the changes into existing events?
  • Should we, once the participant goes online, notify him about the changes to the events that were not accepted by the participant because he was offline? What about the already passed ones?

And so on with an added inherent complexity of dealing with time zones. Calendar is a classical distributed system with participants being people within the same organizations using it simultaneously. Participants can be offline for extended periods of time; they must find consensus on timing of the group events using some reasonably robust protocol; they move around. You probably noted already that part of the problem is the difficulty in specifying it correctly — you’ll have lots of fuzzy and vague sentences with “except” in your specification, rendering almost all your abstraction skills useless. The difficulty of developing a calendar suite differs from the difficulty of your typical job ad’s “hard and interesting problem” in the same way as your morning 2 km jog differs from doing an Iron Man.

If you made a note to yourself never to work on calendar suites, here’s the second most complicated software in the world: library dependency manager. It does not have to deal with people as participants, but is just as full of fuzzy specs: how to handle conflicting (or broken) transitive dependencies, non-mandatory ones, source vs. binary, etc.

Would you want to work on a calendar suite or a dependency manager?

Venice–San Francisco (May 11, 2015)

Everyone who has ever visited Venice as a tourist probably went there with some limited baggage of knowledge about the town. A former center of very powerful Venetian Republic is, as the modern story goes, now full of tourists, almost abandoned by locals, sinking. Reality is, of course, much less grim. When I stayed there in March — it’s the low season there — Venice presented itself as a very lively small town. You stroll around and see kids return from school across the bridge of Calle Bandi in Cannaregio and groups of students from the nearby Academy of Fine Arts playing guitar in Dorsoduro.

Venice Dorsoduro

One comparison held very strongly on my head the entire time I spent in Venice. What I saw around was, figuratively speaking, what San Francisco will look and feel like in three or four centuries. Bounded by water on almost every side, it’s now just as restrained from further growth and almost as wealthy compared to its modern peer cities now as Venice was at its heyday. In the coming centuries the technology — that San Francisco will ultimately represent — will only grow in importance in everyone’s life and will help it prosper further. The best artists will lend their skills to making the future San Francisco the most refined city for those living there. The unstoppable gentrification will continue to purify the city fabric, eventually turning it into something as beautiful and uniform as Venice’s Centro storico.

Venice roofs

It will all be fine for San Franciscans until the technology will stop to matter (because everything eventually does). Maybe the new human psycho-powers will be discovered in, say, Cologne — it for sure will be Germans with their sense of irrational who will do it, — rendering the entire technology industry obsolete. After the period of San Francisco decline we’ll visit it, marvel at the hills and the architecture, and then will all be looking for “non-touristy typical San Franciscan restaurant”.

Apple Watch (May 4, 2015)

Can’t understand what’s the fuss with Apple Watch overwhelming people with constant distracting notifications. Notifications can easily be disabled. This is what my (and any reasonable person’s) notifications settings screen looks like.


Calls, texts, things to do, navigation, updates. That’s all; Twitter mentions can wait, Instagram likes can wait, even email can wait. Below the fold there are something like hundred and fifty apps, all of which were willing to get their share of my attention and which I decided to check on my own schedule. Because it’s the phone that belongs to me, not the other way around.

Deployment-driven development (November 10, 2014)

Die-hard fans of test-driven development advocate writing tests first, when no code of the actual program has been written. Their purpose at this stage is to serve as an (incomplete) specification of the program that also does not capture or express its actual semantics. To me, the described extreme case of TDD is meaningless, but recently I’ve discovered a real situation where one might want to write “code before actual code”. Let’s call it “deployment-driven development”.

The basic idea of DDD is that you always keep your program in a state where it can be deployed anywhere. This means you always start writing deployment automation code before anything. Want to work on a Python webapp? Instead of installing nginx and uWSGI locally, write a Puppet manifest or a Chef recipe first that will apply it on a virtual machine or a Docker container. It may even take less time to do this than to perform an actual installation. Need to add a database or configure log rotation? Update the manifest and redeploy. As a result, you’re always dealing with a software that is ready to be deployed anywhere and is not confined within your development machine.

There are a number of advantages to starting even the small projects with server automation first. When the product will be ready to go live, there will be many less checkboxes to go through in pre-launch checklist, and many less last-minute surprises. Exposure to deployment makes you think about the system architecture of your product at a very early stage and grow it as you code. You are always spawning an isolated machine to run your code, so library dependencies from different projects won’t interfere with each other.

What’s not to like about it?

So you’re moving to London (June 15, 2014)

This post was inspired by Alex Payne’s So you’re moving to San Francisco where the author went to great depths describing the San Francisco city life. I have lived in London for four years now and below are my personal reflections on the time spent here. Just like the original piece, my advice is aimed towards someone who considers moving to a major city with tech industry to work and this major city may be London.

For most Europeans London looks like an obvious choice. No visa hassle, no unfamiliar language to deal with, plenty of tech jobs and due to the compactness of Europe, it’s within a three-hour flight radius of most European capitals, so you’ll never be too far away from home. It’s the city of Beatles and The Gherkin. It’s great.

It’s disputed whether one city or another is better for joining an established tech company or starting a startup. Each city obviously comes with its own unique set of challenges or upsides. San Francisco and the neighboring Silicon Valley, of course, boast offices of the hottest tech companies in the world, an abundance of free capital, the most influential people in our industry and crazy ideas flowing freely around, but some of us can’t or wouldn’t consider living and working there for one reason or another. That reason can be a US visa that year after year remains difficult to obtain, or the general cost of living and getting your company off the ground. Alternatives like Berlin are cheap and cheerful and if you ask your programmer friend from Ljubljana to move and work there they won’t need a visa at all, but living in a German-speaking environment is not for everyone. Another emerging startup center, Tel Aviv, has its own track record of billion dollar exits, a steady supply of Technion graduates and beautiful beaches, but let’s be frank, Israel is not on the radar of most 20-something educated males willing to relocate to work in tech.

And then there’s London.

The Finance

In reality, London revolves around something that is both a curse and a blessing at once: the financial industry, mostly concentrated in an area called the City of London. Every Londoner is very familiar with the gap dividing “City” and “the rest” and often in a conversation it’s enough to simply specify your occupation as “I work in the City”. Even if you’re not planning to join an investment bank or a hedge fund but looking for more traditionally “hip” startup instead, you need to know how hard it is to avoid a distracting noise generated by finance. Tech recruiters will call the positions within established trading houses they advertise “a greenfield startup”. The salary gap between finance and everything else will be striking (even though it’s now shrinking — very slowly) and at times the industry in the City may look very appealing financially, especially when annual bonuses are taken into consideration. Unlike most other old European capitals or San Francisco where “billionaire’s modesty” is mostly a norm, London life can also be flashy, which adds to distraction. You will be turned around at some places’ entrances because of the wrong shoes (if those types of places are your thing) and you will never run into a London’s equivalent of Marissa Mayer wearing jeans in Starbucks around the corner.

The Neighbors

If everything above sounds like a description of city made for the rich, this is not the case. The main reason why so many people enjoy living in London so much is its tolerance. At the height of the British Empire, London was at the intersection of nations, and still continues to be the place where people are almost never judged by their wealth, accent, skin color, lifestyle or anything else. A native Russian myself, I have heard several people saying that this city helped them “develop themselves into who they wanted to be” unlike other, less tolerant places. This tolerance also manifests itself physically, in the layout of London. Thirty-two London boroughs and countless districts within them all carry unique personality that Londoners are very proud of. “Unique” is not an exaggeration here: it’s not uncommon for two neighborhoods within hundred meters from each other to look differently, house different classes of people and offer very different lifestyle choices. This means where you live in London will define how you live to a greater extent than elsewhere. The places you’ll visit and the people you’ll meet will reflect the personality of an area you live in — or the other way around.

There’s one thing that all districts of London have in common though: they are all expensive to live in. Rental costs in London are utterly disproportionate to the inadequate quality and the average size of the housing. The same could be said about public transport: while there are bright patches in this picture, most of the London’s transport is dominated by terribly underinvested and expensive Tube. Most young professionals thus start their life in London with renting shared apartments and spend a significant amount of their free time discussing Tube crowds and delays.

The Industry

For those young professionals London offers lots of possibilities to develop themselves and engage with the strong technical community here. Despite the most influential people of our profession living and working across the ocean, there are regular meetups and conferences that bring the brilliant minds from all over the world to London. Be this software, hardware, business or anything else tech-related, there will be an event to go to and people to meet.

But the omnipresent distraction of finance and high real estate costs shape what the rest of the tech industry look like. If you’re going to join an established tech company like Google, Facebook or Twitter, all of which have offices in London, they will look and feel mostly like their counterparts across the ocean. Running a startup in London may differ significantly from what a person familiar with a startup scene elsewhere would expect.

After visiting Silicon Valley, most people, including me, get the impression that the relationship between investors and founders in the startup world is driven by the same genuine desire to help each other that permeates the air in California. This isn’t surprising: most of the top-tier venture capitalists in the Valley are former hackers themselves, so it’s easy for them to identify themselves with the founders. Moreover, for angels or venture firms helping to make the companies in their portfolio successful is the only way to increase their wealth or the value of their fund. The founders of those companies are thus treated accordingly.

The investors and investment funds in London are different. All tech firm exits I’m aware of as of today are below $500M (with LoveFilm, Playfish and being the largest ones), which by Silicon Valley standards is pretty modest. What is even worse, the money earned from those exits rarely stays in London and almost never gets re-invested. For startup founders this means that your investor most likely will be someone who made their fortune in a hedge fund, trading or a real estate firm. They genuinely want some piece of hot startup action, but you need to be prepared that your company will be just a minor part of the spectrum of your investor’s business activities with their main focus most likely being away from tech.

To me this seems like a major missing ingredient in the startup world: founders meeting the right people at the right time may define the success or failure of the companies they start. Paul Graham famously quoted Sean Parker as an example of an influential figure in the development of Facebook, despite his relatively minor involvement in the daily Facebook activities.

Still, if your goal is to move to London to start your own company, I encourage you to have a closer look at the startup scene yourself. Unlike San Francisco, London is a home to a number of world-class companies in industries other than tech and finance, so it might be that your company could benefit from this diversity.

The Town

Experiencing London outside of work may be both very rewarding and disappointing. For a two thousand year old city, London has no antique palaces or medieval streets; very few buildings predate the great London fire of 1666. For a casual stroller London looks like a loosely linked collection of small villages, some of which may feature rows of Victorian or Georgian terraced houses, some of which may be full of grim warehouse conversions or social housing in different stages of disrepair (and some may have both). The close proximity of good and bad neighborhoods together with the unique ability of Londoners to ignore something they don’t want to see lead to the fact that there are virtually no places in London that offer an uninterrupted beautiful view. You can say anything about London: it’s cool, vibrant, extravagant, fun, tolerant, it’s the city where everything is possible, but one thing it isn’t — it’s not beautiful.

One thing that is truly represented well in this town is everything cultural. Culture in London is one and maybe the only thing I’ll miss very strongly if I ever leave here. London boasts a number of very well presented world-class museums, several of which are state-sponsored and free to visit. They can easily cater both to someone who, like myself, is fond of contemporary art, as well as those who can appreciate classics. Cinema gets surprisingly little public attention in London, and even arthouse cinemas tend to offer just slightly offbeat mainstream selections. Music is, of course, one of the England’s strong sides and good concerts are very frequent. Competition for tickets can be fierce, though, which always leaves the proponent of free markets in me puzzled: if a concert is sold out within twenty minutes, surely there should be a motivation to hold another one? Theatre in London is simply great, especially when it comes to music and drama. London hosts a number of international theatre festivals, so seeing a theatre or ballet troupe from abroad is entirely possible.

You won’t be disappointed when it comes to food either. A couple of decades ago the restaurant scene in London may have been boring, but now it offers the widest selection, both when it comes to hole-in-the-wall places, as well as middle-class, upmarket or luxury restaurants. There are a few caveats to this though. First, the available options of everyday food are still very limited: most Londoners find it acceptable to roll on a run-of-the-mill sandwich from one of the fast food chains most days. Second, when a fast-food industry adopts some food type in London, it completely wipes away all other options to try it, leaving you only with a dull ersatz (this is what happened to sushi, for example). Just like with the music scene, the competition for a spot in a good restaurant can be fierce.

Discussions of London food are impossible without the discussions of London weather. This part of England has the same weather throughout most of the year: windy winters with virtually no snow and no frosty sunny days, and chilly rainy summers with almost permanently overcast sky. Even lifelong Londoners don’t disagree — it’s dire. Any glimpse of clear skies inevitably has people flocking outdoors, filling parks and forests across town.


It’s hard to love London unconditionally. Most people I know — and I include myself here — are balancing between love and hate, most of the time. For me, the famous London tolerance and world-class cultural scene is what keeps me here. There are lots of other places in the world offering a better quality of life, but few that offer (though maybe just marginally) better professional environment for people in technology. It might be that I’ll be heading elsewhere in the coming years. If you’re heading to London soon — good luck.