Sergey Mikhanov  

Deployment-driven development (November 10, 2014)

Die-hard fans of test-driven development advocate writing tests first, when no code of the actual program has been written. Their purpose at this stage is to serve as an (incomplete) specification of the program that also does not capture or express its actual semantics. To me, the described extreme case of TDD is meaningless, but recently I’ve discovered a real situation where one might want to write “code before actual code”. Let’s call it “deployment-driven development”.

The basic idea of DDD is that you always keep your program in a state where it can be deployed anywhere. This means you always start writing deployment automation code before anything. Want to work on a Python webapp? Instead of installing nginx and uWSGI locally, write a Puppet manifest or a Chef recipe first that will apply it on a virtual machine or a Docker container. It may even take less time to do this than to perform an actual installation. Need to add a database or configure log rotation? Update the manifest and redeploy. As a result, you’re always dealing with a software that is ready to be deployed anywhere and is not confined within your development machine.

There are a number of advantages to starting even the small projects with server automation first. When the product will be ready to go live, there will be many less checkboxes to go through in pre-launch checklist, and many less last-minute surprises. Exposure to deployment makes you think about the system architecture of your product at a very early stage and grow it as you code. You are always spawning an isolated machine to run your code, so library dependencies from different projects won’t interfere with each other.

What’s not to like about it?

So you’re moving to London (June 15, 2014)

This post was inspired by Alex Payne’s So you’re moving to San Francisco where the author went to great depths describing the San Francisco city life. I have lived in London for four years now and below are my personal reflections on the time spent here. Just like the original piece, my advice is aimed towards someone who considers moving to a major city with tech industry to work and this major city may be London.

For most Europeans London looks like an obvious choice. No visa hassle, no unfamiliar language to deal with, plenty of tech jobs and due to the compactness of Europe, it’s within a three-hour flight radius of most European capitals, so you’ll never be too far away from home. It’s the city of Beatles and The Gherkin. It’s great.

It’s disputed whether one city or another is better for joining an established tech company or starting a startup. Each city obviously comes with its own unique set of challenges or upsides. San Francisco and the neighboring Silicon Valley, of course, boast offices of the hottest tech companies in the world, an abundance of free capital, the most influential people in our industry and crazy ideas flowing freely around, but some of us can’t or wouldn’t consider living and working there for one reason or another. That reason can be a US visa that year after year remains difficult to obtain, or the general cost of living and getting your company off the ground. Alternatives like Berlin are cheap and cheerful and if you ask your programmer friend from Ljubljana to move and work there they won’t need a visa at all, but living in a German-speaking environment is not for everyone. Another emerging startup center, Tel Aviv, has its own track record of billion dollar exits, a steady supply of Technion graduates and beautiful beaches, but let’s be frank, Israel is not on the radar of most 20-something educated males willing to relocate to work in tech.

And then there’s London.

The Finance

In reality, London revolves around something that is both a curse and a blessing at once: the financial industry, mostly concentrated in an area called the City of London. Every Londoner is very familiar with the gap dividing “City” and “the rest” and often in a conversation it’s enough to simply specify your occupation as “I work in the City”. Even if you’re not planning to join an investment bank or a hedge fund but looking for more traditionally “hip” startup instead, you need to know how hard it is to avoid a distracting noise generated by finance. Tech recruiters will call the positions within established trading houses they advertise “a greenfield startup”. The salary gap between finance and everything else will be striking (even though it’s now shrinking — very slowly) and at times the industry in the City may look very appealing financially, especially when annual bonuses are taken into consideration. Unlike most other old European capitals or San Francisco where “billionaire’s modesty” is mostly a norm, London life can also be flashy, which adds to distraction. You will be turned around at some places’ entrances because of the wrong shoes (if those types of places are your thing) and you will never run into a London’s equivalent of Marissa Mayer wearing jeans in Starbucks around the corner.

The Neighbors

If everything above sounds like a description of city made for the rich, this is not the case. The main reason why so many people enjoy living in London so much is its tolerance. At the height of the British Empire, London was at the intersection of nations, and still continues to be the place where people are almost never judged by their wealth, accent, skin color, lifestyle or anything else. A native Russian myself, I have heard several people saying that this city helped them “develop themselves into who they wanted to be” unlike other, less tolerant places. This tolerance also manifests itself physically, in the layout of London. Thirty-two London boroughs and countless districts within them all carry unique personality that Londoners are very proud of. “Unique” is not an exaggeration here: it’s not uncommon for two neighborhoods within hundred meters from each other to look differently, house different classes of people and offer very different lifestyle choices. This means where you live in London will define how you live to a greater extent than elsewhere. The places you’ll visit and the people you’ll meet will reflect the personality of an area you live in — or the other way around.

There’s one thing that all districts of London have in common though: they are all expensive to live in. Rental costs in London are utterly disproportionate to the inadequate quality and the average size of the housing. The same could be said about public transport: while there are bright patches in this picture, most of the London’s transport is dominated by terribly underinvested and expensive Tube. Most young professionals thus start their life in London with renting shared apartments and spend a significant amount of their free time discussing Tube crowds and delays.

The Industry

For those young professionals London offers lots of possibilities to develop themselves and engage with the strong technical community here. Despite the most influential people of our profession living and working across the ocean, there are regular meetups and conferences that bring the brilliant minds from all over the world to London. Be this software, hardware, business or anything else tech-related, there will be an event to go to and people to meet.

But the omnipresent distraction of finance and high real estate costs shape what the rest of the tech industry look like. If you’re going to join an established tech company like Google, Facebook or Twitter, all of which have offices in London, they will look and feel mostly like their counterparts across the ocean. Running a startup in London may differ significantly from what a person familiar with a startup scene elsewhere would expect.

After visiting Silicon Valley, most people, including me, get the impression that the relationship between investors and founders in the startup world is driven by the same genuine desire to help each other that permeates the air in California. This isn’t surprising: most of the top-tier venture capitalists in the Valley are former hackers themselves, so it’s easy for them to identify themselves with the founders. Moreover, for angels or venture firms helping to make the companies in their portfolio successful is the only way to increase their wealth or the value of their fund. The founders of those companies are thus treated accordingly.

The investors and investment funds in London are different. All tech firm exits I’m aware of as of today are below $500M (with LoveFilm, Playfish and being the largest ones), which by Silicon Valley standards is pretty modest. What is even worse, the money earned from those exits rarely stays in London and almost never gets re-invested. For startup founders this means that your investor most likely will be someone who made their fortune in a hedge fund, trading or a real estate firm. They genuinely want some piece of hot startup action, but you need to be prepared that your company will be just a minor part of the spectrum of your investor’s business activities with their main focus most likely being away from tech.

To me this seems like a major missing ingredient in the startup world: founders meeting the right people at the right time may define the success or failure of the companies they start. Paul Graham famously quoted Sean Parker as an example of an influential figure in the development of Facebook, despite his relatively minor involvement in the daily Facebook activities.

Still, if your goal is to move to London to start your own company, I encourage you to have a closer look at the startup scene yourself. Unlike San Francisco, London is a home to a number of world-class companies in industries other than tech and finance, so it might be that your company could benefit from this diversity.

The Town

Experiencing London outside of work may be both very rewarding and disappointing. For a two thousand year old city, London has no antique palaces or medieval streets; very few buildings predate the great London fire of 1666. For a casual stroller London looks like a loosely linked collection of small villages, some of which may feature rows of Victorian or Georgian terraced houses, some of which may be full of grim warehouse conversions or social housing in different stages of disrepair (and some may have both). The close proximity of good and bad neighborhoods together with the unique ability of Londoners to ignore something they don’t want to see lead to the fact that there are virtually no places in London that offer an uninterrupted beautiful view. You can say anything about London: it’s cool, vibrant, extravagant, fun, tolerant, it’s the city where everything is possible, but one thing it isn’t — it’s not beautiful.

One thing that is truly represented well in this town is everything cultural. Culture in London is one and maybe the only thing I’ll miss very strongly if I ever leave here. London boasts a number of very well presented world-class museums, several of which are state-sponsored and free to visit. They can easily cater both to someone who, like myself, is fond of contemporary art, as well as those who can appreciate classics. Cinema gets surprisingly little public attention in London, and even arthouse cinemas tend to offer just slightly offbeat mainstream selections. Music is, of course, one of the England’s strong sides and good concerts are very frequent. Competition for tickets can be fierce, though, which always leaves the proponent of free markets in me puzzled: if a concert is sold out within twenty minutes, surely there should be a motivation to hold another one? Theatre in London is simply great, especially when it comes to music and drama. London hosts a number of international theatre festivals, so seeing a theatre or ballet troupe from abroad is entirely possible.

You won’t be disappointed when it comes to food either. A couple of decades ago the restaurant scene in London may have been boring, but now it offers the widest selection, both when it comes to hole-in-the-wall places, as well as middle-class, upmarket or luxury restaurants. There are a few caveats to this though. First, the available options of everyday food are still very limited: most Londoners find it acceptable to roll on a run-of-the-mill sandwich from one of the fast food chains most days. Second, when a fast-food industry adopts some food type in London, it completely wipes away all other options to try it, leaving you only with a dull ersatz (this is what happened to sushi, for example). Just like with the music scene, the competition for a spot in a good restaurant can be fierce.

Discussions of London food are impossible without the discussions of London weather. This part of England has the same weather throughout most of the year: windy winters with virtually no snow and no frosty sunny days, and chilly rainy summers with almost permanently overcast sky. Even lifelong Londoners don’t disagree — it’s dire. Any glimpse of clear skies inevitably has people flocking outdoors, filling parks and forests across town.


It’s hard to love London unconditionally. Most people I know — and I include myself here — are balancing between love and hate, most of the time. For me, the famous London tolerance and world-class cultural scene is what keeps me here. There are lots of other places in the world offering a better quality of life, but few that offer (though maybe just marginally) better professional environment for people in technology. It might be that I’ll be heading elsewhere in the coming years. If you’re heading to London soon — good luck.

On the difficulty of thinking big (April 7, 2014)

Yesterday it suddenly struck me what I think is so special about the most successful startup founders.

Let me first say that I haven’t met any of them in person yet. The part of London startup scene that I’m exposed to is not your classic Silicon Valley breed of Stanford and CalTech grads, ex-Googlers and friends of Marc Andreessen. People here build taxi ordering systems and video curation apps. They may be fine pieces of engineering, but this is not the thinking and scale that wins big.

That scale and thinking is, I think, is what differentiates successfull founders. Not alone, though, but combined with the capacity to execute.

Mentioning any of those qualities as differentiators is not unusual at all. Paul Graham wrote about “thinking big”, just as well as Fred Wilson and countless others. Calling founders “execution machines” is also pretty common. What I find most amazing is that those qualities are almost mutually exclusive, and hence so rare. Thinking big, as the authors above mention, is truly difficult.

There is a group that is inherently good at it — children. Here’s, for example, a project from Elif Bilgin, the last year’s global winner of Google Science Fair in her age category. Just think about it: a sixteen year old proposes a way to make plastics from banana peel. She most likely does not know about the political and economical intricacies of the chemical and agricultural industries, about how hard it is to bring a technology like this to a mass market, and about mind-blowingly complicated relationship between different parts of the Earth’s ecosystem. That’s great: all this does not restrain her imagination. You truly believe that you can change the world when you’re so young. This is thinking really big.

If you think you can do this too, try. It’s not that easy to let your imagination go. For every little project detail, your brain does a micro feasibility check, without you even noticing. The wilder your dreams are, the stronger is the pull to the ground.

It’s still possible to imagine a project like being born in the depths of, say, BASF, and given a green light. Then a person responsible for that would drown in the details. Local legislation, partner companies, manufacturing processes, keeping the eco groups at bay, etc. A really big checklist, meticulously followed may help. This is the startup’s execution part that may be less glamorous, but is always absolutely necessary.

Successful founders, I think, are those who can do both thinking big while keeping the execution complexity at bay. Their micro feasibility checks turn into a gut feeling about growing markets. Their execution capacity is big enough for so many variables involved in building of the startup. This is what makes them special.

A love letter to Java (March 31, 2014)

I used to say that when it comes to comparing Ruby and Python, my personal opinion is that Ruby is worse. I absolutely was wrong. To misquote Stalin, they are both worse.

It’s about a month since I switched to working in Python. The project consists of few layers: an infrastructure to control ffmpeg-based video transcoding, elastic cloud running on AWS that hosts those transcoders, and a web front-end to control all that. The more I deal with it, the more I think fondly of the time when I’ve worked on similarly-structured projects in Java.

Don’t get me wrong: what comes below is not a criticism of Python as a language or an ecosystem, it’s more of “what a pity that a great platform like Java gets so little love from hackers.”

For example, a typical setup for a Python web app is nginx, WSGI and some framework (we use Bottle). Now, a typical setup for Java web app is a container (say, Tomcat) and a framework. In Java, when I need to debug my web application, I go to $TOMCAT_HOME/log directory and find all the logging output there. The logging libraries everyone loves to hate, log4j and slf4j, do an enormous amount of heavy lifting to allow a fine-grained control of the logging output not only of your program, but of all the libraries you depend on, plus the container itself. Where do logs from nginx go? Oh, it’s /var/log, whereas WSGI and your app normally log to some different place. Want to reconcile them? Good luck with that.

The famous slogan of Java, “Write once, run everywhere” was a foundation for a mindset that Sun used to enforce. Dependency on the external native libraries was considered a bad taste and a last resort. For example, JDBC drivers were split into four types, from least portable type 1 to pure-Java type 4. Python language is portable, but Python libraries are not — most of them are just wrappers around native binaries, and when Pythonistas say “native”, they mean “Linux”. It took me about an hour to run into a library that our project needs and that does not work on my favourite operating system. That’s pyinotify, a wrapper around a Linux kernel function called inotify, used to watch for changes in the file system. I do understand that it’s hard to develop a library like this that will be portable, but developers of Java managed to.

The situation with native libraries is made even more complex by the fact that Python processes themselves are rarely long-running. They are normally spawned by more stable processes, in our case crond and nginx. Controlling native libraries with LD_LIBRARY_PATH gets even more painful. How do those processes inherit environment variables? How to debug it?

You may ask why I mention Ruby here at all. I think that Ruby and Python together did a very big thing to programming. They made it popular. In parallel with that, they made it seem easy. But it’s not easy, it’s very hard. Programming only seems easy when you focus on some particular detail and ignore the rest, because in most cases it’s the communication with the rest of the world that makes software systems complicated and hard. And as much as everyone likes to say that Java is for people who like to build factories of factories, you have to give Sun credit. They didn’t ignore details they didn’t like.

What a great language.

Couple of good videos I watched lately (March 27, 2014)

C10M: Defending The Internet At Scale. This is a surprisingly deep video about how to program very big machines with huge RAM and lots of CPU cores under Linux disguised as an introduction video to imaginary problem for web developers. You didn’t know how poll or select scale when machine characteristics grow? You can learn about it from the video.

Cloud Haskell. This is a video about how to design a distributed system disguised as a tutorial for a specific library. Really useful to watch even if you don’t practice Haskell. It covers lots of dimensions of a distributed system, such as process execution and distribution, instance allocation and configuration and the way to unify them in a coherent library.