Whilst initially I considered moving to the US, I gave up after learning about how hard it is to get permanent residency there as an Indian citizen and focussed on (western) Europe, so this post is heavily biased towards that region.

Moving is hard and stressful as is, but moving to a completely new country is even more daunting. A lot of folks think it’s impossible and you need to be a genius to pull it off. I can assure you that that’s very much not the case and it’s very doable if you have the patience to stick through the process. Let’s dive in!

Finding a job with visa sponsorship takes research and patience

This topic is what I get almost all the questions about. It’s the hardest and often the most confusing part as it’s not very straightforward to find employers who would sponsor a visa. More so if you’re not looking to work at big tech companies. In almost all countries, you need a job if you wish to relocate.

Caveat: your entire life will be tied to your job and if you lose it for whatever reason, you may be forced to leave the country after a short period of time.

A few companies clearly mention up-front whether or not they sponsor visas. Your level can have an impact - some companies prefer to sponsor relatively senior folks than folks early in their career. Remember that visa sponsorship is a costly and cumbersome process for the company to undertake. However, most big enough startups are open to sponsoring folks at all levels. I’m not sure how Covid-19 will impact this in a remote-first world.

I looked for jobs through a variety of sites: LinkedIn, Twitter, various job boards. Going through a mutual is the quickest method by a long measure, but I cold-emailed and got in touch with folks directly too. Some governments, like the UK, also publish their database of employers who hold a valid sponsorship license, which you can sift through. It is often a very time-consuming process and I haven’t found an easier way to do it. Based on what I remember, there were significantly more options for frontend positions than backend positions. However, this might have changed now.

Edit: To gather data on employers that sponsor visas, I created a repository for folks who are having trouble finding sponsorship.

Be clear about wanting to relocate from the start

Always clarify before interviewing that you’re looking to relocate if you’re not clear whether or not your prospective employer is open to sponsoring you. It’ll be a huge waste of time for everyone involved if you go through the entire process and find out that they’re not willing to sponsor you. Explicitly state what you’re looking for in terms of relocation in one of your initial calls rather than waiting.

Having more than one option maximises your chances

Don’t put all your eggs in one basket. It’s a hectic and lengthy process and you want to have multiple options. I interviewed with 5 companies across 5 different countries in order to make sure I have options in case something goes wrong somewhere.

How was the process for applying for the visa?

Once you’ve found a job, the next logical step is to apply for a visa. This varies a lot country-to-country. In the EU (except UK and Ireland), countries have their own work visas as well as something known as a Bluecard. This is an EU-wide work permit with every member country having their own rules around what is permitted and timelines for getting residency. Some employers prefer to apply for a country-specific visa despite having the option to apply for a Bluecard. 🤷‍♀️

The UK and Ireland have their own separate visa systems that only allow you to work in that country.

This will usually be taken care of by your employer. Typically, they’ll employ an immigration law firm that’ll take care of drafting and submitting your application, and you’ll just need to provide them with the right documents. Timelines vary a lot depending on the country and type of visa and you’ll need a lot of patience to get through this process and keep your anxiety at bay.

Clarify who is paying for what

Check with your employer about whether they’re funding your visa as they cost a lot of $$$$$. You don’t want to be stuck paying that out of pocket.

Read your contract carefully

Also read your contract to make sure you understand if there are any special terms attached to it like having to pay back some of the visa or relocation costs if you leave before X months/years.

What was the actual move like?

Once you have your visa, you’re set to actually make the move. Wooooohoooo, the hard part is over! 🎉 Depending on when you get your visa, you’ll negotiate a start date with your employer.

You’ll need some savings to kickstart your life

Have some savings you can rely on before you get your first paycheck and/or relocation bonus. If you don’t have enough savings, make sure you clarify that with your employer and ask for your bonus upfront. There’s no shame in doing that. 🤗

Ask for a relocation bonus

Negotiate a relocation bonus, especially if you’re coming from a country with a significantly weaker currency, like India, or if you’re moving somewhere with a higher cost of living. Your savings will evaporate in front of your eyes before you know it. Usually, you can use it for all sorts of costs associated with moving: shipping your stuff from back home, furnishing your new place, etc.

Your employer will usually pay for your flight to the new country and some kind of temp accommodation till you find somewhere to live. This may or may not come out of your relocation budget.

Don’t relocate on the day you’re starting your new job

Fly in before you’re due to start so that you can beat jet lag and familiarize yourself with your new home. I flew to London a couple of days before I was due to start my job because I wanted to make sure I could get used to a completely new country (and city!) for a few days beforehand.

It’s a lot of work and it took me around 4-5 months to get it all sorted from arranging interviews to finally moving. Patience is key but at times it can feel impossible. I want to stress that if you’re willing to put in the work, it’s definitely doable.

I tried to distill most of widely-applicable advice I could think of in this post. If you’ve more questions, feel free to email or DM me on Twitter and I’ll try my best to answer by augmenting this post. 😊

Shout out to Miles and Rika for reviewing my drafts. 💜

The goal of these posts would be to share what I wish someone would have told me when I first started working. However, I have had to learn over the course of many years, so I hope this will help folks who are new to the industry and looking for some guidance.

Doing isn’t enough, you need to know how to market your work

This was my biggest and most hard-hitting realization so far in my career. I’ve always been the kind of person that sits in the corner, puts their head down and gets stuff done. It took me a while to realize that that’s not enough. Knowing how to present, talk about and market your work to the right people is a critical skill. Nobody teaches you how to do this, but trust me, you’ll thank yourself further down the line in your career if you invest some time in learning how to do this effectively. It’ll make a huge difference in your career trajectory. Learning to shout about your work is hard, I know, but you’ll often not reap the rewards if you leave it there.

Titles do matter, even if they’d like you to believe that they don’t

There’s a ton of discourse about this on tech Twitter and I often see men who are already senior in tech tell folks just entering the industry that they shouldn’t run after titles. Honestly, I think this is doing them a disservice. My main gripe with it is that it completely fails to consider the fact that titles decide what rooms you are allowed in. Titles/levels give you legitimacy, especially when you don’t enjoy the privilege of being assumed to be competent. It makes people, who otherwise wouldn’t, listen to you and take you seriously. They’re worth it for proving that you indeed know what you’re talking about.

The dangers of blindly climbing a career ladder

Titles/promotions/levels do matter and you shouldn’t let them take a back seat in your career. However, you need to establish a balance that works for you. If you spend all your time and energy chasing a promotion, then it starts to almost feel like having a second job. You want to avoid getting into that situation as it can prevent you from focussing on your actual job and drain you of all the excitement and energy. If you find yourself blindly trying to climb a career ladder while your actual job and growth has taken a backseat, it’s a red flag, and it might be time to start looking for something new. It’s sad that it has to be this way but it’s not by chance that the most common and fastest way to get promoted or a raise in this industry is to switch jobs. Protect your energy to invest in better and more fulfilling pursuits.

Sponsors are like cheat codes in the career game

There’s a broader point here about investing early on in building a solid network which I touched upon in my post from last year. Here, I want to specifically touch upon the importance of building and investing in longer term relationships. If you want to learn more about the concept of mentors vs sponsors and how they differ, Lara Hogan has an excellent blog post and talk that covers it better than I could here. Having folks who are trusted by people around you, are willing to stick their neck out for you and help you grow is like having access to cheat codes while playing the career game. What sponsorship has looked like for me in practice:

• Shouting about me and my work in rooms I am not in
• Trusting me with opportunities that stretch my comfort zone but having my back and making sure I don’t spread myself too thin
• Being a good sounding board for when stuff gets hard and making your sponsee feel heard and understood. Then actually doing something about it

I can’t emphasize the importance of having folks who are eager to open doors for you at every step of the way. It’s not just a confidence boost that there’s someone out there who wants to lend their privilege and help you build credibility, but it also goes a long way in helping you progress much faster.

Programming gets easier over time

Programming is hard. I know people who’ve been doing this for a while don’t like to talk about that but that doesn’t negate it. It took me a while to form mental models of how things work. Now I like to see programming as a way to keep augmenting those mental models, breaking down and forming new ones as we go. Something that has taken me by surprise time again is that you might not see everything coming together but after spending some time, things automatically start to click and those are the moments I live for. It makes sense and becomes a hell of a lot easier after a while, I promise.

I love building things with code. Programming brings me so much joy all these years later. And even though it’s not always been easy, I can’t really imagine doing anything else with my life. 3 years in – many more to go. To bigger and better things. ✨

These points could’ve been self contained blog posts, so if you enjoyed this post and want me to write a post on any or all of the points above, do drop me a line via Twitter/email, I’d love to hear from you. 💜

When working with a distributed system, the only guarantee we have is that things will fail sooner or later. In these circumstances, we want to “design for failure”.

Retries are a technique that helps us deal with transient errors, i.e., errors that are temporary and are likely to disappear soon. Retries help us achieve resiliency by allowing the system to send a request repeatedly until it gets a success response. This is useful if you have some component in the path of the request failing the first time around.

There are two ways to retry a failed request:

1. Manual retries: a failed request prompts the caller which in turn decides whether or not it wants to retry the request
2. Automatic retries: a failed request is automatically retried by the system without any interference from the caller

For example, imagine a service A needs to talk to service B in order to finish the work it is supposed to do. What happens if service B fails when the request gets to it? We have two options here:

• return an error to A and do nothing
• return an error to A but automatically retry the request again

If we go down the second route, we can ensure that the system itself can take care of a failed request due to partial failure (C or D failing) without external intervention. This is a super useful feature to have in a distributed system where the probability of something failing at any given time is non-trivial.

Retries can lead to retry storms which can bring down the entire system

Retries, if employed without careful thought can be pretty devastating for a system as they can lead to retry storms. Let’s break down what happens during a retry storm with a real-world example.

Consider a queue for a customer service center. The representative can take one phone call every three minutes and the queue of callers keeps flowing smoothly. However, if a few customers are taking longer to be serviced, the rep is much slower than we’d expect them to be when taking calls.

Customers, on the other hand, aren’t prepared to wait more than a few minutes and will continue to ring the center from different numbers while being on hold in case the previous call gets through.. This overwhelms the phone line as they can’t figure out which call connections should be kept alive and which should be discarded.

A very similar situation can occur within a distributed system as well. Imagine, we’ve multiple services A, C, D and E all trying to talk to service B at the same time. C, D and E are unaware that A is trying to talk to B and vice-versa. If the request from any of A, C, D or E fails, we’ve the following scenarios:

• Best case: the retry succeeds in the first or second try
• Worst case: the requests can be stuck and will keep getting retried repeatedly if, for example, B is undergoing garbage collection

The worst case scenario can spiral out of hand really quickly if B is being issued lots of requests. All of them will fail and all of them will consequently be retried. It turns into a self-perpetuating cycle where every failed retry in-turn spawns X (X = number of retries your system is configured to use) retries.

We usually don’t retry retry requests (meta, I know) as this can lead to exponential growth and bring down a system really quickly. So only the initial failed request is re-tried within the system. Retry requests aren’t and shouldn’t be issued concurrently, they should be sequential instead in order to avoid increasing unnecessary load on the system.

Let’s dig into what happens a little more to clarify it further. B is now being bombarded by different retry requests from multiple clients at the same time while continuing to receive normal traffic from various clients. The load grows linearly over time. This can quickly exhaust B as it will run out of compute and/or memory in order to cope with all the additional load.

Note: We use X=3 in the illustration, but the value of X will vary from system to system, it’s really hard to come up with a one-size-fits-all value for it. However, if you’re retrying requests in a loop, it’s a good idea to have an upper threshold for it which when reached should break out and terminate the request. This will avoid a scenario where we keep trying in an infinite loop.

This situation is known as a retry storm. If we have multiple of these across our system at the same time, then we can end up DDOSing our own system.

It’s not easy to detect a retry storm. Doing that will require every node to have a decent picture of what’s happening within the system.

Adding latency can work in our favour

As developers, we’re constantly taught that “fast is better”, so the idea of adding latency might seem a little weird at first. But it can be really helpful in distributed systems!

However, just adding the same amount of delay between two requests wouldn’t help. Let’s try to understand why. If, say, a hundred requests fail at the same time and we retry them all with a delay of 10ms, then we’re not solving the problem we had on our hands — we just shifted it 10ms into the future.

Exponential Backoff

Another option might be to delay each retry using an exponential delay: for simplicity, let’s use 2^n ms delay where n = retry count. Continuing from our previous example, we’ll see something like this:

• First retry: 2ms
• Second retry: 4ms
• Third retry: 8ms

It’s always a good idea to have an upper limit for backoff!

So and so forth. Again, this doesn’t solve our problem if multiple requests fail at the exact same time within our system (the chances of something like this happening in a real-world system are non-trivial). We’ll again issue the retry request at the same time and overwhelm the system. All we’ve done by this is added delay between successive re-tries without ensuring that they’re not synchronised across requests. However, it does give affected larger gaps of time to recover.

Jittering

To break this synchronisation, we can add randomness to the time interval by which we delay retries for a failed request. This is also known as “jittering” the request. In order to simplify understanding, let’s consider the following example:

• First retry: 2ms + 0.5 ms
• Second retry: 4ms + 0.8 ms
• Third retry: 8ms + 0.3 ms

By combining exponential backoff and jittering, we introduce enough randomness such that all requests are not retried at the same time. They can be more evenly distributed across time to avoid overwhelming the already exhausted node. This gives it the chance to complete a few in-flight requests without being bombarded by new requests simultaneously.

The illustrations used in this post are from a doodle I posted a couple of days ago:

New comic is all about retry storms! #devdoodles#sketchtogether pic.twitter.com/PcE9kJAbFk

— Shubheksha ✨ (@ScribblingOn) May 2, 2020

Lots of gratitude for Suhail and Ingrid for reviewing drafts of this post. 💜

This time last year I was on a break waiting for my visa to come through. I had just quit my very first job out of college and was incredibly burnt out and delusional. I was just trying to stay afloat and not lose whatever little hope I had left. I couldn’t have excepted how this year turned out in the wildest of my dreams. Some highlights:

• The first thing I did this year was move halfway across the world to the UK to start a new job at Monzo
• I got the first promotion of my career and grew more as an engineer this year than all my previous year combined 🎉
• I learnt so much about the kind of work I like to do, what challenges me and what makes me feel bored
• I also learnt that I really like working outside my comfort zone where I make things up as I go by till I get comfortable
• I discovered for the first time how much you can enjoy your work when you have colleagues you look upto and enjoy working with and also the importance of having a supportive manager

Last year, I was really struggling to detangle my identity from my career and realised what an unhealthy place it is to be in, so I made a deliberate decision to invest in activities outside of work this year and have some semblence of a work life balance. I think I did a decent job at that. Some personal highlights:

• I set myself a reading challenge of completing 50 books this year. I ended up reading 69 books ultimately which I’m honestly very proud of. I’ve been struggling to stick to reading for years and I’m glad I made it happen.
• I was able to focus on my mental health for the first time in my life as I wasn’t operating in crisis mode and putting out fires all the time. I was able to take a step back and actually question things and focus on what’s important to me.
• I learnt to eat and do things by myself. I don’t know why we make this into such a huge taboo and that makes me really sad. Learning to be okay with yourself is such an important stepping stone to pretty much anything in life, really. It was and is still hard at times because it’s something that can fill us with fear or shame or both but I’ve been learning to get through the hard parts and revel in the good parts slowly and steadily.
• I built a life in London from scratch — a completely clean state. I met so many amazing people who made me a better person in one way or another. Some chose to leave while others chose to stick by my side. Either way, I’m grateful.
• I started taking piano lessons — something I really wish I had done back in my childhood
• I invested more in crochet after picking it up last year on a whim from my mom. I made so many handmade things this year 🥰
• I was deliberately single all year after at least 5 years in 2019. It was hard at time because of all the stigma attached to it, but it was great. It allowed me to learn so much about myself and what I actually want from life and relationships. You get to know yourself in such a different way when you’re not in a relationship.
• I actually took vacation this year that wasn’t related to a conference in anyway for the first time since I started working.
• Most importantly, I think, this year I learnt what I actually want and how I wanna live my life without succumbing to the near-constant pressure we’re subjected to by being part of “a society”, especially as women. Pushing back against it is hard, but I don’t think we have much of a choice, really.

I’ve tried so hard to push myself to do things I’ve always wanted to do but put on a backburner for one reason on another. 2019, in that sense, has been a year of “firsts” for me. I’ve been thinking a lot about this during the holidays and this is what I’ve concluded: I want to make every year a “year of firsts” for me till I run out of things I wanna try. I’ll keep you posted on how that turns out. 😂

2019 has been the best year of my life so far and I sincerely hope everyone gets to experience something like that one time or another. I’m still tired but extremely ready to take on whatever 2020 throws my way. ♥️ Here’s to unapologetically building the kind of life you want. ✌️

The importance of good leaders:

If there’s one thing I’ve learnt in my rather short career, it’s that good tech leads/managers are invaluable. It’s an incredibly hard skill set and there are far too many people who suck at it. If you’re lucky enough to find good ones, stick with them for a while and if possible, move around with them because they’re ridiculously hard to find. They make such a HUGE difference in how fast you progress and directly impact your life so much. Here’s what to look for in good leader: kindness, empathy, emotional intelligence. They should be good listeners, be your advocate where possible, help you advocate for yourself and help you do what’s best for your career even when it’s uncomfortable.

Avoid bad managers/jobs (if you can):

This is controversial point and there are lots of factors at play. If you have the privilege of doing this, I’d strongly suggest not waiting it out at a toxic job working for terrible managers especially if you’re early in your career. It actively stunts your growth and hampers your career. In the long run, it does more harm than good in the sense that the effects of a toxic job outlive the job itself. It takes a very long time to get over it and there’s a high chance that you’ll carry it with you. Bad leaders make you question your worth all the time and you end up feeling absolutely helpless and stuck — it’s the classic recipe for burn out.

Building software is so much more than writing code

I realised this pretty quickly but it still bums me out how I had no idea till I started working. Most of the time, the hardest problem is figuring out what code to write and ensuring we’re solving the correct problem rather than figuring out how to write it. The latter is much easier than figuring out the former. Estimation and prioritisation is also super hard. Being a software engineer isn’t about sitting in a corner and writing code all day. It’s an inherently collaborative process that needs a wide variety of skills, especially people skills. It naturally follows from here that being an excellent engineer isn’t just about being very strong technically, even though it’s definitely super important, that’s just one part of the equation. Being an effective communicator, listener and team player are extremely important and valuable skills.

The world is small and the tech industry is smaller

Don’t be a dick and treat people badly. This is very simple and logical advice, I know, but I’m still amazed by how many people underestimate how far just being a decent human being will take them. If you do some shit, people will find out sooner or later. The tech industry is much smaller even though it can feel very big sometimes. Word travels faster than you can imagine. Don’t underestimate the power of backchannels. If you or your companies treat folks from underrepresented groups like shit, we will find out about it one way or another. The secret underground women in tech cabal meets every couple of days to discuss the shit y’all do.

Repeatedly push yourself out of your comfort zone

All of the good stuff happens when you push yourself to do things you’re not very comfortable with or are scared of doing. For a long time, I kept waiting for the “right moment” in order to seize the opportunity and do the thing but it never came. It took me a while to realise that sometimes you’ve to step way out of your comfort zone even if you don’t feel ready (in my case, I know I’ll never feel ready) and do it anyway. Wanna start reviewing code? Jump in and review a PR even if you’re not very familiar with the code being modified. Ask for help & read it until it makes sense. A little while ago, Sunil told me this and it has stuck with me ever since: write a shit ton of code, especially the bits you’re no good at, and sooner than later you’ll find yourself rubbing shoulders with experts as their peer. I like to compare it to learning to play the piano: if you keep practicing the bits you’re already comfortable with, you’ll never learn the ones you’re not good at.

Invest in building a network early on

I can’t stress on this enough, seriously. Nobody told me this and it sorta happened for me accidentally but it’s probably the single most important thing I’ve done for the sake of my career. No matter what people tell you, networking does matter. Treat people with respect and kindness. So many amazing and wonderful opportunities have come my way owing just to the network I built.

As an aside for underrepresented folks: invest in surrounding yourselves with people who are your tireless cheerleaders and will believe in your and your talent even when everything is shit. This industry can be very isolating and lonely on some days and outright shit show on other and it makes ALL THE DIFFERENCE. The best thing I’ve done for myself in the last two years is surrounding myself with amazing, smart, ambitious and highly technical women I deeply respect, admire and aspire to be. On days the tech industry is a shit show (and it’s not an uncommon occurrence at all), that’s all that keeps me going. Back in college, all my role models were white dudes and I constantly wondered if someone who looked like me had done amazing things in tech. Now my role models look more like me and are constantly pushing boundaries and paving the way for the rest of us to follow through.

Shout out to Jess Frazelle, Dizzy Smith & Joe Nash for reviewing a draft version of this post. 💖

The goal of this post is to provide some pointers about how teams can help new engineers deal with this and also folks early in their career can re-frame the way think about and deal with production outages.

Re-assurance from senior members of the team goes a long way

Fun fact: the first big change I worked on broke something in production 😂 I didn’t know what to do and how to deal with it so, me being me, I started panicking instantly. As soon as it was identified that it was my change that messed something up, my tech lead, Daniel Chatfield sent me a reassuring message. This very simple gesture on his part instantly calmed me down and meant a lot to me and is something that I’ll remember for a long time.

(Fincrime = Financial Crime, that’s the team I’m part of at Monzo)

Not all bugs are equal

At Monzo, we believe in deploying and shipping small, incremental changes regularly and our platform enables us to do exactly that. As a result, rolling back most changes in exceptionally easy — you just need to run a single command to revert the change.

However, not all bugs are equal. For example, in the particular case I outlined above, that wasn’t enough since some invalid data had been written to the database in the time before we detected the issue which needed to be fixed. Being new to the team, I was not super familiar with the service, but one of the more experienced members of the team with more context sat with me to explain how can go about fixing the invalid data calmly and I was able to write a fix confidently. 😊

It’s important to design systems and processes that make it difficult to ship large bugs to production. However, bugs will always slip through, and it’s invaluable to have tools that allow you to resolve issues quickly and limit the impact. In this case we used our iterator service (that allows us to run jobs across all of our users) to fix the invalid data quickly.

Talk about the stuff you fucked up

This can take the form of in-depth debriefs after an incident depending on how major it is, knowledge share sessions, detailed incident reports, etc. Detailed investigation reports can prove to be super useful here and they can be key in passing context which usually lives inside people’s heads to folks new to the team and can fill in gaps in documentation.

At Monzo, we recently started doing weekly lightning talks and one of the first ones was about “Stuff I broke in prod” by Matt Heath, one of our most senior engineers. It’s empowering to hear that literally everyone makes mistakes and it’s okay. Talking about things going wrong shouldn’t be something that fills you with shame or guilt. If that’s the case, it’s more likely to be an underlying cultural problem and not so much a technical one.

Communication is key

When you know you messed something up, the most important thing is to ask for help. At the end of the day, managing incidents is a team-sport and not a one-person-show. If there’s one thing you take away from this blog post, let it be this one. Don’t try to hide your mistakes in order to look smart. It’s not about you or your ego, it’s about fixing the mistake with the least amount of customer impact and you need all hands on deck for that as soon as possible. Communicating proactively is key here and it’ll make you a better and stronger team player in the long run.

Treat it as a learning opportunity

Even though this one feels like a no-brainer, I wanna give you some insight into how I like to think about it. When something breaks in production, I usually have mutliple takeaways from it:

• I learn to reason about our code better
• I learn how to debug the problem (or watch how other people reason about the problem which is also super fascinating for me because everyone’s brains work differently)
• People actually noticed when something breaks — this is a good thing! That means people use the stuff you build which is cool!
• It also gives the broader engineering team insight into how to make the processes more resilient to failures — fixing tooling, updating documentation, automating and detecting failure early, etc., so that the person who comes after you can avoid making the same mistake.

It’s never just one person’s fault

When something goes wrong, chances are even though you wrote the code, someone else reviewed it. And they didn’t catch the bug either. This isn’t an invitation to shift the blame on to them for whatever went wrong. It’s another way to look at the same thing: humans are flawed and will make mistakes no matter how perfect the tooling or automation. The goal isn’t to not make mistakes, it’s to learn from them and to avoid making the same mistake twice.

Engineering culture sets the tone for everything

How an individual reacts in such a situation is inherently tied to the team dynamics and the engineering culture of the organisation. If your workplace berates people for making mistakes and singles them out (believe me, this is more common than you think), then chances are very high that you will not feel comfortable talking to your colleagues or peers about your mistakes. This is an overarching point that is tangential to all of the other points I touched. Having a blameless culture where people feel safe to accept that they messed something underpins and is a hallmark of a good engineering team.

I would like to specifically highlight here that it can be really hard to folks in their first or second job to know what a good engineering culture looks like because they don’t have have the luxury to fallback on experience. It can leave folks completely disillusioned if their first job is a cultural dumpster fire and yet this isn’t uncommon at all. I’ve written more about toxic jobs and their symptoms in a different post. It is not your fault if you end up in a company with a terrible culture. 💖

To wrap up, I’ll try and summarise a conversation I had with a colleague about “humanising” bugs: instead of treating outages as an invitation to throw an individual under the bus, we should instead try and treat them as opportunity to make our systems better, more robust and more resilient to failure. 😊

Shoutout to Danielle Tomlinson, Daniel Chatfield and Suhail Patel for reviewing an early draft and providing useful feedback and everyone who chimed in with their suggestions in the Twitter thread on this topic! 😊

The short version is: I love living in London, I love my job and I feel more stable and happy than I’ve ever been and ¹⁰⁄₁₀ would do it again.

It’d be useful to have some context about my situation and mindset before diving in. I started actively looking around ~June-July last year. I was determined about moving out of India, so I didn’t even apply for anything in India at all (bold move on my part, I know). By that time I had travelled quite a bit and I believe that really accelerated my desire to move. I’ve always wanted to move out, but my initial plan was to stick around in India for around 4-5 years. I didn’t apply for anything in the US because the visa process is a complete shit show for Indians and I knew it’d be a waste of everyone’s time. It’s important to understand this because I was knew I had a decent idea of the kind of position I was looking for and that I wanted to relocate to Europe (Canada was my second option). I wasn’t doing it as an experiment, I was very determined and knew exactly what I wanted and so I went all in.

Why London?

At the end of my search, I ended up with 5 offers in 5 different countries (4 in Europe and 1 in Canada). I ended up choosing London because I really liked the company, my interview experience was positive and overall it seemed like a really good option for me. This was of course the primary reason but after thinking a bit more, there were lots of other secondary reasons:

• London is more culturally and ethnically diverse than any other European city
• English is the main language, one I speak already
• I already had a few friends here so it wouldn’t be walking into a voluntary social exile (it’s hard to make new friends as an adult)
• The easy availability of veggie (and Indian) food
• Having lived in small cities all my life, the idea of living in a big city fascinated me!

All of these things definitely helped me make a sound decision!

It’s a lengthy process

Applying for a visa sucks universally. To give you an idea of timelines, I got my offer in late October and I accepted in early November. After waiting to get a few approvals, we started gathering the paperwork to apply for my visa. We were able to finally apply for my visa around mid-December after getting everything in order and I got my visa right after the holidays. It’s a lengthy and tiring process, no doubt. It really helped that my employer put me in touch with an immigration agency who knew what they were doing and took care of everything without me having to worry too much.

Culture shock (but not really)

A lot of people ask me if it was a culture shock to move to the “west” from India. I think the answer to that depends on what you were expecting from the move. I haven’t really had any culture shocks so far that were jarring and something that I didn’t expect. (Except for using toilet paper — how have y’all not moved to bidets/hand showers is beyond me.)

Some things I really appreciate are people being polite, people being mindful of others’ personal space, and people actually have a vibrant and exciting life outside of their jobs. I think I had a fair idea of what to expect when moving which made adjusting easier. In fact, some of the very reasons I wanted to move would probably classify as a “culture shock” for a lot of people. 🤷🏻‍♀️

Life outside of work

Ease of getting around

This one was probably a huge motivation for me. I can’t begin to put in words how much I appreciate having footpaths and pedestrian crossings everywhere. I know a lot of people take that kinda stuff for granted, but having not grown up with them, I’ve come to appreciate them a lot more. I enjoy walking a lot and I absolutely love how walkable London is. Slightly related to that is public transport. Again, having never lived in a city with a functional transport system, I constantly felt trapped in the house because my only option was to take an Uber — something I didn’t feel comfortable doing at night. I don’t have to make plans thinking “oh, I need to get back by 11pm in an Uber and that’s not particularly safe or fun”. I don’t feel completely terrified of walking at after 9pm and depending on the neighbourhood, it’s usually fine too. This something I couldn’t even imagine doing ever in India.

So many things to do!

When I was living in Hyderabad and was absolutely miserable, a lot of it could be attributed to having absolutely nothing to do outside of work unless you wanted to go and spend all your money on booze every weekend or you were willing to travel 1-1.5hr each way and deal with traffic jams. I didn’t like doing that. I face the exact opposite problem here: I love how much London has to offer to you outside of work — more than you’ll ever have time and money to do. It’s a good problem for me to have right now. I can do so many things I’ve always wanted to do but never had the option before: watch classical music performances, ballet, go to spoken word nights, stand up comedy shows, music concerts and what not.

Ample availability of public spaces

Something else in the same vain: the easy availability of well-maintained and well-functional public spaces. I love parks, waterfronts, ferries. They make me super happy. I love that I’ve the option to just have lunch in a park if I feel like it or go read there for fun on a sunny day.

All of the above things might seem trivial to some people, but they’ve made a huge difference in the quality of my life. 😊

The day-to-day life

This was probably one of the main reasons I wanted to move out of India ASAP. I wanted to get an idea of what self-sufficiency looks like in practice: being able to cook a decent meal for yourself and clean your own mess. When I started talking about moving out, the most common retort was: life is too difficult without a maid and/or cleaner and that I won’t be able to adjust. I’ve strong opinions about it but that’s a controversial topic and a different blogpost altogether, so we’ll leave it out of this one. But guess what, it’s not fun to do those things every time, but I’m glad that I’ve had to pick up essential life skills that’ll serve me well in the long run. I actually enjoy cooking for myself more than I thought I would and I’m getting better at it with time. 💪🏼 I’d pick cooking for myself and cleaning any day if I get to live the life I have right now.

Everything isn’t rosy

Of course I miss home from time to time. I miss my family being ~2hr away instead of 15hr and the food. I miss Indian food so much even tho London is probably the best place for Indian food outside of India. 😂

The biggest adjustment for me has come in the form of getting used to spending money and paying so much more for literally the same goods and services. This basically meant being forced to define my relationship with money and the value of work (which is a good thing!). It’s been a journey and I’m still learning to get comfortable.

It’s not all rosy, but I like it. I think it comes down to what you value in your life. I value access to the things I have now a lot. My job also plays a big role in this: my work is super important to me and I genuinely love my job. I’ve been learning so much ever since I joined and I get to work with and learn from some incredibly smart, kind and nice people. ✨

To wrap up, being able to move to another country is a huge privilege — something a lot of people don’t have the option to do at all. I’m lucky that I was able to find an employer than was willing to sponsor me, that I’m regarded as a skilled worker, I had a relevant degree which made getting a visa way easier, I didn’t have dependents I had to take care of and had the freedom to move my entire life halfway across the world. A recurring theme in my daily gratitude journal since I moved has been “I am very grateful for being able to live the kind of life I’ve always wanted” and that would be a great way to sum up London so far for me. ✨

It’s not rocket science, you can do it too

Seriously, it’s not. For a long time, I constantly thought of people who review code as people who have all the context to review the piece of code they were looking at, so of course, they’d find it easy and effortless to do that. This is rarely true, even for smaller companies, with systems that have lots of moving parts. Nobody knows everything about every part of the system. It’s good to remember that. Reviewing code is a skill that can be learnt just like writing code. The more you do it, the better you get at it. Reviewing code also helps you write better code because you know what kind of things you need to watch out for. Don’t let the fear of not knowing everything about the code you’re reviewing hold you back.

Be prepared to make mistakes

Mistakes will be made. Something or the other will always slip through the cracks because humans write code and humans review code and we’re all inherently prone to all sorts of errors. That is okay. Just because you can’t do it perfectly doesn’t mean you shouldn’t do it at all. It’s also completely okay to ask someone to take a second look if you’re not sure about something. Having a second pair of eyes never hurts.

Tangentially related to this point is having a blameless culture where people don’t hunt you down if you mess something up in production. The primary focus is to diagnose and fix the problem rather than assigning blame (who wrote/reviewed the faulty code). There is room to make mistakes and what’s important is learning from them. I’m lucky to work somewhere that has this culture, so it doesn’t feel as daunting. 😊

Remember to be kind and empathetic

Code reviews are very ripe for misunderstanding and lack of empathy on either side. At the heart of code reviews is collaboration. It is as important to remind yourself as a reviewer that you’re reviewing someone’s code and not passing judgments on them as a person and it is equally important to remember that whatever your reviewer tells you is not meant as a personal attack. I feel a lot of this is dictated by the engineering culture of your team and the company overall. Being mindful of this before posting comments can greatly reduce any possibility of friction.

A very simple way to do this:

Asking questions about why someone did something one way is so much more effective than suggesting another

— Adam Brocklehurst 🐧 (@AdamBrockle) March 6, 2019

The junior-senior dynamic and how to deal with it

Another point I want to touch on is the junior-senior dynamic that comes into play when reviewing code. As someone early in their career, it can be very daunting to review code for someone who has a lot more experience and/or context than you. However, this shouldn’t discourage you from reviewing their code. Some ways you can do this:

• Shadowing reviews on existing pull requests: this is a great way to learn to spot what kind of things folks watch out for when reviewing code and also how they communicate it and how it is received by the author (how you communicate feedback on a change is equally important, if not more than what you communicate)
• Pairing with another engineer or the author when reviewing something more complex: this let’s you ask questions in real time and gain more context about what’s going on
  

Ask for clarification when you need it

Asking for more context is okay. This can be scary at first when you’re brand new to a team and/or domain and everyone around you seems to know so much more than you. If something doesn’t make sense even after looking the relevant bits and doing your research, it’s okay to ask for more clarification.

Let me give you an example of this: Last week, I was asked to review a small PR for a service I have never touched or used before and my first instinct was to ask someone on my team with more context to do it instead. However, I took it as a learning opportunity and tried to figure out what was happening in the proposed change. After looking at it, even though the overall pull request made sense, some things didn’t, so I gathered the courage to state that I’m new to this service and ask the author for more clarification. And guess what - they were more than happy to provide that! Once I had that, I was able to approve their change and at the same time, walked away with several things:

• I’ll be more confident about asking for clarification after this positive experience
• I’ve a better understanding of what that service does and can review related changes more easily
• We identified how we can improve the documentation for the functions we were looking at so that other folks can understand it more easily

It was a win-win in every way! 🎉

Once I started looking at code reviews as just another skill that I can learn and get better at over time, I felt much more confident about doing them. It’s less scary now and I look forward to doing more of them. 💪🏼

Credit

Have you ever felt like you don’t know what you’re doing? Have you ever felt like everyone is going to find out any time now that you’re a complete fraud? Have you ever thought you somehow managed to convince everyone you’re smart and know your stuff? Welcome to the club, my friend! You suffer from imposter syndrome, and you’re one of us! Yay! 🎉

I don’t think it’s hyperbole to state that the very best of us suffer from imposter syndrome. At this point, I’ve come to believe that it’s part of the human condition, mainly if you work in any creative field. If someone claims that they don’t suffer from imposter syndrome, they probably have an ego too gigantic to admit it. And you know what? Suffering from it is okay.

Credit

I’ve suffered from crippling imposter syndrome over the years. So I wanted to talk about some of the ways I’ve tried to tame it and how it has helped me become a better person and engineer.

This was one of the issues of my newsletter “Life Reliability Engineering.” But I thought it deserved space of its own as a blog post.

Writing your accomplishments down

I can’t even begin to stress how much this has helped me. Having a handy list of tangible professional achievements that you can refer to when you feel like a fraud works wonders. Write down your accomplishments — small or big. Write down stuff you’re proud of, stuff you didn’t think you could do but did anyway and things like that. Spoke at a conference? Got a scholarship? Started a new job? Shipped something at work? Write it all down!

Go back to it when you’re feeling like crap. Remind yourself that if you were a total fraud, you wouldn’t have been able to accomplish all this in the first place. Remembering that it is in your head isn’t enough, because your brain won’t recall it fast enough when it’s in imposter mode. So writing it down is important.

Accepting and reveling in the fact that you’re not the smartest person in the world

(Side note: how, if at all, you can measure this and what does it even mean????) I guess everyone has their own timeline for this. Finally accepting it is, frankly, so relieving. And it has positive side effects! I’m nearly constantly striving to work with folks who’re smarter than I am, and accepting this has enabled me to learn better (and more!) from them!

Don’t be that person who is constantly trying to prove they’re the smartest person in the room by putting other people down. Nobody likes that person. Being humble and ready to learn from other folks helps you grow better and faster. It also makes you a good co-worker.

Surround yourself with people who recognize your work and worth

This one is a hit or miss in the sense that you don’t really realize the importance of doing this till you actually do it. Spoiler alert: it’s magical.

Being surrounded by folks (friends/co-workers/community) who recognize and/or reward your work and reassure you of everything you bring to the table from time-to-time is one of the best things you can do for yourself. It’s truly hard to overstate how big a difference it makes. Invest time in this. It’s very hard to find these folks, but once you do, make sure you stick with them.

Everyone excels at different things and it’s NOT a zero-sum game

This point is sort of tangentially related to the second one but still deserves its own space. There is no superhuman person out there who is good at everything. That’s simply not possible. Different people have different areas of expertise and that’s a very good thing! It gives all of us something to learn from one another and balances things out.

Maybe your strength is someone’s weakness and vice-versa. Someone else being good at something that’s not your area of expertise doesn’t take away anything from you. If anything, it gives you a chance to learn something from them and improve. Recognize and celebrate that.

NOT comparing yourself with people who have a lot more experience than you

Comparing yourself to someone who has been around for a long time is kind of like comparing apples to oranges. It’s a recipe for disaster, so don’t do it. It’s a bad idea.

If you’re someone who is just starting out and you compare yourself to someone who has had a decade-long career, then, of course, they’re gonna be doing way better than you! Simply because they’ve been around much longer than you have. Of course, they’ve put in a lot more effort than you, built a solid network, spoken at major conferences and what not. (I’m not claiming that everyone does this, but simply picking one of the most common parameters people tend to size themselves up against).

This is doomed even before you start doing it. Whenever you start doing this, take a step back, pause and think: is this a valid comparison? Do I need to be comparing myself to anyone at all? Am I happy with where I am? Because that’s what matters in the end.

Realize it’s a waste of time and energy

I’m a firm believer in healthy introspection, but more often than not, this isn’t healthy. Obsessing over this doesn’t really get you anything when all is said and done. Mostly there are no concrete takeaways. You end up being miserable and down, feeling like a fraud and no matter how much you do, it’ll never be enough.

You don’t need or deserve to feel that way. Following this chain of thought is probably not a good use of your time because it doesn’t do anything to actually help you fix anything. Your time will be better utilized in doing or thinking about something more meaningful and actionable. Acknowledging that can at least help you not descend down the “thinking rabbit-hole”.

Final thoughts

I can go on writing about this topic endlessly because I’ve battled with it a lot over the years and I know it’s not a battle that’s going to end any time soon (or possibly ever!).

So, for now, I’ve been trying to devise ways to deal with it so that I don’t waste time doing this. Instead I put my energies into being the best software engineer I can be. 💪🏼 It has been working out well so far.

This has been a long post. I sincerely hope this will help you deal with random bouts of imposter syndrome as well. ✨

You go to work everyday and you want to feel excited but something always feel amiss. Some day you’re full of dread showing up to work, some days you wanna skip it altogether. Rather than being rewarding, it ends up taking a huge mental toll on you everyday. You are scared of what it’s doing to your mental and eventually, even physical health. You try sticking around hoping it’ll get better eventually, but it doesn’t. You wonder if something is wrong with you? Did you manage to screw it up somehow? It ends up being worse if you’ve to deal with this stuff early in your career because you don’t have the luxury of relying on experience to inform your assessment of the situation you’re trapped in.

Credit

This story might sound very familiar to folks belonging to underrepresented minorities in the tech industry.

Let’s dig in and start from the basics.

What is a toxic job?

A toxic job is one that leaves you burnt out and drained every other day. It can cause you trauma affecting your physical or mental health. Constantly feeling tired or dreading going to work are all signs that point to it. Alarm bells should start ringing when it becomes a “normal” occurrence or something you find yourself getting used to instead of something that happens once in a while.

What leads to a job becoming toxic?

It’s very difficult to pin-point a single reason and usually it ends up being a combination of multiple reasons. I can list some of them here:

• Unsupportive managers — a manager that doesn’t support you can be a huge source of discontentment at work. Be it supporting your career aspirations in the long-term or day-to-day development, clarity on your immediate and long-term goals, having a supportive manager is key.
• Toxic working relationships — a set of coworkers that are negative, completely demotivated and spiteful can also significantly decrease your day-to-day morale at work. Having a healthy amount of mutual accountability, respect and support is paramount for a good working environment. Being deprived of that, being condescending to junior folks or indulging in petty politics in the work place can also lead to jobs being toxic.
• Chaotic and disruptive workplace practices— everything around you being in a constant state of disarray is also not very helpful. Worse if nothing is being done to alleviate them and it’s the status quo. Nothing kills your spirit more than feeling that what you do doesn’t matter and there is nothing you can do to make it matter. This can be applied to a large chunk of practices: code reviews, sprint planning, releases, etc.
• Uninteresting/non-challenging work — this can depend greatly from person to person, but for some folks if you’re not doing work that’s challenging and you end up making no use of your brain for a continued period of time, it can be the source of a lot of internal commotion. It can easily lead to a depleted sense of self-worth and cause you to wonder “what the hell am I doing with my life” every other night.

Side note: A lot of people think that the main cause that leads to burn out is often too much work when you’ve no time to do anything but work, work, work 24X7. However, the flip-side of it is that having no opportunities for growth, no recognition of the kind of career trajectory you want to pursue and feeling stagnated because you’re not learning anything new and being unable to improve the situation around you can also severely impact someone’s sense of self-worth and in turn mental health paving the way for burn out. This is another blog post altogether.

This is also by research into how humans interact socially.

Credit

Of course there can be multiple other factors, but I’ve tried to list some of the most common ones. The list is way worse and much longer for folks from underrepresented folks who have to deal with a ton of crap just to be able to their jobs.

What can you do if you’re stuck in a toxic job?

• Get. Out. Run. No, seriously, especially if it’s affecting your health (yes, even mental health). If you’ve been stuck in a terrible situation like months, it might be time to accept that it’s not you and it’s not going to get better. You can’t fix everything and that’s okay. It’s not your fault, you haven’t failed — it’s your manager/employer who has failed you by not being able to provide a safe and productive working environment. If you can change teams within the company, great, evaluate your options. If not, it might be time to start looking for something new.
  (I fully realize that being able to choose this option comes with a lot of privilege and most folks wait to already have something concrete before quitting, if it comes to that)
• Don’t let it become your “new normal” no matter what. Don’t get complacent. This is probably the worst side-effect of a toxic job: it’s so easy to normalize your situation by reasoning around it — “this is how it’s supposed to be” — don’t do that to yourself. It’s not normal. Don’t let it fool yourself and stagnate you for years.
• Don’t let it consume you. Surround yourself with people who’ve healthy jobs and are somewhat happy with them. Tell them to remind you that what you’re going through isn’t normal in any way and that you deserve better.
• If you’re early in your career and don’t the luxury of falling back on experience, invest time in finding mentors who’ve spent more time here and know better.
• Talk to your colleagues if possible. This will be highly dependent on the kind of relationship you’ve with your colleagues, but if possible, you should talk to them about the issues you’re facing at work and ask if they’ve experienced/noticed them as well. This can be a great way to ascertain that there is a much larger problem at hand and it isn’t *you*. This is also a great way to find out what has been done to deal with such problems in the past. If these same problems have existed for years with nothing being done about them, you know it’s a lost cause.
• If you’re suffering, please seek help. There is absolutely no shame in asking for help when you’re going through a terrible time. Talk to your friends, family and you’ve access to therapy, please do seek professional help.
• Lastly, don’t lose hope. Despite all the crap, there are good people in this industry who’re working hard every day to create a good, inclusive working environment. They’re out there and it gets better, I promise. ❤️

In closing, please take excellent care of your mind and body because:

P.S.- These views are my own and if you don’t agree with them, it’s completely okay.

This paper argues that a common design paradigm for systems is fundamentally flawed, resulting in unstable, unpredictable behavior as the complexity of the system grows.

The “common design paradigm” refers to the practice of predicting the environment the system will operate in and its failure modes. The paper states that a system will deal with conditions that weren’t predicted as it becomes more complex, hence it should be designed to cope with failure gracefully. The paper explores these ideas with the help of “distributed data structures (DDD), a scalable, cluster-based storage server”.

By their very nature, large systems operate through the complex interaction of many components. This interaction leads to a pervasive coupling of the elements of the system; this coupling may be strong (e.g., packets sent between adjacent routers in a network) or subtle (e.g., synchronization of routing advertisements across a wide area network).

A common characteristic that such large systems exhibit is something known as the Butterfly Effect — a small unexpected disturbance in the system resulting from the intricate interaction of various components can result in a widespread change.

A common goal for system design is robustness: the ability of a system to operate correctly in various conditions and fail gracefully in an unexpected situation. The paper argues against the common pattern of trying to predict a certain set of operation conditions for the system and architecting it to work well in only those conditions.

It is also effectively impossible to predict all of the perturbations that a system will experience as a result of changes in environmental conditions, such as hardware failures, load bursts, or the introduction of misbehaving software. Given this, we believe that any system that attempts to gain robustness solely through precognition is prone to fragility.

DDS: A Case Study

The hypothesis stated above is explored using a scalable, cluster-based storage system Distributed Data Structures (DDD) — “a high-capacity, high-throughput virtual hash table that is partitioned and replicated across many individual storage nodes called bricks.”

This system was built using a predictive design philosophy as the one described above.

Based on extensive experience with such systems, we attempted to reason about the behavior of the software components, algorithms, protocols, and hardware elements of the system, as well as the workloads it would receive.

When the system operated within the scope of the assumptions made by the designers, it worked fine. They were able to scale it & improve performance. However, in the case when one or more of the assumptions about the operating conditions were violated, the system behaved in unexpected ways resulting in data loss or inconsistencies.

Next, we talk about several such anomalies.

1. Garbage Collection Thrashing and Bounded Synchrony

The system designers used timeouts to detect failure of components in the system. If a particular component didn’t respond within the specified time, it was considered dead. They assumed bounded synchrony in the system.

The DDS was implemented in Java, and therefore made use of garbage collection. The garbage collector in our JVM was a mark-and-sweep collector; as a result, as more active objects were resident in the JVM heap, the duration that the garbage collector would run in order to reclaim a fixed amount of memory would increase.

When the system was at saturation, even slight variations in load on the bricks would increase the time taken by the garbage collector in turn dropping the throughput of the brick. This is called GC thrashing. The affected bricks would lag behind their counterparts leading to a further degradation in performance of the system.

Hence, garbage collection violated the assumption of bounded synchrony when it was nearing or beyond the saturation point.

2. Slow Leaks and Co-related Failure

Another assumption made while designing the system was that the failures are independent. DDS used replication to make the system fault-tolerant. The probability of multiple replicas failing simultaneously was very small.

However, this assumption was violated when they encountered a race condition in their code that caused a memory leak without affecting correctness.

Whenever we launched our system, we would tend to launch all bricks at the same time. Given roughly balanced load across the system, all bricks therefore would run out of heap space at nearly the same time, several days after they were launched. We also speculated that our automatic failover mechanisms exacerbated this situation by increasing the load on a replica after a peer had failed, increase the rate at which the replica leaked memory.

Since all the replicas were subjected to a uniform load without taking performance degradation and other issues into consideration.This created a coupling between the replicas and…

…when combined with a slow memory leak, lead to the violation of our assumption of independent failures, which in turn caused our system to experience unavailability and partial data loss

3. Unchecked Dependencies and Fail-stop

Based on the assumption that if a component timed out, it has failed, the designers also assumed “fail-stop” failures, i.e., a component that has failed will not resume functioning after a while. The bricks in the system performed all long-latency work (disk I/O) in an asynchronous way. However, they failed to notice that some parts of their code made use of blocking function calls. This caused the main event-handling thread to be randomly borrowed leading to bricks seizing inexplicably for a couple of minutes and resuming post.

While this error was due to our own failure to verify the behavior of code we were using, it serves to demonstrate that the low-level interaction between independently built components can have profound implications on the overall behavior of the system. A very subtle change in behavior resulted in the violation of our fail-stop assumption across the entire cluster, which eventually lead to the corruption of data in our system.

Towards Robust Systems

..small changes to a complex, coupled system can result in large, unexpected changes in behavior, possibly taking the system outside of its designers’ expected operating regime.

A few solutions which can help us make more robust systems:

Systematic Over-provisioning

When approaching the saturation point, systems tend to become fragile to accommodate unexpected behavior. One way to combat this is to deliberately over-provision the system.

However, this has its own set of issues: it leads to the under-utilization of resources. It also requires predicting the expected operating environment and hence the saturation point of the system. This can’t be done in an accurate manner in most cases.

Use Admission Control

Another technique is to start rejecting load once the system starts approaching the saturation point. However, this requires predicting the saturation point — something that’s not possible always, especially with large systems which have a lot of contributing variables.

Rejecting requests also consumes some resources from the system. Services designed with admission control in mind usually have two operating modes: normal where the requests are processed and an extremely lightweight mode where they’re rejected.

Build Introspection into the system

an introspective system is one in which the ability to monitor the system is designed in from the beginning.

A system which can be monitored and designers and operators can derive meaningful measurements about its operation is much more robust than one a black-box system. It’s easier to adapt such a system to change in its environment, manage and maintain it.

Introduce adaptivity by closing the control loop

An example of a control loop is a human designers and operators adapting the design in response to a change in its operating environment indicated through various measurements. However, the timeline for such a control loop isn’t very predictable. The authors argue that systems should be built with internal control loops.

These systems incorporate the results of introspection, and attempt to adapt control variables dynamically to keep the system operating in a stable or well-performing regime.

All such systems have the property that the component performing the adaptation is able to hypothesize somewhat precisely about the effects of the adaptation; without this ability, the system would be “operating in the dark”, and likely would become unpredictable. A new, interesting approach to hypothesizing about the effects of adaptation is to use statistical machine learning; given this, a system can experiment with changes in order to build up a model of their effects.

Plan for failure

Complex systems must expect failure and plan for it accordingly.

A couple of techniques to do this:

1. decoupling of components to contain failures locally
2. minimize damage by using robust abstractions such as transactions
3. minimize amount of time in failure state (using checkpointing to recover rapidly)

In this paper, the authors argue that designing systems by assuming the constraints and nature of its operation, failures and behavior often leads to fragile and unpredictable systems. We need a radically different approach to build systems that are more robust in the face of failure.

This different design paradigm is one in which systems are given the best possible chance of stable behavior (through techniques such as over-provisioning, admission control, and introspection), as well as the ability to adapt to unexpected situations (by treating introspection as feedback to a closed control loop). Ultimately, systems must be designed to handle failures gracefully, as complexity seems to lead to an inevitable unpredictability.

Credit: logz.io

If you’ve worked with Docker even for a little while, I bet you’ve come across tags. They often look like “my_image_name:1” where the part after the colon is known as a tag. The tag is not always specified when tagging images, but we’ll get to the bottom of that later.

Ever since I started using Docker, I’ve been very confused about tags. The documentation doesn’t explain them very well, and there really aren’t any thorough explanations on the topic. That’s why I decided to write this post.

What are Docker tags?

So, what exactly are Docker tags? In simple words, Docker tags convey useful information about a specific image version/variant. They are aliases to the ID of your image which often look like this: f1477ec11d12. It’s just a way of referring to your image. A good analogy is how Git tags refer to a particular commit in your history.

The two most common cases where tags come into play are:

1. When building an image, we use the following command:

docker build -t username/image_name:tag_name .

Let’s try to unpack what this command does for a bit. We tell the Docker daemon to fetch the Docker file present in the current directory (that’s what the . at the end does). Next, we tell the Docker daemon to build the image and give it the specified tag. If you run docker images, you should see an image whose repository is username/image_name and tag is tag_name.

username/image_name is not a mandatory format for specifying the name of the image. It’s just a useful convention to avoid tagging your image again when you need to push it to a registry.

Your image can be named anything you want. For the public Docker registry, you’re restricted to a two level hierarchy while naming images. For example, your image cannot have the name a/b/c:1. This restriction usually doesn’t exist in private registries. As stated before, it’s not mandatory to specify a tag_name. We’ll see what happens in that case soon.

2. Explicitly tagging an image through the tag command.

docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]

This command just creates an alias (a reference) by the name of the TARGET_IMAGE that refers to the SOURCE_IMAGE. That’s all it does. It’s like assigning an existing image another name to refer to it. Notice how the tag is specified as optional here as well, by the [:TAG] .

What happens when you don’t specify a tag?

Alright, now let’s uncover what happens when you don’t specify a tag while tagging an image. This is where the latest tag comes into the picture. Whenever an image is tagged without an explicit tag, it’s given the latest tag by default. It’s an unfortunate naming choice that causes a lot of confusion. But I like to think of it as the default tag that’s given to images when you don’t specify one.

A lot of confusion around latest is caused due to the expectation that it’s the latest version of the image, especially in Dockerfiles. Let’s consider the various scenarios with an example:

Scenario 1:

Suppose the following statement is present in our Dockerfile:

FROM debian

Since we didn’t specify any tag, Docker will add the latest tag and try to pull the image debian:latest .

Scenario 2:

FROM debian:9.3

Since the tag is explicitly mentioned here, Docker will pull the Debian image tagged 9.3

Another thing to keep in mind is that there is no rule which states that an image needs to have just one tag. An image can have multiple tags and they’re usually used to specify major and minor versions. For example, consider this:

Docker Hub page for Debian

At the time of writing this post, the latest tag for the Debian image points to the 9.3 release and the 9 release. This will most likely change in the future whenever the major or minor version is bumped for the image.

Please note that tags being used for semantic versioning is a convention that’s followed, but tags weren’t designed just for that purpose.

In conclusion, latest is not a special tag

The main takeaway from what we’ve covered so far is that latest is just like any other tag. The onus is on the developer to tag the images properly such that latest always points to the latest stable release of the image.

Hence, we don’t explicitly specify a tag in our Dockerfiles when pulling images, since we might end up with a completely different version of the base image than what we had used before. There is no guarantees about whether it’ll be a major bump or minor bump. Even an old release can be tagged as latest.

P.S. If you found any misconceptions/errors in the post, please feel free to tweet to me @ScribbingOn.

Thanks to Jérôme Petazzoni for helping me make sense of some of this.

Credit

The focus of this paper is on how to make reliable software systems and what techniques can help us achieve that. Reliability here implies that a system works as expected under a given set of conditions.

The unfortunate fact is that the standard approach to building
systems, involving extensive debugging, has not proved successful in producing reliable software, and there is no reason to suppose it ever will. Although improvements in debugging techniques may lead to the detection of more errors, this does not imply that all errors will be found. There certainly is no guarantee of this implicit in debugging: as Dijkstra said, “Program testing can be used to show the presence of bugs, but never to show their absence.”

To be confident that our system works correctly, we need testing that meets the following conditions:

1. We can generate a minimal set of relevant test cases
2. All test cases in the set can be generated

The solutions to these problems do not lie in the domain of debugging, which has no control over the sources of the problems. Instead, since it is the system design which determines how many test cases there are and how easily they can be identified, the problems can be solved most effectively during the design process: The need for exhaustive testing must influence the design.

The paper further argues that reliability is a major issue with complex systems. It goes on to define complex systems as follows:

1. The system has many states and it’s difficult to organize program logic to handle all of them correctly
2. It requires several people working together in a coordinated manner

Criteria for a Good Design:

To tame the design of a complex system, we need to use modularization and divide the program into several modules (sub programs, later on referred to as partitions in the paper to avoid overloading the term “modules”) which can be compiled separately but are connected to other modules.

The connections are defined by Parnas as follows:

The connections between modules are the assumptions which the modules make about each other.

Although the idea of modularity sounds like a great tool for building large complex software systems, it can introduce additional complexity if not done right.

The success of modularity depends directly on how well modules are chosen.

Some common issues are:

1. A module does too many things
2. A common function is distributed among many different modules
3. A module behaves unexpectedly with common data

The next question that arises: What is good modularity?

We use two techniques to answer that: levels of abstraction to tackle the inherent complexity of the system and structured programming to represent the design in software.

Levels of abstraction:

Levels of abstraction…provide a conceptual framework for achieving a clear and logical design for a system. The entire system is conceived as a hierarchy of levels, the lowest levels being those closest to the machine.

A group of related functions make up a level of abstraction. Each level can have the following two types of functions:

1. External: These functions can be called by functions in other levels
2. Internal: These functions do a common task within the level and cannot be called by other functions in a different level

Levels of abstraction are governed by the following two rules:

1. Each level has exclusive control over some kind of resource
2. Lower levels aren’t aware of higher levels and can’t reference them in any way. However, higher levels can ask lower levels to perform an action or for info.

Structured Programming:

A structured program defines the way control passes among various partitions in a system. It is defined by the following two rules:

1. The program is developed in a top down fashion and divided into levels (the notion of levels here is different from that of levels of abstraction because the first rule isn’t satisfied)
2. Only the following control structures can be used: concatenation, selection
   of the next statement based on the testing of a condition,
   and iteration. Jumping using goto isn’t permitted.

Back to the question that was posed earlier: how do we define good modularity?

In a modular system that is also reliable, the connections between partitions are limited as follows:

1. They need to follow the rules imposed by levels of abstraction and the structured programming
2. Passing of data between partitions should be done using explicit arguments passed to external functions of another partition
3. Partitions should be logically independent — the functions within a partition should support its own abstraction only

The next question that arises after we’ve figured out how to defined good modularity is — how do we achieve it in our design?

The traditional technique for modularization is to analyze the execution-time flow of the system and organize the system structure around each major sequential task. This technique leads to a structure which has very simple connections in control, but the connections in data tend to be complex.

Partitions supports abstractions that a system designer finds helpful when thinking about the system.

Abstractions are introduced in order to make what the system is doing clearer and more understandable; an abstraction is a conceptual simplification because it expresses what is being done without specifying how it is done.

The paper then presents some guidelines for identifying different types of abstractions while designing a system:

1. Abstraction of resources: for every hardware resource on the system, we can map characteristics of the abstract resource to the underlying resource
2. Abstract characteristics of data: how it is stored
3. Simplification via limiting information the partition needs to know or has access to
4. Simplification via generalization by identifying functions that perform a common task. Such functions can be grouped together in one partition. “The existence of such a group simplifies other partitions, which need only appeal to the functions of the lower partition rather than perform the tasks themselves.”
5. System maintenance and modification: functions performing a task whose definition is prone to change in the future should be part of independent partitions. For example, functions which deal with connecting to a particular kind of storage back end so that if a different back end is used in the future, only functions in that partition will be affected.

Now that we have some idea about how we can achieve good modularity while designing our system, how do we proceed with it?

The first phase is to identify a set of abstractions that represent the eventual behavior of the system in a general way. The next phase “establishes the data connections between the partitions and describes the flow of control among the partitions”.

The second phase occurs concurrently with the first; as abstractions
are proposed, their utility and practicality are immediately investigated.

A partition has been adequately investigated when
its connections with the rest of the system are known
and when the designers are confident that they understand
exactly what its effect on the system will be.

The next question one would ask is:** how do we identify when the design is finished**?

1. All major abstractions have been identified and been linked to a partition. The system resources have been divided among the various partitions and their positions in the hierarchy have been defined
2. The interfaces and flow of control among the partitions is clearly defined. The test cases for each partition have been identified
3. A basic user guide for the system can be written

Today, let’s take a look at “Program Design in the Unix Environment” published in 1983 by Pike and Kernighan.

The paper opens by listing why Unix has been successful and is a commentary on the Unix philosophy and its benefits. It does so by taking examples and discussing trade offs where programs diverged from the Unix philosophy.

The reasons for Unix’s success:

1. Portability: the kernel & applications are written in C, hence they can be moved from system to system without being re-written in the assembly language particular to that system.
2. The same OS runs on different hardware, so the users are already familiar and don’t have to relearn when new hardware is released.
3. The vendors can ship same software with each machine despite changes in hardware.
4. The system was not too big and easy to modify since everything was written in C.
5. It provided a new philosophy based on the use of general purpose tools which did one thing well and could be combined to do a particular task instead of creating giant monolithic tools that serve only one purpose.

The paper argues that the use and design of tools is closely related — how they fit together is the main subject of this essay.

The paper then dives into cat— the Unix command line utility for concatenating and printing files — it copies its input to its output. The input is usually a sequence of one more of files or the standard input. The output is a file or the standard output.

The main purpose of cat was to act as a utility to concatenate files. It can be combined with the pipe (|) operator to further enhance to extend its utility through output redirection.

Other systems, on the other hand, try to dump a bunch of related functionality into a single command which is against the Unix philosophy. It also creates a lock-in of functionality that might to useful to other programs.

Advantages of the Unix approach:

1. The shell and the programs it can invoke provide a uniform access to system facilities. Eg: the filename arguments are expanded by the shell in a similar fashion for each command. Because of pipes, we don’t need every command to deal with pre- and post- processing of input.
2. Growth is easy when functions are well separated.

Example: the ` (backquote) operator was added to convert the output of one program into input of another without requiring changes in any other program as it is interpreted by the shell. All programs the shell invokes acquire this feature automatically. If each program that required this feature interpreted it, it’d be very hard to enforce uniformity and carry out further experimentation as each new idea would affects all the programs that would want to use it.

However, in the future versions of cat, many new options were introduced, like for printing line numbers and non-printable characters.

The authors argue that instead of adding those options to cat itself, either existing programs should’ve been used or new programs should’ve been created. For example, line number functionality could’ve been provided by using pr. However, there was no program which allowed printing of non-printable characters hence warranting the creation of a new one.

Such a modification confuses what cat’s job is concatenating files with what it happens to do in a common special case showing a file on the terminal. A UNIX program should do one thing well, and leave unrelated tasks to other programs. cat’s job is to collect the data in files. Programs that collect data shouldn’t change the data; cat therefore shouldn’t transform its input.

Whenever we split something into multiple programs, we sacrifice some efficiency. But since cat is usually used without any options, it makes sense to have the most common cases be the most efficient.

Separate programs are not always better than wider options; which is better depends on the problem. Whenever one needs a way to perform a new function, one faces the choice of whether to add a new option or write a new program (assuming that none of the programmable tools will do the job conveniently). The guiding principle for making the choice should be that each program does one thing. Options are appropriately added to a program that already has the right functionality. If there is no such program, then a new program is called for. In that case, the usual criteria for program design should be used: the program should be as general as possible, its default behavior should match the most common usage, and it should cooperate with other programs.

Let’s consider another issue: dealing with fast terminal lines. How to deal with output from cat scrolling off the top of the screen?

There are two approaches:

1. Tell each command about terminal properties so it does the right thing
2. Write a command that handles only terminals without modifying other programs

Let’s consider examples of both approaches: lsc and ls which prints out the list of files in a directory.

lsc varies its output depending on the input — it displays the list in a columnar fashion across the screen so that the o/p fits if it’s outputting to the terminal whereas ls displays everything in a single column.

By retaining single column output to files or pipes, lscensures compatibility with programs like grepor wcthat expect things to be printed one per line. This ad-hoc adjustment of the output format depending on the destination is not only distasteful, it is unique no standard UNIX command has this property.

The authors argue that the columnation facility is useful in general & shouldn’t be locked away in just lsc and be inaccessible to other programs. They advocate for a different program whose primary job is columnation.

Similar reasoning suggests a solution for the general problem of data flowing off screens (columnated or not): a separate program to take any input and print it a screen at a time. Such programs are by now widely available, under names like pg and more. This solution affects no other programs, but can be used with all of them. As usual, once the basic feature is right, the program can be enhanced with options…

Based on the previous example, the authors also talk about different cases where some functionality is locked away in a specific program, like input history in the terminal, which would be better off as a central service. All interactive programs could benefit from it.

They conclude with how augmenting existing commands with features/options is not desirable in Unix, it goes against its basic philosophy which is to make a program do one thing well and several such programs can be composed to accomplish a more complex task.

The key to problem solving on the UNIX system is to identify the right primitive operations and to put them at the right place. UNIX programs tend to solve general problems rather than special cases. In a very loose sense, the programs are orthogonal, spanning the space of jobs to be done (although with a fair
amount of overlap for reasons of history, convenience or efficiency). Functions are placed where they will do the most good: there shouldn’t be a pager in every program that produces output any more than there should be filename pattern matching in every program that uses filenames.

One thing that UNIX does not need is more features. It is successful in part because it has a small number of good ideas that work well together. Merely adding features does not make it easier for users to do things it just makes the manual thicker. The right solution in the right place is always more effective than haphazard hacking.

The paper focuses on the design of large, fault-tolerant, replicated distributed systems and how it’s evolving based on changing requirements over time. It starts of by stating “Reliable systems have always been built out of unreliable components”.

As the granularity of the unreliable component grows (from a mirrored disk to a system to a data center), the latency to communicate with a backup becomes unpalatable. This leads to a more relaxed model for fault tolerance. The primary system will acknowledge the work request and its actions without waiting to ensure that the backup is notified of the work. This improves the responsiveness of the system because the user is not delayed behind a slow interaction with the backup.

Fault-tolerant systems can be made of many components and their goal is keep functioning correctly when one of those components fail. We don’t consider Byzantine failures in this discussion, but instead the fail fast model where either a component works correctly or it fails.

The paper goes on to compare two versions of the Tandem NonStop system — one that used synchronous checkpointing and one that used asynchronous checkpointing. Refer section 3 of the paper for all the details. I’d like to briefly touch upon the difference between the two checkpointing strategies.

• Synchronous checkpointing: in this case, with every write to the primary, state needed to be sent to the backup. Only after the write was acknowledged by the backup did the primary send a response to the client who issued the write request. This ensured that when the primary fails, the backup can take over without losing any work.
• Asynchronous checkpointing: in this strategy, the primary acknowledges & commits the write as soon as it processes it without waiting for a reply from the backup. This technique has improved latency but also poses other challenges addressed later.

Log Shipping

A classic database system has a process that reads the log and ships it to a backup data-center. The normal implementation of this mechanism commits transactions at the primary system (acknowledging the user’s commit request) and asynchronously ships the log. The backup database replays the log, constantly playing catch-up.

The mechanism described above is termed as log shipping. The main problem this poses is that when the primary fails and the back up takes over, some recent transactions might be lost.

This inherently opens up a window in which the work is acknowledged to the client but it has not yet been shipped to the backup. A failure of the primary during this window will lock the work inside the primary for an unknown period of time. The backup will move ahead without knowledge of the locked up work.

The introduction of asynchrony into the system hasan advantage in terms of latency, response time and performance but it makes the system more prone to the possibility of losing work when the primary fails. There are two ways to deal with this:

1. Discard the work locked in the primary when it fails. Whether a system can do that or not depends on the requirements and business rules.
2. Have a recovery mechanism to sync the primary with backups when it comes back up and retry lost work. This is possible only if the operations can be retried in an idempotent way and the out-of-order retries are possible.

The system loses the notion of what the authors call “an authoritative truth”. Nobody knows the accurate state of the system at any given point of time if the work is allowed to be locked in an unavailable backup or primary.

This leads to the conclusion that business rules in a system with asynchronous checkpointing are probabilistic:

If a primary uses asynchronous checkpointing and applies a business rule on the incoming work, it is necessarily a probabilistic rule. The primary, despite its best intentions cannot know it will be alive to enforce the business rules. When the backup system that participates in the enforcement of these business rules is asynchronously tied to the primary, the enforcement of these rules inevitably becomes probabilistic!

The authors state that commutative operations (operations that can be reordered) can be allowed to execute independently and reordered as long as business rules are preserved. However, this is hard to do with storage systems because the write operation isn’t commutative.

Another consideration is that work of a single operation is idempotent, i.e., executing the operation any number of time should result in the same state of the system.

To ensure this, applications typically assign a unique number or ID to the work. This is assigned at the ingress to the system (i.e. whichever replica first handles the work). As the work request rattles around the network, it is easy for a replica to detect that it has already seen that operation and, hence, not do the work twice.

The authors suggest that different operations within a system can provide different consistency guarantees depending on the business requirements. Some operations can choose classic consistency over availability and vice versa.

Next, the authors argue that as soon there is no notion of authoritative truth in a system, all of computing boils down to three things: memories, guesses and apologies.

1. Memories: you can only hope that your replica remembers what it has already seen.
2. Guesses: Due to only partial knowledge being available, the replicas take actions based on local state and may be wrong. “In any system which allows a degradation of the absolute truth, any action is, at best, a guess.” Any action in such a system has a high probability of being successful, but it’s still just a guess.
3. Apologies: Mistakes are inevitable, hence every business needs to have an apology mechanism in place either through human intervention or by automating it.

The paper next discusses the topic of eventual consistency by taking the Amazon shopping cart built using Dynamo & a system for clearing checks as examples. The work coming into these systems is uniquely identified and is processed by a single replica. It flows to other replicas as and when connectivity permits. The requests coming into these systems are commutative (reorderable) and can be processed at different replicas in different orders.

Storage systems alone cannot provide the commutativity we need to create robust systems that function with asynchronous checkpointing. We need the business operations to reorder. Amazon’s Dynamo does not do this by itself. The shopping cart application on top of the Dynamo storage system is responsible for the semantics of eventual consistency and commutativity. The authors think it is time for us to move past the examination of eventual consistency in terms of updates and storage systems. The real action comes when examining application based operation semantics.

Next they discuss two strategies for allocating resources in replicas that might not be able to communicate with each other:

1. Over-provisioning: the resources are partitioned between replicas such that each has a fixed subset of resources they can allocate. No replica can allocate a resource that’s not actually available.
2. Over-booking: the resources are can be individually allocated without ensuring strict partitioning. This may lead to the replicas allocating a resource that’s not truly available, promising something they can’t deliver.

The paper talks also about something termed as the “seat reservation pattern” which is a compromise between over-provisioning and over-booking:

Anyone who has purchased tickets online will recognize the “Seat Reservation” pattern where you can identify potential seats and then you have a bounded period of time, (typically minutes), to complete the transaction. If the transaction is not successfully concluded within the time period, the seats are once again marked as “available”.

ACID 2.0

The classic definition of ACID stands for “Atomic, Consistent, Isolated, and Durable”, it’s goal is to make the application think that there is a single computer which isn’t doing anything else when the transaction is being processed. The authors talk about a new definition for ACID which stands for: Associative, Commutative, Idempotent, and Distributed.

The goal for ACID2.0 is to succeed if the pieces of the work happen: At least once, anywhere in the system, in any order. This defines a new KIND of consistency. The individual steps happen at one or more system. The application is explicitly tolerant of work happening out of order. It is tolerant of the work happening more than once per machine, too.

Going by the classic definition of ACID, fault tolerance is based on having a linear history. If we want to achieve the same guarantees in a distributed system, it’ll require concurrency control mechanisms which “tend to be fragile”.

When the application is constrained to the additional requirements of commutativity and associativity, the world gets a LOT easier. No longer must the state be checkpointed across failure units in a synchronous fashion. Instead, it is possible to be very lazy about the sharing of information. This opens up offline, slow links, low quality datacenters, and more.

In conclusion,

We have attempted to describe the patterns in use by many applications today as they cope with failures in widely distributed systems. It is the reorderability of work and repeatability of work that is essential to allowing successful application execution on top of the chaos of a distributed world in which systems come and go when they feel like it.

This post presents a summary of the paper “Harvest, Yield, and Scalable Tolerant Systems” published by Eric Brewer & Amando Fox in 1999.

This paper deals with the trade offs between consistency and availability for large systems. It’s very easy to point to CAP and assert that no system can have consistency and availability. However, there is a catch. CAP has been misunderstood in a variety of ways. As Coda Hale explains in his excellent blog post “You Can’t Sacrifice Partition Tolerance”:

Of the CAP theorem’s Consistency, Availability, and Partition Tolerance, Partition Tolerance is mandatory in distributed systems. You cannot not choose it. Instead of CAP, you should think about your availability in terms of yield (percent of requests answered successfully) and harvest (percent of required data actually included in the responses) and which of these two your system will sacrifice when failures happen.

This paper focuses on increasing the availability of large scale systems by fault toleration, containment & isolation.

This paper focuses on increasing the availability of large scale systems by fault toleration, containment & isolation.

We assume that clients make queries to servers, in which case there are at least two metrics for correct behavior: yield, which is the probability of completing a request, and harvest, which measures the fraction of the data reflected in the response, i.e. the completeness of the answer to the query.

The two metrics, harvest and yield can be summarized as follows:

• Harvest : data in response/total data

Example: If one of the nodes is down in a 100 node cluster, the harvest is 99% for the duration of the fault.

• Yield : requests completed with success/total number of requests

Note: Yield is different from uptime. Yield specifically deals with the number of requests, not just the time the system wasn’t able to respond to requests.

The paper argues that while there are certain systems which require perfect responses to queries every single time, there are systems that can tolerate imperfect answers once in a while. To increase the overall availability of our systems, we need to carefully think through the required consistency and availability guarantees it needs to provide.

Trading Harvest for Yield — Probabilistic Availability

Nearly all systems are probabilistic whether they realize it or not. In particular, any system that is 100% available under single faults is probabilistically available overall (since there is a non-zero probability of multiple failures)

Fox and Brewer talk about understanding the probabilistic nature of availability. This helps in understanding and limiting the impact of faults by making decisions about what needs to be available and what kind of faults can the system deal with.

They outline the linear degradation of harvest in case of multiple node faults. The harvest is directly proportional to the number of nodes that are functioning correctly, hence it’s decreases/increases linearly. Two strategies are suggested for increasing the yield:

1. Random distribution of data on the nodes. If one of the nodes goes down, the average-case and worst-case fault behavior doesn’t change. Instead if the distribution isn’t random, then depending type of data, the impact of a fault maybe variable. For example, if only one of the nodes stored info related to a user’s account balance and it goes down, he entire banking system will not be able to work.
2. Replicating the most important data. This reduces the impact in case one of the nodes containing a subset of high-priority data goes down and also improves harvest.

Another notable observation made in the paper is that while it is possible to replicate all of your data, it doesn’t do a lot to improve your harvest/yield while increases the cost of operation substantially. This is because the internet works based on best-in-effort protocols which can never guarantee 100% harvest/yield.

Application Decomposition and Orthogonal Mechanisms

The second strategy focuses on the benefits of orthogonal system design. It starts out stating that large systems are composed of sub systems which independently cannot tolerate failures but fail in a way that allows the entire system to continue functioning with some impact on utility.

The actual benefit is the ability to provision each subsystem’s state management separately, providing strong consistency or persistent state only for the subsystems that need it, not for the entire application. The savings can be significant if only a few small subsystems require the extra complexity.

The paper states that orthogonal components are completely independent of each other having no run time interface to other components except a configuration interface maybe. This allows each individual component to fail independently minimizing its impact on the overall system.

Composition of orthogonal subsystems shifts the burden of checking for possibly harmful interactions from runtime to compile time, and deployment of orthogonal guard mechanisms improves robustness for the runtime interactions that do occur, by providing improved fault containment.

Over all, the goal of this paper was to motivate research in the field of designing fault-tolerant and highly available large scale systems. Also, to think carefully about the consistency and availability guarantees the application needs to provide and the trade offs it is capable of making in terms of harvest vs yield.

Raft:

Raft is a distributed consensus algorithm. It was designed to be easily understood. It solves the problem of getting multiple servers to agree on a shared state even in the face of failures. The shared status is usually a data structure supported by a replicated log. We need the system to be fully operational as long as a majority of the servers are up.

Raft works by electing a leader in the cluster. The leader is responsible for accepting client requests and managing the replication of the log to other servers. The data flows only in one direction: from leader to other servers.

Raft decomposes consensus into three sub-problems:

• Leader Election: A new leader needs to be elected in case of the failure of an existing one.
• Log replication: The leader needs to keep the logs of all servers in sync with its own through replication.
• Safety: If one of the servers has committed a log entry at a particular index, no other server can apply a different log entry for that index.

Basics:

Each server exists in one of the three states: leader, follower, or candidate.

In normal operation there is exactly one leader and all of the other servers are followers. Followers are passive: they issue no requests on their own but simply respond to requests from leaders and candidates. The leader handles all client requests (if a client contacts a follower, the follower redirects it to the leader). The third state, candidate, is used to elect a new leader.

Raft divides time into terms of arbitrary length, each beginning with an election. If a candidate wins the election, it remains the leader for the rest of the term. If the vote is split, then that term ends without a leader.

The term number increases monotonically. Each server stores the current term number which is also exchanged in every communication.

.. if one server’s current term is smaller than the other’s, then it updates its current term to the larger value. If a candidate or leader discovers that its term is out of date, it immediately reverts to follower state. If a server receives a request with a stale term number, it rejects the request.

Raft makes use of two remote procedure calls (RPCs) to carry out its basic operation.

• RequestVotes is used by candidates during elections
• AppendEntries is used by leaders for replicating log entries and also as a heartbeat (a signal to check if a server is up or not — it doesn’t contain any log entries)

Leader election

The leader periodically sends a heartbeat to its followers to maintain authority. A leader election is triggered when a follower times out after waiting for a heartbeat from the leader. This follower transitions to the candidate state and increments its term number. After voting for itself, it issues RequestVotes RPC in parallel to others in the cluster. Three outcomes are possible:

1. The candidate receives votes from the majority of the servers and becomes the leader. It then sends a heartbeat message to others in the cluster to establish authority.
2. If other candidates receive AppendEntries RPC, they check for the term number. If the term number is greater than their own, they accept the server as the leader and return to follower state. If the term number is smaller, they reject the RPC and still remain a candidate.
3. The candidate neither loses nor wins. If more than one server becomes a candidate at the same time, the vote can be split with no clear majority. In this case a new election begins after one of the candidates times out. > Raft uses randomized election timeouts to ensure that split votes are rare and that they are resolved quickly. To prevent split votes in the first place, election timeouts are chosen randomly from a fixed interval (e.g., 150–300ms). This spreads out the servers so that in most cases only a single server will time out; it wins the election and sends heartbeats before any other servers time out. The same mechanism is used to handle split votes. Each candidate restarts its randomized election timeout at the start of an election, and it waits for that timeout to elapse before starting the next election; this reduces the likelihood of another split vote in the new election.

Log Replication:

The client requests are assumed to be write-only for now. Each request consists of a command to be executed ideally by the replicated state machines of all the servers. When a leader gets a client request, it adds it to its own log as a new entry. Each entry in a log:

• Contains the client specified command
• Has an index to identify the position of entry in the log (the index starts from 1)
• Has a term number to logically identify when the entry was written

It needs to replicate the entry to all the follower nodes in order to keep the logs consistent. The leader issues AppendEntries RPCs to all other servers in parallel. The leader retries this until all followers safely replicate the new entry.

When the entry is replicated to a majority of servers by the leader that created it, it is considered committed. All the previous entries, including those created by earlier leaders, are also considered committed. The leader executes the entry once it is committed and returns the result to the client.

The leader maintains the highest index it knows to be committed in its log and sends it out with the AppendEntries RPCs to its followers. Once the followers find out that the entry has been committed, it applies the entry to its state machine in order.

Raft maintains the following properties, which together constitute the Log Matching Property> • If two entries in different logs have the same index and term, then they store the same command.> • If two entries in different logs have the same index and term, then the logs are identical in all preceding entries.

When sending an AppendEntries RPC, the leader includes the term number and index of the entry that immediately precedes the new entry. If the follower cannot find a match for this entry in its own log, it rejects the request to append the new entry.

This consistency check lets the leader conclude that whenever AppendEntries returns successfully from a follower, they have identical logs until the index included in the RPC.

But the logs of leaders and followers may become inconsistent in the face of leader crashes.

In Raft, the leader handles inconsistencies by forcing the followers’ logs to duplicate its own. This means that conflicting entries in follower logs will be overwritten with entries from the leader’s log.

The leader tries to find the last index where its log matches that of the follower, deletes extra entries if any, and adds the new ones.

The leader maintains a nextIndex for each follower, which is the index of the next log entry the leader will send to that follower. When a leader first comes to power, it initializes all nextIndex values to the index just after the last one in its log.

Whenever AppendRPC returns with a failure for a follower, the leader decrements the nextIndex and issues another AppendEntries RPC. Eventually, nextIndex will reach a value where the logs converge. AppendEntries will succeed when this happens and it can remove extraneous entries (if any) and add new ones from the leaders log (if any). Hence, a successful AppendEntries from a follower guarantees that the leader’s log is consistent with it.

With this mechanism, a leader does not need to take any special actions to restore log consistency when it comes to power. It just begins normal operation, and the logs automatically converge in response to failures of the Append-Entries consistency check. A leader never overwrites or deletes entries in its own log.

Safety:

Raft makes sure that the leader for a term has committed entries from all previous terms in its log. This is needed to ensure that all logs are consistent and the state machines execute the same set of commands.

During a leader election, the RequestVote RPC includes information about the candidate’s log. If the voter finds that its log it more up-to-date that the candidate, it doesn’t vote for it.

Raft determines which of two logs is more up-to-date by comparing the index and term of the last entries in the logs. If the logs have last entries with different terms, then the log with the later term is more up-to-date. If the logs end with the same term, then whichever log is longer is more up-to-date.

Cluster membership:

For the configuration change mechanism to be safe, there must be no point during the transition where it is possible for two leaders to be elected for the same term. Unfortunately, any approach where servers switch directly from the old configuration to the new configuration is unsafe.

Raft uses a two-phase approach for altering cluster membership. First, it switches to an intermediate configuration called joint consensus. Then, once that is committed, it switches over to the new configuration.

The joint consensus allows individual servers to transition between configurations at different times without compromising safety. Furthermore, joint consensus allows the cluster to continue servicing client requests throughout the configuration change.

Joint consensus combines the new and old configurations as follows:

• Log entries are replicated to all servers in both the configurations
• Any server from old or new can become the leader
• Agreement requires separate majorities from both old and new configurations

When a leader receives a configuration change message, it stores and replicates the entry for join consensus C<old, new>. A server always uses the latest configuration in its log to make decisions even if it isn’t committed. When joint consensus is committed, only servers with C<old, new> in their logs can become leaders.

It is now safe for the leader to create a log entry describing C<new> and replicate it to the cluster. Again, this configuration will take effect on each server as soon as it is seen. When the new configuration has been committed under the rules of C<new>, the old configuration is irrelevant and servers not in the new configuration can be shut down.

A fantastic visualization of how Raft works can be found here.

More material such as talks, presentations, related papers and open-source implementations can be found here.

I have dug only into the details of the basic algorithm that make up Raft and the safety guarantees it provides. The paper contains lot more details and it is super approachable as the primary goal of the authors was understandability. I definitely recommend you read it even if you’ve never read any other paper before.

It presents an updated explanation of Viewstamped Replication, a replication technique that handles failures in which nodes crash. It describes how client requests are handled, how the group reorganizes when a replica fails, and how a failed replica is able to rejoin the group.

Introduction

The Viewstamped Replication protocol, referred to as VR, is used for replicated services that run on many nodes known as replicas. VR uses state machine replication: it maintains state and makes it accessible to the clients consuming that service.

Some features of VR:

• VR is primarily a replication protocol, but it provides consensus too.
• VR doesn’t use any disk I/O — it uses replicated state for persistence.
• VR deals only with crash failures: a node is either functioning or it completely stops.
• VR works in an asynchronous network like the internet where nothing can be concluded about a message that doesn’t arrive. It may be lost, delivered out of order, or delivered many times.

Replica Groups

VR ensures reliability and availability when no more than a threshold of f replicas are faulty. It does this by using replica groups of size 2f + 1; this is the minimal number of replicas in an asynchronous network under the crash failure model.

We can provide a simple proof for the above statement: in a system with f crashed nodes, we need at least the majority of f+1 nodes that can mutually agree to keep the system functioning.

A group of f+1 replicas is often known as a quorum. The protocol needs the quorum intersection property to be true to work correctly. This property states that:

The quorum of replicas that processes a particular step of the protocol must have a non-empty intersection with the group of replicas available to handle the next step, since this way we can ensure that at each next step at least one participant knows what happened in the previous step.

Architecture:

The architecture of VR is as follows:

1. The user code is run on client machines on top of a VR proxy.
2. The proxy communicates with the replicas to carry out the operations requested by the client. It returns the computed results from the replicas back to the client.
3. The VR code on the side of the replicas accepts client requests from the proxy, executes the protocol, and executes the request by making an up-call to the service code.
4. The service code returns the result to the VR code which in turn sends a message to the client proxy that requested the operation.

Overview

The challenge for the replication protocol is to ensure that operations execute in the same order at all replicas in spite of concurrent requests from clients and in spite of failures.

If all the replicas should end in the same state, it is important that the above condition is met.

VR deals with the replicas as follows:

Primary : Decides the order in which the operations will be executed

Secondary: Carries out the operations in the same order as selected by the primary

What if the primary fails?

• VR allows different replicas to assume the role of primary if it fails over time.
• The system moves through a series of views. In each view, one replica assumes the role of primary.
• The other replicas watch the primary. If it appears to be faulty, then they carry out a view-change to select a new primary.

We consider the following three scenarios of the VR protocol:

• Normal case processing of user requests
• View changes to select a new primary
• Recovery of a failed replica so that it can rejoin the group

VR protocol

The state maintained by each replica is presented in the figure above. Some points to note:

• The identity of the primary isn’t stored but computed using the view number and the configuration.
• The replica with the smallest IP is replica 1 and so on.

The client side proxy also maintains some state:

• It records the configuration.
• It records the current view number to track the primary.
• It has a client id and an incrementing client request number.

Normal Operation

• Replicas participate in processing of client requests only when their status is normal.
• Each message sent contains the sender’s view number. Replicas process only those requests which have a view number that matches what they know. If the sender replica is ahead, it drops the message. If it’s behind, it performs a state transfer.

The normal operation of VR can be broken down into the following steps:

1. The client sends a REQUEST message to the primary asking it to perform some operation , passing it the client-id and the request number.
2. The primary cross-checks the info present in the client table. If the request number is smaller than the one present in the table, it discards it. It re-sends the response if the request was the most recently executed one.
3. The primary increases the op-number , appends the request to its log, and updates the client table with the new request number. It sends a PREPARE message to the replicas with the current view-number, the operation-number, the client’s message, and the commit-number (the operation number of the most recently committed operation).
4. The replicas won’t accept a message with an op-number until they have all operations preceding it. They use state transfer to catch up if required. Then they add the operation to their log, update the client table, and send a PREPAREOK message to the primary. This message indicates that the operation, including all the preceding ones, has been prepared successfully.
5. The primary waits for a response from f replicas before committing the operation. It increments the commit-number . After making sure all operations preceding the current one have been executed, it makes an up-call to the service code to execute the current operation. A REPLY message is sent to the client containing the view-number, request-number, and the result of the up-call.

Usually the PREPARE message is used to inform the backup replicas of the committed operations. It can also do so by sending a COMMIT message.

To execute a request, a backup has to make sure that the operation is present in its log and that all the previous operations have been executed. Then it executes the said operation, increments its commit-number , and updates the client’s entry in the client-table. But it doesn’t send a reply to the client, as the primary has already done that.

If a client doesn’t receive a timely response to a request, it re-sends the request to all replicas. This way if the group has moved to a later view, its message will reach the new primary. Backups ignore client requests; only the primary processes them.

View change operation

Backups monitor the primary: they expect to hear from it regularly. Normally the primary is sending PREPARE messages, but if it is idle (due to no requests) it sends COMMIT messages instead. If a timeout expires without a communication from the primary, the replicas carry out a view change to switch to a new primary.

There is no leader election in this protocol. The primary is selected in a round robin fashion. Each member has a unique IP address. The next primary is the backup replica with the smallest IP that is functioning. Each number in the group is already aware of who is expected to be the next primary.

Every executed operation at the replicas must survive the view change in the order specified when it was executed. The up-call is carried out at the primary only after it receives f PREPAREOK messages. Thus the operation has been recorded in the logs of at least f+1 replicas (the old primary and f replicas).

Therefore the view change protocol obtains information from the logs of at least f + 1 replicas. This is sufficient to ensure that all committed operations will be known, since each must be recorded in at least one of these logs; here we are relying on the quorum intersection property. Operations that had not committed might also survive, but this is not a problem: it is beneficial to have as many operations survive as possible.

1. A replica that notices the need for a view change advances its view-number , sets its status to view-change , and sends a START-VIEW-CHANGE message. A replica identifies the need for a view change based on its own timer, or because it receives a START-VIEW-CHANGE or a DO-VIEW-CHANGE from others with a view-number higher than its own.
2. When a replica receives f START-VIEW-CHANGE messages for its view-number, it sends a DO-VIEW-CHANGE to the node expected to be the primary. The messages contain the state of the replica: the log, most recent operation-number and commit-number, and the number of the last view in which its status was normal.
3. The new primary waits to receive f+1 DO-VIEW-CHANGE messages from the replicas (including itself). Then it updates its state to the most recent based on the info from replicas (see paper for all rules). It sets its number as the view-number in the messages, and changes its status to normal. It informs all other replicas by sending a STARTVIEW message with the most recent state including the new log, commit-number and op-number .
4. The primary can now accept client requests. It executes any committed operations and sends the replies to clients.
5. When the replicas receive a STARTVIEW message, they update their state based on the message. They send PREPAREOK messages for all uncommitted operations present in their log after the update. They execute these operations to to be in sync with the primary.

To make the view change operation more efficient, the paper describes the following approach:

The protocol described has a small number of steps, but big messages. We can make these messages smaller, but if we do, there is always a chance that more messages will be required. A reasonable way to get good behavior most of the time is for replicas to include a suffix of their log in their DO-VIEW-CHANGE messages. The amount sent can be small since the most likely case is that the new primary is up to date. Therefore sending the latest log entry, or perhaps the latest two entries, should be sufficient. Occasionally, this information won’t be enough; in this case the primary can ask for more information, and it might even need to first use application state to bring itself up to date.

Recovery

When a replica recovers after a crash it cannot participate in request processing and view changes until it has a state at least as recent as when it failed. If it could participate sooner than this, the system can fail.

The replica should not “forget” anything it has already done. One way to ensure this is to persist the state on disk — but this will slow down the whole system. This isn’t necessary in VR because the state is persisted at other replicas. It can be obtained by using a recovery protocol provided that the replicas are failure independent.

When a node comes back up after a crash it sets its status to recovering and carries out the recovery protocol. While a replica’s status is recovering it does not participate in either the request processing protocol or the view change protocol.

The recovery protocol is as follows:

1. The recovering replica sends a RECOVERY message to all other replicas with a nonce.
2. Only if the replica’s status is normal does it reply to the recovering replica with a RECOVERY-RESPONSE message. This message contains its view number and the nonce it received. If it’s the primary, it also sends its log, op-number, and commit-number.

3. When the replica has received f+1 RECOVERY-RESPONSE messages, including one from the primary, it updates its state and changes its status to normal. > The protocol uses the nonce to ensure that the recovering replica accepts only RECOVERY-RESPONSE messages that are for this recovery and not an earlier one.

   Reconfiguration

Reconfiguration deals with epochs. The epoch represents the group of replicas processing client requests. If the threshold for failures, f, is adjusted, the system can either add or remove replicas and transition to a new epoch. It keeps track of epochs through the epoch-number .

Another status, namely transitioning, is used to signify that a system is moving between epochs.

The approach to handling reconfiguration is as follows. A reconfiguration is triggered by a special client request. This request is run through the normal case protocol by the old group. When the request commits, the system moves to a new epoch, in which responsibility for processing client requests shifts to the new group. However, the new group cannot process client requests until its replicas are up to date: the new replicas must know all operations that committed in the previous epoch. To get up to date they transfer state from the old replicas, which do not shut down until the state transfer is complete.

The VR sub protocols need to be modified to deal with epochs. A replica doesn’t accept messages from an older epoch compared to what it knows, such as those with an older epoch-number. It informs the sender about the new epoch.

During a view-change, the primary cannot accept client requests when the system is transitioning between epochs. It does this by checking if the topmost request in its log is a RECONFIGURATION request. A recovering replica in an older epoch is informed of the epoch if it is part of the new epoch or if it shuts down.

The issue that comes to mind is that the client requests can’t be served while the system is moving to a new epoch.

The old group stops accepting client requests the moment the primary of the old group receives the RECONFIGURATION request; the new group can start processing client requests only when at least f + 1 new replicas have completed state transfer.

This can be dealt with by “warming up” the nodes before reconfiguration happens. The nodes can be brought up-to-date using state transfer while the old group continues to reply to client requests. This reduces the delay caused during reconfiguration.

This paper has presented an improved version of Viewstamped Replication, a protocol used to build replicated systems that are able to tolerate crash failures. The protocol does not require any disk writes as client requests are processed or even during view changes, yet it allows nodes to recover from failures and rejoin the group.

The paper also presents a protocol to allow for reconfigurations that change the members of the replica group, and even the failure threshold. A reconfiguration technique is necessary for the protocol to be deployed in practice since the systems of interest are typically long lived.

This paper focuses on the inclusion of Inter Process Communication (IPC) primitives and the consequences of doing so. It explores, in particular, the time-out and the insertion property feature described in detail below with respect to distributed systems of sequential processes without system buffering & interrupts.

It also touches upon the two generals problem which states that it’s impossible for two processes to agree on a decision over an unreliable network.

Introduction:

The design of an Inter Process Communication Mechanism (IPCM) can be described by stating the behavior of the system & the required services. The features to be included in the IPCM are very critical as they might be interdependent, hence the design process should begin with a detailed spec. This involves thorough understanding of the consequences of each decision.

The major aim of the paper is to point out the interdependence of the features to be incorporated in the system.

The paper states that at times the incompatibility between features is visible from the start. Yet, sometimes two features which seem completely unrelated end up affecting each other significantly. If the trade-offs involved aren’t explored at the beginning, it might not be possible to include desirable features. Trying to accommodate conflicting features results into messy code at the cost of elegance.

Intermediate Processes:

Let’s suppose a system doesn’t allow indirect communication between processes that cannot establish a connection. The users just care about the logical sender and receiver of the messages: they don’t care what path the messages take or how many processes they travel through to reach their final destination. In such a situation, intermediate processes come to our rescue. They’re not a part of the IPCM but are inserted between two processes that can’t communicate directly through a directory or broker process when the connection is set up. They’re the only ones aware of the indirect nature of communication between the processes.

Centralized vs Distributed Systems:

Centralized Communication Facility

1. Has a single agent which is able to maintain all state information related to the communication happening in the system
2. The agent can also change the state of the system in a well-defined manner

For example, if we consider the IPCM to be the centralized agent, it’ll be responsible for matching the SEND & RECEIVE requests of two processes, transferring data between their buffers and relaying appropriate status to both.

Distributed Communication Facility

1. No single agent has the complete state information at any time
2. The IPCM is made of several individual components which coordinate, exchange and work with parts of state information they possess.
3. A global change can take a considerable amount of time
4. If one of the components crashes, the activity of other components still interests us

Case 1:

In Figure 1, P1 and P2 are the two communicating processes on different machines over a network with their own IPCMs and P is the interface which enables this, with parts that lie on both machines. P handles the details of the network lines.

If one machine or a communication link crashes, we want the surviving IPCM’s to continue their operation. At least one component should detect a failure and be able to communicate. (In the case of a communication link failure, both ends must know.)

Case 2:

Distributed communication can also happen on the same machine given that there are one or more intermediate processes taking part in the system. In that case, P, P1 & P2 will be processes on the same system with identical IPCMs. P is an intermediate processes which facilitates the communication between P1 & P2.

Transactions between P1 & P2 consist of two steps: P1 to P and P to P2. Normally, the status returned to P1 would reflect the result of the P1 to P transfer, but P1 is interested in the status of the over all transaction from P1 to P2. One way to deal with this is a delayed status return. The status isn’t sent to the sender immediately after the transaction occurs but only when the sender issues a SEND STATUS primitive. In the example above, after receiving the message from P1, P further sends it to P2, doesn’t send any status to P1 & waits to receive a status from P2. When it receives the appropriate status from P2, it relays it to P1 using the SEND STATUS primitive.

Special Cases of Distributed Facility

This section starts out by stating some facts and reasoning around them.

FACT 0: A perfectly reliable distributed system can be made to behave as a centralized system.

Theoretically, this is possible if:

1. The state of different components of the system is known at any given time
2. After every transaction, the status is relayed properly between the processes through their IPCMs using reliable communication.

However, this isn’t possible in practice because we don’t have a perfect reliable network. Hence, the more realistic version of the above fact is:

FACT I: A distributed IPCM can be made to simulate a centralized system provided that:> 1. The overall system remains connected at all times, and> 2. When a communication link fails, the component IPCM’s that are connected to it know about it, and> 3. The mean time between two consecutive failures is large compared to the mean transaction time across the network.

The paper states that if the above conditions are met, we can establish communication links that are reliable enough to simulate a centralized systems because:

1. There is always a path from the sender to the receiver
2. Only one copy of an undelivered message will be retained by the system in case of a failure due to link failure detection. Hence a message cannot be lost if undelivered and will be removed from the system when delivered.
3. A routing strategy and a bound on the failure rate ensures that a message moving around in a subset of nodes will eventually get out in finite time if the target node isn’t present in the subset.

The cases described above are special cases because they make a lot of assumptions, use inefficient algorithms and don’t take into account network partitions leading to disconnected components.

Status in Distributed Systems

Complete Status

A complete status is one that relays the final outcome of the message, i.e., whether it reached its destination.

FACT 2: In an arbitrary distributed facility, it is impossible to provide complete status.

Case 1:

Assume that a system is partitioned into two disjoint networks, leaving the IPCMs disconnected. Now, if IPCM1 was awaiting a status from IPCM2, there is no way to get it and relay the result to P1.

Case 2:

Consider figure 2, if there isn’t a reliable failure detection mechanism present in the system and IPCM2 sends a status message to IPCM1, then it can never be sure it reached or not without an acknowledgement. This leads to an infinite exchange of messages.

Time-outs

Time-outs are required because the system has finite resources and can’t afford to be deadlocked forever. The paper states that:

FACT 3: In a distributed system with timeouts, it is impossible to provide complete status (even if the system is absolutely reliable).

In figure 3, P1 is trying to send P2 a message through a chain of IPCMs.

Suppose if I1 takes data from P1 but before it hears about the status of the transaction, P1’s request times out. IPCM1 has now knowledge about the final outcome whether the data was successfully received by P2. Whatever status it returns to P1, it may prove to be incorrect. Hence, it’s impossible to provide complete status in a distributed facility with time-outs.

Insertion Property

An IPCM has insertion property if we insert an intermediate process P between two processes P1 & P2 that wish to communicate such that:

1. P is invisible to both P1 & P2
2. The status relayed to P1 & P2 is the same they’d get if directly connected > FACT 4: In a distributed system with timeouts, the insertion property can be possessed only if the IPCM withholds some status information that is known to it.

Delayed status is required to fulfill the insertion property. Consider that the message is sent from P1 to P2. What happens if P receives P1’s message, it goes into await-status state but it times out before P could learn about the status?

We can’t tell P1 the final outcome of the exchange as that’s not available yet. We also can’t let P know that it’s in await-status state because that would mean that the message was received by someone. It’s also not possible that P2 never received the data because such a situation cannot arise if P1 & P2 are directly connected & hence violates the insertion property.

The solution to this is to provide an ambiguous status to P1, one that is as likely to be possible if the two processes were connected directly.

Thus, a deliberate suppression of what happened is introduced by providing the same status to cover a time-out which occurs while awaiting status and, say, a transmission error.

Logical & Physical Messages

The basic function of an IPCM is the transfer and synchronization of data between two processes. This may happen by dividing the physical messages originally sent by the sender process as a part of a single operation into smaller messages, also known as logical message for the ease of transfer.

Buffer Size Considerations

As depicted in figure 5, if a buffer mismatch arises, we can take the following approaches to fix it:

1. Define a system-wide buffer size. This is extremely restrictive, especially within a network of heterogenous systems
2. Satisfy the request with the small buffer size & inform both the processes involved what happened. This approach requires that the processes are aware of the low level details of the communication.
3. Allowing partial transfers. In this approach, only the process that issued the smaller request (50 words) is woken up. All other processes remain asleep awaiting further transfers. If the receiver’s buffer isn’t full, an EOM (End Of Message) indicator is required to wake it up.

Partial Transfers and Well-Known Ports

In figure 6, a service process using a well-known port is accepting requests for sever user processes, P1…Pn. If P1 sends a message to the service process that isn’t complete and doesn’t fill its buffer, we need to consider the following situations:

1. The well-known port is reserved for P1. No other process can communicate with the service process using it till P1 is done.
2. When the service process times out while P1 is preparing to send the second and final part of the message, we need to handle it without informing P1 that the first part has been ignored. P1 isn’t listening for incoming messages from the service process.

Since none of these problems arise without partial transfers, one solution is to ban them altogether. For example:

This is the approach taken in ARPANET where the communication to well known ports are restricted to short, complete messages which are used to setup a separate connection for subsequent communication.

Buffer Processes

This solution is modeled around the creation of dynamic processes.

Whenever P1 wishes to transfer data to the service process, a new process S1 is created and receives messages from P1 till the logical message is completed, sleeping as and when required. Then it sends the complete physical message to the service process with EOM flag set. Thus no partial transfers happen between S1 and the service process, they’re all filtered out before that.

However, this kind of a solution isn’t possible with well-known ports. S1 is inserted between P1 and the service process when the connection is initiailized. However, in the case of well-known ports, no initialization takes place.

In discussing status returned to the users, we have indicated how the presence of certain other features limits the information that can be provided.> In fact, we have shown situations in which uncertain status had to be returned, providing almost no information as to the outcome of the transaction.

Even though the inclusion of insertion property complicates things, it is beneficial to use the weaker version of it.

Finally, we list a set of features which may be combined in a working IPCM:> (1) Time-outs> (2) Weak insertion property and partial transfer> (3) Buffer processes to allow> (4) Well-known ports — with appropriate methods to deal with partial transfers to them.

This post distills the material presented in the paper titled “A Note on Distributed Systems” published in 1994 by Jim Waldo et al.

The paper presents the differences between local & distributed computing in the context of Object Oriented Programming, explaining why treating them to the be same is incorrect and leads to applications that aren’t robust or reliable.

Introduction:

The paper kicks off by stating that the current work in distributed systems is modeled around objects, more specifically, a unified view of objects: objects are defined by their supported interfaces & the operations they support. Naturally, this can be extended to imply that objects in the same address space, or in a different address space on the same machine, or on a different machine all behave in a similar manner: their location is an implementation detail.

Let’s define the most common terms in this paper:

Local Computing:

It deals with programs that are confined to a single address space only.

Distributed Computing:

It deals with programs that can make calls to objects in different address spaces either on the same machine or on a different machine.

The Vision of Unified Objects:

Implicit in this vision is that the system will be “objects all the way down”; that is, that all current invocations or calls for system services will be eventually converted into calls that might be to an object residing on some other machine. There is a single paradigm of object use and communication used no matter what the location of the object might be.

This refers to the assumption that all objects are defined only in terms of their interfaces. Their implementation which also includes location of the object (remote/local), is independent of their interfaces and is hidden from the programmer. Hence, as far the programmer is concerned, they write the same type of call for every object, whether local or remote and the system takes care of actually sending the message by figuring out the underlying mechanisms which aren’t visible to the programmer writing the application.

The hard problems in distributed computing are not the problems of how to get things on and off the wire.

The paper goes on to define what are the toughest challenges of building a distributed systems:

1. Latency
2. Memory Access
3. Partial failure & concurrency

Ensuring reasonable performance while dealing with all of the above doesn’t make the life of the a distributed systems engineer any easier. Moreover, the lack of any central resource or state manager just adds on to the various challenges. Let’s observe each of these one by one.

Latency:

This is perhaps the fundamental difference between local & distributed object invocation. The paper claims that a remote call is four to five times slower than a local call. If the design of a system fails to recognize this fundamental difference, it is bound to suffer from serious performance problems especially if it relies heavily on remote communication. A thorough understanding of the application being designed is required to decide which objects should be kept together and which can be placed remotely.

If the goal is to unify the difference in latency, then we’ve two options:

• Rely on the hardware to get faster with time in order to eliminate the difference in effeciency
• Develop tools which allow us to visualize communication patterns between different objects and move them around as required. Since location is an implementation detail, this shouldn’t be too hard to achieve.

Memory:

Another difference that’s very relevant to the design of distributed systems is the pattern of memory access between local and remote objects. A pointer in the local address space isn’t valid in a remote address space. We’re left with two choices:

• The developer must be made aware of the difference between the access patterns
• The system handles all memory access

To unify the differences in access between local & remote access, we need to let the system handle all aspects of access to memory. There are several way to do that:

• Distributed shared memory
• Using the OOP paradigm, compose a system entirely of objects, i.e., dealing only with object references. The transfer of data between address spaces can be dealt with by marshalling & unmarshalling the data by the layer underneath. This approach, however, makes the use of address-space-relative pointers obsolete. > The danger lies in promoting the myth that “remote access and local access are exactly the same” and not enforcing the myth. An underlying mechanism that does not unify all memory accesses while still promoting this myth is both misleading and prone to error.

Hence, it’s important for programmers to be made aware of the various differences between accessing local & remote objects so that they don’t get bitten by not knowing what’s happening under the covers.

Partial failure & concurrency

Partial failure is a central reality of distributed computing.

The paper argues that while both local & distributed systems are subject to failure, it’s a lot harder to discover what went wrong in case of distributed systems. For a local systems, either everything is shut down or there is some central authority which can detect what went wrong (the OS, for example).

However, in the case of a distributed system, determining what has failed is extremely difficult since there is no global state or resource manager available to keep track of everything happening in and across the system, hence there is no way to inform other components which may be functioning correctly which ones have failed. Components in a distributed system fail independently.

A central problem in distributed computing is insuring that the state of the whole system is consistent after such a failure; this is a problem that simply does not occur in local computing.

In order for a system to withstand partial failure, it’s important that it deals with indeterminacy and the objects react to it in a consistent manner. The interfaces must be able to state the cause of failure if possible and allow the reconstruction of a “reasonable state” in case the cause can’t be determined.

The question is not “can you make remote method invocation look like local method invocation?” but rather “what is the price of making remote method invocation identical to local method invocation?”

Two approaches come to mind:

1. Treat all interfaces & objects as local. The problem with this approach is that it doesn’t take into account the failure models associated with distributed systems and hence it’s indeterministic by nature.
2. Treat all interfaces & objects as remote. The flaw with this approach is that over-complicates local computing. It adds on a ton of work for objects that are never accessed remotely. > A better approach is to accept that there are irreconcilable differences between local and distributed computing, and to be conscious of those differences at all stages of the design and implementation of distributed applications. * * *

It started off with great summer at Microsoft India campus as a software engineering intern, working with amazing people. I learned more in those 8 weeks than I did in 3 years of college till then. I realized how much college doesn’t teach you about “software engineering” and how software is made in the real world. This is what got me thinking about how I can improve my skills in the one year I’ve before I start working full time. I had a choice between two options: working on open source or working on side-projects. For reasons that aren’t relevant to this post, I chose open source. As I look back today, I’ve zero regrets about it.

In August, I got my first job offer as a software engineer from Microsoft and will be joining them next year. I started contributed meaningfully to web-ext around the same time and published this blogpost:

A Beginner’s Very Bumpy Journey Through The World of Open Source
_Did you land on this story looking for advice on how to start contributing to open source? There are tons of these…_medium.freecodecamp.com[](https://medium.freecodecamp.com/a-beginners-very-bumpy-journey-through-the-world-of-open-source-4d108d540b39)

The reaction to this blog post was unprecedented. I could have never imagined that it’ll receive the kind of attention it did. So many people reached out to me with kind words of compassion. Some people told me their story motivated them to be more kind and empathetic. Some of them went back to contributing to open source, some made their first contribution. Moreover, this post also put me in touch with some of the most wonderful people I’ve had the pleasure of interacting this year: Dan Abramov, Quincy Larson and Kent C. Dodds.

The most important thing I’ve done is overcome my fear and all other barriers of contributing to open source. Furthermore, I also got selected for the present Outreachy cohort with Mozilla which was like a cherry on the top of the cake. Along the way, I’ve met some absolutely wonderful people I’d like to thank.

Starting with Kumar McMillan and Luca Greco. Y’all are the most amazing mentors one can ask for. Thank you for all your patience, empathy and all the time you’ve invested in teaching me and reviewing my code. It’s been a pleasure to learn from and work with you people.

Quincy Larson, thank you for the way you dealt with the issue I had faced with one of the FCC maintainers. Also, for the countless hours you put into the FCC Medium publication. You’re enthusiasm for open source is amazing and something I definitely look up to.

Henry Zhu, thank you for the work you do with Babel. You’re incredibly inspiring to me and I’m sure a ton of other people as well. I hope I can keep contributing the way you do despite having a full time job outside of OSS.

Kent C. Dodds, thank you for all your support throughout. I really appreciate your passion for open source and teaching.

Jessie Frazelle, you’ve been a role model for me even before I got into open source. As I’ve told you, I really hope someday I know my shit like you know yours. You’re simply amazing & I really look up to you.

Lastly, but not the least, Dan Abramov. I don’t think anyone has inspired me as much you’ve this year. It’s been wonderful to work with you and learn from you in whatever I did for CRA and Javascript in general. Thank you for being so kind, patient and empathetic always.

Huge shout out to everyone who helped me get over my fears. Anna Ossowski and VM Brasseur for working with me and pushing me to submit my first ever conference proposal. Also, to everyone who reviewed it and gave feedback.

I started technical blogging and sharing what I learn and how I learn it with others. Shout out to everyone to reviewed my posts and helped me improve: ashley williams, Dan Abramov, Kent C. Dodds, Quincy Larson and everyone else!

I also won a scholarship to attend the Grace Hopper Celebration in Bangalore this year where I got to meet a ton a amazing women in tech. I was also fortunate enough to interact with a lot of senior women who’ve been in tech for over 15–20 years. I could finally look up to someone and say — yes, that’s where I see myself in 15 years — doing amazing work and building great products. It was a memorable experience in more than one way.

Lastly, some people who’ve really inspired me this year:

Sara Mauskopf, you’ve you been a role model for me. Thank you for sharing your life with us. You’re absolutely badass in every possible way. I’m and will always be in complete awe of how you balance work and family. I’m sure you give a lot of hope to folks who want to do the same.

Julia Evans, I’m in awe of you and how you think! I love your zines and you’ve’ inspired me to draw some of my own. Your curiosity about systems programming is greatly inspiring as well. Thank you for sharing your work and how your approach problems with us. :)

Finally, a shout out to Joe Nash & Helen for just being amazing people. Also, thank you for being my zine beta-testers, haha! ;)

Wrapping this year on a positive note, I really hope to learn even more next year. Here’s hoping for a great 2017 for everyone!

Mozilla’s Silicon Valley headquarters. Photo Bernard Andre

I recently joined Mozilla’s Outreachy program as an intern. If you haven’t heard of this program before, let me start by answering some basic questions.

What is Outreachy?

Outreachy is a program for folks from underrepresented groups. It aims to getting them more involved with open source through remote internships.

Organizations like Mozilla, Wikimedia, Gnome, and Linux Kernel take part in the program. They hire interns to work on their projects.

It’s kind of like Google Summer of Code, except that both organizations and corporate sponsors both help pay interns.

Another major difference is that it happens twice a year. There are both summer and winter cohorts. Participation isn’t limited to just students like Google Summer of Code, either.

Every intern works on a project for an organization under the supervision of a mentor. The duration of the program is three months.

You can read the full details of the program here.

I’m working for Mozilla on web-ext, a command line utility to help with the development of WebExtensions. WebExtensions are cross platform addons, the future of browser addons. They can run in browsers like Firefox, Chrome, and Opera with only minor changes.

How did I get involved with Outreachy?

My involvement with Outreachy has got a lot to do with how I started contributing to open source. I wrote all about this in a previous story.

A Beginner’s Very Bumpy Journey Through The World of Open Source
_Did you land on this story looking for advice on how to start contributing to open source? There are tons of these…_medium.freecodecamp.com[](https://medium.freecodecamp.com/a-beginners-very-bumpy-journey-through-the-world-of-open-source-4d108d540b39)

I had long known about both Outreachy and Google Summer of Code, and I’d always wanted to take part. But thought I wouldn’t stand a chance. I doubted why anyone would pay me to work on their project. I didn’t have a lot of prior experience. Nor could I prove that I had a decent level of expertise in the technologies I was working with.

If you’re in the same boat, all I’d like to say to you is: stop. Don’t be so hard on yourself.

It may surprise you to know how I actually applied and got selected for Outreachy. It might as well be labelled an accident.

I had been contributing to the web-ext tool for about a month or so. I happened to ping one of the Outreachy interns from the summer cohort of 2016, Anjana Vakil, to ask for advice on how to prepare and apply for Outreachy.

This was quite sometime before the organizations were announced, let alone projects. I wanted to start preparing as early as possible to maximize my chances of getting in. So I told her about my background and involvement with Mozilla. She told me if I had a project in mind that I’d like to work on during Outreachy, I shouldn’t hesitate in talking to the maintainers. I should definitely ask them if it’ll be a viable fit for Outreachy.

I had been contributing to web-ext for a month or so by then while juggling my usual college and having a great time. After sitting on that thought for quite sometime, I finally mustered enough courage. I asked Kumar McMillan, my mentor, if web-ext could use some more help and time over a span of three months. And he seemed delighted by the idea!

In the meanwhile, I continued contributing to web-ext.

If not for Anjana, the idea of proposing a project I’d like to work on wouldn’t have ever crossed my mind. A huge thanks to her! :)

The next step was waiting for the announcement of the organizations and their projects. I spoke to my mentor and prioritized the tasks that were to be accomplished, then submitted an application. Then I just waited for the announcement of the results on the Outreachy page. And one day I looked there and saw my name. I had made it.

A lot of you might be wondering at this point if you need to start as early I did to get selected. The answer is no. When I started contributing to web-ext, Outreachy wasn’t anywhere in my mind. I picked it because it seemed approachable to me. The maintainers were extremely kind, emphatic and ready to help me contribute in anyway possible. A huge shout out to Kumar McMillan and Luca Greco — you guys are amazing! :)

What do you get out of Outreachy?

1. Learning:

This is the primary reason why I’d encourage anyone eligible to take part in Outreachy. The amount of learning involved is invaluable. You get to work with some of the smartest people out there on real world projects which people use. You learn valuable software engineering skills that you’ll use in your day to day job as a developer.

As is often the case with people trying to contribute to OSS, you won’t get lost midway because of lack of help/direction. Your mentor is there to help you. You learn how to work with remote teams which are spread geographically over the world.

2. Money:

Outreachy isn’t just limited to students and they expect to work full time for 40 hours/week. The money can serve as an incentive to folks who’re older looking for a career change or getting more involved with open source.

Each intern gets $5,500 USD, and an extra $500 USD travel grant to use within a year of their internship period.

3. Confidence:

Working on a real world project which is used by people gives a lot of confidence in your skills and your work as a developer.

A lot of past interns from these programs who were complete newbies at the start of it have gone to help other newcomers get started with open source.

Outreachy helps lay a foundation for your involvement in open source as it gives you credibility and experience.

4. Mozilla specific perks

All Mozilla interns get LDAP credentials. This means you get a @mozilla.com e-mail address like all regular employees. You get to take part in team meetings and have access to pretty much everything.

All interns get a laptop to keep. What you get varies from year to year. This year they gave a 15” Macbook Pro retina 2015 model to all the interns.

In the past, interns were invited to attend All-Hands, the Mozilla company work week, which happens twice every year in various parts of the world.

I’m super excited about Outreachy.

All in all, I’m super excited to be working with Mozilla on the web-ext project. It’s a really exciting time to be working on tooling for WebExtensions as they’re going to be the future going forward.

Last but not the least, a huge shout out to everyone who helps in making Outreachy a reality: Larissa Shapiro and Elizabeth Noonan at Mozilla, and Marina, Sarah Sharp, and other Outreachy admins as well.

If you’re thinking of applying for the next round of Outreachy but don’t feel confident enough or have doubts about navigating the application process, feel free to drop me a line. I’d love to chat with you. :)

It’s hard to attract contributors to your FOSS project — especially contributors who are new to open source.

Project maintainers often discuss making their open source projects more beginner-friendly, so they they can attract new contributors and make them feel welcome. But this dialogue often gets trapped in the echo chamber of maintainers themselves. Instead, it needs to involve the other party: the new contributors themselves.

During my short time as an open source contributor, I’ve browsed through tons of projects, and contributed to a few of them as well. In this post, I’ll try to list what appealed to me as a new contributor, and what didn’t. And I’ll use a few projects as case studies.

Tip #1: Label beginner issues appropriately

This is definitely the most important consideration to me as a potential contributor.

Labels make it easier to find issues that can serve as a beach head for a first contribution. If your project doesn’t have these, it significantly raises the bar for contributing to it.

It is very difficult for someone unfamiliar with your codebase to gauge the difficulty of an issue. So a generic label like help wanted isn’t useful enough on its own. Try using more specific labels, such as good first bug, easy, low hanging fruit, etc. to communicate that an issue is easy enough for an initial contribution.

Tip #2: Establish proper contributing guidelines

A well-written Contributing.md can describe the workflow your project’s maintainers expected contributors to follow. This is easy to document, and will save both sides a significant amount of time.

Specify whether the contributor is required to work on a separate branch for each issue, or to squash their commits and rebase their changes before submitting a PR. Try to link to an appropriate tutorials for each of these processes in case the contributor isn’t yet familiar with them.

Tip #3: Document your project’s design, architecture, and directory structure

Having a document that gives a high-level overview of the design and architecture of your project can save both parties a lot of time.

Instead of explaining the same thing over and over again to every new contributor, take note of questions that are frequently and create an FAQ right there on your README.md.

The steepest barrier for most new contributors is navigating dense codebases. Help newcomers find their way by offering a description of your folder structure.

A lot new contributors are junior developers who may not yet know a lot of common architecture and design patters. Create documents that highlight these decisions and the reasoning behind them.

Tip #4: Put in place a clear Code of Conduct

Create a proper Code of Conduct that explicitly states rules within your FOSS community. A lot of new contributors, including me, are scared of being treated badly, or looked down upon while trying to contribute.

Your Code of Conduct that clarifies how to report violations can help folks feel safe. This also entails calling out bad behavior at every step, irrespective of the stature of the person involved, and taking appropriate action.

Tip #5: Create templates for pull requests and issues

A proper issue template can help people better describe the environment required to reproduce a bug. Contributors can then start working on the issue immediately, without needing to gather information from disparate places. This saves time for both parties.

A similar argument can be made for pull request templates. Clearly lay out what’s expected when a contributor submits a PR, including the format of their commit message, test plan, the changes made. This will help with code review as well, saving everyone even more time.

Tip #6: Prioritize responding to pull requests

Being the maintainer of a popular FOSS project is an incredible amount of work. Most people don’t get paid to contribute to FOSS — maintainers and contributors alike. Most maintainers don’t have enough time to review all PRs with the same amount of scrutiny.

Prioritize PRs so contributors can understand beforehand whether or not they can expect to receive feedback for a low priority/easy bug fix.

Tip #7: Welcome all kinds of contributions

Our field has a nasty tendency to look down on non-code translations. Please don’t let your project fall prey to this mentality.

Encourage all kinds of contributions, be they documentation, code, fixing typos, tests — anything at all.

Tip #8: Reward new contributors

If you have the budget, reward new contributors by sending them swag such as stickers or shirts.

If your budget doesn’t permit that, then a simple shoutout or mention in a blogpost or on social media can also go a long way. It’ll make them realize that their contributions — no matter big or small — aren’t overlooked, and that they are valued.

This establishes a feeling of belonging that may inspire them to contribute more.

Kent C. Dodds made a nifty open source specification for this: All Contributors.

A good start can be maintaining a list of contributors on your project website and/or repository.

Lastly, it never hurts to remember that everyone was once a junior developer. At some point, they needed help to get to the level they’re at today.

This simple fact can go a long way in establishing a healthy culture of mentorship within your open source organization. Treating everyone with respect — irrespective of their background — is critical to running a successful open source project.

I tweeted the above a while back, when I was just getting started with contributing to FOSS. But since then, a lot has changed. I’ve finally concluded:

Open source is way more about people than it’ll ever be about code.

P.S. I am trying to create a list of beginner-friendly FOSS projects. If your open source project has any material relevant to new contributors, please consider opening an issue on this repository.

A shout out to Dan Abramov, Kent C. Dodds and Quincy Larson for helping me with this piece by providing their perspectives as maintainers. 😄

If you have any further tips to add to this post, feel free to reach out to me or reply below.

And if you found it useful, please tap or click “︎❤” to help to promote this piece to others.

A few days ago, ashley williams, one of the leaders of the Node.js community, tweeted this:

I didn’t really understand what she meant, so I decided to dig in deeper and read about how package managers work.

This was right when the newest kid on the JavaScript package manager block — Yarn — had just arrived and was generating a lot of buzz.

So I used this opportunity to also understand how and why Yarn does things differently from npm.

I had so much fun researching this. I wish I’d done so a long time ago. So I wrote this simple introduction to npm and Yarn to share what I’ve learned.

Let’s start with some definitions:

What is a package?

A package is a reusable piece of software which can be downloaded from a global registry into a developer’s local environment. Each package may or may not depend on other packages.

What is a Package Manager?

Simply put — a package manager is a piece of software that lets you manage the dependencies (external code written by you or someone else) that your project needs to work correctly.

Most package managers juggle the following pieces of your project:

Project Code

This is the code of your project for which you need to manage various dependencies. Typically, all of this code is checked into a version control system like Git.

Manifest file

This is a file that keeps track of all your dependencies (the packages to be managed). It also contains other metadata about your project. In the JavaScript world, this file is your [package.json](https://docs.npmjs.com/files/package.json)

Dependency code

This code constitutes your dependencies. It shouldn’t be mutated during the lifetime of your application, and should be accessible by your project code in memory when it’s needed.

Lock file

This file is written automatically by the package manager itself. It contains all the information needed to reproduce the full dependency source tree. It contains information about each of your project’s dependencies, along with their respective versions.

It’s worth pointing out at this point that Yarn uses a lockfile, while npm doesn’t. We’ll talk about the consequences of this distinction in a bit.

Now that I’ve introduced you to the parts of a package manager, let’s discuss dependencies themselves.

Flat versus Nested Dependencies

To understand the difference between the Flat versus Nested dependency schemes, let’s try visualizing a dependency graph of dependencies in your project.

It’s important to keep in mind that the dependencies your project depends on might have dependencies of their own. And these dependencies may in turn have some dependencies in common.

To make this clear, let’s say our application depends on dependencies A, B and C, and C depends on A.

Flat Dependencies

Dependency graph in case of flat dependencies

As shown in the image, both the app and C have A as their dependency. For dependency resolution in a flat dependency scheme, there is only one layer of dependencies that your package manager needs to traverse.

Long story short — you can have only one version of a particular package in your source tree, as there is one common namespace for all your dependencies.

Suppose that package A is upgraded to version 2.0. If your app is compatible with version 2.0, but package C isn’t, then we need two versions of package A in order to make our app work correctly. This is known an Dependency Hell.

Nested Dependencies

Dependency graph in case of nested dependencies

One simple solution to deal with the problem of Dependency Hell is to have two different versions of package A — version 1.0 and version 2.0.

This is where nested dependencies come into play. In case of nested dependencies, every dependency can isolate its own dependencies from other dependencies, in a different namespace.

The package manager needs to traverse multiple levels for dependency resolution.

We can have several copies of a single dependency in such a scheme.

But as you might have guessed, this leads to a few problems too. What if we add another package — package D — and it also depends on version 1.0 of package A?

So with this scheme, we can end up with duplication of version 1.0 of package A. This can cause confusion, and takes up unnecessary disk space.

One solution to the above problem is to have two versions of package A, v1.0 and v2.0, but only one copy of v1.0 in order to avoid unnecessary duplication. This is the approach taken by npm v3, which reduces the time taken to traverse the dependency tree considerably.

As ashley williams explains, npm v2 installs dependencies in a nested manner. That’s why npm v3 is considerably faster by comparison.

Determinism vs Non-determinism

Another important concept in package managers is that of determinism. In the context of the JavaScript ecosystem, determinism means that all computers with a given package.json file will all have the exact same source tree of dependencies installed on them in their node_modules folder.

But with a non-deterministic package manager, this isn’t guaranteed. Even if you have the exact same package.json on two different computers, the layout of your node_modules may differ between them.

Determinism is desirable. It helps you avoid “worked on my machine but it broke when we deployed it” issues, which arise when you have different node_modules on different computers.

This popular developer meme illustrates the problems with non-determinism.

npm v3, by default has non-deterministic installs and offers a shrinkwrap feature to make installs deterministic. This writes all the packages on the disk to a lockfile, along with their respective versions.

Yarn offers deterministic installs because it uses a lockfile to lockdown all the dependencies recursively at the application level. So if package A depends on v1.0 of package C, and package B depends on v2.0 of package A, both of them will be written to the lockfile separately.

When you know the exact versions of the dependencies you’re working with, you can easily reproduce builds, then track down and isolate bugs.

“To make it more clear, your package.json states “what I want” for the project whereas your lockfile says “what I had” in terms of dependencies. — Dan Abramov

So now we can return to the original question that started me on this learning spree in the first place: Why is it considered a good practice to have lockfiles for applications, but not for libraries?

The main reason is that you actually deploy applications. So you need to have deterministic dependencies that lead to reproducible builds in different environments — testing, staging, and production.

But the same isn’t true for libraries. Libraries aren’t deployed. They’re used to build other libraries, or in application themselves. Libraries need to be flexible so that they can maximize compatibility.

If we had a lockfile for each dependency (library) that we used in an application, and the application was forced to respect these lockfiles, it would be impossible to get anywhere close to a flat dependency structure we talked about earlier, with the semantic versioning flexibility, which is the best case scenario for dependency resolution.

Here’s why: if your application has to recursively honor the lockfiles of all your dependencies, there would be version conflicts all over the place — even in relatively small projects. This would cause a large amount of unavoidable duplication due to semantic versioning.

This is not to say that libraries can’t have lockfiles. They certainly can. But the main takeaway is that package managers like Yarn and npm — which consume these libraries — will not respect those lockfiles.

When you’re new to open source, you’ll find yourself asking:

I know some [programming language]. I want to get some practice, while helping out. How do I find an open source project where I can contribute? Hm… I don’t know where to start. This seems complicated.

I’ve asked this same question over and over to a lot of developers. And their answers can be categorized as one of three approaches:

Approach #1: Contribute to something you love

The most common answer I get is to contribute to something you already use everyday. Something that interests you.

Approach #2: Specifically seek out beginner-friendly projects

Here are a few characteristics of beginner-friendly open source projects:

• Well-defined, detailed contribution guidelines that include setting up their project locally, their Git workflow, and their coding style guidelines
• Proper classification of issues using labels like “good-first-bug”, “beginner”, or “first-timers-only”
• Activity on those beginner issues, with previous questions answered quickly

Approach #3: Stop searching for projects and start searching for bugs.

This is the approach I chose, and the focus of this article.

After trying approaches #1 and #2, I stopped thinking in terms of projects. I focused instead on finding bugs that I thought I could fix.

Every bug is associated with a project, so when finding bugs, you’ll inevitably discover projects, anyway.

This approach works if you want to get started immediately. I can’t guarantee that it will inspire you to stick with a project after your first few contributions. Maybe you won’t be interested after all. But maybe you’ll dive into the project and discover that you really like it.

Either way, once you’ve fixed a few bugs, you’ll have the confidence to venture out there and explore more on your own.

So how do you find the bugs to begin with?

Deciding which bugs to work on isn’t easy. There are a ton of projects out there, and each has plenty of open issues. But you need to start somewhere.

So I’ll share all the resources and tips I’ve used to find bugs. First I’ll focus on finding good starter bugs in general in various bug trackers and code hosting sites. Then I’ll share some resources specific to the Mozilla ecosystem, where I’ve been contributing regularly.

Finding good bugs for beginners

A good place to start your bug hunt is Up For Grabs. The whole purpose of the site is to help new contributors get their feet wet by maintaining a list of projects with beginner-friendly issues. It’s a great place to get started if you feel completely lost.

GitHub has a powerful search engine where you can customize your search in a variety of ways. The easiest way to search is by issue label.

A lot of open source projects label their issues to conveniently track them. A lot of projects use labels like beginner, easy, starter, good first bug, low hanging fruit, bitesize, trivial, easy fix, and new contributor.

You can further narrow down your search based on the programming language you’re comfortable with, by adding language: name to your search query. For example, here are all issues labelled as “beginner” in JavaScript.

Issuehub.io is a tool for searching issues by label and language, in case you find it tedious to remember the GitHub search syntax.

If you’re completely new to open source, you should definitely start with First Timers Only. It’s an initiative by Kent C. Dodds, based on his own First Timers Only post and Scott Hanselman’s Bring Kindness Back to Open Source. The bugs are labelled first-timers-only.

You might also find this Twitter bot helpful. It tweets out all issues labelled as “first-timers-only”.

Another great way to find issues is YourFirstPR by Charlotte Spencer. They showcase starter issues on GitHub that can be easily tackled by new contributors.

Awesome-for-beginners is a GitHub repo that amasses projects with good bugs for new contributors, and applies labels to describe them.

Openhatch is a non-profit organization that helps lower barriers of entry into open source. You can find bugs and projects here, as well.

The Mozilla Contributor Ecosystem

A lot of Mozilla’s projects are hosted on GitHub. For these projects, everything I listed above is still useful. They use the label “good first bug” for starter issues.

But Mozilla also uses its own tool called Bugzilla as its primary issue tracker. They host some of their issues here, and use Mercurial for version control instead of Git.

Firefox is one of the projects that uses Bugzilla and Mercurial. It’s a bit scary, to be honest. It’s a lot to take in. So I recommend this excellent blog post and video, which does a great job at demystifying these tools.

Over the years, Mozillians have tried to make it as simple as possible to contribute to Mozilla. Here are their efforts:

• Good First Bugs: These are bugs that developers have identified as a good introduction to the project. They are often (but not always) relatively easy to solve
• Mentored Bugs: These bugs have a mentor assigned who will be there on IRC to help you when you get stuck while working on fix. They often review your patch and give feedback. If you don’t know where to begin with contributing to Mozilla projects, this is the best place to start. You’ll have someone who can answer your questions when you feel you’ve run up against a wall. All the mentors I’ve worked with have been super responsive, supportive, and helpful throughout.
• Bugs Ahoy: This is a site dedicated to finding bugs on Bugzilla. It has a friendly UI, where you can filter by language.
• Firefox DevTools: This site is dedicated to bugs filed for the developer tools in the Firefox browser. You can sort based on the DevTools components you want to work on.
• What Can I Do For Mozilla — This is a great way to explore and figure out what you can work on by answering a bunch of questions about your skill set and interests.
• Start Mozilla: This is a Twitter account that tweets about issues fit for contributors new to the Mozilla ecosystem.

If you know of any other resources for finding good bugs for newbie contributors, please let me know in the comments. I will be more than happy to extend this list.

Image credit

As a newbie open source contributor, I often felt lost and dejected. I couldn’t figure out how different modules fit. I couldn’t find my way around huge codebases.

A lot of us are in this same boat. And it’s okay to be there.

I stuck with it. Eventually, things started to make sense. Project maintainers started accepting my pull requests. And my confidence rebounded.

I wrote about my experience hoping to encourage other newbie open source contributors. And was blown away by the attention and positive responses my article received.

So many people reached out to me saying my post encouraged them to start (or resume) contributing to open source. A few maintainers also went back and reviewed some pull requests I’d previously submitted. I couldn’t have asked for a better response.

After some thought, I’ve concluded that the main reason my article triggered such a big response was because the dialogue around contributing to open source has been one-sided for a long time.

Maintainers often write posts about how new contributors can get involved in their projects. They share all their efforts to make their project beginner-friendly. They also write tons of guides on how to start contributing, and answer questions on Quora and other platforms.

But there aren’t as many posts by newcomers on how things pan out once they start contributing. I’ve gone through many of those guides myself, but they aren’t as helpful as a fellow newbie contributor’s firsthand experience.

It’s difficult to strike a balance when only one party is communicating. I’d like to hear more about which projects newbies contribute to, the path they follow, what they expected VS what came out of it.

Thus, I want to encourage all new contributors to document their experience. Help us establish this balance. I assure you it’s worth your time and patience.

This will help you in a number of ways. When you’re writing, you can’t brush details under the carpet. Writing forces you to perceive things as they are, and not how you think they are. Writing also puts things into perspective. It helps you remember where you started, and how far you’ve come.

You may worry that you don’t know enough to write about a particular topic. I’m here to tell you that you don’t have to be an expert on a topic to write about it. Write what you know.

The worst thing that can happen is that you get something wrong. And if you do, someone will probably point it out to you. And this will help you fill in the gaps in your knowledge. It’s a win-win situation.

Finding Inspiration in Other Developers’ Blogs

I have a few drafts that I’m waiting to publish, because they aren’t good enough just yet. They can be more polished, more technical.

When I feel this way about a draft, I surf the internet for inspiration. One day when I was doing this, I stumbled across Write An Excellent Programming Blog by A. Jesse Jiryu Davis. I keep going back to it and re-reading it when I feel like I don’t have a topic to write about, or I don’t feel qualified to write about what I know.

Another place I frequently visit looking for inspiration is Julia Evans’ blog. Her posts are short, simple, and a joy to read. I learn something from almost every post.

And I occasionally end up at the Stack Overflow co-founders blogs = Joel On Software and Coding Horror. They have great articles on all sorts of things related to tech.

Once you start looking around for inspiration, you’ll find lots of it. Developers publish thoughtful articles on various topics every day.

How to Find Topics to Write About

It’s OK if you don’t yet feel confident enough to start writing on technical topics. You’ll get there eventually. Start by just documenting your journey =

• How you got involved with the organization you’re contributing to
• What kind of community they have and how welcoming they have been to new contributors
• The project you chose to work on, and what went into making that decision
• How difficult it was to set up your local development environment and clone their repo, where you got stuck, and how you cleared that hurdle
• How you found your first issue to work on
• If you had a mentor assigned, how your experience was working with them, and the kind of help you received
• What you expected open source to be like, versus how it actually turned out
• Any advice you’d give fellow newbie contributors once you’ve gotten your feet sufficiently wet
• How to behave, and what kinds of questions to ask in a organization’s IRC channel or Gitter room

These are just some topics I can think of off the top of my head. There are undoubtedly many more aspects of contributing to open source that you can come up with and write about.

You don’t have to write daily or weekly, or follow any strict time-bound schedule at all. Write when you feel like you have something worth telling others — be it a small victory or a large contribution.

As you gain more experience, you can begin to write on more technical topics. These can be something you’ve worked on for a long time. They can also just be your journey to acquire knowledge of a new language/framework/library.

At this point, you might wonder whether too much has been written about a particular topic already. You might ask what more you could possibly add to it that hasn’t already been said?

Well, it doesn’t matter how much has been written or by whom. That’s never stopped any determined writer from writing yet another article from their own viewpoint, based on their own understanding.

Everything you write offers your own perspective, which may greatly differ from another person’s. It’s all in the details = how you break a topic down, how you approach it. So don’t let the fact that there’s already been a lot written about a particular topic deter you from writing about it yourself.

Your mission, if you choose to accept it = document your experience as you contribute to open source. Write about concepts as you learn them. As a result, you’ll learn faster and become a better developer.

You’ll also help other lost, intimidated newbie contributors find their way through the vast world of open source software.

Believe me, nothing feels better than knowing that something you wrote helped a person who is struggling.

If you have any additional ideas for what newbie open source contributors can write about, or have a favorite programming blog you visit often for inspiration, please let me know in the comments.  =)

Me running away from the world of open source software

Did you land on this story looking for advice on how to start contributing to open source? There are tons of these stories on the inter-webs, aren’t there?

And I am sure you must have read a lot of them by now, because you’ve been trying to start contributing for quite sometime. And you still feel like you’ve not progressed at all.

I get that feeling. I was in the exact same position until a few weeks ago. Let me tell you my story.

I’ve been trying to contribute to open source for about two years now.

Yes. Two years.

And there is one thing I can tell you with a lot of certainty — it’s intimidating. It’s tough to get started. You have to learn how to work within a large code base. You have to learn and adhere to a project’s coding style guides.

Nothing makes sense. The control flow, how different modules interact, how and why the code is organized the way it is — it’s all one big maze.

I feel like this all the time because I am, after all, just an amateur trying to learn as much as I can.

So I tried to take the easy way out: fixing typos in docs or comments in the code, doing trivial bug fixes because it was obvious what needed to be changed and where. I didn’t want to ask too many questions or try to understand the code base.

Every time I wanted to contribute, I went to Github — or any bug tracker — and tried searching for issues with labels “easy”, “beginner”, “good first bug”, “low hanging fruit.” After sifting through hundreds of them, I found something trivial enough to do without any major external help.

Now, this worked for a while, until I realized that I could make better use of the skill set I was building up. I’d learned so many new things, but I couldn’t figure out where to apply them. Just learning — without applying — counts for very little. I was stuck on a plateau, and I wasn’t moving forward at all.

Then something happened that made me terribly scared about being a new contributor, trying to wade my way through the world of open source. I picked out an issue that seemed easy enough from a large, popular project.

I thought it would be better to ask clarifying questions before making any changes for fear of messing up. So I posted a comment saying that I was a new contributor, and asking how a particular piece of text should be altered so as to close the issue.

The reply I got was: “If you can’t figure out how to make the change, you’re not qualified to make the change.” This response baffled me, and made me even more scared to ask questions when I didn’t understand something about a project.

Maybe I wasn’t wanted because I didn’t know enough? Maybe I needed to work more on my skills instead of asking stupid, lame questions to experienced people who’re super busy?

This is when my quest for a mentor began as well. I felt that maybe if I knew someone with whom I was comfortable asking questions, things would be OK and I can make myself more useful.

So I emailed a bunch of people, asking them to help me get started since I was feeling particularly intimated after the aforementioned experience. I received a lot of positive replies full of encouragement, but I still didn’t find exactly what I was looking for.

I felt like I was bumping into a closed environment in the world of open source.

Everything seemed to suggest that I just put myself out there and not be scared. But I just wasn’t up for this at the time.

Discovering Mozilla

I landed on this Mozilla project which helps you test web extensions one fine evening searching for issues to work on. I was glad to find a few issues labelled as “good first bugs”, but none of them were as simple as fixing a small typo.

Boy, I am so glad about that right now.

I started working on one of them, but quickly realized that I had to ask questions if I wanted to be able to close the issue. I skimmed through the code base. Once I had some sense of what was going on, I asked for more direction. And voilà! I was able to solve the issue after getting all the relevant details I needed.

Now that I’ve opened three pull requests — one merged, and the other two on their way to being merged — I’m glad I took the plunge. I’m glad I didn’t back down when it came time to ask relevant questions, even if it meant risking looking stupid sometimes.

It’s okay to not know everything, and take one step at a time to learn something new.

The folks at Mozilla mentoring these issues have been nothing less than super helpful and supportive. They’ve guided me all the way, taking time out to break things down and explain them in incredible detail. This is despite the fact that they could’ve fixed these issues themselves in a few hours instead of spending time mentoring me toward a solution of my own over the course of several days.

I’ve learnt and discovered so many new things by working on just three basic issues. And I’m super excited to work on even more challenging bugs and expand my understanding and knowledge.

I can’t thank them enough for being such a positive experience, because it led to setting up Firefox locally and scouring for bugs on Bugzilla every other day. (I’m saving the how’s and why’s for a more in-depth post).

I plan to continue contributing to Mozilla as regularly as possible. Every time I’ve asked a relevant question — be it on IRC, Github or Bugzilla — I’ve gotten a very friendly response.

As of now, I’ve closed three issues in web-ext, and gotten a patch accepted and landed for Firefox.

My contributions were noticed by the community, and I made it into the Addons Contribution Recognition document, as well.

All in all, my experience over the past few weeks has been nothing short of wonderful. I’ve learned so many things, big and small, that no engineering textbook can ever teach me.

My advice to fellow beginning developers who want to contribute to open source:

Tip #1: Don’t be afraid to ask questions.

I can’t stress on this enough. I lost a lot of time because I kept holding myself back, and this was my biggest inhibition.

Everyone is scared of looking stupid. But don’t let that crippling fear result in stunted growth.

It’s okay to ask if you don’t understand something related to the project. The maintainers of the project have been well-versed in the project for years. They can help you fairly quickly. The alternative is you spending hours lost in their codebase trying to figure something out that you’re not even supposed to know in the first place.

But be careful about asking for information that is already readily available to you via some documentation or Google searches. You want to be respectful of project maintainers’ time.

Tip #2: It’s OK to have holes in your knowledge.

You’re not expected to know everything in and out if you want to start contributing to a project. The idea is to learn and grow as you start solving harder issues becoming more familiar with the project and the tooling they use. The time it takes for this to happen varies from project to project and person to person.

Tip #3: Just start

Don’t waste a lot of time choosing the “right” project. If you know a project or a organization with a beginner-friendly community, just start there.

Find an issue you’re comfortable taking up — preferably in a language you’ve been working with for a while — and try to figure out what needs to be done. Ask for the relevant information to fill the gaps in your knowledge, then get to work. Don’t wait.

Thank you, open source maintainers!

A huge shout-out to all the open source maintainers who have been super responsive and encourage of new contributors. You are helping newcomers navigate huge code bases and contribute in maybe a small yet meaningful ways. Your efforts are truly appreciated and needed.

As a newbie and a junior developer, I’m just trying to find my way through this vast and incredible tech industry. A few minutes of your mentorship — be it being introducing to a simple debugging technique or showing me how to write proper test cases — will help me become a better developer in the long run.

You have the experience, and I have the incessant desire to learn as much as I can.

A special thanks to Guido, Kumar McMillan, and Luca for being amazing mentors throughout this journey, following up every time, and answering my various questions. I really appreciate all the time and effort you’ve invested in me. :)

If you’re a fellow struggling newcomer to open source, I’d like to hear about your experience and story. If I can help you in any way, please don’t hesitate to reach out!

I plan to document my journey of contributing to open source, so if there is anything in particular you’d like me to write about, please drop a comment.

Thanks to Pawan Dubey and Quincy Larson for helping me refine this article.