Tales from the Git Keeper: It’s My Party and I Can Uninvite You If I Want To

choose-the-high-road-my-dearshit-rolls-downhill-c2146
via https://www.someecards.com/

Background

I had told my manager and his manager (my skip manager*) that I was quitting. I gave them a date and they told me to sit tight while they hash out the details with HR. I gave them a date 1 month out and described the work I intended to complete during that time. I was anxiously awaiting their verdict. I got the impression they might require that I leave immediately, which would incur a significant financial penalty to me due to the terms of the work contract.

Meanwhile, the team brain trust (my skip manager and a senior engineer) had been planning an all day team offsite to craft our road map (i.e. be told what we should do). This was to build hype for our technical challenges and build team cohesion. Also, there would be free lunch.

Timeline: I told my managers I was leaving on Friday. The team offsite was on the following Tuesday. We were asked for our lunch selection on the previous Wednesday.

Conversation 1: Direct Manager

Timeline: Monday, the day of nail-biting anxiety over whether I’d be asked to leave immediately

Manager: We had the discussion with Skip Manager and HR.

Me: And…?

Manager: Well, you can leave whenever you want and you will keep getting paid until next month. You won’t have to pay back the $X for breaking the contract.

Me: That’s amazing! I didn’t even realize that was an option.

Manager: I was surprised to. About the offsite though: I think Skip Manager is going to talk to you and encourage you not to go.

Me: I guess that’s his decision but why? Wouldn’t that be unsettling to the rest of the team?

Manager: Well, he says it’s to give you an extra day off since you won’t be involved in it but it might be because he doesn’t want your interference. He’s very concerned about any morale impact it might have on the team.

Me: Okay… so, what is he going to tell the team? And wouldn’t be suddenly not being there be more of a morale impact?

Manager: We’re not going to tell the team you are leaving until next week so I don’t know what he’ll tell them.

Me: Overall this is good though that offsite thing is weird. Thank you for your help.

That went really well for the quitting part. I personally felt I would be able to help the junior engineers on my team with brainstorming during the offsite but it’s not my circus anymore. Still, Skip Manager’s choice seemed more morale damaging than having me there.

Conversation 2: Skip Manager

Timeline: Later Monday

Skip Manager: As you know, we’ve discussed your departure.

Me: Yes, Manager mentioned that.

Skip Manager: Glad to hear that. Anyway, about the offsite: why don’t you take that day off? It’s not going to be relevant for you after all.

Me: If you think that’s best. I would be happy to contribute my experience from past work or not if you’d prefer.

Skip Manager: Great, so what will you tell the team why you aren’t going?

Me: Um… I could say I’m sick?

Skip Manager: No, I don’t want you to lie.

Me: Er… okay… I have to go see the doctor and since I will have the time I can bump my appointment up.

Skip Manger: Great, go with that. Bye.

Recap: Skip Manager gives me less than 24 hours to come up with a reason why I suddenly can’t come to work tomorrow that isn’t a lie because he decided I shouldn’t go to his stupid planning party. Oh, and make sure it doesn’t give away that you’re leaving. And who’s going to eat my BLTA?

Conversation 3: The White Lie

Timeline: Tuesday, day of The Offsite

Dear Team,

I just found out that there’s an opening at my podiatrist’s office tomorrow and I’ve been waiting a few weeks to see him. Since it’s midday and may take a while to do x-rays, I will have to miss the offsite. Have fun and I look forward to the great ideas coming out of it!

Alacritical

This wasn’t a complete lie since I did have to go to the podiatrist to get fitted for some foot related problem. They weren’t sure if they’d need to x-ray. It didn’t have to be Tuesday though. It also took 30 min. Still not a complete lie.

Conversation 4: Human Resources Lady

Timeline: The Friday after The Offsite

Me: Thank you for all your help with negotiating me leaving the company!

HR Lady: No problem, I was happy to help… by the way, how did the conversation go with your managers?

Me: Well, Manager was pretty clear about it and made sure I had some choice in when I left. I wanted to stay a bit to make sure I didn’t leave anyone with garbage to deal with.

HR Lady: Yes, he mentioned he expected that from you. What about Skip Mansger?

Me: Manager gave me a heads up but Skip Manager asked me not to go to a team offsite, which is fine, but I really didn’t like that he asked me to make up a reason the day before why I couldn’t go.

HR Lady: *tsk*, I told him I recommended he not do that. He asked you to come up with the reason? *shakes head*

Me: He also asked me not to lie and I managed to come up with something reasonably truthful but I didn’t like cleaning up his mess. Still, if that’s the worst of it, I’m happy with how this turned out.

HR Lady: I’m glad it worked out and feel free to reach out if you need anything.

Even HR thinks Skip Manager is a little crazy. That makes 3 of us (HR, Manager, and myself). By the way, I gave the HR Lady a gift basket to show my appreciation. I like to reinforce good behavior.

Things I didn’t do that were suggested:

  • Instead of making up a white lie, tell my coworkers that I’m leaving and Skip Manager banned me from the offsite
  • Send the “can’t make it” email but attend anyway
  • Refuse to not attend

Since I had been generously offered pay without having to work, I considered being the scapegoat for Skip Manager’s convenience a fair trade-off.

What did I learn from this?

  • Managers don’t have to listen to HR and HR gets irritated about that too
  • Sometimes managers can’t find a good rationalization so they make you do it
  • I will never know who got my BLTA

*A skip manager refers to your managers manager. Also known as: senior manager, grandfather manager

Title reference

Exit to Stage Left: Thoughts on Quitting

Is quitting the easy way out? I think quitting can be hard and more effort than sticking with something that’s just okay or tolerable. There’s a lot of stress and formalities to go through to quit a job and even more so if you want to stay on good terms. Here I’ll talk about different ways to quit a job or team.

Clarification: For the purposes of this post, by “quit” or “quitting” I mean the voluntary resignation or termination of employment by the employed individual. I am not talking about lay-offs or other involuntary termination of employment.

Stages of Quitting

I think of different stages of quitting as the different points at which I can engage my current employer in discussions:

  1. I’m thinking of quitting or am interviewing for other companies.
  2. I’ve decided to quit but have no set date on when my last day is.
  3. I know the rough day I want to quit and I’ve started casually removing personal items from the office.
  4. It’s the day which is the minimum amount of notice my company requires before quitting.
  5. It’s my last day.
  6. I’m gone.

When you start talking to your employer about quitting depends on your relationship with your direct manager, your company policy on resignation, and the terms of your contract. I have primarily worked at “at-will employment” shops so, technically, I could email my manager from my bed and say I’m never coming in to work again. I have never been in a situation where that has needed to happen but I always check to make sure it’s an option.

Factors of quitting

  1. Your finances, health care, next job, etc.
  2. Your manager/employer
  3. Your coworkers
  4. Your HR department, if you have one
  5. Your legal obligations

Each of these factors can impact when you quit, what your life is like before you quit, how much notice you are obligated to give, and how you feel about quitting. When making a plan to quit, I would recommend assessing the risk presented by each of these factors. Any rules you fail to follow can compromise legal contracts and may result in financial consequences. Any human involved may become unreasonably emotional or you may find yourself needing to defend your decision to quit against unexpectedly intense interrogation. On the other hand, your managers could see that you will be more successful elsewhere and wish you the best. This has happened.

Approach 1: Burn the Ground and Salt the Earth

In this approach, you will wait until your last day or the day after your last day to leave. This will greatly inconvenience your employer for several reasons:

  • Your work is in an unknown state and your manager or team needs to pick it up
  • Your manager needs to explain to his managers, perhaps while being raked over coals, why you left so suddenly. He also has to report this to HR.
  • HR systems need to suddenly destroy your access to everything, stop your pay, not let you pass go, and take that $200 away from you.
  • Your team is suddenly less one person without warning or explanation. This is disruptive to team productivity, road map, and morale. Also, you won’t be able to make that ping pong tournament.

I would not recommend taking this approach unless you never plan to work for that company again and potentially any companies affiliated with that company or that could be acquiring or acquired by that company. That aside, before you plan on taking this option, make sure it is viable. Some contracts will specify a penalty for leaving without notice and others are “at-will”.

Approach 2: I’ll Stay but Do Very Little

In this approach, you’ve decided to follow the cultural or prescribed “notice” before leaving. Let’s call this 2 weeks for the sake of simplicity. You give your boss your 2 weeks notice and then you decide you won’t give anything more to your company than you already have. This involves some level of not doing your job ranging from not taking on new projects to spinning in your chair for your remaining time.

I don’t recommend belligerence but at the same time, if your company is sucking the life out of you, keystroke by keystroke, maybe toning down the amount of effort you put into it isn’t a bad thing. Going back to the factors that affect quitting: you’ve taken care of legal requirements and the HR department by following the rules. The HR department might be notified of inappropriate behavior and flag you to not be hired again. That leaves you, your manager, and your coworkers.

If you are leaving for medical or urgent personal reasons, sometimes the company will make an exception on the 2 week notice rule without any penalty to you. It will still be disruptive for your manager and your team but as long as people can rationalize something, they tend to be okay with it. Now, if you are stuck with them for the full 2 weeks remaining, you will be subject to the emotional reactions of your managers, your coworker, and yourself.

What does this mean? In some cases, your manager will ask you to lie about why you are leaving to your coworkers. In other cases, your manager will ask you to say nothing and then say who-knows-what behind your back after you leave. I consider these to be some of the less pleasant cases. This means that your manager has chosen to treat you as a risk to be mitigated and now you’re supposed to do what he says. However, you’re already leaving, right? Of course, you’re not obligated to lie for your managers but you’ll see similar consequences of openly going against them as in Approach 1. Even managers I thought were on my side the whole way have shown me a new, less appealing side of themselves when I quit.

Approach 3: I’d Like to Quit, Please

If you want to minimize the amount of bridge burning and water under the bridge or other bridge related metaphorical garbage from piling up, you can give your management a heads up that you are considering leaving. I don’t recommend doing this more than a month ahead of when you plan to leave. If you give them a heads up with more than a month, typically you are asking them to change something for you or it’s so much time they can’t do much with it so they ignore you. This will map to about stage 2 or 3 in the stages of quitting I’ve listed. Either you’re getting prepared to leave or you’re actively interviewing and have a date to quit by.

This strategy opens up possibilities for negotiation. In Approach 2, you might get some push back from management but you’ll end up burning a lot of goodwill that would otherwise make them want to change the workplace for you. In this approach, you are stating your desire to leave with ample amounts of time and they may be able to accommodate changes you need to stay.

This can be good or bad. It can be bad if you weren’t prepared for them to give you a counteroffer. So be prepared for counteroffers and come up with a list of things they would have to change for you to stay. If these things are “double vacation time”, they probably can’t meet that and it will give them an understandable reason why you are leaving. If you are able to give reasons that you know the company can’t accommodate, it somehow makes it easier for them to accept you leaving. If you can’t provide a good enough reason, they tend to be bitter and passive aggressive. This isn’t a “rule” but a tendency I’ve noticed.

Here is an example of a conversation where an employee tells a manager they are leaving:

Employee: I would like to discuss my career path.

Manager: Okay. What are you thinking of?

Employee: I’ve looked at opportunities here and where I want to be in the future. I don’t see myself achieving my goals here so I am planning to leave.

Manager: Oh, this is sudden. I didn’t expect this. Is there anything we can do to make you stay or is something in particular wrong?

Employee: I’m want to become a fullstack developer and all the opportunities here are for backend development. I know there are some front-end teams but I really want to learn from senior fullstack developers I work with day-to-day. Since there aren’t any people or teams like that, I’ve decided to move elsewhere.

Manager: I see. I don’t think we’d be able to find a senior fullstack developer to keep you here. Do you have a job already lined up?

Employee: Not yet but I’m waiting for a few different responses on interviews and I’d like to leave in the next month.

Note that the employee provided a reason why they wanted to leave and showed that they researched whether or not their current position could accommodate that. If you don’t show you considered making your current employer work, your manager might become defensive and passive aggressive saying things like “we could have made a place for you here but I guess it’s your career.”

Make sure to focus on things you want rather than things your employer doesn’t have. Saying “you don’t offer enough autonomy” can put people on the defensive and you’ve shut down the possibility for a productive conversation. Instead saying, “I’d like to work on a team with more freedom to experiment and less pressure to deliver” is you talking about yourself and typically people can’t tell you how you think and feel. This is part of Crucial Conversations if you’re interested in more behind the psychology of these types of conversations.

There Will Be Bullshit

Whenever there’s an event that can be emotional, it will be emotional (a play on Murphy’s Law). When quitting, you might see managers who you thought were reasonable start showing their bad side. Coworkers who you thought were almost your friends start being stand-offish and acting superior. Be prepared to see negative emotional reactions from coworkers. This doesn’t always happen but it’s better to consider it before. If you consider it beforehand, you are less likely to feel bad or overreact in the moment.

The “bullshit” can extend quite far. If you are doing an internal transfer, your managers can take action to block that transfer if they are displeased with you. Some companies require a testimony to good performance from your old manager before transferring so unfortunately, they can hold your fate in their hands. If you are quitting the company, this won’t be a problem. If it does come up, it’s tricky to get out of the situation unless you have some level of proof that you are actually a good performer (ex. past reviews, feedback from coworkers, projects that went well). With this, you might be able to go to HR and claim your move is being wrongfully blocked. I haven’t tried this before.

Odds and Ends

What do you say to your coworkers

If you plan to be working with your coworkers during this time, it’s good to think up a cover story. I’m not saying it doesn’t have to be true but it does have to be palatable and consistent. People tend to stop asking or thinking about something if they get closure.

Examples of palatable reasons with closure:

  • You are looking for different challenges to broaden your industry experience
  • You are looking for a hybrid position where you can be a manager and designer at the same time
  • You are looking for a position closer to your family
  • You are looking for a position that will allow you to pursue further education while working

These are all things that give a good reason why you can’t be here and also don’t convince someone they should also leave. Even if you have a good reason, people will start saying things that indicate they think they are superior because they feel suited to their current position. This is them rationalizing why they aren’t leaving or “sweet lemon/sour grapes”. You may also hear some passive aggressive comments from managers about you leaving. It’s an unfortunate side effect of working with humans.

Keeping in touch with coworkers

If you like your coworkers or might want to work with them again in the future, feel free to offer to stay in touch with them. I recommend LinkedIn as a good way to stay in touch. If you want to be more personal, ask them if they’d be okay adding you on Facebook as a friend. I stay away from asking for personal emails or phone numbers though there’s no reason not to offer or accept them. Just keep in mind that they could steal your rewards points.

The only thing to be careful of is any anti-poaching rules in the company you are quitting or the company you will work for. If you are planning to stay in touch with coworkers in order to pick them off into your new company, there is sometimes a grace period of several months to a year where you cannot do this. The consequences are usually against your new company, who will not appreciate that you brought those consequences down upon them.

Can I keep this?

Your company almost certainly has rules around company property, both intellectual and physical. Make sure you return any tracked items. In terms of intellectual property, you could be considered stealing if you downloaded anything company related onto your personal computer or any mobile device. In several companies I’ve worked at they have a device policy where they can claim any device, personal or otherwise, that has ever had company data on it. If you work on sensitive information and are allowed to use personal devices, this might apply to you.

Save Before Quitting

Before you start talking to your manager about quitting, make sure to collect and print or save externally all the documentation you may need if your manager decides to terminate your employment immediately. I was escorted out of the building the first time I mentioned I was quitting and this is how I prepared for it:

  • Print out the following pieces of information if accessible:
    • Proof of employment (employment verification and immigration purposes)
    • Last Paystub (employment verification and immigration purposes)
    • Employment contract
    • HR department phone number
    • Contact information for your health plan provider and any other benefits providers
    • Employee Resignation or Termination guide if there is one
    • Your manager and skip manager’s name and contact information
    • The emails of anyone you want to stay in touch with at work
  • Clear all your temp data, browsing data, and cached data from all your work devices
  • Ensure at least 1 other person on your team has access to every resource you have access to. Not required but nice to do.
  • If you still have stuff at work, make sure you have enough bags or boxes to get it out of your building and a way to transport it home
  • If you are using a public transit card funded by your employer, you will need to give that back. Have a backup fare if you took the bus to work that day.

You can do all of this without alerting anyone that you are quitting unless you’re careless in the print room about leaving your print job unattended.

Good luck quitting!

 

 

Book Report: Kaizen Express

20180625_150222

Kaizen Express: Fundamentals for Your Lean Journey

Amazon book link

Themes: Kaizen, Toyota Production Systems, Lean, Efficiency

Would I recommend this: No. This was hard to follow. I don’t think this book was meant to be read through cover to cover. It seemed like a companion textbook to a class. Half of each page was in English and the other half in Japanese, a translation of the content or maybe the other way around.

Note: Any examples below are generated by me.

Key Concepts

Waste

Waste is any activity consuming resources without producing customer value. The resources can be equipment time, human work time, or raw materials. Customer value in this case is any product or service the customer is willing to pay for.

Example

Not-waste: Engineers writing code for product or code for testing the product. The product is sold to the customer and increasing or maintaining the quality of the product ensures the customer will keep paying for it.

Waste: Engineers waiting for the computer to update during the time when they could be developing code. This could be done during a time period where developers aren’t using their computers.

Muda

There are different types of waste. The Japanese term for this is “muda” and each type of waste is written in a different form of Japanese script.

Type 1( むだ): Waste within a process that can be resolved with point Kaizen or within the process by an individual. Example: Blocking software updates that could be scheduled to run off hours.

Type 2( ムダ): System level waste that needs to be addressed via changes in different components. Example: High levels of inactive time between code review and production deployment.

Type 3 (無駄): Organizational or policy level waste that needs to be addressed by management. Example: Enforced time sheet logging for all employees when there is no variation in day to day time reports.

Accumulation, Batching, or Stockpiling

Accumulation between different processes, batching in large quantities, or stockpiling products or product components leads to waste. When you have large batches, it can lead to accumulation between processes or you may end up stockpiling a lot of one resource or product in anticipation of demand. Any type of accumulation hides defects.

Example

You have a table factory and a production line creating screws for the tables. You’ve run the screw line to produce a large amount of screws so you can shut off the line when you have several hundreds of thousands and start up the table assembly line to use this accumulation or stockpile of screws. Unfortunately, you find out shortly after starting the table assembly line that one of the dies was misaligned and all the screws aren’t usable. You may have found out minutes later, days later, or months later. Not only did this waste the material in the screws, but also the time on the machines, potentially some material from the tables, and the time of the humans operating the machinery.

Jidoka

The idea of Jidoka is self-stopping machinery such that it does not need to be constantly monitored by a human. One of the common examples of this in everyday life is motion detectors. Cars, garage doors, and assembly lines use motion detection to detect anomalies in places they should be and automatically stop or create an alert.

Example

Garage doors in the past were known to crush children or animals that were caught under them and needed to be watched carefully as they lowered. These days (in North American houses), new garage door openers have pressure detection when lowering as well as detection of objects near the ground that would interfere with closing. We don’t need to watch the garage door close anymore. Similarly, with Jidoka, it frees up human work time to allow us to do other things and check back only when needed.

Just-In-Time

As named, just-in-time describes only producing items or taking action as they are needed. Recall in the “Waste” section that accumulation can hide defects. Instead of batching and stockpiling, you only create things just-in-time.

Example

Your team is expanding and you know you needs lots of engineers so you hire as many as you can in anticipation of future growth. Later on, after hiring 20 engineers, you find what you actually needed was 10 engineers, 5 data scientists, and 5 quality assurance engineers but you weren’t aware that the demand or requirements would change in the future. If you had hired “just-in-time”, you would have been able to save time and money.

Discussion

What if some things can’t be done just-in-time? Take the example of packing for travel. If you’re at home, you are fine to buy toothpaste as you need it but when you travel, you want to pack your full supply for that duration because you may not be able to find the toothpaste you are looking for. This ties into more of the philosophy of Toyota Production Systems, lean, and kaizen. This restriction should push you to find a way to build a system that can support just-in-time. In the toothpaste example, this might mean that you learn how to make your own toothpaste out of baking soda, which is more readily available internationally than your specific brand.

Efficiency

There are different types of efficiency:

  • Apparent Efficiency: You can increase production without increasing the number of operators or equipment.
  • True Efficiency: Producing the number of parts that can be sold with the minimum number of operators and equipment.

Examples

Apparent Efficiency: You are in a cookie factory. Apparent efficiency is being able to double your batch of cookies without needing more workers or bigger bowls.

True Efficiency: If you can only guarantee selling 10 cookies per hour, you operate with the minimum required operators and equipment to make 10 cookies per hour. This also applies to resources. If you only consume the flour, eggs, and other ingredients needed to produce 10 cookies per hour, you reduce waste in a system that has enough apparent efficiency to produce 20 cookies per hour.

Local vs. Total Efficiency

Efficiency should always be looked at in terms of the whole system. If you have high local efficiency, this leads to an accumulation between the efficient process and the next process downstream. As we know by now, accumulation leads to waste.

My favorite example of this is when software developers punch out code review after code review but there aren’t enough reviewers to keep up. This results in a build up of unreviewed code. The volume of code to review makes reviewers less likely to focus on individual details and more likely to miss bugs. This also increases the chances of conflicts with other local development and creates wasted time for other developers that later need to merge the changes as they come through. Perhaps counter-intuitively, this local efficiency of producing code results in less global efficiency because the high volume of code to review leads to more bugs and more time to develop for other developers.

Machine Efficiency

A machine’s operating rate is the time a machine is used in a given time frame. The operating ability of a machine is the time the machine works well when it is needed. If you optimize operating rate, you will get overproduction and accumulation. Accumulation leads to waste.

The overall machine effectiveness is the availability (rate) times the performance (rate) times the quality (rate). Translates to: is it working (performance) well (quality) when I need it (availability)?

Continuous Flow

Continuous flow shows the production process as the life-cycle of a single item through a series of steps. In contrast, there is batch processing, which sees a collection of sub-processes that produce batches of items that move to other sub-processes.

Why is this good? Consider the idea of just-in-time above: if we are able to optimize our end-to-end system to produce a single item quickly, we will be able to produce items just-in-time more effective. Further consider the idea of efficiency above: if we are able to understand the true efficiency of producing a unit and we have actual efficiency in our system, we can scale to only produce items according to demand and reduce waste.

From Flow to Pure Continuous Flow

So you’ve got a continuous flow system and you want to make it better or achieve pure continuous flow. There are a few ways to look at your continuous flow system to try and improve it further. A lot of these concepts are phrased for manufacturing but it can be translated to other applications.

  • Process Village Layout: Relocate sequential processes to be together in product families. Example: put all your assembly stations for tables close together in a table factory rather than intermixing them with chair assembly. You can see this concept in how houses are laid out. We have kitchens for food related activities and bathrooms for hygiene related activities.
  • Minimize Worker Movement: If you see workers or operators spending a lot of time walking and not producing something, you may want to consider your layout. An example of this is when bathroom layouts place sinks or wash basins far from the toilet stalls. In software, this can be physical distance, like the space between offices and meeting rooms. It can also be virtual distance. How many times do you need to SSH and enter your password through various redirects to find the document you need to review? That could be minimized with dashboards or single-sign on (SSO) solutions.

Pull vs. Push

When it comes to pull vs. push, it helps me to visualize a rope with markers every few inches or centimeters. If you are in a tug-of-war style situation where there are people pulling every few meters or feet on the rope, how do you make sure as many markers make it to the end of the line of people as possible? If you push the rope onto the next person, it will bunch up and only create movement where you are. If you pull the rope from the person down the line, it will not only pull your section but also the sections all the way down the line to the beginning. So, if you want to get a steady stream of markers making it to the end of the line, should you be pushing or pulling?

Pulling means you are starting a processes based on a signal from the “end of the line” or the customer. When you are pushing, you are pushing what you produce downstream and eventually onto the customer whether they want it or not. If they don’t want the product at the rate you are producing, you end up with overproduction and accumulation. I think at this point if you’ve been reading sequentially, you know what that means: waste!

The 5 Ss

The 5 Ss are a guide to organizing the work space for maximum efficiency and effectiveness.

  • Seiri: Separate needed from unneeded.
  • Seitan: Arrange in order of consumption.
  • Seiso: Keep clean and inspect regularly.
  • Seiketsu: Don’t clutter or litter.
  • Sitsuke: Sustain and maintain discipline.

The Wikipedia page explains this much more clearly than the book did. I’ll provide some examples in the context of a software development process following sprints:

  • Seiri or Sorting: Only consider work or bugs that can be accomplished within a sprint and only work that is ready to be picked up.
  • Seitan or Order and Simplify: Prioritize and ensure all items are understandable and as small as they can be.
  • Seiso or Shine: Regularly revisit your sprint tasks to ensure they are still relevant and have been updated to reflect current state.
  • Seiketsu or Standardize: Ensure all individuals are participating in maintaining the first 3 Ss and have daily stand-ups to go through the exercise of maintaining your sprint work.
  • Sitsuke or Sustain: Continue to follow and maintain regular processes on the team such as spring planning, stand-up, and retrospective.

Kanban

Kanban is a tool for visualizing a pull system. It is not synonymous with Toyota Production System. The way kanban is used in production systems is a way to signal that work needs to be done and each signal or kanban card contains instructions and details needed to complete the work order. It doesn’t always need to be a card but it does need to indicate a “pull” is needed and should contain just the information needed, not more.

Rules of Kanban:

  • Processes should be triggered by customer action and a pull from the customer side.
  • Suppliers only produce what is requested on the card and not more.
  • No stock or items should move in the system unless they are accompanied by a kanban card.
  • Defects should not make it downstream.
  • Limit the number of kanban in progress to lower stored inventory and find problems more quickly.
  • Each item should not pass through the hands of the same person more than twice.

Heijunka

The idea of heijunka is to target producing product quantities in shippable units at each stage. In batching models, you might product all components of type A, then all of type B, and then pack them up. As you know, accumulation causes waste and hides defects. Heijunka will have the effect of shorter lead times, which helps with just-in-time production, and smaller inventories since you don’t need to stockpile as much before shipping a unit to a customer. This might seem like an obvious thing to do unless you consider that several production lines need to be converted between products. You can have a line that can do either screws or bolts but not both at the same time. This is similar to having one engineer doing both development and quality assurance.

Example: Waterfall vs. Agile

You can see this idea in waterfall vs. agile software development methodologies. In waterfall, you will do all design, then all development, then all testing. In agile, you do design, development, and testing for one deliverable unit or product and then start on the others. In waterfall, you see waste when implementation needs to change the design and a cascade of alterations that may result in a redesign of all components. In agile, by completing one unit entirely, you are able to adjust the development of other units before design and implementation has started. You just eliminated waste!

Pacemaker

You may think “oh, not the cardiac pacemaker” but it’s not so far off. The pacemaker in kaizen regulates continuous flow, as a heart regulates blood flow. All requests from the customer go through the pacemaker so it can regulate rates of demand and enforce heijunka. It is the first place in the system where heijunka can be enforced.

In the software world, the Project or Program Manager would be the pacemaker, taking all customer requests and scheduling them into a roadmap for the product or project the development team is working on. The thing that’s key in this example is that the manager will not dump all the work on the development team at once and instead will try to create a sequence that will then be given to the development team either monthly or quarterly to deliver. In an agile development setting, this would happen in sprints as well and the manager of the project/program would contribute work and assist in prioritization.

Setup Reduction

One of the driving factors for batching in some production systems is the time cost of switching equipment over to make a different part. To reduce the need to batch and accumulate (oh no! waste!), aim to reduce and shorten changeover times as much as possible. Changeover time is defined as the difference in time from the last good old item produced to the first good new item produced.

Example

You are running a catering company that makes cookies and meatballs. As such a small and specific company, you have only one working surface and you need to make sure none of the cookies or meat products touch each other. As a result, each time you change between making meatballs and cookies, you have to clean your entire surface. This will drive you to instead do all of one and then all of the other. However, as we know by now, this will lead to accumulation and potential defects. If it turns out one of your meat sections or set of eggs for your cookies were rotten, the entire set made on the shared surface with shared equipment might need to be thrown away. If the batch were smaller, this would be less waste. To reduce setup time, you can consider speeding up the cleaning process by wearing latex gloves that can be switched between batches and using parchment paper to protect the surfaces being used so it can easily be tossed and replaced.

Great, so we’re not making cookies and meatballs, we’re making software. Let’s talk about that.

Some companies have dedicated architects, dedicated development teams, and dedicated quality assurance teams. Moving a product from one to the other takes time with hand-offs, back and forth communication, scheduling of meetings, and reviews, especially in a waterfall model. This is a way to interpret the changeover time in software. How do we make this faster? What if the same person were the designer, implementer, and tester. This way, we wouldn’t pay any cost of hand-offs for switch over and testing or redesign can be done as the product develops at shorter intervals, looking more like agile. Another way to reduce the time is making the hand-offs smaller in incremental components rather than all at once, again, more agile. If there is accumulation (waste!) between the design and development or development and test phases, this is likely to hide bugs.

Andon

Andons are tools meant to visually highlight anomalies and operational status of each station. Through use of these you should be able to visually survey the work space and see if anything is going wrong. On the operator side, andons are activated simply and quickly by a button or signal cord that sometimes will stop the local production line to prevent further propagation of error.

Day to day we can see andon-like signals all over the place:

  • Grocery store check-out areas have a green light above them if they are open and are off if not. With a glance you can identify which ones are working. Associates can turn off the light if there is a problem at the register.
  • Highways with dynamic lanes show which lanes are operational and which ones are not. If an accident occurs, a lane can be shut down by using the signal to communicate to drivers not to use it.
  • Your car’s dashboard lights that tell you if your tire is flat, your oil needs changing, or if your gas tank is empty.

These are strictly andons because they are themselves a way to communicate to customers.

In software we typically see these mechanisms in operational areas with dashboards for our testing suites, environment health, or deployment health. The dashboards typically have green for “good”, red for “needs attention immediately”, and yellow for “needs attention but probably not immediately”. Further, these states are automatically detected or manually set when something wrong occurs. The faster you detect anomalies and stop the production line or rogue zombie process, the fewer defects and waste go down the line.

Maximize Human Work Time

You can get more efficiency by maximizing human work time. By watching for gaps in activity, you can find ways to creatively fill them. An example in the book shows that production lines that auto-eject boxes and wait for them to be taken allow one worker to operate two lines. By contrast, lines that need a worker to watch for the right timing to remove a product from a conveyor belt can only have one line per worker.

I don’t see this a lot in software because micromanaging at this level is typically stifling. However, I often see this when there is a “planning phase” of little development, they fill the development team time with operational improvement tasks to ensure they are keeping busy. In organizations that are more agile, there is typically an interleaving of operational and product work. That’s a different discussion though.

Zone Control

The idea of zone control is to make all defects localized and not allow them to propagate to other zones. Between zones there should be a defined inspection or quality assurance. This can be likened to running integration or health tests before deploying code to a new service host. This reduces the impact of a bug to a single host or stage instead of all versions of the system.

Standardized Work

All repeated work should be standardized and part of a cycle or defined procedure. Any work that occurs out of the cycle destroys continuous flow. Standardizing a cycle or procedure can be through written example or through enforced checks, like a machine not starting if a prerequisite is not met.

An example of how non-standardized work destroys flow is when the standard development process is skipped for a bug fix. Often when I’ve seen this done it means all tracking data for why that change was made is lost and often validation has been skipped. This inevitably leads to regressions later on and very complex bugs.

Visual Management

All components should be in plain view with their status clearly visible. The standardized process should be defined visually within this system. This visual management system should make it easy to assess status, error areas, defect rate, and performance of each area.

Kanban is a great example of a visual management system. In children’s schools, visual management systems are typically well-used to show where each student’s belonging are and what activities each student is doing. Old hotels use key hanging racks to show whether a room is occupied and if the occupant is in.

Genryo Management

Genryo management refers to how you can expand or contract your production with efficiency. It allows you to make profits and lower costs as needed. It contains a few sub-concepts:

  • Shojinka or Labor Linearity: Adding labor will increase production and decreasing labor will decrease production.
  • Capital Linearity: Create a production line such that adding more machinery will increase production and removing machinery will decrease it. This is meant to avoid the initial cost of a lot of machinery leading to over investment and potentially to overproduction (and accumulation… and waste!)

Why would this be good? Consider working on a development team of 5 people that produces 1 feature per month. If I add 5 more people, does that mean I will get 2 features per month? In practice, no, we don’t see this happening unless the two features are completely independent of each other. In an ideal world, this is how it would work. In this example, the effect of adding a new developer is more complex that just adding more work capacity. This illustrates how all the other aspects of kaizen like standardization through documentation, continuous flow, and others need to be in play for genryo management to be effective.

Examples

  • Shojinka: At the motor vehicle licensing office, adding an attendant to accept applications will result in more processed applications. Removing an attendant will decrease processed applications.
  • Capital linearity: This is often seen in scaling services with commodity hardware. Adding more hosts means more customer requests can be processed. Decreasing the number of hosts results in less.

Process Study

A process study will help you understand where you can improve. This involves listing all the steps in your process and tracking all the items that pass through it. By timing all the steps and understanding the touch points of items through the system, you can get an idea of where and how much you can improve. Process studies should be repeated until consistency in timings and performance are reached.

Employee Involvement

Employees should be motivated to connect to the flow of the production system. They will be able to contribute improvements with their expertise and familiarity that managers and designers will not see. Understand that you can develop abilities in employees that will help with improvement and growth of kaizen. Remember the discipline or sustain aspect of the 5Ss: every employee should be contributing to sustain. Hold kaizen workshops to allow any and all employees to participate regularly. Kaizen isn’t a fix once and forget solution; continually maintain kaizen and grow an improvement culture.

Glossary

Kaizen: All of this together makes up kaizen.

Takt time:  How often to produce a component based on customer demand. Takt time = (available time per shift) / (demand rate per shift). Ex. 8 hour work day / 4 code reviews per day = 2 hour takt time for completing code reviews or a developer averages completing a code review every 2 hours.

Toyota Production System (TPS): a philosophy of lean production founded by Taiichi Ohno

5 Whys: Keep asking ‘why’ until you get past the obvious problems and reach the real root cause of an issue.

Poka-Yoke: Any process or procedure should be designed to be hard to get wrong. This is a design style to reduce defects through human error. Make it hard for mistakes to happen.

Other things about this book:

  • There are tons of forms and worksheets showing how to implement some of these guidelines in the Appendices

An Independent Venture: Before You Go…

At this point, I’ve decided to stop working full time. I considered a few different things before starting self-employment.

Disclaimer: I haven’t fully tested this advice so stayed tuned to see if this worked or not.

Motivation

If I’m actually going to do this thing, I’ve made sure I have thought out, defensible reasons to drive me and keep going through the hard parts. It is scary to quit something stable that works well and that “everyone” tells me is the greatest thing that ever happened to me. There will be a lot to learn with surprises along the way. People before me have been successful in this and I can be too… as long as I have the fortitude and grit to keep going.

Goals

Have them. This feeds into motivation as well and can help with some unexpected decisions. By choosing work based on goals, I might actually achieve them. For example, if I want to be primarily remote, I’m going to have to keep that in mind when Some Large Tech Company offers me a wonderland of benefits, high pay, and perks… in exchange for on-site work only. Then I’m back to square one.

Examples of goals I’ve considered:

  • A flexible work schedule
  • Freedom to choose projects and clients you want to work with
  • Freedom to learn a new skill and use it on the job right away
  • Able to work in any physical location
  • Variety and excitement in your work life
  • Being able to meet a diverse group of different people regularly
  • Far more interesting tax returns

You might think, “these don’t sound like goals, they sound like tenets“. Call it want you want – “North Star”, “Tenet”, “Guiding Principle”, or “Goal” – you (or I) need to be aware of the factors that help make or break decisions with arranging work,  life, and the fact that they’re about to become the same thing… if they weren’t already.

Digression on Goal Specificity: Flexibility

To clarify the level of detail in my goals, I will go into a more thorough explanation of what flexibility as a goal looks like to me. I have achieved the goal of flexibility if:

  • I can choose the number of hours I work proportional to how much money I want to make. If I work 10 hours, I get $10; if I work 20 hours, I get $20.
  • I can shift my work hours by days, weeks or months in order to create significant gaps in my work schedule for something like travel. I can work January to March and take April to travel to Japan for cherry blossom viewing.
  • I am able to choose my location of work 80% of the time. I can work from my home, someone else’s home, or a hotel for a large duration of a project (8 days of a 10 day project).

By working towards this goal I want to find out the following things:

  • How many hours per week is ideal for me within a range of +/- 5 hours
  • What are my optimal work times during the day or week
  • Am I more productive with short bursts of concentrated work or steady streams of work over longer periods of time.

As I drill down into these ideas of what I want to accomplish, I turn it into an experiment to make me a productive human in a way the full-time workplace couldn’t. If by some ironic coincidence I find out that I’m optimal for a concentrated, non-stop, 40-hour work week year-round, I suppose I’d go back to working for large tech companies.

As with development, it helps to establish a target even if I don’t get there. It helps orient  a scale, time frame, and organizes my values so I can say ‘no’.

Money

It turns out I need money or “funding” stashed away somewhere if I want to quit my job and not work for an extended period of time. Short story: I determined how much money I spend in a given period of time (let’s go with a month) and how much time I ramp up time I need. So, if I spend $10 per month and plan a month of travel and then 3 months before earning sustained income, I get $40. There are many “rules of thumb” out there that will give values like “3 months worth of pay”.

I also considered running out of money with the goals unmet. This means finding a safety net or backup plan, like Mom’s basement. I also needed an idea of when to hit the “eject” button. I don’t wait until you have $1 left before realizing I need $500 to move my stuff to Mom’s basement. Or more realistically, pay for my internet bill to do that phone interview to get the job I now can’t afford to turn down.

There is a ton of discussion on finances out there and I’m not an expert so I won’t go into it more. If you have recommendations on your favorite finance resources, I’d love to hear about them.

As a side note, starting a low-rent apartment complex called “Mom’s Basement” just for kicks suddenly sounds like a great idea…

Brand

“What is your personal brand?” is a question I usually avoid. However, if I am selling my skills directly to clients, I need to have one and it better be the one I picked. Picking a brand usually relates to “what do you want to be known for?” or “what do you want your customers to ask you to do?”. If people keep requesting SQL database migrations even though I really want to be a Ruby on Rails developer, something is wrong with my goals and my brand. People will learn about me from word of mouth and my work portfolio. If I accept and do jobs in an area, that area will be part of my brand.

Work style also contributes to how clients and customers see you. Do I insist on fully remote? How often do I meet with customers? Do I often deal with individuals directly or through others? Am I charismatic or overly technical? Whatever I want to be seen as, I try to be intentional about how my actions form my brand. It will impact the type of work and clients I get. It’s hard to change a personal brand once it has started to morph into a monster of its own.

All Your Base

Finally, once I have motivation, goals, money, and brand more or less ironed out, I need to get going. I think of this as “establishing a base” and it consists of the following:

  • Physical work location
  • Work setup (computer, developer accounts/software, contract templates, work email, comfy chair, fancy desk, desk toys…)
  • Client pipeline and advertising
  • Methods of professional development

Honestly, I really just need the first three. I added the professional development because I will be better able to advertise and network if you are current in your area. It’s also yet another venue to get work through conferences or courses.

Another disclaimer: this is how I got into the mindset of being ready to embark on self-employment. I heard other people reach it accidentally or unintentionally. I’ve heard stories of less preparation. I’ve never heard stories of more preparation. I suppose that says more about me than anyone else.

 

Tales from the Git Keeper: No Professional Growth for You

This is a post about wanting to grow as an engineer and having my management block me from doing so.

I frequently go to tech events for networking, learning, or some mix and I’m an active member in several online tech communities as well. I came across a tech event that was being advertised in several of my groups and it turned out that it was in the same city I worked. I had just started at the company I was working at a few months before and hadn’t yet encountered conferences as part of developer growth. I was about to find out what the organization I was in thought of that.

Firstly, I thought I should make sure the content of the conference was relevant to my company in an effort to convince them to sponsor my attendance or at least let me take that day as a work day and not a vacation day. I collected the following data about my organization and company goals:

  • My company at the time had an company Town Hall describing how moving to “the cloud” (AWS in this case) was going to help them go global.
  • My organization at the time was working on reducing the overhead of devops in their service maintenance and was looking into solutions for implementing Continuous Integration and Continuous Deployment
  • The organization needs to maintain both their on-premises datacenter and new AWS clusters at the same time.
  • The organization wants to move to building a platform instead of a product.

The bold words are meant to prepare you for the next segment of my proposal.

Second, I tried to relate the content of the talk to the organization goals. These are some of the titles of the talks scheduled at the conference:

  •  What does it take to build an enterprise SaaS product?
  • I may need hybrid cloud, but where do I start?
  • DevOps in an immutable world
  • Serverless and Containers: a Match Made in the Cloud
  • Building a Notification System at Scale
  • From Infra Ops to Platform Development: T-Mobile’s Approach to Cloud
  • Does your SaaS automate enough?
  • Not Your Mother’s Cloud: Best Practices for Enterprise Hybrid Cloud – from On-prem, Cloud, Containers, and Beyond

I’ve highlighted some terms in case it wasn’t clear that just the titles of the talks directly related to what my organization was advertising to be it’s goals. If you had, for some reason, built a heuristic function to rank conferences on how relevant to your organization they were based on your organizational goals, I thought this conference would rank pretty highly.

Thirdly, I tried to quantify the cost of this so I could be transparent to the company what I was asking for. Here’s what I got:

  • Cost at the time: $345 for a single ticket
  • Duration: 1 day
  • Location: Across the street kitty-corner to my office building (I could see the conference building from my desk if I chose to stand up)

As a finishing touch, I noted that my company was listed as one of the Gold Sponsors of the event. That’s right. I was asking to attend a nearby event at relatively low cost to learn about technology related to the goals of my organization that was sponsored by my company. Let’s see how that went.

[Instant Messaging]

Me to direct manager (DM): Hi DM, I’m really interested in this tech conference that’s across the street from us and sponsored by us. You can find the info here at http://www.notareallink.com. Do you think I’d be able to either get funding to go or be able to go without taking time off work?

DM: That looks great! I think we should try to send more people to that conference and even I want to go. It looks really relevant and it’s right nextdoor. I will talk to our director to see how funding for conferences work.

That went really well. I didn’t have to negotiate or bargain at all. Unfortunately, my direct manager, DM, at the time had only started 2 weeks after me and was optimistic that such a prominent tech company would of course sponsor sending its employees to go to a conference they themselves are funding. Sadly for both of us, he was wrong.

[A week later]

DM: I brought up the conference thing at our manager strategy meeting.

Me: How did that go?

DM: Well… I tried asking about it and then our director said it wasn’t important so we’d talk about it at the next meeting.

Me: I see… well, I bought my tickets because early bird pricing was only on until 2 days ago.

DM: That’s okay, at the very least our Sr Manager said we can’t fund it but we can look the other way if you go on your own.

Me:… I’ll do that.

Okay, so that wasn’t the best news but there’s still hope that they might reimburse my tickets and possibly send a few other people, right?

[Another week later]

DM: So, I asked about the conference again…

Me: Ah… that sounds like it didn’t go well.

DM: Well, I was a little surprised by the response. The director said we would need to make a case for its relevance and we’d consider sending people but we’d have limited budget.

Me: Does he know we’re sponsoring this?

DM: He didn’t seem to care.

Me: Awesome.

While both my manager and I remained confused at the lack of investment my organization had in its developers, I carried on and made sure to give ample notice to my managers and teammates that I would be out all day for one Wednesday to go to the tech conference next door. Tickets sponsored by myself.

I thought that was it. Sure, it was disappointing that the company wasn’t supportive of professional development but at least I would be able to go and not lose a vacation day. A week before the event, I overheard a manager say we were doing a mandatory all day off-site the following week. Knowing I had sent out my notice of my absence a month before, I was sure it wouldn’t conflict with my plan to go to the conference. Just in case, I sent a reminder to my managers:

[Email]

Me: Dear Managers, The agenda for the conference is up and can be found here: http://www.notarealsite.com. Some of the topics are fairly relevant to our organization, particularly those about migration from on-premises to cloud and hybrid cloud. If there is a talk that you’d like me to attend, please let me know and I will try to make it.

Skip Manager (SM): That conference conflicts with our all day off-site that I was planning to schedule tomorrow. Do you really have to go to this conference or can you go to another one?

Me: Given that this one is very close (across the street), only one day long where others on this topic are typically 3 days long, and I’ve already paid for it, yes, I will have to insist on going.

[Time Passes]

SM: I’ve rescheduled the off-site to the day after. The talk on hybrid architecture sounds good to me.

Not only was it a bit of a struggle to get people to tolerate my going to a job relevant conference, they also try to stop me from going because in favor of plans that don’t really exist yet. So much for the CEO’s proclamation that the company “attracts and develops top tech talent”.

I ended up going to the conference and enjoyed it greatly. I wrote up notes and made sure to send them out to my teammates and managers. No one cared. I didn’t go for them though so that’s fine by me. As for the offsite, it was rescheduled and delayed two more times before they eventually decided I no longer needed to attend. I’m glad they had all that hammered out before they asked me to cancel my conference attendance.

Event Review: GeekWire Cloud Tech Summit 2018

Screen Shot 2018-06-27 at 9.10.06 AM.png

Event Info

Event Location: Maydenbauer Center, Bellevue, WA

Event Cost: $350 (early bird + group) to $500 (late bird solo) USD. PyLadies and Stripe were generous enough to sponsor my attendance.

Approximate number of attendees: 300 – 500

Event Duration: 9 hours (8am to 5pm) + after party

Slides and videos

Highlights

  • Talks were short and manageable in length (30 min each) so I could move around
  • All topics were relevant and understandable
  • I learned about cloud-edge computing and more about Kubernetes
  • This was my favorite swag: Tech Tarot Cards
  • Great dedication to diversity by giving out gender pronoun pins, providing diversity activities and events
  • Excellent venue: lots of space, tables, electric outlets, and Wi-Fi

Lowlights

  • Nothing. I fully enjoyed this event.

Would I go again?

Definitely. I recommend this to anyone wanting to get breadth on current cloud topics in the Seattle area.

Details

Caution: This is a long one.

Fireside Chat with Mark Russinovich, CTO Microsoft Azure

  • New Azure features announced recently: Azure IoT Edge, Data Lake integration, New China regions
  • Azure IoT edge: a lot of computing will move out to the “edge” to optimize for bandwidth consumption and responsiveness of computed results.
  • Github: Microsoft will not change what’s already there and fully invests in and supports the open source community
  • How are people using blockchain in non cryptocurrency applications? Non-centralized security where multiple parties can see and verify a set of actions in a distributed networks.
  • What is Quantum? It is a type of computer that can solve intractable problems in computer science that cannot be solved by a traditional computer.
  • Is Microsoft planning to invest in quantum computing? Yes, Microsoft is getting ready for it along with an SDK and Q#, a language for quantum computing.
  • Where do you think the best space to invest in technologically is? AI and ML – we’re at a breakthrough point where it’s still unknown how much we can do with it.
  • Live poll: People are very unsure about what to think of Microsoft acquiring GitHub.

Did I like the talk?

No. This seemed like a giant advertisement for Microsoft and I could have done without. The comments on block-chain and quantum compute were interesting but very much sidelines.

3 Back-to-the-Future Trends in AI Systems – Carlos Guestrin, Apple and Amazon Professor Of Machine Learning at University of Washington

  • We are in a period of resurgence in AI. Previously we saw a dip in its popularity.
  • In the 80s, AI was primarily rule based systems that were not as useful or correct.
  • Now, we are using machine learned models we are able to go into applications where we cannot enumerate all rules in spaces where there are a lot of possibilities. For example, we previously could beat humans in chess and now we are able to move to use Go without needing to describe the full space of the game.
  • Spark uses large amounts of data with low compute per datum and massively scales out. We are moving towards Deep Learning where we have huge amounts of data (where huge > large) and there is high compute per datum that requires compute grids and node scalability.
  • We moved from specialized hardware to specialized with Spark. Now with deep learning we need to move to more specialized hardware once again.
  • AutoTVM: How do you make sure you choose the right coding patterns to optimize for your hardware when training models using a tensor language over different hardware? This project is meant to learn the right parameters to maximize use of hardware and optimize efficiency.
  • We are trying to move to simpler adoption of machine learning. Currently you need a lot of expertise in order to understand data, use the tools, and manage the hardware needed for training.
  • Two trends: pre-trained models where generic data is sufficient and the need for developers to create models for their specific scenario.
  • Turi Create: allow developers to create their own domain specific data to easily train models via transfer learning from existing generic model training.
  • Another trend is moving towards making the internal workings of machine learning and AI available to everyone instead of being a black box.
  • Providing transparency into how a result was reached helps understand what features of the input are affecting the output as well as understanding where error might be coming from.
  • The data you choose to use with define the user experience.
  • Example: initially when film was being developed, they created chemical exposure guides that optimized for only light skinned models. As a result, dark skin colored people in photos were not visible due to over exposure. We need to think about the culture and values the data we select to train on shows.

Did I like this talk?

Yes, I enjoyed the examples provided and the organization of the talk. I may have already know the content but it was phrased in a organized and easily understood way.

Rook: Portable Storage on Kubernetes – Bassam Tabbara, Founder and CEO of Upbound, Inc.

  • Kubernetes can be thought of as the cluster operating system or the container orchestrator. It also does life-cycle management, scaling, load balancing, scheduling, etc.
  • It exposes a portable cloud API and thus supports the “write once, run everywhere”. You can for the most part containerize and deploy using Kubernetes and run in AWS, Azure, or other places fairly easily.
  • When we start to manage storage beyond the lifecycle of a container (or other unit), we have to look outside of Kubernetes. Similarly, different forms of storage also need to be managed: blocks, volumes, queues, key-value stores, time-series, etc.
  • Kubernetes allows portable volume extraction that is independent of pods. This is currently the only stateful workload abstraction.
  • You need to interact with storage services that exist outside of your Kubernetes managed cluster for the most part.
  • Rook is a cloud native storage orchestrator. It is an extension to Kubernetes hosted by the Cloud Native Foundation.
  • It helps storage systems run like a service on top of a Kubernetes cluster by encapsulating storage system management logic.
  • Can create a configuration that specifies size, monitors, and scaling that will create all components within the cluster that is needed using the operator pattern via Kubernetes operators.
  • Rook operator defines a desired state and loops through state activating different components until the desired state is achieved and continues to do so to maintain desired state.
  • This is an open source project looking for contributors!

Did I like the talk?

No. I understood what the problem and solution were but somehow I felt like I was missing something when he was explaining the examples. Most of them were “here is the configuration and then it just happens”.

Serverless and Kubernetes: The best of both worlds by Aparna Sinha, Google

  • They key to a serverless platform is such that not many people are needed to maintain the servers and more time can be spent on developing the application.
  • A lot of time in application development is spent on understanding monitoring, logging, orchestration, etc. that the application developer needs to spend time on.
  • What is serverless at Google? Zero ops, auto-scaling, managed security; event-driven programming model, horizontally scaling with microservices; pay for usage.
  • Serverless can be applied to more than compute including data storage and other components.
  • Two main ideas: decouple applications and automate infrastructure
  • This translates to: containerize the application + automate deployment + build in metering
  • Kubernetes is container infrastructure as a service:
    • It contains load balancing, networking, DNS and server discovery
    • Environment agnostic management
    • Has flexible, composable primitives that can be built together to craft your application. Ex. application + storage + deployments
  • Kubernetes is a platform for automating management
    • Can provide a desired number of replicas and Kubernetes will make this happen
    • Provides auto-scaling based on system state like queue length or traffic rate
  • Kubernetes is a declarative extensible API
    • You can define different components or controllers you want to use in your own application
    • Example: you could create a custom Spark controller that will allow you to treat Kubernetes as a Spark orchestrator
  • Demo: the demo goes through creating an extension deploy a function via Kubernetes in such a way that makes the extension appear as part of the Kubernetes API. Using that function, write a hello.py function and deploy it via Kubernetes.

Did I like the talk?

Yes. I finally know what Kubernetes does (you’d think I would have looked it up by now). However, having worked with these types of orchestrators before it wasn’t too hard to figure out.

From Infra Ops to Platform Development: T-Mobile‘s Approach to Cloud – Nicholas Criss, Sr. Manager, Cloud Center of Excellence, T-Mobile

  • Background: T-Mobile aims to be a market disrupting mobile carrier that strongly favors magenta the color
  • Moving from waterfall development to agile development
  • In 6 months moved up from last to first in JD Power survey
  • Moved to the cloud and didn’t just follow “lift and shift” but tried to adopt the new philosophy of cloud development to make sure they didn’t fall behind competition
  • Having already moved to the cloud, the company was seeing that they were able to adjust very quickly to sudden demands from business needs. Example: They were able to spin up tons of video processing servers very quickly and scale down as needed. Without being on the cloud it wouldn’t have been possible.
  • Migrated through different stages:
    • Before cloud: no cloud integration
    • AWS console management: Manual management of cloud infrastructure
    • Some automation of cloud infrastructure
    • Managing infrastructure as code: all infrastructure was automated in code
    • Software Developers: The software developers can easily (with minimal error) leverage fully automated infrastructure and eliminate the need for infrastructure teams
    • Platforms: We got tired of having to relearn through repeating the same mistakes on different teams – make all these learnings a part of the platform. For example: scaling, fault tolerance, and general operational readiness.
  • We moved from “we’re mostly security, compliance, and operations” to “we are software developers”
  • Build our compliance and operations into code rather than Sharepoint docs that no one will ever read
  • If you can’t implement something as code, it won’t happen or will happen incorrectly
  • No more projects, only products – focus on building customer focused value instead of meeting deadlines and completing tasks
  • Moved from being a telecom company that sometimes has interesting technology to a software company that happens to build telecommunications software
  • Platforms that T-Mobile built internally to make software development better: Security management, policy enforcement platform, enhanced Hashicorp Vault service that optimizes for T-Mobile integration, enhanced container management platform, deployment platform
  • Building out platforms to take away the painful parts of developing makes people more excited to develop and come to work
  • They see these growth points as ways to interact with and give back to the community by open sourcing their projects
  • Question: How did T-Mobile make this happen instead of just talk and no management support?
    • From top level, management is supporting training and education
    • From the bottom, there is the ability to innovate and remove pain by moving to a continuous improvement culture. There is a push to adopt new things to make old pain go away.
    • There are still some holdovers of the old model with devops teams continuing to do deployments but there is progress and benefits being seen.
  • Question: How do you get started by targeting small things to start building interest?
    • Break off small pieces and build up the great infrastructure
    • However, speaker encourages moving directly to going fully to containers and platform and don’t waste time through the in betweens
  • Question: How do you migrate people to your platforms without the additional cost of making it work?
    • We don’t force people to use a particular solution but instead we encourage them to use our solutions where they make sense
    • It’s a continuous journey to make this work

Did I like the talk?

Yes. It was a rough start with some of the disjointed examples but it improved a lot by highlighting the journey steps and the growth that happened each time.

Diversity and Inclusion Panel Sponsored by Stripe

  • Panel:
    • Uma from Stripe Infrastructure Team manager
    • Amy is a system software engineer at a local startup
    • Dana is a manager at Google
  • Q: How did you choose your field?
    • Uma: I was one of the “accidental tech people”. In India, my grades were not high enough for medicine and the only other option was engineering. Software was chosen because it was less intense than the hard engineering. I found that I enjoyed it and continued.
    • Amy: Tried to do pre-med and didn’t like it. Through a chat roulette interaction via Omegle, decided to try computer science!
    • Dana: Father introduced her to computers and started playing with programming and got into that way.
  • Q: What’s a problem in your field that you are excited about?
    • Uma: Since we are growing to be global we are solving a lot of problems of scale and growth.
    • Amy: Working on Kubernetes, we want to enable larger companies to be able to migrate easily to new technology.
    • Dana: We focus on talking about how many 9s we want to support. We need to push harder to have better quality because with millions of users, even 0.01% is a lot of people.
  • Q: Amy, what are some ways you can demonstrate leadership without being in a formal leadership role?
    • Amy: I am not a manager and I don’t want to be one. I still want the ability to influence others. I achieve that through deep understanding and making my knowledge available through youtube or automating and sharing.
  • Q: Dana, as you started moving into your tech lead role, what kinds of pros and cons were you thinking about? What advice was helpful?
    • Dana: Tech lead can mean a lot of things and in silicon valley it can mean a tech manager. What I considered before moving into this position was to talk to other women in a tech lead manager position at Google. I recommend you make sure to talk to other women in these positions before making this move. What I got was: the hybrid tech manager role is a challenging one at Google. You won’t be able to do everything you want to do and you need to learn to let go of some things. However, you get a broad impact. You need to be comfortable not coding and still having an impact.
  • Uma: You moved from being an engineer to manager and then from manager of engineers to manager of managers. What motivated you to do this switch?
    • Uma: First, I didn’t plan to be a manager and was offered the opportunity to try being a manager after coming back from maternity leave. My manager supported me trying it out and would keep the IC option open if it didn’t work. I found that I really liked it and wanted to keep mentoring and growing other developers. Moving to the manager of managers position was due to seeking a new challenge with a level of abstraction. However, people don’t tell you how lonely it is being a manager since you lose the casual chat of other ICs.
  • Q: When you were earlier on in your career was there support that you wished you had and how would you recommend people find it?
    • Uma: Women need to ask more for access to things. Try to share what you learn with others to help them be aware of what it out there.
    • Amy: The money that people are willing to offer you is a measure of how much they value you. Having a diverse group of people you can talk to to understand the potential range of values and opportunities that could exist. Try to find mentors or sponsors in different demographics and in different pay levels.
    • Dana: I was lucky enough to get interesting products until I moved to Google. You need to find the thing you want to work on and make a difference on your own. Look for opportunities to do something you haven’t done before.
  • Q: When it comes to negotiating, what advice do you have to figure out what to ask for?
    • Dana: I was underpaid at my last job. Checked Glassdoor at a point where I was ready to potentially leave and asked them to pay me competitively or I would go. And they paid me more. Use recruiters as an opportunity to reassess you value and gain leverage.
    • Uma: Continually research and update your data on what you should be paid. Don’t base your targets on your emotional response to the money. A lot of things are negotiable like working from home and you can negotiate for what is important for you.
    • Amy: Always be on the job market to keep collecting data on what you are valued at. Make sure to try to get a pay band so that you are aware of how much you can negotiate for and where the lower end of tradeoffs might be. If you can, be on social media and get yourself out there.
  • Q: Do you notice any behaviors that you wish under-represented groups would display more?
    • Dana: If you haven’t noticed, I’m trans and had spent some of my career as a different gender. I have been socialized to have some of that other gender’s traits and as a result I don’t back down even though I have a feminine communication style. I will call people out and stand up for myself. I would like to see more women doing that. The way people behaved towards me when I changed to female. It was harder and I had to adapt. I encourage everyone to stand up for themselves.
    • Amy: Ask to be treated equally and have the same as what your peers. Don’t change your communication style because other people don’t like it.
    • Uma: Risk-taking in a more calculated way is something I’d like to see more people doing. Build a support network so that you can take risks.
  • Audience Question: Do you think that offering training to help people learn these skills that would allow them to get an even footing culturally would help?
    • Uma: Yes, at Stripe we had a training around negotiating that really helped understand that everything can be negotiable. Further, in backgrounds where saying no to managers is hard, building 1:1 relationships that helps understand the new ways interact with peers and manager.
    • Amy: Managers should constantly be emphasizing that they are equals and not superior so that directs are comfortable say no or constructive discussion. It is really helpful that my managers communicates that I’m allowed to challenge them.
  • Audience Question: How do you thank people for treating you like an equal?
    • Dana: Thank them. It’s okay to tell them and explain to them why it matters to you.

Did I like this Panel?

Not particularly but that’s only because I’ve already heard this advice a number of times.

Fireside Panel with CIO of Providence Health Janice Newell and Charu Jain CIO at Alaska Airlines

  • Q: Can you tell us about a service story that went horribly wrong?
    • Charu: a lot of our cloud providers are on their own journey as well and even though you do everything right on your end, one day you find out that something isn’t there when you expect it to. We’ve learned to trust but verify even with some of the large providers?
    • Q: How do you verify?
    • Charu: Work with them to prove that it is working and pair on testing.
    • Q: What about a great thing?
    • Charu: We were building a cloud infrastructure to support more bookings. After moving to a cloud based solution we found that we were able to do things instantly that weren’t possible before.
    • Janice: I have a horror story from us. During the Wannacry virus, we found out that one of our cloud providers that had advertised high security and a whole bunch of other features. It turns out they didn’t actually have those features. The virus then took down voice transcription for doctors for over a month. This created a huge problem for the customers. We ended up moving all of our physicians off of the SaaS based solution to an on-prem solution. We then went back to verify and inspect all the features and fault tolerance the company said they offered and were horrified to see how terrible it was. We pushed back and had to get them to improve their systems. This is an example of the trust but verify example is similar to what Alaska had to do.
  • Q: We’ve seen a trend of a shift from Hybrid (on-prem + cloud) to an idea of multi-cloud where you use multiple cloud solutions (Azure + AWS). Are you thinking of multi-cloud and how would you use it?
    • Charu: Our strategy is stated as multi-cloud. We are currently on Azure and have multiple SaaS providers. We are trying to get better at one thing before maturing into a multi-cloud environment.
    • Janice: We are using containers to try to adopt that flexibility.
  • Q: How has cloud adoption impacted things organizationally and culturally? Devops? Team makeup between specialist, generalist, or devops?
    • Charu: We are working on a phased plan where we move one team and then take the learnings and roll that out to more teams. We are choosing teams first that have the most legacy technology that needs to be moved further. Developers feel empowered to drive these changes. We stood up automation teams to help start up the infrastructure and that has migrated to a cloud center where we help teams understand developing for the cloud.
    • Q: Has there been resistance along the way?
    • Charu: Change has always met resistance. People who have embraced it are setting the example.
  • Q: What is your approach to monitoring?
    • Charu: We are still in early stages. We are trying to make our monitoring complete but also not too noisy.
    • We need to measure to know how our services are doing.

Did I like this panel?

No, it seems more managerial/business level than technical. There was a lot of repetitive back and forth so it was hard to pull out relevant information.

Machine Learning At Scale On Microsoft Azure, Paige Bailey, Microsoft

  • Artificial Intelligence: any technique that allows computers to mimic human behavior
  • This means anything that is coded to behave as a human which started off as rule based systems
  • As the behavior becomes more complex and can’t be enumerated, you can use machine learning to let the computer learn a large space than the rules defined by a human

Types of Machine learning:

  • Do you have labelled data?
      • Yes – Cluster Analysis – Example: K-means
        • These problems are harder
        • Do you want to Group Data?
          • Yes – Cluster Analysis – Example: K-means
          • Categories: classification – determine what type of thing and object is. Example: KNN.
          • Quantities: regression – estimate an output based on input. Example – linear regression.
  • No – Unsupervised Learning
  • These problems are harder
  • Do you want to Group Data?
    • Yes – Cluster Analysis – Example: K-means
    • No – Dimensionality reduction – Example: PCA

     

  • Example walkthrough: you are given a CSV with age, credit history, and employment; determine if you should give someone a loan
  • You can make a decision tree to classify this person into yes or no buckets
  • Aside: Jupyter notebooks are excellent for machine learning
  • Feature: think of this as a column in a CSV
  • Record: think of this as a row in a CSV
  • Demo: showing a simple training example in a Jupyter notebook. (Author’s note: many tutorials are available online that will walk you through this)
  • Machine learning is helping us learn without explicit rules; deep learning is now helping us identify features to learn on and extrapolate huge amounts of potential cases. Example: Magenta
  • After this there were more examples and a discussion of how you can save your company money by learning to do machine learning on your own and also bump your salary due to the additional skills
  • GPUs are very good for training neural networks
  • Data science virtual machine using X2Go: if you don’t want to have to worry about managing your own machine learning environment, this will maintain all versions of libraries and languages to help you move faster with tons of educational Jupyter notebooks out of the box including topics like TensorFlow. They are ready, waiting, and free!

Did I like this talk?

Yes and no. I already knew all of this so it wasn’t super helpful but the presenter did a great job starting from 0 and showing people good examples in the time given.

Container and Serverless at Edge Connected to Cloud – Ying Xiong, Chief Architect, Cloud Platform, Huawei

  • What is edge cloud? Computing resources at the edge managed by the cloud owned by provider or customers with bidirectional communication between both.
  • Example scenario: security cameras where processing can be done on the camera itself and send data back to cloud
  • Challenges: resource constraints and frequent network partitioning with no public access; we also need to support edge to edge communication
  • Kubernetes on the Edge: what happens when and edge node loses the network connection? We can’t update it or support serverless on it.
  • To solve these problems:
    • We need to create robust communication between edge and cloud with high reliability and scalability
    • Build an edge resource controller that manages the edge from cloud
    • Support asynchronous metadata syncing and local storage to allow edge to operate on it’s own until connectivity is available
    • Similarly, build a component called KubeEdge that will manage the edge applications and listen to events to allow self maintenance on edge nodes
    • Use an event hub as a centralized listener to aggregate and queue events locally on the edge
  • Created a custom TCP/IP to optimize for edge-cloud communication for reliable bi-directional multiplex channels that can scale
  • Needs to support AuthN with certificates
  • This provides a Kubernetes plugin to manage and maintain edge nodes and also provides Kubernetes on the node to support app status syncing
  • KubeEdge is an optimized version of kubelet on edge for managing the edge node lifecycle and maintenance. It can work offline.
  • Not yet open source but that is the plan.

Did I like this talk?

Yes. Before going to this conference I hadn’t hear the term “edge cloud” and it turns out it is very aligned with my technical interests.

Fireside Chat with Matthew Prince, Co-founder & CEO, Cloudflare

  • When Cloudflare started we were competing with companies that were trying to find newer and better hardware to accomplish their goals; we did the opposite: we searched for ways of making the commodity hardware scale cheaply and linearly.
  • Over 150 datacenters for Cloudflare with fully owned hardware in rented spaces; ISPs are interested in moving Cloudflare closer to them since it creates a faster internet and better customer experience.
  • We have a world-wide infrastructure and we are suffering from low performance that isn’t improving from existing DNS providers due to lack of competition. We were able to create a secure and performant competitor in response.
  • Discussion on opening business to public offering, costs and profits, competitive advantages, customer successes, various marketing and business data
  • Insider threats are a big security challenge going forward
  • “Our scale is so large that attackers can’t DDOS attack”
  • Question: You made a decision to stop supporting a Neo-Nazi customer. How have you dealt with this internally?
    • “I never thought I’d know so much about Neo-Nazis”
    • “It’s lots of fun to kick Neo-Nazis off your service”
    • There’s no way for consumers to understand when they are using CloudFlare. It seems creepy if we curate what users can or cannot see.
    • We had to decide what the role of deep cloud infrastructure companies is
    • The point at which we decided to do this is when the customer started to attribute their political view to the company
    • There is still no clear answer on how to handle these cases

Did I like this talk?

No. This appeared to be a very long advertisement for Cloudflare. There were tidbits of interesting notes about network structure. Some really great quotes came out of this one.

Power Talk by Peter DeSantis Vice President, AWS Global Infrastructure and Customer Support followed by a Fireside Chat

  • At a large scale you are able to invest in things that wouldn’t be cost efficient at lower scale
  • AWS has expanded to many region and at an accelerating rate
  • What is a region? A set of availability zones?
  • What is an AZ (availability zone)? A fully isolated infrastructure with one or more datacenters, a unique power center, and a meaningful distance from each other
  • Every region has at least 3 and up to 6 AZs
  • Connectivity:
    • Between datacenters within the AZ
    • Between AZs within the region
    • Between AZs and the public internet across the region
  • Built on top of a dense fibre backbone with high redundancy than Amazon manages
  • Also creating inter-region Direct Connect that allows customer to get a fast lane between regions
  • Compute with AWS EC2: Security, Performance, and Familiarity (look and feel is like any other server)
  • Going over the history of EC2 architecture
    • First started off with customer instances on top of a hypervisor that managed security, network, and hardware
    • In the next big iteration, they moved over to Nitro infrastructure
    • Through these moves they were able to improve network latency by 20%
    • This also enabled moving to commodity hardware – except for security and some other core features
    • Could continue to look for hardware solution, use FPGA chips, or go with a custom ASIC – eventually acquired Annapurna labs to realize the last option
    • Thus equipped, security improved and allowed nearly 100% of compute resources to be available to customers on the instance. This achieved the familiarity goal.
  • Now have also opened offerings for bare metal instances as well as Nitro VMs

Did I like this talk?

Yes. Despite knowing a lot of this information at a high level, I hadn’t seen in presented in this particular nutshell. I liked how it was phrased and the progression through the topics.

Event Infrastructure

  • The event used a phone app to support planning your agenda, quick connect with other attendees, integration with sli.do for live polls
  • Regular video on projectors but also a live keyword note taking screen which was actually very helpful
  • The breakfast was great: hot toasted bagels and a hot oatmeal bar for breakfast
  • Event support for developers is great: they provide desks and tables with a high density of plugs available
  • The sponsors presented before, between, and after each talk. It was a little overwhelming and felt like I was watching Youtube with ads turned on. However, the comfort of the event was worth it. Similarly, all “ads” were relevant so it wasn’t too bad.

Networking Notes

  • Very easy to network at this event: all I needed to do was sit down at a table and people would start up conversations about technical topics
  • Despite trying to avoid doing so, people are very interested in hearing the terrible things about the company I currently work for
  • I encountered some discussion because I perhaps foolishly brought my motorcycle gear with me (I did ride the motorcycle to the event) and started some conversations about whether or not to lock your helmet to your bike. This then lead to discussions about companies that let you work from home and some nice connections. If you learn one thing from this: bring or wear an item that will inspire people to talk to you.

Book Report: Six Sigma For Dummies

I’m a huge fan of continuous improvement, the “lean enterprise” philosophy, and just making things more efficient. There are different things that draw people to software development and one of them is automating repeated and wasteful tasks in their lives. This is part of what Six Sigma is about applied to an entire organization. In this book review I won’t give all the content of the book but rather an overview of topics covered and how well I think it was communicated.

The book.

51ixq96w5sL

Summary

  • This book goes over what Six Sigma is and how it is a data driven and criteria focused methodology for improvement
  • The topic is a business and organizational topic but can be applied to any process including doing chores or planning a road trip
  • There are a lot of relatable examples that can be used to teach the topic if you’d like to share these ideas with your team, friends, or family
  • The second half of the book goes into statistics, data presentation, and data interpretation
  • Measurement is the first step to improvement
  • There are some good Lord Kelvin quotes about measurement referenced in the book
  • Overview of the DMAIC process

Recommend?

I recommend this book to those who:

  • Already have a Six Sigma leader in their organization and want to know what it’s about in a single reference
  • Want a deeper understanding of which data visualizations are relevant and which are not
  • Those who want an overview of the key components of Six Sigma (only read the first half)

Detailed overview

  • Six Sigma is about measuring inputs to your system, identifying variance, and controlling or reducing the variance in your process.
  • This can be applied to any type of process including manufacturing, software development, or hiring.
  • You must measure your all targets whether they are a quality bar (ex. physical measurement or latency) or the time it takes to produce an item.
  • Both the quality and efficiency measurements are needed along with other factors you want to improve or maintain at a certain level.
  • This is a practical and understandable use of statistical and experimental methods.
  • Focus on eliminating special cases of variance or error rather than start with the common cases. Once you eliminate special cases then any improvements from common cases will be more impactful and less overshadowed by exception scenarios of variance.
  • You need to measure all inputs and outputs along with your goal to figure out what your variance is and to see improvement over time.
  • Try to find the factors that go into your process that have the greatest impact. Improving these first will provide you with more leverage to convince the organization to change.
  • Follow the DMAIC process in order to achieve improvements.
  • There are different types of improvements or change:
    • Thinking for breakthrough – affecting cultural or thought change
    • Processing for breakthrough – improving existing processes
    • Designing for breakthrough – change or improve how new processes or products are created
    • Managing for breakthrough – change leadership and hiring to create a culture of improvement
  • Six Sigma makes everyone a leader for improvement who cuts through assumptions and fights for betterment.
  • This type of organizational change needs to be driven from top down. You may have small scale success in an isolated team but you can’t do anything on a larger scale without management driving it.
  • All aspects of an organization must be involved including non-production roles like finance or HR to define their expectations and results.

DMAIC

  • A tool to turn an everyday problem into a logical plan and solution
  1. Problem definition
  2. Identify impacted processes
  3. Create a data driven definition for the problem
  4. Create a data driven solution goal
  5. Create a verification plan for the solution
  6. Design a practical solution based on the data driven goal
  7. Implement and see results!
  • Typically improvements from this process should benefit all customers and stakeholders.
  • There are different brainstorming techniques the book goes into that can be used with different groups of people to identify problem areas that feed into the definition phase.

Identifying the problem and target:

  1. Determine what needs to be improved (or the Y)
  2. Identify the impacted projects or processes
  3. Determine a baseline for Y of what is currently observed
  4. Quantify the cost and impact of this problem
  5. Write a problem statement: improve [metric] from [baseline] to [goal] in [time] with [impact] to [business goal]
  6. Identify key individuals and build a team: the best person to lead an improvement is the owner of the process.
  7. Obtain approvals from stakeholders and launch the project.
  • If you have more than one Y to improve this may indicate your scope is too broad.
  • Initial baselines may not be accurate but will improve with more measurements over time.
  • Long term metrics are more valuable and relevant than short term or short-timespan metrics. Snapshots can be misleading and you’ll only identify trends of improvement over longer periods of time.
  • Avoid biasing the project by putting parts of the solution into the problem statement.
  • Create specific and measurable statements to convince management of the value and to ensure all parties impacted can understand the value.
  • Identify the entitlement in your process: this is the best possible performance you can achieve by adjusting the inputs you can control.
  • Work to identify hidden processes which likely represent areas for improvement.

Statistics

This book went over various concepts in detail with examples of each. I will only list them out here:

At this point in the book they started to go into different ways of plotting the data, different types of visualizations, the pros and cons of each type of visualization, how to interpret the data from each visualization and I became very bored, very quickly. I stopped reading.