The root cause of all things complex is simple (or how to explain the state of US economy)

December 22, 2012 Leave a comment

While attending a recent holiday party and discussing the US Presidential elections, my dear friend suddenly asked me,

“You are a CTO, Leon.  You solve very difficult problems every day.   What’s wrong with the US economy?”

I began my career as a software engineer designing and engineering “must never fail” financial settlement systems.  The core components of the system I was responsible for have been written in Assembler, despite the availability of higher level languages.  Why?  To ensure maximum awareness of the problem context and environment.  This particular system even had its own access method to maximize I/O performance.

I learned a lot in my early career.  But what I am most grateful for is an early encounter with the universal truth that the root cause of all things complex is simple.   I learned the value of this universal truth while spending a few days looking for the root cause of a resilient memory leak.

The state of US economy resembles a classic memory leak – and I believe you will agree.   The virtual address space is fairly large (think of it as the ability of US government to raise the debt ceiling and borrow more money).    So ‘we’ allocated – or created – $16.3 trillion worth of debt.

But why can’t we seem to reduce our national debt?  Referring to what every software engineer knows:  in a classic address space model, there are memory release mechanisms.  Translating to our fiscal reality, the US government collects tax revenue and use some of this tax revenue to pay the interest and principal on national debt, ideally reducing the debt to a manageable level.   That’s why this is classic memory leak:  debt grows faster than the payments towards the interest and principal.

But the question is – why?

I believe the answer is actually quite simple and lies in the labor participation rate which is now at the lowest point since 1981 at 63.5%.

To understand the sense of urgency, one does not even need to be an economist.   You are a homeowner in a neighborhood with 100 houses.  All 100 homeowners pay monthly association fees of $10 towards a monthly association management budget of $1000.  Let’s place 50 homeowners in financial distress who cannot pay $10 a month.   The remaining 50 homeowners still have to cover the basic maintenance needs of the association.  Their monthly dues are now $20 (times 50 = $1000).

The state of US economy is largely equivalent to the problems of the above homeowner’s association.  The US economy lost entire industries in 30 years and did not replace departing jobs with anything that carries the same economic benefit.   Today, fewer and fewer Americans are able to participate in the economy and pay taxes.   Which means that taxes for all of us in the US will go up without any additional economic benefits.

So what’s the solution?

When your US Senator or US Congressional Representative promises to create jobs, ask them to explain how their legislative agenda will foster growth first.   Anyone who talks about job creation without first discussing how to generate growth ought to resign from the Congressional seat.   Better yet, please exercise your right to critical thinking while voting.

Arguably, one of the biggest economic growth drivers of US economy has been the decision to build the interstate highway system, or The Dwight D. Eisenhower National System of Interstate and Defense Highways. Find some time to read this article from

I hope US Congress will find the lessons from the past just as applicable when trying to solve today’s problems.

When great Apple software becomes a bad apple and what to do (whether Apple listens or not)

November 5, 2012 1 comment

In February 2010, I wrote about the importance of negative testing and the biggest impact of insufficient negative testing:  customer’s confidence in your software.  ”If this simple functionality fails, what else may broken”.

Apple is a very successful company by almost every financial and non-financial measure.   I have been an early adopter of Apple products and still use Apple products every day.   However, Apple is beginning to exhibit early signs of trouble, the kind of trouble that every leader begins to lose sleep over because the issues are broad enough to involve the three elements of success:  people, process,  and technology.   When all three appear to be broken , it’s no longer business as usual.

In the past, Apple software update has been simple and uneventful.   The update appears as a notification.   Few minutes later, the update process is done and the user continues to be productive while enjoying new features.

Why Apple should be very concerned:  from

“I’m really only here to complain. I have learned not to expect a response from Apple or timeline for fixing the issue. I just want Apple to know that it’s alienating some of its most loyal customers. For years, I used your products to keep myself and my media organized. I don’t want to change that and do not look forward to changing it. Still, I was a PC user once and I can certainly go back to being a PC user again. The inconvenience of attempting to organize my files on a PC will be offset by the time saved not having to deal with features you introduce that only work some of the time. I’m begging Apple to STOP BREAKING THINGS THAT ARE NOT BROKEN and to GO BACK AND FIX THE THINGS YOU BROKE. I would much prefer that you concentrate on fixing these glitches than introducing useless new UI features like different shades of color at the top of the iOS task bar.”

The problem could not be stated any better.


The departure of Scott Forstall has been discussed in detail, including possible reasons.    Scott’s departure highlights a much bigger problem at Apple.  If senior executives, like Scott Forstall and Sir Jonathan Ive do not collaborate effectively to develop products which should – by design – work perfectly in Apple’s’ own ecosystem, what does it say about Tim Cook who – as a leader – could not recognize this problem early enough and its impact on products, quality, and customer good will?


There are many complaints in Apple discussion forums about iPhone 5 and battery drain problems and Contacts disappearing after upgrading to IOS 6.

How long does it take for a quality engineering team to create a soak test where iPhone 5 – and 90% of popular apps – would be subjected to a 72 hour typical usage tests?   This simple test would most certainly produce some results worthy of discussion and corrective action – for the benefit of the customer.


iPhone is a platform where greater value for the customer is enabled by new software-based capabilities.   Get it right – and Apple gains new customers.  Get it wrong – and at some point revenues and margins will be impacted.

What to do

If you are an Apple user, you may want to slow down the adoption of new IOS and OS X releases.   And always have good backups – several backups.   Sharing my own experience:

– OS X Lion reached a stable point with 10.7.3 release
– Do not plan to install OS X Mountain Lion until 10.8.5 release and only on an external bootable clone of my hard drive to perform extensive testing
– Still using iPhone 4S and IOS 5.1.1;  will not upgrade to IOS 6 in the foreseeable future
– Do not plan to purchase iPhone 5 for the same reason why iPhone 4S is a better iPhone 4
– Will however purchase iPad Mini

Categories: Software Engineering

Hiring a mobile apps software engineer: what to look for

September 25, 2012 Leave a comment

One of my former managers called me last night.   “Steve” worked for me several years ago.  Three months ago,  “Steve” was actively recruiting 2  mobile apps engineers to support both iPhone and Android platforms.   The search has been very difficult due to a shortage of qualified engineers.

“Steve” still had many resumes from great engineers who had limited experience developing mobile applications.   Could the right candidate be among these engineers?  I shared with “Steve” several suggestions to help identify the right talent.   There months later, “Steve” happily extended  offers to 2 engineers.   Both turned out to be fantastic members of the team.

What should a hiring manager look for in a software engineer who will develop mobile applications?   Believe it or not, the number one item is not whether the engineer knows Objective C or Java.

#1:  Does the software engineer understand the five dimensions of mobile design context: location, locomotion, immediacy, intimacy, and device?  (source:  Forrester Research)

Location:  mobile users can use their devices wherever they are

Locomotion:  mobile users can use their device while on the go:  in the car, running, walking.

Immediacy:  mobile users can user their device at a moment’s notice

Intimacy:  mobile users can use their device for different purposes.   Device use can vary from a digital appendage to an occasionally used device for a specific personal or work task.

Device: mobile devices vary greatly in form factor and capabilities

#2: Does the candidate show a deep appreciation for solving problems that may arise while the software is operating in a mobile environment?

– What if the network is not available or signal is lost every 2-3 minutes?
– Are there any functional scenarios sensitive to latency issues?
– Security?  Authentication?
– Diagnostic capabilities?

#3:  Can the candidate select (and defend) a solution approach while considering three implementation paths?  What implementation path might be a better option for a given set of functional requirements?

– Native
– Hybrid:  where HTML5 is rendered inside the native application

#4:  Can the candidate describe, design, and implement a testing framework for the mobile application?

#5:  Can the candidate give examples of mobile applications which can serve as an example of user interface “done right”?   Why?

Then – and only then – it’s helpful to learn about specific technical skills and experience:  Objective C, Cocoa Touch, HTML5, SQLite (for Android) …

The next fantastic mobile engineering hire is only a few questions away.

Do Google style interview questions illuminate the talent in front of you?

July 31, 2012 3 comments

In December 2011, The Wall Street Journal published an article How to Ace a Google Interview. But do Google style interview questions help determine if the candidate represents the best fit for Google?

The best fit … any hiring manager looks for the best fit.   But can Google – and anyone else asking the same style questions – determine the best fit without creating a practical framework for the hiring decision incorporating both tangible  factors (experience, skills, problem solving depth) as well as  intangible factors (personality, composure, ethos)?

The best fit is not a mystery. Determining the best fit by the hiring equivalent of alchemy creates an enormous disservice to both the candidate and the company.  At stake:  the success of a company, its products, and customers.

Hiring decision is a highly educated, fact based hypothesis that the person receiving an offer will create significant value in the organization while working with others.

I will highlight at the end why the best fit is not a mystery and should never be a mystery.

Why don’t we create a virtual interview and ask the same question being discussed in the Wall Street Journal article?

“You are shrunk to the height of a nickel and thrown into a blender.  Your mass is reduced so that your density is the same as usual.  The blades start moving in 60 seconds.  What do you do?”

I suggest reading  How to Ace a Google Interview to appreciate the answers to the above question.   Throwing loose coins from the pocket with hopes to jam the blades isn’t likely to impress the interviewer.

Before providing my own answer, it’s helpful to understand the motivation behind the question.   If a software engineer is faced with a seemingly impossible situation, what will he / she do?    The truth:  direct answers to an impossible situation do not provide any valuable information about the candidate to the interviewer.  The person will only become a victim of his or her own failure while trying to answer the question.  And the interviewer will not recognize depth of one’s talent.

I also ask questions that may seem impossible to answer during my own interviews.   But I try to work with the candidate by providing clues and alternatives to learn more about the candidate’s problem solving approach.

So why would one ask these questions?   I have a completely different motivation for asking impossible question.   I want to see candidate demonstrate the following  ‘genes’:

–       Accept failure / it happens

–       Learn from the failure / convert lessons learned into actionable plan

–       Quickly change the conversation and refocus on either preparing for this situation or avoiding it all together? Remember one of the best books written on the subject, “Only The Paranoid Survive” by Andy Grove, former CEO of Intel Corporation?

–       Tell me in detail how they would prepare for this situation and be better equipped to handle it

–       The best candidates will even prepare an action plan for dealing with any aftermath of the problem being addressed

It’s time to put myself in the spotlight and answer the same question.

“I am very glad you asked me this question.   This particular problem already happened to a colleague of mine and I learned from the unfortunate incident.   Should this problem happen to me, I have 3 methods to deal with this problem:

–       I always have a portable laser device with me.   It will cut through the blender glass within 10 seconds allowing me to exit before the blades begin spinning.

–       If the laser device fails, I will use a high frequency sound generator.    The glass will shutter within 8 seconds.   I prefer to use the first option and prevent the broken pieces of glass flying in all directions.

–       If the first 2 methods fail, my portable electro magnetic pulse (EMP) generator will fry the motor inside the blender but will also affect the microwave nearby

–       All three devices have already received appropriate regulatory approvals

As you can see, the blender will be inoperable after my successful exit.   So I took the liberty to order 2 replacement blender glass containers after each incident.  The blender can be immediately returned to service for the benefit of our customers.  Do you have any additional questions for me?”

Google hires very smart people.   But in the business of software, even smart people can still fail by failing to create commercial value from an idea.

Here is a partial list of of products Google launched and discontinued:

–       Aardvark

–       Google Answers

–       Google Buzz;  launched in February 2010;  discontinued by the end of 2011;  less than 2 years in operation.

–       Google Desktop

–       Google Directory

–       Google Fast Flip

–       Google Health

–       Google Gears

–       Google Squared

–       … and others …

Many Google employees (including product managers who ultimately set the product direction and determine commercial roadmap) worked very hard on products listed above.    Imagine the aggregate investment of time and money invested in these products.   It’s not trivial.    This investment can also be seen as a lost opportunity because these bright product managers and engineers could have worked on something with a much greater commercial potential.

Would the outcome be different if Google focused on a different set of questions during the interview process?

“We are thinking of developing Google Buzz.   What do you think about it’s potential?  What about the competitive landscape and risks?   How would you maximize the chances of Google Buzz succeeding?

That’s why the best fit is not a mystery.   The questions during the interview process must be relevant, including the motivation behind seemingly impossible questions.

Final point: never leave the candidate alone with their own sense of failure – no matter how small or big  – during the interview.   Because you still have to coach and mentor a newly hired superstar who happened to answer the seemingly impossible question to your satisfaction.   Career long coaching and mentoring is a better predictor of success than an answer to an impossible situation during an interview.

Which mode does your technology organization operate: yesterday, today, or tomorrow?

April 8, 2012 1 comment

Long flights can have unexpected benefits:  the person sitting next to you and how much you can learn from this person.

While in transit over the Atlantic some time ago, my neighbor was a CEO of a high technology company.   Twenty minutes into our conversation, he asked me, “as a CTO, what is the single most important concern that keeps me up at night”.

I paused for a moment and answered,

… whether my technology organization operates in yesterday, today, or tomorrow mode.

He smiled, signaling that he understood exactly what it meant.

The question of operating in yesterday, today, or tomorrow is everywhere – not just in software or high technology market alone.   Everyone heard of the expression, “do not delay until tomorrow what can be done today”.

I will illustrate several helpful examples – very familiar to all CTOs:

– The competitor unexpectedly launches a new product which instantly places your own product about 9 months behind
– Your technology organization does not have the skills to refactor the product prior to accelerating the delivery of new features
– Your budget has been set 10 months ago and the conversation about incremental funding is moving too slowly
– The team is clearly stressed and very reluctant to work even harder, despite working at a breaking point for quite some time

By this point, it is clear that something is wrong.   If the CTO does not respond to these alarms, the technology organization will fail to execute – reasons no longer being important after the failure – and equally fail to support the change needed to reposition the business towards a position of competitive parity or – better yet – a clear position of strength.

In my experience, a small percentage of technology organizations operate in ‘tomorrow’ mode:  for some reason, everything that needs to happen today, happens tomorrow.   The product management organization produces requirements late.  Designs are not completed on time, or worse – poor designs still become the norm in subsequent releases creating or expanding technical debt.   Critical releases miss dates.   These organizations do not survive for too long and the right change in leadership will move the organization to operate in ‘today’ mode.

The vast majority of technology organizations operate in ‘today’ mode and spend too much time celebrating the fact that a release planned 6 months ago has been delivered on time.   What if a competitor launches a market-breaking, disruptive solution offering and the technology organization now has to deliver the current release 3 months earlier and deliver a new product at the same time?   Suddenly, being very good at tactical execution of a release planned 6 months ago is no longer the reason to celebrate.

High performance technology organization always – always – operate in ‘yesterday’ mode.   These organizations deliberately plan and execute critical initiatives many months ago in order to get ahead of the competition.


– Planning a complex refactoring effort well in advance in order to create more flexibility in the architecture or component model and be well positioned to accelerate the delivery of important features.
– Creating a succession plan for critical skills and proactively hiring ahead.   It takes at least 12 months of deliberate organizational transformation focus to upgrade the skill profile of a technology organization, while managing the natural disruption of under performers leaving and new stars arriving.
– Creating a global development organization with 4 teams working in 4 global locations and accelerating product development velocity by at least 60%.    This typically takes 18-24 months.

So what did I learn from the CEO sitting next to me on a long flight?

“Today is meaningless because today is mainly influenced by what happened yesterday.   If we are lucky, we may spend some time today to plan a better tomorrow”

“Yesterday is therefore the most important day.   If we do the right things yesterday, today and tomorrow will automatically benefit”

There is a lot of truth in what I learned from this CEO.

High performance technology organizations are fanatical about operating in an anticipatory mode (yesterday) and are better than their closest competitor in responding quickly to changes in business  climate.

Which mode does your technology organization operate?

Categories: Uncategorized

How to practically guarantee an offer to become the new CTO

March 29, 2012 1 comment

Highly admired, rapidly growing technology company has been searching for a new CTO who would play a critical role in the company’s next growth cycle.

You – the reader – were selected as one of the top 10 candidates and later as one of the two final candidates.   However, the company decided to hire the other candidate.  Why?  The truth is – you will never know.   But with hopes to help anyone being considered for a senior technology leadership position, I wanted to provide more visibility about the reasons why a seemingly terrific candidate ultimately did not receive an offer.

The top 3 reasons why the offer has not been extended to you:

1.  You did not ask the right questions, in the right sequence.

2.  You did not explain correctly what you would doing next Monday in your new role as a CTO

3.  You did not exhibit enough evidence of being able to quickly win the respect of both technology organization and other executive peers.

You did not ask the right questions, in the right sequence.

The job of any CTO in a software company is not to manage technology but to ensure that technology strategy supports the economic goals of a business, or 100% alignment between different phases of growth and how technology must evolve to support each phase of growth, while thoughtfully recognizing that some key initiatives must be sponsored and financed well in advance.

You must learn as much as possible about the company’s business during the interview process – in the right sequence.

First – learn about the company from the CEO (not an exhaustive list):  roots, key evolution milestones leading to the present, broad market conditions, major competitors (VP of Sales will provide more details), capital structure, investor profiles, investor goals, and resulting financial targets as well as time constraints.   It may or may not be possible to grow revenues 200% in 3 years.   These matters will have a significant impact on you as a CTO.

Second – ask to speak with the CFO.   That’s right – the CFO.   He / she will share with you how the company’s strategic plans are translated into operational plans and budgets which include all other organizations.   Is there a history of underinvestment in the technology organization and product engineering?   What’s the budget history of the technology organization during the last 3-4 years?    Are you walking into an organization where you may not have the right budget to ensure the success of the product development goals?  Or – the organization is mature enough to have a meaningful discussion and agree with you (hopefully the very leader being sought), even at the price of reducing investment in other areas?

Third – speak to VP of HR.   Learn about the state of technology organization / ask for the organizational chart.  Compensation – is it competitive?  Are there any open requisitions?   Imagine a growing organization where a principal level engineer role has been open for 6 months.   Why couldn’t the right person be hired?    Start making notes.   Can you have a meaningful conversation with VP of HR if the compensation structure in the technology organization you are about to inherit and lead is inadequate and about to trigger significant attrition?

Fourth – speak with VP of Sales.    What are we selling?   How?   Are the customers happy?  If not – what are the reasons?   Is there a detailed win-loss analysis process?   Can you get a copy of the latest report?

Fifth – speak with VP of Marketing and Product Management.   Is the requirements management process rigorous?   Are the requirements collected from all the channels:   customers, competitive intelligence sources, professional services?   How are the requirements managed:  documents or requirements management system?  When was the last time the product roadmap was changed?   What were the reasons?   How did the organization respond to some critical features being delayed instead of accelerated?   Ask for the product roadmap presentation.

Sixth – speak with VP of Professional Services to validate the state of the customer.   Is the customer community satisfied?   What is the typical deployment process:  duration, complexity?   How can the product be improved to reduce the complexity of the customer deployment process?

By this point, you have demonstrated 2 items the executive team is looking for:

– Ability to learn about the business from key executive peers (something you will have to do anyway but the key is to demonstrate this competency during the interview process)
–  Ability to extract key insights which will allow you to begin thinking about aligning the goals of the technology organization with the goals of the company

You did not explain correctly what you would doing next Monday in your new role as a CTO

If someone interviewing you asks, “what would you do during the first 90 days”, chances are your value as a candidate has already eroded.   Do not wait until this question is asked.

Find the right moment to mention that you are already thinking about the plan.   But before proposing the plan, you need to learn a lot about the business, your own organization, as well as every other organization that are critical to your success.

What every CEO is looking for in a new CTO when the CTO discusses the 90-day plan:

– 0 – 30 days:  make no changes in the technology organization.   Engage every employee.  Listen & take notes.   Resist every temptation to make significant changes.   Focus on creating a culture of transparency and accountability.   Team superstars will respect you from day one.  Underperforms will realize that the time for a new job search has arrived.

– 31 – 60 days:   become a silent participant in all other meetings.   Attend sales pipeline reviews.   Attend customer account planning calls.   Learn the pain points in every other organization that may be addressed later by adjusting the product roadmap or simply correcting product functionality.   Win the respect of your executive peers by sharing their daily challenges.

– 61 – 90 days:   Invest time in creating a very clinical perspective on what you learned.   Present it to the executive team.   They may not even realize certain issues or opportunities exist.   Then begin working on a plan to meet current operational goals while introducing methodical changes to position the company for growth.   Present it to your own organization.   Do not hold back any information.   Your commitment to transparency and clear goals will be the reason why everyone will accept you as the new leader.

You did not exhibit enough evidence of being able to quickly win the respect of both technology organization and other executive peers

The best part about item 3:   if you have done well with items 1 and 2, this one becomes a non event.  You have already demonstrated your ability to win the respect of everyone you will be closely working with.

If you act as a CTO while being interviewed for a CTO role, the offer to become the new CTO will be practically guaranteed.


5 mistakes growing companies make when hiring a CTO (the results could be fatal)

The examples of hiring the right person for the job – outside of the software business – are everywhere.

– Passengers on a Boeing 737-800 know that the pilot in command has been properly trained to fly this aircraft
– Passengers on an Airbus A320 know the same
– The country or state where you reside asked you to pass a driver license exam in order to drive

Yet growing technology companies make mistakes – often fatal mistakes – when hiring a CTO, in this case someone who will play a leading, critical role in formulating a technology strategy (with a plan to deliver of course) in growing the business.

Mistake 1:  Failure to ask the right questions and assess how you – as a company – arrived at the present moment in time

“There is a reason why today is meaningless and meaningful at the same time.  Meaningless – because today is simply an outcome of everything we have done yesterday.   Meaningful – because everything we do today will influence tomorrow.”

– Are you a company that has evolved via acquisitions or organic growth?   How would you rate yourself against the industry and closest competitors?
– Who is your competition?  How well do you know your competitors and their technology leadership team?    Will your new CTO be as capable as their CTOs?
– What is your growth strategy?  Is there an organizational strategy to support the growth strategy?
– What is the true state of the product?  Zero technical debt or lots of technical debt?
– Is there tension between members of the executive team?   There is always some healthy tension.  But – is there a fundamental misalignment between the strategic agenda and the ability to execute?  Would the new CTO simply add to the tension and – despite their best efforts – fail early?

Mistake 2:  Failure to create a clinical competency matrix which clearly defines who we want as the new CTO

Yes – the competency matrix has to be clinical (or “binary clear”).

Imagine the following scenario.   The new CTO joins the company and quickly notices that the sales team is actually very average.   What’s the point of building a fantastic product if the sales team cannot sell it?   Removing the constraints of organizational boundaries, what is the right thing to do?   Will the new CTO have the organizational courage and maturity to engage and help the sales team sell?  And if this approach does not work, approach the CEO and explain the problem?

That’s why Mistake 1 above is the one that almost always leads to a fatal outcome.   Growing companies that do not know themselves – or fail to learn about themselves – have very little chance of successfully hiring the right CTO.   Unless the company knows itself, it cannot create the very matrix which will define the right CTO against which the candidates will be compared during the search efforts.

Mistake 3:   Failure to hire the right candidate vs seeking “safe” candidate

“Safe” candidates are very attractive:  do not have to relocate anyone, may be coming from the competitor (but are they capable of quantum innovation or simply able to avoid mistakes that the competitive circles already know).

The right candidate are much more difficult to identify unless the competency matrix is “binary clear”.   Once identified however, open all doors to get this person onboard.

Mistake 4:   Failure to accommodate the friction which is always associated with progress

The right candidate will cause friction.  It’s inevitable.  Progress is impossible with friction.

Does the executive team understand the strategic direction and how the newly hired CTO will help achieve the desired business outcome?  Change will be inevitable and with progress friction will arrive.   Is everyone truly ready for change and some healthy incremental friction?

Mistake 5:   Failure to support the newly hired CTO

Newly hired CTOs fail for many reasons.   One of the major reasons is failure to understand and engage their team.  I always suggest that any new CTO should spend the first 30-45 days in the trenches to understand everyone’s challenges and needs.  Only then the new CTO will have enough organizational capital to  describe the mission and inspire everyone to follow.   The second major reason why a new CTO fails is lack of support from the executive team that invested an incredible amount of time (and sometimes money) to find the right person and not the “safest” candidate for the job.  As an agent of change and progress, the new CTO will be a messenger of disappointing news in the beginning.   One of their early goals is to understand what works and does not work before suggesting a specific course of action.   Do not shoot the messenger.   If the new CTO fails, you will fail with him or her.