Archive for the ‘Software Engineering’ Category

How to start DevOps in your organization – the right way

November 16, 2014 Leave a comment

The topic of DevOps has reached a point where any discussion about any improvement – whether accelerating engineering velocity, improving quality, or maintaining exceptional availability and performance – will quickly reach a conclusion that DevOps is required.

The truth – very few organizations implemented DevOps.  But why?

Before defining DevOps and making suggestions that any organization can begin implementing immediately, it’s important to become students of history for a few moments.

In 2013, annual revenues reached $75B.   This level of activity will keep any CTO busy, especially Werner Vogels.   How does he do it?

In 2006, Werner was interviewed by ACM.    It doesn’t mention DevOps.   DevOps was coined as a term   3 years later as DevOps Days conference,  or The Conference that brings development and operations together.

Even before DevOps became an industry accepted term, Werner said what I believe is the most important foundation of DevOps, or the basic truth:  “You build it.  You run it”.

It’s often difficult to take a basic truth and then create an implementation of this basic truth in a complex organization, where for example PCI-DSS compliance requires segregation of duties.

I heard many times that DevOps is a culture and Engineering and Operations have to work together to implement DevOps culture.

DevOps is not a culture by itself.   It’s the biggest mistake companies make when declaring that DevOps is just a culture and trying to implement it.  “Something” has to exist and operate effectively for DevOps culture to be born and sustained.

An organization with zero team members has no culture.  There is no one present to even plant the seeds for any culture to be felt and appreciated by other team members joining the organization.

DevOps is … (a definition for large, complex organizations trying to implement DevOps)

1. Is a team with a leader who understands both the needs of Engineering and Operations with executive support from all other organizations
2. Is a team with the right budget
3. Is a t
eam of specialists with one mission:  to support a continuous delivery model by …
a.  Building a maintaining a platform where engineers can build, test, and deploy their components a manner which is identical to production environment – before deploying to production environment
b.  Building and maintaining an integrated set of tools to enable engineers to perform 3.a effectively
c.  Providing support to engineers to move to this model
d.  Reminding and encouraging everyone to believe in the value of “You build it.  You run it”

What are the signs that your large, complex organization is not ready for DevOps?

Believing that DevOps is just a culture without an empowered team, led by the right leader, and with the right budget.

Unable to discuss how to fund DevOps.

Unable to agree about the reporting structure for the DevOps team in an organization which never had DevOps in the past

Deploying code through multiple, segregated environments instead of increasing focus on automated deployment, automated verification, and automated simulation of critical business activities in order to deploy as many times per day as needed.

Taking segregation of duties to such an extreme that engineers do not have access to logs in the production environment.

Unable to create the compensation structure which will attract the very best and the very brightest DevOps practitioners.

Unwilling to recognize that even if all of the above are in place, taking one step backwards in order to take 3 steps forward is often a necessary, difficult decision.

DevOps is not easy. It becomes easier when “You build it. You run it” becomes the overarching basic truth which drives everything else.

Categories: Software Engineering Tags:

Why the first phone interview with the hiring manager is the most important one

I wanted to share my advice with all great engineers who are seeking a rare opportunity.   Lets begin.

You noticed that Company A is looking for a Principal Software Engineer.   The job description is perfect and the company seems like a great place to work.

You applied.  The internal recruiter called you immediately.  Good news – your resume generated interest.   More good news!  The hiring manager scheduled a phone interview with you.

Your primary objective:  make the phone interview the most convincing discussion where future in-person interviews will be only a confirmation that you are the right person for this position (not a candidate – more on that later in the closing argument).

First – take the phone interview very seriously.  Prepare, prepare, prepare.

– Create time and environment where you can be 100% focused on the discussion.   Standing outside on a busy street while trucks are passing by is not a good idea.

– Ensure that your phone works.  Taking the call in a building where your mobile phone is guaranteed to have very poor reception is not advisable.

–  Research the company.   Read the last annual and quarterly reports (if the company is public).    Research competition, business milestones (acquisitions, divestitures), product lines, revenue by product line.  Is the company growing?  Is the growth slowing down?   Which products may not be competitive?  Which product YOU may working on and what kind of problems YOU may be asked to solve?  Would you rather work on a product which generates 10% or 80% of the company’s revenue?

– Do not be late if you have been provided with a conference call number and a specific time to dial in.   My suggestion:  dial 2 minutes before and wait for the hiring manager to join.  As a hiring manager, I dial in 3 minutes early and try to learn whether the candidate is prompt.

The hiring manager is looking for 3 things, although not every hiring manager will clearly explain these 3 things to you or even ask questions in a manner indicative of these 3 things.

These 3 things are nevertheless essential for you to prove beyond any doubt that you are the right person for the Principal Software Engineer position.

1.  Are you a methodical problem solver capable of dealing very complex problems?

2.  Are you a master of your craft (technology) and do you believe that your code speaks for yourself?  Hint:   if asked to participate in an unscheduled code review, do you welcome it or run aware from it?

3.  Are you good enough to lead by example and teach others by example?  Will you fit in the team?

Even if the hiring manager does not ask you about the last particularly difficult problem you solved, volunteer to discuss it in a manner relevant to the business of the company you are hiring.

Talk about technology like you designed and built it.   Go deep.  Discuss advantages and disadvantages of solution approaches you proposed recently. Share why you made a certain decision.  Defend it.  Your ability to make balanced decisions in an imperfect environment is what the hiring manager is looking for.

Get excited about sharing knowledge and replicating excellence.   The right hiring manager will look for more than just your technical depth.  He / she will also look for your ability as a Principal Software Engineer to lead by example and help grow the team by sharing knowledge.

The next in-person interview will be just a confirmation that you are the right person for this position.   Stop being a candidate.   Be the very best in what you do and share it with passion for delighting the customer because the customer is using a software product you built – AND signed your name on it.

If you follow these suggestions and get the job you want, drop me a line.   I will be glad to hear from you.



What makes a great CTO

A few years ago, one of my senior software engineers, “D”, and I had a career discussion.   “D” was a terrific engineer, wanting to advance his career very quickly.  Perhaps too quickly.   Yet I knew “D” was under compensated and asked him to be patient.

“D” was considering a very good offer:  well compensated with the role and title exactly what “D” wanted to have.   “D” asked for my advice.  Some of you are wondering why would an employee be asking his boss for career advice which may in fact cause the employee to leave.   The answer is simple:  if I fail to build the most desirable culture and a great place to work, then I fail as a leader.   Employee would leave anyway.   The best part of my job is to see great people grow, leave, and then … return to my organization.

Despite my advice, ‘you are not ready’, “D” left.  We stayed in touch.   6 months later, “D” called me told me that his job was in trouble.   “D” admitted the organization was very complex and many urgent issues had very little to someone’s ability to apply their technical depth.  “D” was clearly not succeeding.

‘May I come back?’, asked “D”.   “D” returned, learned a lot more, and became one of the most committed employees in my organization.   I cannot thank “D” enough for all contributions over time.    In every customer success story, “D’s” contributions can be easily noticed.   By the way, “D” reported to a Sr. Director of Engineering in my organization, or “G”.   “G” was a stellar engineering leader, capable of so much more.  “G” left for a better opportunity but we stayed in touch.

My phone rang.  It’s “G”.  “G” is one of the candidates being considered for a CTO position.  However, “G” also shared with me that the executive recruiter considered “G” to be candidate number 10 our of 10 candidates.  Not a good position to be in.

“G” asked for my advice, “what makes a great CTO?  How can I impress the company?”

I told “G” the challenge was not to impress anyone.   Simply tell your story, knowing that you are the best.   Then you will no longer be a candidate, but someone who will be hired almost immediately.  Stop being a candidate.  Stop.

Great CTO is someone who can …

1.   Create and execute a multi faceted plan which supports business strategy.   This plan is not just a technology plan.  This plan must address three elements of success:  people, process, and then technology.  In this very specific order.

2.  Practice innovation / speak innovation / inspire innovation.   This is the most difficult task of any CTO.  How does one innovate in the era of constrained budgets and tactical priorities?  Don’t stop trying and always be transparent.   The world of software hasn’t forgiven many CTOs who failed to match the rhetoric with the right culture and realistic budgets.  As one engineer told me, “how can I innovate if I work 60 hours a week.   My innovation time is now sleeping time”.

3.  Deliver great results in a thoughtful, anticipatory manner.    Delivering something that works on time and on budget is not enough. Deliver something which doesn’t need refactoring and can support the needs of future customers without 5 emergency releases.   Have the vision to do it and ignore those who say this is not necessary.

4.  Govern by serving your people.  Set clear goals, empower, enable, and then get out of the way.

“G” finally had the interview and called me 2 weeks later with disappointing news.   The company elected to hire someone else.

“G” did everything right.   He told his story which was a story about a great CTO.   I advised “G” to be patient.

Four months later, “G” called me with terrific news.   “G” had an interview with another company.

“How did it go?”, I asked.

“Well, the CEO stopped the interview and asked me when I could start”, “G” answered.

“What did you tell him?”, I asked with great interest.

“I told him a story about “D” who left, realized it was a mistake, came back, and become an amazing employee.  Do you remember that story?”.

Yes, I do.  That’s what makes a great CTO.  Great CTOs build amazing cultures, where engineers build amazing products.



Categories: Software Engineering

How to determine if you are ready to manage – or ready to lead

In not so distant past, “N” was one of the most capable principal software engineers.  “N” could solve any technical problem, deconstruct something very complex into a series of simple solution steps, and commanded respect from everyone in the organization.

“N” and I had a regularly scheduled lunch every month.  The topic of the day was “N”‘s prior request to be considered for a promotion. “N” felt strongly about being promoted to an engineering manager role during the next 6 months.

“Tell me why you are ready to become an engineering manager”, I asked.

“N” responded, “I am ready.  I am already acting as a manager of an important initiative where engineers from other teams are virtually reporting to me”.

“Do you know the difference between a manager and a leader?”, I asked.

“N” paused and acknowledged that difference was difficult to define.

“Why don’t you do some research and then let’s talk again”, I asked.  “N” agreed.

Our next lunch was one of the most memorable experience I had as a leader with an opportunity to identify and develop the next leader.

“N” researched my question well.  “N” also provided many examples.

“That’s very good.  In your opinion, what is the single most important difference between a manager and a leader”, I asked.

“N” took time to provide an answer, clearly uncertain about the single difference. “N” also felt that the answer would be one of the reasons the promotion would arrive sooner than later.

“N” finally provided the very answer I was looking for.  “The manager does things right; the leader does the right thing”, N responded with confidence.

“But can you do things right if the objective isn’t the right objective?”, I asked.   ‘N” said, “No, I can’t execute something that isn’t right by definition,  I also can’t follow a leader who doesn’t do the right thing”.

“N” proved to me that he was ready to be a manager.

Managers do things right.  Leaders do the right thing which inspires managers to execute the mission correctly – or do it right.   Every manager is a leader in training.   Help your leader to do the right thing by asking the right questions.

Managers who simply do – lose their privilege to to be a managers of others.  These managers will also never become leaders who inspire others by doing the right thing.

“N” became a great engineering manager and later one of the best engineering directors I had the privilege of managing in the past.

2014 New Year resolutions for everyone in a CTO role

December 31, 2013 Leave a comment

This year, I will only focus on  ‘people’ component of ‘people, process, and technology’ equation for success.

Sustainable competitive advantage can only become an achievable goal when an organization is being built – by design – to operate at the highest possible level.  Without it, sustainable competitive advantage is only a dream.

Retain like you attract.   

Attracting a star performance is exceedingly difficult.   Once hired, the difficult work continues.   Are your star performers happy?  How do you know they are working on the right tasks?  How much time are you spending developing your organization?  Is the vision clear?  Is the plan to realize the vision clear?

Employees do not leave companies.  They leave managers.   Know your managers.

When was the last time you took the time to assess your own management team?  What should the bar be for the management team?   Do your managers know the “Right Side Up” organizational principle where managers work for their own employees, set clear objectives & measures of success, empower & equip, and then get out of the way?  Do they know the difference between leading and managing?

Plan without action is futile.  Action without a plan is fatal.   Plan well. 

The best plans enjoy the support of the entire team.  Even the most difficult plans with clear business outcomes will be supported by your team.  Yet no one will follow you if the plan doesn’t explain how all the hard work ahead will create sustainable benefits for the business.


It’s very easy to describe something very complex. It’s incredibly difficult to transform a very complex topic (or goal) and make it simple.   Once simplified, the effort to build viable plan becomes equally simpler.

Only happy people build great software.   Build a dialog with every team member.

Happy people are happy at work and at home.  They are free to create and innovate.  How well do you know every team member?   Get to know them again in 2014.

Is your talent acquisition process working?

November 24, 2013 Leave a comment

While waiting to board a delayed flight from Dallas, TX to Washington, DC, I had an opportunity to speak with a fellow software executive about what works and doesn’t work in his organization.

“I cannot hire great people fast enough”, he said.

“How do you know if you talent acquisition process is working?”, I asked.

“Well – we have internal recruiters who are working very hard to find the right people”, he countered.

“What is your rejection rate after in person interviews?”, I asked.

“Hmmm … it’s quite high but I don’t know the details”, he answered with a touch of disappointment.

The number one problem – by far without an exception – why talent acquisition process is not working is inability to build a funnel of candidates that show potential.

This is especially true in companies that rely on internal recruiters who haven’ spent enough time with hiring managers or – when hiring managers are too busy to provide meaningful feedback to internal recruiters.

Talent acquisition problem – by the numbers:

– 1 principal level engineer role open
– 4 weeks later:  100 candidates in the pipeline;  20 candidates in the “funnel”
– 6 weeks later:  10 phone screens and 5 candidates reaching in-person interview phase
– 10 weeks later:  all 5 candidates fail in-person interviews

Almost 3 months later, the process continues and a critical  position remains open.

What  anyone can do in the same situation to begin solving the talent acquisition problem:

– Develop a practical interview guide for internal recruiters who can ask specific questions and capture answers.  The questions have to be 100% relevant to what a new hire would experience on Day:  problems he / she solved, using the same or applicable technology and tools, and how he / she worked with everyone in the organization.

For example:  if the new hire will be working on critical production support problems, ask about similar efforts in the past.   Don’t just ask about common Java questions.  Ask how the candidate used specific Java relevant technology to solve problems of interest.   Did he / she used a profiler?   What was the problem?  What did the profiler identify?  What was the root cause?  How did you verify that the problem was solved?   Was 6 hour soak test sufficient?  Why?

This will help identify candidates who will advance to in-person interview phase with a much better chance of success.

– Develop a broader interview guide for in-person interviews.  This guide should simulate how someone could engage with the team without being hired.   Make these interviews as long as needed to spend enough time with the candidate while working on a specific problem.  Get a conference room with a whiteboard.  Ask multiple team members to attend and simulate a team meeting where a candidate is expected to contribute to the problem solving process.  Continue to ask probing questions to determine what the candidate knows and doesn’t know.  Determine learning style:  self motivated or something else?  Can he / she learn?
Can he / she advance in the organization and become an asset in the succession plan for key roles?

The success of your talent acquisition process will directly influence your own success as an engineering leader.  Do not “outsource” it to someone else.

Steve Ballmer’s resignation through former Microsoft superstars’ resignation letters

August 23, 2013 Leave a comment

The resignation of Steve Ballmer as Microsoft’s CEO will undoubtedly create an endless stream of comments.

I will step aside for a moment (and refrain from comments) by referring to the resignation letters from two former Microsoft superstars:  Dave Stutz and Ray Ozzie.

In 2003, Dave Stutz resigned and published a resignation letter which – with full credit going to Dave – follows below.  The letter is titled Advice to Microsoft regarding commodity services.   For those that have little time, the last sentence in Dave’s letter says it all.   “Stop looking over your shoulder and invent something!”

In 2010, Ray Ozzie resigned from Microsoft and wrote a memo called Dawn of a New Day where he asked everyone at Microsoft to imagine a post-PC world.

Full text can be found at

Advice to Microsoft regarding commodity software – Copyright 2003 by Dave Stutz
The market for shrink-wrap PC software began its slow upmarket ooze into Christensen obsolescence right around the time that Microsoft really hit its stride. That was also the time of the Internet wave, a phenomenon that Microsoft co-opted without ever really internalizing into product wisdom. While those qualified to move the state of the art forward went down in the millennial flames of the dotcom crash, Microsoft’s rigorous belief in the physics of business reality saved both the day and the profits. But the tide had turned, and a realization that “the net” was a far more interesting place than “the PC” began to creep into the heads of consumers and enterprises alike.

During this period, most core Microsoft products missed the Internet wave, even while claiming to be leading the parade. Office has yet to move past the document abstraction, despite the world’s widespread understanding that websites (HTML, HTTP, various embedded content types, and Apache mods) are very useful things. Windows has yet to move past its PC-centric roots to capture a significant part of the larger network space, although it makes a hell of a good client. Microsoft developer tools have yet to embrace the loosely coupled mindset that today’s leading edge developers apply to work and play.

Microsoft’s reluctance to adopt networked ways is understandable. Their advantaged position has been built over the years by adhering to the tenet that software running on a PC is the natural point at which to integrate hardware and applications. Unfortunately, network protocols have turned out to be a far better fit for this middleman role, and Microsoft, intent on propping up the PC franchise, has had to resist fully embracing the network integration model. This corporate case of denial has left a vacuum, of course, into which hardware companies, enterprises, and disgruntled Microsoft wannabes have poured huge quantities of often inferior, but nonetheless requirements-driven, open source software. Microsoft still builds the world’s best client software, but the biggest opportunity is no longer the client. It still commands the biggest margin, but networked software will eventually eclipse client-only software.

As networked computing infrastructure matures, the PC client business will remain important in the same way that automotive manufacturers, rail carriers, and phone companies remained important while their own networks matured. The PC form factor will push forward; the Pocket PC, the Tablet PC, and other forms will emerge. But automakers, railroads, and phone companies actually manufacture their products, rather than selling intangible bits on a CD to hardware partners. Will Microsoft continue to convince its partners that software is distinctly valuable by itself? Or will the commodity nature of software turn the industry on its head? The hardware companies, who actually manufacture the machines, smell blood in the water, and the open source software movement is the result.

Especially in a maturing market, software expertise still matters, and Microsoft may very well be able to sidestep irrelevance as it has in the past. The term “PC franchise” is not just a soundbite; the number of programs written for the PC that do something useful (drive a loom, control a milling machine, create a spreadsheet template, edit a recording…) is tremendous. But to continue leading the pack, Microsoft must innovate quickly. If the PC is all that the future holds, then growth prospects are bleak. I’ve spent a lot of time during the last few years participating in damage-control of various sorts, and I respect the need for serious adult supervision. Recovering from current external perceptions of Microsoft as a paranoid, untrustworthy, greedy, petty, and politically inept organization will take years. Being the lowest cost commodity producer during such a recovery will be arduous, and will have the side-effect of changing Microsoft into a place where creative managers and accountants, rather than visionaries, will call the shots.

If Microsoft is unable to innovate quickly enough, or to adapt to embrace network-based integration, the threat that it faces is the erosion of the economic value of software being caused by the open source software movement. This is not just Linux. Linux is certainly a threat to Microsoft’s less-than-perfect server software right now (and to its desktop in the not-too-distant future), but open source software in general, running especially on the Windows operating system, is a much bigger threat. As the quality of this software improves, there will be less and less reason to pay for core software-only assets that have become stylized categories over the years: Microsoft sells OFFICE (the suite) while people may only need a small part of Word or a bit of Access. Microsoft sells WINDOWS (the platform) but a small org might just need a website, or a fileserver. It no longer fits Microsoft’s business model to have many individual offerings and to innovate with new application software. Unfortunately, this is exactly where free software excels and is making inroads. One-size-fits-all, one-app-is-all-you-need, one-api-and-damn-the-torpedoes has turned out to be an imperfect strategy for the long haul.

Digging in against open source commoditization won’t work – it would be like digging in against the Internet, which Microsoft tried for a while before getting wise. Any move towards cutting off alternatives by limiting interoperability or integration options would be fraught with danger, since it would enrage customers, accelerate the divergence of the open source platform, and have other undesirable results. Despite this, Microsoft is at risk of following this path, due to the corporate delusion that goes by many names: “better together,” “unified platform,” and “integrated software.” There is false hope in Redmond that these outmoded approaches to software integration will attract and keep international markets, governments, academics, and most importantly, innovators, safely within the Microsoft sphere of influence. But they won’t .

Exciting new networked applications are being written. Time is not standing still. Microsoft must survive and prosper by learning from the open source software movement and by borrowing from and improving its techniques. Open source software is as large and powerful a wave as the Internet was, and is rapidly accreting into a legitimate alternative to Windows. It can and should be harnessed. To avoid dire consequences, Microsoft should favor an approach that tolerates and embraces the diversity of the open source approach, especially when network-based integration is involved. There are many clever and motivated people out there, who have many different reasons to avoid buying directly into a Microsoft proprietary stack. Microsoft must employ diplomacy to woo these accounts; stubborn insistence will be both counterproductive and ineffective. Microsoft cannot prosper during the open source wave as an island, with a defenses built out of litigation and proprietary protocols.

Why be distracted into looking backwards by the commodity cloners of open source? Useful as cloning may be for price-sensitive consumers, the commodity business is low-margin and high-risk. There is a new frontier, where software “collectives” are being built with ad hoc protocols and with clustered devices. Robotics and automation of all sorts is exposing a demand for sophisticated new ways of thinking. Consumers have an unslakable thirst for new forms of entertainment. And hardware vendors continue to push towards architectures that will fundamentally change the way that software is built by introducing fine-grained concurrency that simply cannot be ignored. There is no clear consensus on systems or application models for these areas. Useful software written above the level of the single device will command high margins for a long time to come.

Stop looking over your shoulder and invent something!