Mark's Stories: "The carpets are so clean, we don't need janitors!"

(edit: Hello to the folks who are coming here from YCombinator Hacker News.  Feel free to comment here as well as on YCHN.  I've posted a few more details about this story at YCHN.)

At one company I worked at, one of the problems it didn't have was IT.

When someone was hired, by the time they got to their new desk, there was a computer on it with the correct image on it, their desk phone worked, their email worked, the calendaring and scheduling worked, and all necessary passwords and ACLs were configured.  The internal ethernet networks all worked, were fast, and were properly isolated from each other.  The wall ports were all correctly labeled, and there where the right kinds of wall ports in each cubical and conference room. The presentation projectors and conference room speaker phones all worked. The printers all worked, printed cleanly, were kept stocked, and were consistently named. The internet connections were fast and well managed.  Internal and external security incidents were quickly recognized and dealt with. Broken machines were immediately replaced with working and newly imaged replacements. If someone accidentally deleted a file, getting it back from backup typically took less than an hour. Software updates were announced ahead of time, and usually happened without issue.

The IT staff did not seem noticeably bitter, angry, harried, or otherwise suffering from the emotional costs traditionally endemic to that job role.  In fact, they were almost invisible in their skill and competence.

So, of course, came the day when the senior executives said "the carpets are just naturally clean all the time, we don't need all these janitors!". IT was "reorganized" into a smaller staff of younger and much less experienced (and probably cheaper) people.

Of course, it all went to shit.  New employees would go a week before they had machines, phones, passwords, and ACLs.  Printers ran out of paper, projectors ran out of lightbulbs, servers ran out of storage, networks got misconfigured, and so forth.  The total time lost and wasted across the whole company was most certainly greater than the savings of laying off the expensive and skilled IT staff.

This is not to say that the reorganized IT staff were stupid or lazy.  They worked very hard and ran themselves ragged trying to keep up with the cycle of operations, while trying to skill themselves up in their "spare time" and with a slashed training budget.

The lessons I learned from this experience speak for themselves.

What lessons that may have been learned by any of the other people involved, especially the executives who made these decisions, I cannot say.


  1. The lesson the executives learned is "I can make my bonus by causing other people pain".

  2. This is exactly why we felt it was so important to have an entire chapter on how to be visible when things are going well.

    Tom Limoncelli
    co-author of "The Practice of System and Network Administration"

  3. ordered, Tom. That chapter alone is something I need.

    I've put a lot of time and effort into this program to make sure everything runs well, and while it's not primarily Systems or Network Administration, I expect the content will generally align with what I should be doing. I find my resources are getting trimmed more and more because "we don't have to do anything".

  4. The executive knew the company didn't need to run as well as it did. Sometimes those decisions need to be made.

  5. The real issue is, that executive probably got a bonus for reducing costs because the cuts to IT were easily measurable. The damage and lost productivity is a bit harder to measure. If the executive failed to analyze the data on it, no one is the wiser and he is a hero to his bosses.

    Seen it happen many times during my 21 years at a fortune 50 company.

  6. The Rise and Fall of Civilizations. quote: "Of course, it all went to shit." Why did the British have a great 'empire'?
    1.)Sewage systems kept the water pure. Bad sanitation leads
    to bad health as Haitai (with NO trees and NO sewer system)
    The British nobles drank mostly wine and spring water
    and the Management made fun of the 'Invisible Germs."
    2.)How did the English sailors beat everyone in the world?
    the LIMEYS, so-called, required weekly doses of lemons.
    This prevented scurvy. Low scurvey reduces personal efficiency.
    3.)'management consultants - the former ENRON accountants? - selling 'quick fix' solutions. I.T. DOES NOT
    MATTER - according to past article by Harvard Business
    School Professor.
    4.)short term orientation of the CEO and the CIO. The
    average tenure is less than three years or sometimes TWO
    YEARS. Golden parachute compensation leads to the
    an example:
    King George waged war on the USA colonies on the cheap.
    He hired Hessian (German) mercenaries. They were
    surprised (drunk and sleepy with no guards) on Christmas
    Day by General George Washington.

    THE CRITICAL CODE SECTION of the USA capital markets is THE MARKET MAKER. Did Knight Capital Group
    test their code?
    Knight Capital Group computers nearly BLEW UP THE
    MARKET and lost the firm $440 million in 45 minutes.
    warning: adults only. a 'controversial word' is used.
    Many of the sub-organizations HAVE THEIR OWN SYSTEM
    ADMINISTRATORS, but they 'mislabel the title.' This is a
    variation of the economic LAW:
    Tragedy of the commons. Why bother planting trees
    in the GREAT DUST BOWL of the USA BEFORE it became
    a great ecosystem catastrophe?
    Warning this is controversial. Many of the sub-organizations
    as MORE competitive against one another INSIDE the
    corporation rather than OUTSIDE.

    analogy with teaching profession:
    those with Masters have little or NO job security and
    are treated with disrespect.
    Major NURSING organizations are TURNING DOWN
    job applicants. Why? Because there are not enough teachers
    with a PhD in NURSING.

    Google does not appear to be 'too stupid.'
    1.)plenty of free coffee and tea
    2.)free 'schoolbus' to work
    3.)exercise rooms - yes, injuries like carpal tunnel syndrome
    are a problem. As a former system administrator, even
    the BAD, BAD JET LAG from too much emergency and shift
    work is particularly hard on the OLDER WORKERS.
    4.)clean, simple food instead of the FAST FOOD chains, with
    suspect food cleanliness and potential for food poisoning.

    Lastly, this is a fictional story. Chatted with 'contract cleaners'
    to CLEAN THE COFFEE POTS with soap. The soap residue
    will cause 'the runs' and YES, the washrooms seem to be
    completely full. Appears some people were trying to get rid
    of shit.

  7. At first I thought your description of a well-run IT was tongue-in-cheek, as I've never quite seen anything like that. Even in not-so-perfect scenario though, I've seen how a junior sysadmin, with the best intentions in the world, high motivation and great enthusiasm can cause so much damage and ending up costing a fortune to the company in the long run. Fixing mistakes and poor architectural decisions is just so much more difficult than making the right choices in the first place. Or as my grandmother used to say, and I try to badly paraphrase "one fool can cause damage that 100 smart people cannot fix". Paying high for highly skilled people is sadly very hard to justify upfront though.

  8. @Voav Aner

    Actually, it isn't all that hard to justify. There is a fair bit of (perhaps anecdotal) evidence that it's the right thing to do.

    Any time you have systems or things where mistakes can be deadly, you want to have the best people possible working on them. After all, this is the reasoning behind some of the massively high CEO (and related tier) remuneration -- "they want the best possible to guide the corporation" and they could get hired away if someone else offers more.

    I'd focus more on the critical operations and programs of the business. IT is often one of them, since everything else depends on it.

    It's not hard to *justify*. People just don't want to do it because

    1. "it's a cost centre"
    2. "it doesn't produce anything"
    3. when done well, it's invisible

  9. I had an absolutely awful onboarding experience at [a major airline with local headquarters] (pun intended) when I went to work there on a contract. It took about a week just to get a computer, which I was told to set up but not to get too attached to... because it was a temporary machine, and I would have to go through the entire setup process all over again.

    Add to that stupid firewall restrictions that would break the various Java and Groovy/Grails build tools (especially Maven), a total lack of documentation, and even longer delays for just about everything else, and I was glad to leave after about a month (for a better contract elsewhere). By which point, I was still waiting on my "real" workstation.

    As my manager was walking me out on my last day, he actually apologized for the tortuous onboarding process they had, and wished aloud that they had made it more efficient. But this is a company that sees IT as a necessary evil and doesn't ever spend money on resources... like having a dev and test environment so we weren't pushing code updates directly into production. I guess they figured it was cheaper and easier to force everyone on the team to carry a pager so they could be called when untested code broke on Easter weekend.

  10. Thanks for this great post. I especially enjoyed the point about a new hire's first day - it's a great reminder to stay on top of onboarding procedures. Thanks!