When Bad is Good

When bad things happen you have the ability to turn them into something good.  The choices you make when something goes seriously wrong are what determines if you come out of it for the better or you make it worse.

Bad into Good:

If there is a better way to finally get rid of orphaned servers and VM sprawl leftovers than a data center wide power outage I haven't seen it. ...if...IF you have the flexibility or buy-in to make the decision not to just universally power everything back on.  Assuming you, like most companies I have seen, do not do a great job of life cycle management and you are given the opportunity to power servers back on one by one as, and only as, they are requested by your customers, you will be amazed at the number of servers you get to leave powered off. 

Bad into Worse:

The tendency when something bad happens is to reflect and regret what could and should have been done.  This is one of the proper and needed responses so that we can learn from our mistakes and oversights.  The danger is in the desire to "fix" these past discretions with a knee jerk and immediate reaction.  The reason this is dangerous is that the "could have"/"should have" actions that you think you are fixing are ultimately there because of a failure to plan.  By implementing all of the things that should have been done before an incident immediately after an incident is only inviting a new and more deadly disaster.  As much as you want to jump right in and fix things so that a similar incident doesn't happen again this is the time to properly plan changes to set yourself up for future success.

Focusing on the Good:

Now, you have made some good from bad and left a number of servers off, ultimately decommissioning them after a respectable amount of time, it is time to take that on demand lesson and expand its practice.  With more talk about cloud and on demand there are plenty of products and ideologies centered around the making of server provisioning, life cycle and deprovisioning into a self serve process.


Think about where that entirely self serve method goes wrong.  If David Developer gets an email asking if his server is still needed the vast majority of the time he will say yes.  This is ultimately the same issue as asking for one up approval, most leaders are going to approve requests from their team that they don't understand or that don't directly affect them.  


Consider now one of the most time consuming aspects of manually deploying applications which is troubleshooting the install.  If you were deploying a web application to a farm of servers and had to do each one manually the chances of something getting missed goes up with each additional server you need to deploy to.  The solution of course is a tool to automate the deployment.  Once you have that deployment tool the act of deploying software becomes trivial.  You are now in a place where both the provisioning of servers and the deployment of software is trivial and you can implement practices that incorporate self serve and automation.


Holy grail time.  David Developer needs a farm of servers to develop and test on.  He puts in the request and the servers are automatically provisioned.  He then automatically, through a tool or script, configures the roles and features he needs.  Then his code gets deployed to the entire farm of servers at the push of a button.  So far this is all the same but this is where it changes.   Why does D.D. need to get an email after 90 days to see if he still needs his servers?  Why don't they automatically get retired and the resources returned to the pool after the predetermined life of the servers?  Once D.D.'s servers disappear he puts in a new request and the same thing happens all over again.  Besides the benefits of never having a server you don't need consider the patching implications.  With a 90 day life of your servers you essentially have a quarterly patch management process out of the box without ever having to install a single patch.


If even some of this good can come from bad then I say bring on the power outage!  Thoughts?


- Z

Why do we buy your software?

Why do businesses continue to buy software from companies that have no interest in developing software properly?

How many times have you, as a system admin, been asked to give out the local admin/root password to install or run software?  I'm not talking about giving up an account with local admin access, i'm talking about software claiming to need "the" local admin account.  Do you care so little about the people that need to run your software and the servers that it runs on that you can't even be bothered to implement some kind of proper coding practices?  Is it so hard to setup your software to write to the locations that software should write to, instead of writing everywhere? Do you care so little that you try to mandate all users in Active Directory/LDAP need to be in a certain location instead of properly using LDAP?

• Please stop assuming that you are the only software out there and that people will accept whatever you tell them and bend any kind of best practice to run your crap. (I'm looking at you Cisco)
• Please stop using ridiculous requirements in your software to sell your hardware (Still looking at you Cisco) 
• Please understand that AntiVirus software is a way of life now on Windows systems and realistically there is no difference between AV products.  Don't come up with some stupid requirement to only support a couple of products, just provide the needed exclusions and move on. (Again, CISCO)

It is a simple equation.  Some of the current system admins and application support people will climb that corporate ladder and end up in a place that they have the power to make the decisions.  If they come across a choice that is between your company that has been a pain in the ass or a company that has "worked with" the people supporting their software, who do you think they will choose?

- Z

The New "Emergency"

In our on demand, self serve world, the word Emergency has taken on new meaning. Emergency now means anytime you can't get exactly what you want exactly when you want it.  The first world problems meme that has made its way around the interwebs pretty much has it nailed.  We have all come to expect a certain level of service from companies and technology in our personal lives and that expectation finds its way into our work lives as well.


The old adage "The lack of planning on your part, does not constitute an emergency on mine" has apparently gone the way of the floppy disk.  We are now programmed to expect everything to be on demand and that includes our interactions with various different teams in a work environment.  Someone needs a new server for whatever purpose and the inevitable dialog is as follows:

I need a new server for project X with Y specs (wouldn't that be nice, to get specs with requests! but I digress)

OK, when do you need it by?

Oh, no rush, this afternoon should be fine. (Somewhere back in the depths of history, this kind of turnaround would have been an emergency, now? Normal.)

That is about the time when I need to bite my lip and start the slow nod of hatred.  It's not that technically we can't get it done in that time frame.  Anything is possible given the correct balance of time, money and expertise.  There are a couple of problems with a request coming in like this.

First, if I need anything from other teams to build this server I now have to pay this lack of respect forward by telling them that my request for storage or IP addresses is a rush and will need to make its way to the top of their pile of work.
Second, you are ass-u-me-ing that your work is the only or most important work that I am already working on.  If I was just sitting around waiting for your request and not doing anything else then the company wouldn't be paying me for very long.
Third, it says to me that someone in the chain has no idea what amount of work is involved in whatever type of request was made.  Maybe it wasn't you, maybe it wasn't even the person that tasked you with it.  But it was someone, there is a person someone that uttered the words, "Ya, they should be able to do that"

All we can hope in this ever increasing pace of expected turnaround is that people at least try to reduce the number of these things that once in the distant past were known as "Emergencies"

-Z

Role of Eye-Tee

To me the role of eye-tee is a very black and white thing. At the core we exist to make the work easier for the end user, the person that does the work of the business. This applies whether you work in an eye-tee department or an eye-tee company.

Almost any business function can exist without us, in some capacity. Granted it is made orders of magnitude easier and more efficient by us. Because of this firm middle ground stance, I sometimes find myself disagreeing with both other eye-tee roles and the business side of companies.

One of my big peeves is when companies separate groups based on revenue generating vs non-revenue generating, typically to cater more to the needs and wants of the revenue generators. Really? If all of the other groups weren't needed then wouldn't you just get rid of them? Revenue enabling or better yet revenue enhancing is a much better term than non-revenue generating.

Consider a brokerage firm, they could still have traders on the floor of the exchange.  They may even still be able to run paper tickets and stock certificates.  It's conceivable that said traders could survive with little or no technical help.  Now surround those same traders with a (non-revenue generating) eye-tee department to bring them the latest electronic advances in trading applications and information consolidation via Bloomberg, Reuters or the like. Will they not be better informed with the latest news and information.  Will they not be better servicing the customer by having access to real time exchange data and allowing their clients to view constantly updated accounts of their holdings.  If you had a choice between a stock trader working with real time data and one using 20 minute delayed information, which one would you choose?

IBM, like them or hate them, ran an interesting commercial with an analogy about using real time data in decisions.  The gist is, try taking a picture of a busy intersection now, then in 5 minutes decide when to cross that intersection based on that photo.  If you break it down to that level, technology, when properly implemented and utilized, allows us to make better decisions and provide better service.

Now the flip side. The eye-tee department that views themselves as indispensable and always tries to mold the customer to their way of thinking is no longer making the work easier for the end user.  Many a department has lost sight of what should be the ultimate goal and by doing so ends up complicating the ability to get the customers core work done. This is not to say that every decision an eye-tee department or group makes is going to be viewed as the correct one by the business, but if we keep the core goal of helping the business work smarter instead of harder then we are going our job.

-Z

Double-Edged Sword of Kindness

Have you ever sat and watched an email chain go on in front of you.  You see it quickly gathering steam, getting longer and more convoluted with each "Reply to All".  You sit there knowing that one response from you and the problem would be resolved, but you hesitate.  You resist the urge to be helpful knowing it is the right thing to do.  There are two likely reasons to not respond.

1. You are an arse.  While this is technically possible and entirely feasible, I have found that "most" people are generally not spiteful just for the sake of being spiteful.
2. You have been previously cut by the Double-Edged Sword of Kindness.

Those of you that have been sliced by this nasty beast know what I'm talking about.  It comes in many forms:

• It is the innocent tech help that you give a friend.  
• It is the short email you decide to jump in on and help
• It is the familiar phone number that comes up on the call display at 2 minutes to quitting time

Regardless of the method this nastiness chooses to present itself to you the outcome is always the same.  You provide what you see as a quick bit of assistance to a person in need and you become their default action in the future.

Even if you try to take the teach a man to fish approach and show the person how they can resolve their own issue, the only part they seem to remember is that you are the person that knows how to fix it.  The most frustrating part about this to me is that even though you started out with good intentions you end up being an ass because you eventually have to tell people to do their own work.

I'm not sure the moral of this.  It's not like anyone is going to tell you not to help people, especially since that should be the primary role of eye-tee.  Just beware that this monster is out there and don't let it get you too many times.

-Z

The Missing Work

Oh short work week, why have you forsaken me?

I used to love the short work week.  The joy of having an extra day on the weekend is still a fantastic thing, something to be cherished and adored.  Recently though I have been increasingly disillusioned with the short week that precedes or follows the glorious holiday day.

I have come to realize that the more I slowly claw my way up the "rope of responsibility" (more on that later), the more I despise the short week.  It used to stand for all that is right with the world, now it means extra work and longer hours.  I'm not sure how I didn't see it before but all the short week means now is that there are more meetings and more work, that you would have had 5 days to do and now you only have 4.

I have no idea where the work on that 5th day used to go, it seemed like it was just gone, it never needed to be done.  Now it gets spread across all of the other 4 days, making them more painful than they need to be.

Oh days of the Missing Work, where have you gone and how do I get you back?

-Z

Non-Tech People Scare Me

Sometimes I'm terrified of the non-technological people.

I think it is because I'm not really sure what goes on in their heads.  Currently it is the "letter writers" that are at the top of the freaky list.  I'm not really sure if these people exist outside my life or if it is only me that has the ability to find them.

Have you ever listened to a person complain about a company or a product and somewhere in the conversation they manage to work in that they are, "going to write a letter"?
That is usually when I perk up, "Do you mean send an email?"
"Nope, write a letter."
And they've lost me.  How can it possibly be more effective to write a letter on a piece of paper and mail it to a corporation that seems more likely are just going to junk it than forward it to the person you want to see it.  Do companies even have protocols anymore to handle the paper letter?  In the days of contact us email addresses and feedback forms why write a letter?

It seems that my dad has taken this to the extreme.  Mom was telling me about this feud, for lack of a better word, that dad had with another person in the smallish town they are from.  They were taking shots at each other by writing letters to the newspaper and then sending rebuttals.

Lets break this down for a minute.

1. 1 hour = Get paper and pen, sit down and write your letter.
2. 1 hour = Find or buy envelope and stamps.
3. 30 min = Take letter to mailbox.
4. 2 days = Wait for letter to get to the newspaper.
5. 1 day   = Wait for newspaper to decide if it is worthy of putting in the paper.
6. 1 day   = Wait for newspaper to come out the next day with your comment in it.  

Adds up to 4 days and 2.5 hours, just to get your comment in front of the person it was intended for.  Double that now to see how they respond.  It would be like watching the "electric company" word makers send youtube clips back and forth.  Imagine a 4 day delay between each part of the word, yikes.

The reason this is scary to me is, who stays angry enough, long enough to use this as a method of arguing with someone?  That is a person that I don't want angry at me.

-Z