Press J to jump to the feed. Press question mark to learn the rest of the keyboard shortcuts
28
Archived

"This has worked for years"

Hi guys,

How do you respond to this comment (or similar comments)? I received this comment from a customer today and it made me think, I dont really have an answer that, not only I believe but one that I think is convincing for a customer.

Would be interested to hear how you respond to this. Apologies if this has been posted before.

Thanks

62 comments
84% Upvoted
This thread is archived
New comments cannot be posted and votes cannot be cast
dev-oops
36 points · 1 year ago

Yesterday it worked

Today it does not work

Windows is like that

-- Kobiashu, 1840

[deleted]
27 points · 1 year ago

"Everything works until it doesn't."

Regular maintenance helps us to make sure the "doesn't " does not occur.

7 points · 1 year ago

"You promised me I'd never have a problem! What is ransomware?" --Customer

"Hey! he said HELPS not guarantees!" "Now back to your cubicle lowly user!"

Elbow deep in teh cloud!!!11
41 points · 1 year ago

So did chernobyl

6 points · 1 year ago

Not really good comparison, since what went wrong at Chernobyl was as a consequence to a chain of stupid decisions + bad communication between different teams - there was nothing inherently wrong with the design or systems (apart from them not being restrictive enough to block some of the stupid decisions that people could have made(and did make)).

Linux Admin
14 points · 1 year ago

a consequence to a chain of stupid decisions + bad communication between different teams - there was nothing inherently wrong with the design or systems (apart from them not being restrictive enough to block some of the stupid decisions that people could have made(and did make))

Are you sure you're not talking about an IT system? Because that sounds like many IT systems and support teams I've seen. And that gets back to the "it works until somebody makes a stupid decision it wasn't designed to block".

The difference being that the people running the Nuclear Power Plant should have more experience and education with the system then pretty much any end user has with their PC

Linux Admin
3 points · 1 year ago

You mean in contrast to the people running the company who have more experience and education with the water cooler than they do with their PC, much less the IT systems they're in charge of?

are you sure this isn't a good comparison? We decide to maintain systems, management decides whether or not to budget for critical systems/upgrades... the only real difference is it was a nuclear power plant and a government agency.

We respect: rough consensus and running code.
1 point · 1 year ago · edited 1 year ago

budget for critical systems/upgrades

Proper maintenance isn't always about money, as is often implied in /r/sysadmin. You should be careful about giving business sponsors and decision-makers the impression that the choice is always a dichotomy between doing what the techs want and economy.

I've designed and run infrastructures that were very up to date and well maintained for very little money and not too much labor. It helps when you avoid vendors who view their customers as money spigots.

but proper maintenance is about money.

Maintenance is a cost. You can work it into your budget - you can make sure that your accountants know ahead of time that you are replacing systems every x years and to factor that into their balance books... but even if you design a cheaper maintenance plan - or a cheaper replacement solution - that is just making it easier to cut through the red tape because it takes less approval to approve a 20k spend than a 70k spend for instance...

And of course you should be finding the cheapest solution that makes reasonable sense and meets the needs and requirements of your business anyways - if you are recomending a more expensive solution just because it has bells and whistles that you will never use - or because you are best friends with the sales guy you aren't doing your job, and probably aren't trying to negotiate.

We respect: rough consensus and running code.
1 point · 1 year ago

Setting aside labor cost for the moment, maintenance is not mostly about money. A 7-week old machine works the same as a 7-year old machine in most cases -- and when it doesn't you analyze why that is.

Maintenance becomes about money when you let vendors get you into a purchasing cycle. This is what I meant when I mentioned vendors who treat their customers as money spigots.

If you're already in the purchasing cycle then you've got work to do to get out of it. Vendor F knows your machines are going to be out of support soon and wants to sell you shiny new models even though you don't use half the features of the current ones yet. Vendor O and Vendor M keep raising prices at four times the speed of inflation and desupporting anything old and stable. Vendor A and M are railroading everyone into monthly license plans which end up costing three times as much as when you bought every other software release for full retail price.

There are ways to opt out of all these cycles but you have to start by accepting that you're in control of your own fate, and not just there to make the least bad decision between two options that are presented to you every day. When you decide to do it, one thing you can do is post to /r/sysadmin about what you're trying to dump and why, and go from there.

A few comments...

  1. you cannot set aside labor costs as they aren't insignificant both from a user standpoint and from an IT standpoint if you intend to do cleaning / repairs internally.

  2. I am by no means saying that you are "at the whim of your vendor" You just have to understand what it means to have that type of relationship... When contract negotiations come up - use the fact that you are buying hundreds or thousands of machines from them - and signing service agreements from them to negotiate a better discount. Shop around before you negotiate - sales reps will be more than happy to talk to you... using the power of buying in bulk does save you money whether it is over 3 5 or 7 years...

  3. The original comment was about laptops so lets talk about laptops - for us laptops get replaced based on a pretty complex policy but it works out to about every 4-5 years, we could extend that but here is the thing. Employees start to feel like they are gettign an old laptop and it affects how they percieve you - give the wrong manager an old laptop and you create a shit storm... give him a 2 year old laptop and it is still relatively new and not a big deal... Beyond that from the way our users treat laptops - after about 4ish years is when poor treatment of the laptop starts to take its toll and you start seeing more requests come in about broken dvd drives, or screens that flicker, or overheating from fans that are dieing etc... So while we could repair those laptops - and make the machines last another 2 -3 years, the benefit becomes less worthwhile after that time frame...

We respect: rough consensus and running code.
1 point · 1 year ago
  1. you cannot set aside labor costs as they aren't insignificant both from a user standpoint and from an IT standpoint if you intend to do cleaning / repairs internally.

If it's FTE labor that isn't an opportunity cost of something else, we can often simplify this out of the equation.

2) You just have to understand what it means to have that type of relationship...

Trust me, I do. However, I don't run business units, and salespersons are trained to start selling at the top and only work their way down very reluctantly. I'm not going to write extensively on the topic of vendor relations unless I'm being paid to do so.

You're also talking about buying in bulk whereas the implicit limitations you described earlier are a smaller organization that can't easily afford its own R&D, however modest.

3) The original comment was about laptops so lets talk about laptops

I didn't see laptops mentioned in the OP or in any comment between there and here. Did I miss it?

Keeping users happy can be cheap enough, but that doesn't mean they get to indirectly choose a strategic direction. In other comments I've talked about publicly redirecting savings from a desktop-software migration into buying new hardware for users. In that case it was mostly user-visible desktop machines. Non-laptops give a lot of flexibility and can yield a lot of savings when you can make the customer-acceptance factors work.

Even so, buying good, durable laptops isn't enough money to make me upset about the money, if laptops are the right choice for the business. Good for working offsite and Disaster Recovery relocation, worse for everything else.

[deleted]
7 points · 1 year ago

While there were definitely operational problems, there is a reason they went and modified all of the other reactors to reduce the void coefficient. The reactor could be kept stable with very careful monitoring and control but it was a poor design that was at high risk of failure. So probably not a bad comparison to some computer systems.

It wasn't a poor design for its time, but things evolve so it was a somewhat outdated design(outdated meaning it didn't have some safeguards that its newer counterparts did have) by the time of the accident.

But the original reactor on which the Soviet/Russian VVER reactors are based, built in the 1950s, is still in operation today. Which shows that there's nothing wrong with the design itself, it just could be better protected and better documented, which it is today(current model Russian reactors are still of the VVER design, just an evolved version of the same one built in the 50s, and the one in Chernobyl). On the same note, the other 3 reactors in Chernobyl were in exploitation until the early 2000s.

PC LOAD LETTER?!?, The Fuck does that mean?!?
5 points · 1 year ago

Chernobyl was more of a problem with the Communist system, all the operators were told that their systems and reactors were best in class and never had a problem, even though there were people in the KGB that knew differently, but sharing of nuclear secrets such as reactor flaws was restricted

10 points · 1 year ago

Yes, actually there was a similar issue in a similar reactor ~1 year before, but it wasn't made public and the reactor operators in Chernobyl weren't aware of it - which isn't a problem with the Communist system per se, because any sort of political system may decide to withold information from the public when it considers it to be dangerous. "national security" and "homeland security" ring any bells? The amount of shit swept under the rug in the USA(not a Communist country) with those reasons is astounding.

PC LOAD LETTER?!?, The Fuck does that mean?!?
5 points · 1 year ago

The reactor you're referring to was Ignalina, in December 1983, it first showed how unstable the RBMK design was, basically, when it's control rods (which were tipped with graphite) were entered into the reactor, it caused a power surge, and not an immediate power reduction. the whole 'national security' and 'homeland security' thing is right, that happens here in the UK too, the internet spying bill my government tried to create (which, ironically enough, got stopped by the EU) was listed under 'national security' so they wouldn't discuss any features of the program

what went wrong at Chernobyl was as a consequence to a chain of stupid decisions + bad communication between different teams - there was nothing inherently wrong with the design or systems (apart from them not being restrictive enough to block some of the stupid decisions that people could have made(and did make)).

That sounds like a perfect metaphor for most IT issues, to me.

Linux Admin
1 point · 1 year ago

iirc that claim was revoked sometime ago - apparently there actually was a flaw in the design allowing control rods to be removed during situations that would cause more harm than leaving them in. Later redesigns included a safeguard against that.

2 points · 1 year ago

Nope, the reason for the incident was a chain of stupid non-communicated desicions, which were exacerbate by the fact that in this design, VVER, when you scram the graphite rods in the reactor, for a short time there's an elevation of the temperature, which coupled with the desactivation of the emergency cooling system, was disastrous. This wasn't a flaw of the design per se, it just wasn't known(there was an incident ~1 year before in another reactor due to this, but it was swept under the rug) by the operators, but would have been handled(as it was in the aforementioned incident) properly if there wasn't a poorly handled drill at the same time.

0 points · 1 year ago

Not sure if sarcasm...

jack of all web services
13 points · 1 year ago

Educate them on entropy and the 2nd law of thermodynamics

¯\ (ツ) /¯

Jack of All Trades
9 points · 1 year ago

I hate that statement. I don't care how long it has worked.

jack of all web services
4 points · 1 year ago

In fairness, it's actually useful troubleshooting information to get you thinking:

"Okay if it had worked for that long a time, what might have changed recently from that long status quo ante to make it stop working?"

'Okay, now let me fix the problem for you.'

Highly Qualified Googler
2 points · 1 year ago

That's my usual way of handling it. Same with "I swear I rebooted already!" and systeminfo tells me it has been booted for 4 months.

"It says it hasn't been rebooted in 4 months. I'm starting there. Please hold." and then I mute them and listen to them cuss to their coworkers how the computer is lying.

12 points · 1 year ago

So it's long over due for failing catastrophically? "My car's still running after 15 years--it'll never break!"

[deleted]
12 points · 1 year ago

It's got that "never change" oil from bmw. Good stuff that. Perfect material for /r/justrolledintotheshop

Contractor
5 points · 1 year ago

Obviously just ran out of magic beans. Everything in IT runs of magic beans.

6 points · 1 year ago

I had an Instructor in college who had a similar saying for Electricity.

"Electricity is magic and all magic is smoke and mirrors. If the mirrors break and the smoke gets out the magic doesn't work anymore. That's why electronics smell funny when something stops working. That's the smell of the magic leaking out."

7 points · 1 year ago · edited 1 year ago

Magic smoke, I bet everyone has seen some of it before.

https://en.wikipedia.org/wiki/Magic_smoke

1 point · 1 year ago

Ha! That's great, thanks for sharing that.

I ask if they've ever heard the story about what the farmer said when his horse died:

"Huh. It's never done that before."

2 points · 1 year ago

Thanks - that one will definitely come in handy.

"Ever go out to your car and have it not start the next morning after driving it home the night before? Computers are just like a car, they are electrical/mechanical machines and sometimes, they just break."

Ah, right up there with the good 'ol 'nothing's changed'. Yes... something has... that's why it's no longer working.

Digital Cleaner
3 points · 1 year ago

1% of the time, it broke down 100%! True stats bro.

BOFH | CISSP
3 points · 1 year ago

This has worked for years...

"And now it doesn't. Finding out why it broke is what we do here. Tell me how it's supposed to work and now show me you trying to make it work and it failing."

My favorite from electronic studies was: "It stopped working because you let the magic black smoke escape" Perfect answer for when someone has fried a component.

For computers it is much simpler, I explain that the computer has remained the same but the software keeps needing more resources with every update, eventually, the computer, won't be able to handle running that software after enough updates occur. It's an over simplification, but one that people can easily accept.

Sr. Sysadmin
5 points · 1 year ago

When electronics are involved the magic smoke tends to have a bluish tint. ;)

bit monkey
1 point · 1 year ago

Turns blue after it makes contact with Earth's atmosphere. ;)

Get Schwifty!
2 points · 1 year ago

"I don't care if it worked yesterday, it doesn't work today and shouldn't have been configured that way to begin with."

2 points · 1 year ago

When your car breaks and stops in the highway, it was working til a minute earlier.

[deleted]
2 points · 1 year ago

Are you obliged by contract that said service runs? Else, if it should be, it can offcourse be added for $$$.

[deleted]
2 points · 1 year ago

If you do it wrong, it may work some of the time. It may even work most of the time. If you do it right, it will work all of the time.

And who has been performing regular maintenance on it?!

But really I like the car example further up in this post.

2 points · 1 year ago

"Just because your car started this morning and got you to the office doesn't mean it will start later today."

Jack of All Trades
2 points · 1 year ago

When I had customers tell me this I would tell them that yesterday I had a fully functioning exhaust system. Today I do not. Things break and or change. Either that, or there is a failure rate to everything. Or for the movie buff:

“On a long enough time line, the survival rate for every[thing] drops to zero.”

― Chuck Palahniuk, Fight Club

Netadmin
2 points · 1 year ago

What happens to your car, your washer and dryer, your health? It's all been fine for years and now it isn't...Things break. If everything in IT always worked, there would be much less of us employed.

2 points · 1 year ago

I hate people that have out of date hardware and OS say this. I had windows updates break some IOgear embedded print servers where a firmware update had not need issues since 2006! At least I think it was hardware but consumer hardware running 24/7 for the last 10 years could be it too. I don't know. I advise you to buy an up to date printer with an embedded Ethernet port and proper vendor support.

Jr. Sysadmin
7 points · 1 year ago

Shouldn't have done windows updates, it worked for years before them, right? lol

I disagree -- I would resign verses sour my reputation if they won't give me the resources to do at least basic patch management.

Jr. Sysadmin
1 point · 1 year ago

Woosh, mate. Woosh.

Encrypt Everything
1 point · 1 year ago

Introduce them to the concept of entropy.

1 point · 1 year ago

There is a kind of logic at work here that is unfounded - the logic is that since it worked yesterday, it should work today. Philosopher David Hume postulated that even the laws of physics cannot be granted that just because they hold today does not mean they must hold tomorrow.

It is in fact not logical to expect something to work today just because it worked yesterday.

token ring worked for years too!

1 point · 1 year ago

I just tell them I'll look into it and then I find out what's going on.

Certified Linux Miracle Worker (tm)
1 point · 1 year ago

I've run in this a lot at my work where systems aren't changed because "they work" and eventually they discover they can't do updates because of lack of knowledge or tracking upstream so it gets punted/patched....

And eventually it becomes a big trainwreck and requires 6 to 12 months of project planning to forklift it out and replace it with something that works for [ insert new tech/policy here ].

That statement is an implied question (why did it break?) and you shouldn't directly answer it because you probably don't know the answer yet.

"It worked for years," is an expression of frustration and disbelief. That is what you need to address. Give them a plan, a time frame, and remove them from the situation so that you can work and they can start accepting the fact that it's broken but being taken care of.

Community Details

253k

Subscribers

4.8k

Online

A reddit dedicated to the profession of Computer System Administration.

Create Post
r/sysadmin Rules
1.
Your account must be 24 hours old in order to post
2.
Be professional in conduct.
3.
Do not expressly advertise your product.
4.
No Low-quality content.
Associated Subreddits
r/ITCareerQuestions

41,217 subscribers

r/SysAdminBlogs

4,072 subscribers

r/sysadminjobs

11,459 subscribers

Useful Links

A collection of useful references

Cookies help us deliver our Services. By using our Services or clicking I agree, you agree to our use of cookies. Learn More.