Manager worry: catch 22

[Aristos Queue]

I disturbed my release manager today with the following bit of reasoning...

There's a bug. We have a fix. Should we make the change? Release manager is worried about a fix going in this late. But changes do go in late. Given that it is the release manager's job to always worry about changes going in this late, then there must be some other principle that we use to evaluate whether or not changes actually go in this late. If the release manager's worry is a constant, then we can just factor it out. In other words, we should always just ignore the release manager's concerns when submitting changes this late. QED, right? :-)

[crossrulz]

Isn't it the Release Manager's job to worry about changes at all times?  Why should the lateness mean anything?  As long as your process handles the regression testing and proves it didn't break other things, I say you have done your due diligence and throw it in.  I agree the Release Manager's feelings should not matter.

[Aristos Queue]

Excellent thinking, crossrulz! I had restricted myself only to the immediately relevant case of late fixes! Now, consider that the release manager's sole job is to worry about the release. If, as we have demonstrated, we should exclude such worrying from our concerns, it now becomes apparent that we don't need a release manager! Everyone is free to submit code! Freedom! Woohoo!

--- Please note that using logic such as this with your own release manager may be a so-called "career limiting move." Use with caution! :-)

[ShaunR]

That's what risk assessments are for. Fire the manager and distribute a spreadsheet.

Edited February 5, 2015 by ShaunR

[dannyt]

I have to strongly disagree the timing of changes against actual release date is a very key. Also as you get nearer the release date the types of changes allowed into the release should change.

 

In the early part of the release cycle the release manager should be locked away in a cage with a cover over it; the only gatekeeping should be your standard  SCC control process, which should include things like nightly builds and automatic regression testing. At this time new "agreed" functionality is added and bugs fixes applied and yes it can and in some cases should be a free for all.

 

As the time to release gets nearer the process changes, there comes a point that new functionality should not longer be allowed, functionality added that may not really work as expected or at all, may need to be pulled from the release, this sort of decision is hard and based on a mixture of risk against what the customer has been promised. The release manager at this point is out of his cage and working up to full paranoid mode.

 

By now your regression testing suite should be extensive, however you will also need to be doing manual testing and stressing of system as regression testing can never cover the full functionality of a system.

 

In the final run up to the release no change other than bug fixes should be allowed and even then the developer should be made to walk over blazing hot coals first to see that they really are serious in their commitment to them.

 

In my past I have worked as a release manger on several high value financial systems (oil trading and Europe bond markets)  The reality is that even following due diligence and good regression test does not mean you have not introduced an unexpected problem. On the systems I was working on a full test cycle to run through all the test case was in the order of a week.

 

As a release manger, I have had many experiences of developers saying "I have found a bug and this is the fix, do not worry nothing else will be affected"  followed a week later by "ahh I did not see that coming".

 

As a developer I have had experiences where I have found a bug and thought of simple small fix only to be caught out by and unexpected side effect or impact.  Programming is not easy especially in modern large systems and even the very best developer can and at time will get it wrong.

 

So lets hear it for release mangers they are human too.

[ShaunR]

the developer should be made to walk over blazing hot coals first to see that they really are serious in their commitment to them.

 

Yeah. I don't agree with this management technique. It's basically mistrust of your engineers and lack of technical understanding by the manager. It's actually a  negotiation tactic which has been ported across to exert control and, in extreme cases, abuse. It's predicated on the requesting engineer having some sort of vested interest in gaining acceptance of an opinion from the superior rather than technical facts  It tends to fall to pieces with East Asian engineers that just say OK as superiors' opinions are considered absolute regardless of the outcome.

Edited February 6, 2015 by ShaunR

[dannyt]

 

 

In the final run up to the release no change other than bug fixes should be allowed and even then the developer should be made to walk over blazing hot coals first to see that they really are serious in their commitment to them.

 

 I should have added a   to the above line as I was aiming for humour. 

 

I have never seen the Release Engineer role as simply a bureaucratic role, that is a very dated idea like the "Quality Manager" roles of the good old defence industry days. A  Release manger should be working hand in hand with a Chief Engineering and the Test Manager should such a position exist. A Release Manager should have a technical understand of the project, technology and code and testing and be fully a part of the decision making process.

 

Your trust comment is valid.. so not hot coals, but the engineer requesting the change does need to be able to explain and justify the requested change in a reasonable robust manner, cover why it is needed, what the basic change is any have a reasonable discussion about the risks involved. But even people I trust make mistakes.

 

 

I would say that in general I am a very trusting and open person, but in large developed teams or even multi-development teams there can be many agendas in play. I have actually experienced developers if not actually lie to me, certainly bend the true as they see their feature / change as more important than the project as a whole.  I have even caught developers trying to circumnavigate fixed procedures and SCC  systems.

 

I might now have a proper job, but the pay is not nearly as good

 

Finally the Release Manager needs to take into account the impact of the system they a releasing and scale their paranoia accordingly. The checks and balances for a small stand alone test system are totally difference for a system that handles a large percentage of the world Oil Trades over three times zones that takes a weekend to deploy. In the later case if it screws up saying he is a very good developer and I trust him does nobody any favours.

[ShaunR]

Rease manger, I have had many experiences of developers saying "I have found a bug and this is the fix, do not worry nothing else will be affected"  followed a week later by "ahh I did not see that coming".

 

As a developer I have had experiences where I have found a bug and thought of simple small fix only to be caught out by and unexpected side effect or impact.  Programming is not easy especially in modern large systems and even the very best developer can and at time will get it wrong.

 

 

Yeah. But a release manager isn't going to address that, is he? The way to stop that is via testing so its a case of "OK, you've found a fix where're the documents?" However, a Principle Engineer, at the very least, is much more likely to error guess or spot implications than a Release Manger.

 

The Release Manager is really just a checklist that you drink coffee with. Here's the risk assessments, here's the test results, here's the bug fixes, now sign. The only thing he or she brings to the table is somewhere to apportion blame. Well. That's the Project Managers job

 

Even is you think in terms of gates (RFQ, RFT, RFP etc) a Release Manager doesn't bring anything more than a signature to say "We really, really think it works now and here are all the documents written by others that say it too".

 

Replace him or her with one of these 

 

Edited February 6, 2015 by ShaunR

[Michael Aivaliotis]

So basically what you're saying is that we need to let an arbitrary length of time pass between making software changes and releasing it to the public. But we can't find out if the changes caused problems unless we release it to the public. - What does time give you? More testing? If you cut corners at the last minute to squeeze in a fix, then I can see it as a problem. But if you follow the same process, what's the harm?

 

All software has bugs. It really depends on how rigorous your internal and external testing is. Having a public open beta or some kind of user involvement helps. It's also important to have a fast turnaround process on releasing new versions when a bug is found in the wild.

 

Isn't that what "F" patches are for?

[Rolf Kalbermatter]

Yeah. I don't agree with this management technique. It's basically mistrust of your engineers and lack of technical understanding by the manager. It's actually a  negotiation tactic which has been ported across to exert control and, in extreme cases, abuse. It's predicated on the requesting engineer having some sort of vested interest in gaining acceptance of an opinion from the superior rather than technical facts  It tends to fall to pieces with East Asian engineers that just say OK as superiors' opinions are considered absolute regardless of the outcome.

 

It does depend a bit on the system you are working on. If it is the traffic control system of the National Railway company your better make sure that you don't add a fix that adds some regression in some seemingly unrelated area or you are likely to have been in the job for good. 

 

Our Dutch railway support organization just managed to create a country wide chaos this week when they installed a new version of the software in the system over the weekend which was supposed to fix  something (or why wouldn't they even change the software?). Monday, the control system for the signals in the most important railway station of the country decided to shut down leaving the whole station virtually inaccessible for every train, causing a chaos over the whole country and beyond.

 

The comment the next day in the newspapers was in the line of: We are very sorry, but we saw that already coming on Sunday after we installed the update!

 

I'm sure the new software passed all regression tests they have (I can't believe they wouldn't do that for such a system  ) but somewhere somehow something fell through the cracks that when it was stress tested on Monday morning in the commuter traffic, simply failed.

[Michael Aivaliotis]

I'm sure the new software passed all regression tests they have (I can't believe they wouldn't do that for such a system  ) but somewhere somehow something fell through the cracks that when it was stress tested on Monday morning in the commuter traffic, simply failed.

 

There are strategies for this. You can release small and gradual until full capacity. People get cocky, and they ignore obvious signs.

 

I'm a fan of releasing early and often. Yearly monolithic release cycles are big, scary and should be avoided. But hey, that's how most are used to working.

 

Microsoft is going to release Windows 10 with frequent updates for free. But if you're a corporate user and want no updates, you have to pay.

[ShaunR]

Hmm. Not sure you quoted the right bit by the context of your response, but I will assume it is the correct quote.
 

It does depend a bit on the system you are working on.

 
 
No it doesn't.
 
A manager requiring a passionate argument from a subordinate to make a decision is a test of belief and has nothing to do with the software or hardware. It's about one party convincing another about their conviction irrespective of fact. This is why I don't agree with the technique as a valid decision making process..
 

I'm sure the new software passed all regression tests they have (I can't believe they wouldn't do that for such a system ) but somewhere somehow something fell through the cracks that when it was stress tested on Monday morning in the commuter traffic, simply failed.

Sure but would a release manager have prevented it? (assuming they don't have one). I don't think so. From experience, those sort of foobars are usually attributable to a business decision to release in the face of opposition from the technical expertise. May not be the case in that instance but since the knew it was about to happen I bet it was. On many occasions I have been  that technical expertise that said "I told you so"

Edited February 7, 2015 by ShaunR

[Rolf Kalbermatter]

Sure but would a release manager have prevented it? (assuming they don't have one). I don't think so. From experience, those sort of foobars are usually attributable to a business decision to release in the face of opposition from the technical expertise. May not be the case in that instance but since the knew it was about to happen I bet it was. On many occasions I have been  that technical expertise that said "I told you so"

 

A release manager wouldn't necessarily help. But the whole process of upgrading software certainly needs to be adjusted to the possible impact when something fails. Release small and often isn't the fix for this either. Even the smallest bug can cause such a mishap. And releasing small and often does increase that chance rather than decrease it. It also makes rigorous testing even more unlikely.

[dannyt]

So basically what you're saying is that we need to let an arbitrary length of time pass between making software changes and releasing it to the public.

 

No not an arbitrary length of time, but a delay defined by your test cycle time else you are sending out untested software. So you have to decide if the bug your tring to fix is something your can live with or is a must fix issue. There are legitimate reasons to allow know issues with a system to go out rather than risk a release that is not fully tested.

 

I am all for customer beta release and testing however there are many situation where that is just not possilbe.  It is also not always possible to push out a fast release even if the deveoplers can get the fix in place and test OK. There are plenty of real world situation where a release and deplyment of software is a very expensive and time consuming process.

 

 

I'm a fan of releasing early and often. Yearly monolithic release cycles are big, scary and should be avoided. But hey, that's how most are used to working.

 

I totally agree with your statement above, but in the real world even small functionality drops to a large system can invloved a lot of time and money so they had better be right.

 

 

 

A manager requiring a passionate argument from a subordinate to make a decision is a test of belief and has nothing to do with the software or hardware. It's about one party convincing another about their conviction irrespective of fact. This is why I don't agree with the technique as a valid decision making process..

 

Who said anything about a passionate argument or belief's, I do not.  But they have to be able to present the facts and evidence and testing.

[ShaunR]

Who said anything about a passionate argument or belief's, I do not. 

the developer should be made to walk over blazing hot coals first to see that they really are serious in their commitment to them

[ShaunR]

A release manager wouldn't necessarily help. But the whole process of upgrading software certainly needs to be adjusted to the possible impact when something fails. Release small and often isn't the fix for this either. Even the smallest bug can cause such a mishap. And releasing small and often does increase that chance rather than decrease it. It also makes rigorous testing even more unlikely.

 

That was Michaels comment. I don't have any preference for small and often or big and rare. That mentality is sales driven. Release when ready and as few as possible is my view but that's a whole other subject on what ready means.

Edited February 8, 2015 by ShaunR

[Rolf Kalbermatter]

That was Michaels comment. I don't have any preference for small and often or big and rare. That mentality is sales driven. Release when ready and as few as possible is my view but that's a whole other subject on what ready means.

 

I know it was Michaels post and it wasn't really directly towards you. I fully agree that Release when Ready is the more accurate term in this. And Ready can mean a lot of things as you allude to. For a consumer application it can mean the application doesn't crash if started and used normally. For a mission critical system it really means, we have tested everything we could think of and then some more and could not find anything wrong with it and every engineer working on the changes has evaluated every effect that change could have as far as humanly possible. And ideally there was a real field test on some small scale to actually test the real world interactions too.

 

A release manager can help manage that, but in many cases is just another person in a shorter or longer list of people who exists to dilute the responsibilities if something goes wrong.

[mje]

I'm don't particularly like the confrontational relationship I often hear about between the engineers or architects of a project and the levels of management. Though it can produce good software, I think when it does it's in spite of rather than a result of these relationships. It really only serves to advance the project through to completion and has nothing to do with producing "good" software.

 

Fortunately, despite working at a large company, I work with a small team of experienced and talented people at all levels of the project. Our managers are extremely deft at dealing with the demands of marketing, quality, documentation, phase gates, and other aspects of the nine levels of hell bureaucracy that need to be dealt with to release anything at a large company. They serve some very important roles, but when it comes to details of implementation they defer to those who know best: the architects and developers. Without our management team I wouldn't be able to do what I do, and wouldn't be nearly as satisfied with my job as I am.

[JKSH]

In the final run up to the release no change other than bug fixes should be allowed and even then the developer should be made to walk over blazing hot coals first to see that they really are serious in their commitment to them.

 

To be fair, he did say it was a comment made in jest (see post #7)

[dannyt]

 

Your trust comment is valid.. so not hot coals, but the engineer requesting the change does need to be able to explain and justify the requested change in a reasonable robust manner, cover why it is needed, what the basic change is any have a reasonable discussion about the risks involved. But even people I trust make mistakes.

.

 

As I had already tried to point out before your comment, the hot coals was in jest and I did go on to say "justify" the change and in my understanding the word justify implies evidence and rationalisation not emotive argument, but maybe it is not seen to mean that by other.

 

I think there is a very narrow view or experience of a Release Manager's role being used here, of somebody who lives outside of the development team and development process and has no technical ability or validity, but that has not been my experience, I can remember a time when that was the case but that is going back several decades.

[ShaunR]

I think there is a very narrow view or experience of a Release Manager's role being used here, of somebody who lives outside of the development team and development process and has no technical ability or validity, but that has not been my experience, I can remember a time when that was the case but that is going back several decades.

 

You maybe right, but several decades ago I think only Alan Turin was into computers.

 

So what are the responsibilities of a Release Manager in your experience?

[dannyt]

You maybe right, but several decades ago I think only Alan Turin was into computers.

 

 

I am very nearly that old certainly used punch cards when I started.

 

My experience covered that of a stand alone Release Manager and as part of a Release Management Team at two major financial institutions.  In one we released  around every 1-2 weeks between 6pm and 6am while the trading markets were down and in the other we did release approx every six months over a weekend again while trading markets were down.

 

I both case the role covered doing the format builds and release of software to both the test environment and live production systems, this was done mainly by automated scripts write by myself and the release team.  We were the first port of call for the test teams when there were problems with the builds or environment as we knew how the system went together and what changes had just been added and which programmers to go to if required. We coordinated and controlled the software changes by the programmers with  the required database changes by the Oracle db admin team and or unix system changes required. Finally we had to ensure we were happy we could in the case or a problem with a live production release roll the release back to the previous working system.

 

The key thing thought was to act as the glue between the test teams, the various principle engineers that were responsible for different aspects of the system and project managers. We were a team where everybody had a part to play to make a successful release occur.

 

I did not look at the test managers test results but did check he was happy that the testing cycle was complete and passed, I did not look at the programmers code, the principle engineering made sure coding standards etc were adhered to, I was not a unix admin or an Oracle dbA  but I still need to ensure they were in place and ready if needed.

 

At the end of it all I actually did the physical releases I knew what when wrong and needed closer supervision next time , who knew who cut corners (always with the best of intentions)  and needed to be encouraged not to, I know who I could be relied on and who would always forget to provide an SQL roll-back script, or forget to tell the Unix admin's of some dependency package needed as part of the deployment.

 

Another un-thankfull task I have done is that of  SCC control Manager (ClearCase), I am old enough to have had the arguments with programmer who directly refused to keep their code in the SCC system  "why do we need to bother with SSC have just copy and rename my folders"

 

Finally I am sorry that I seem to have gone of on a bit of a rant with this subject, and possibly bored people, this is not something I usually do. I just feel there are many roles involved in getting good quality software released to customers, some are obvious in their impact and glory, other less so and I do not like to see parts of the team that make it happen dismissed.