Odd IT and Corporate Policies

[hooovahh]

So based on conversations in another thread we've come to find several instances in our career where IT and corporate policies that mean well, get in the way or getting work done.  Policies intended to keep IP from leaking, or viruses from taking over our PCs, and ransom-ware, tend to make software developers find work arounds so they can get their job done.  So I thought we could make a thread where we share some of our IT horror stories, focusing on previous work for obvious reasons...

One company I worked with had locked down the work laptops to where you couldn't really do anything other then answer emails and write Word documents.  So when we needed to install LabVIEW which required administrator privileges, there was a whole approval process to get the IDE and some of the DAQ drivers installed.  This included several levels of approvals, and justification for why you wanted to be an admin.  You would apply to have administrator privileges given to you but you only had a 2 hour time slot, and after that the credentials you were given wouldn't work, and you had to apply again.  Of course the first thing we would do in that 2 hours, is use the administrator account we were given to promote our user as a local administrator, so we never had to apply for it again.  This company also had an app store for PC software.  To me the concept of having a PC app store seemed odd.  I sorta consider the whole internet the app store, and the concept that an offline app store can be up to date with all of the internet, with the newest software seemed silly.

[jcarmody]

I remember the good old days where all I needed to do was get added to an admin group and I could do anything I wanted.  The pendulum swung so far away from that extreme to the point where I had to launch a two-week workflow to get the simplest of things done.  The pendulum is near a sane point now where I can do stuff (I'm installing NXG right now).  Have you heard of Viewfinity?  It makes the approval to install something take as long as your boss takes to respond to an email.  Life is good again.

 

[Tim_S]

Our IT blocks access sites where can download shareware and freeware. It seems this includes sites that have commercial free-use icons (have not gotten a story as to why but I expect it has to do with how McAfee categorizes websites) thus I either spend time developing my own icons (which don't look as good) or purchasing a package of icons... or I would but those packages tend to be on the same websites. So when I want professional looking icons, I have to go home to search for what I need and then get purchasing involved.

Edited June 14, 2017 by Tim_S

[hooovahh]

Have you heard of Viewfinity?  It makes the approval to install something take as long as your boss takes to respond to an email.

Use off the shelf software?  Heavens no.  Why don't we just create our own approval software that doesn't integrate with anything, requires extra training, is supported by no one that speaks English, is buggy, and charges a licensing cost calculated by number of clicks of the mouse used during the approval...

Another story before I forget it.  I was deploying a test system to a production facility.  The system involved two GigE gigabit cameras, and was looking at the DUTs display to determine if it was working correctly.  Everything worked great before we got to the facility, but then all the sudden the images from the ethernet camera would sometimes get black bars, obscuring the image to where the vision system would fail to work.  We looked into all kinds of jumbo packet settings, Windows network settings, and just before going to buy a new network card I asked if anyone had used the machine and a worker mentioned someone from IT had used it.  I talked to them and they installed network spying software to make sure it was locked down.  Turns out it was also disrupting all network traffic, including the cameras.  We asked if it could not sniff the two ethernet ports going to the cameras and they said the software didn't allow for that level of control.  We tried to uninstall their software but it required a password during the uninstall, and IT wouldn't give it to us and wouldn't allow us to uninstall it.  At this point the customer is getting increasingly frustrated with their own IT department, especially since it was costing time and money for me and others to be there when the system was crippled.  We were still local administrators on the PC, so we just disabled the Windows services, and had their software not run on startup anymore and production was running again.  Customer was happy the system was running, and IT didn't really care as long as it was installed on the PC

[smithd]

I have a control system that streams lots of data across a local network. If I configure the operator's computer so that they can also log onto the organization network as well (via firewalls, etc) the packet sniffing and a/v software drives CPU usage from 10% up to 50%

 

[Taylorh140]

There is often a disconnect from what IT wants and what is needed for Testing.

One of those things is windows updates, IT really needs interconnected computers to be up to date. But often this means a policy that forces computers to shutdown and reboot, which is awful for longer test. (the solution for this one is difficult to say the least).

Another is forced logoff and locking screens and such. IT policy on power saving is usually the more the merrier. However when running Test this kind of thing needs to be disabled. I certainly do not want my computer hibernating when I am running a test. (this doesn't include weird USB power saving settings)

I have seen some interesting solutions to this one:

• Mouse Jigglers -> These exist as both hardware and software solutions. (what has the world come to?) https://www.amazon.com/dp/B00MTZY7Y4/ref=cm_sw_r_cp_dp_T2_1YOqzbZXHNQW2
• Disabling power settings in windows (requires admin) (sometimes group policy can reset this)
• My personal favorite:The Prevent Screen Saver which is attached. (I don't remember where I got it from) It uses the same call that the windows media center does, to prevent the system from sleeping. Its also nice that when the application closes the screen can lock again which I find better. Also it doesn't require admin.

Testing requires the system configuration to remain largely static. however for security reasons this rarely happens. Testing would like internet access to send notifications and pass information to outside data-servers.

• Some solutions like an air-gap are not really a complete solution. what is really needed is a one way communication valve. So the testing computer can notify and send data but cannot necessarily receive it back. This also implies that data exiting the computer must be guaranteed to be collision-less to prevent data-loss since handshaking would not be possible.
• Perhaps Virtual images will provide a solution in the future but I still think they integrate poorly with hardware.

I think that in general Windows standard settings align more with IT than what is required for good LabVIEW stuff.

There is a great many things that need to be considered for long test using LabVIEW on windows, and the number is increasing every new windows release, and IT isn't helping. It would be nice to have library to handle these needs (maybe the windows people have a plan ).

 

PreventScreenSaver.zip

[mje]

23 hours ago, hooovahh said:

...we've come to find several instances in our career where IT and corporate policies that mean well, get in the way or getting work done...

This amuses me. The policies I'm exposed to originate from nothing other than malice.

[drjdpowell]

1 hour ago, Taylorh140 said:

My personal favorite:The Prevent Screen Saver which is attached. (I don't remember where I got it from) It uses the same call that the windows media center does, to prevent the system from sleeping. Its also nice that when the application closes the screen can lock again which I find better. Also it doesn't require admin.

I use the tool from here, which is probably similar.

[Taylorh140]

Quote

I use the tool from here, which is probably similar.

I think you rediscovered the source! Thanks! (this seems like an updated version of mine.)

[ned]

My (least) favorite conflict with IT: we had a site-wide, network-based solution for printing product labels, which was accessed through Internet Explorer. We wanted some in-house software (that I was writing, in LabVIEW) to talk to that system, but we weren't willing to pay the exorbitant vendor fee for the SDK, so I dropped IE into an ActiveX container on the LabVIEW front panel and got everything working happily.

Later, IT wanted to upgrade the version of Internet Explorer, and needed to confirm that it wouldn't cause problems for any existing applications. I mentioned that several people were using this tool I wrote in LabVIEW, and was promptly told that there was a strict policy against an application accessing any company database without prior permission from IT. My boss had to schedule a meeting that involved his boss, the local IT person, and someone from corporate IT, to explain that my code only took the same actions that any authorized user could take by clicking buttons within the label-printing site, and that the program did not directly access any corporate database.

[hooovahh]

I'm loving this thread, it is stirring up so many memories and stories I mostly forgot.  

I was at a customer's site that had to production lines going which a previous vendor had made using LabVIEW and TestStand.  The company I was working for had been tasked with updating the systems by adding a few new features, and updating to the latest LabVIEW/TestStand versions as well as updating the OS.  Luckily they sent the 3rd system to our facility to update and upgrade while they were still using the other two.  We updated the system and made the changes they asked for, but there was still some onsite work required because we couldn't interface with their network databases until we were there.  Once onsite I started setting up the 3rd stand right next to the other two that weren't upgraded yet.  My first day there I started asking around about the credentials used to access the network and database that I needed to have my software log to.  No one seemed to know and even the IT guys were confused about what I was talking about.  So with the permission of the program manager I looked at the source on the 1st stand to see how it was logging to the databases.  Sure enough there was the user name and password, and it was something like User:TestUser and Password:TestUserPassword.  So I updated our code got it working and the customer was happy.  I saw the IT guy walking around so I let him know.  "Oh yeah I found the credentials used for the production systems it is the TestUser account."  His eyes got really large and said "No one should be using that."  And I'm like "Oh okay well we're just using the same credentials the last vendor was using and it can be changed later."  He said okay and left.  About 10 minutes later I noticed the other two production systems were having all kinds of problems, failing every part it tested.  Just then the IT guy came up and said "By the way I deleted that user account so no one can use it."  Luckily the program manager was in ear shot and started blaming him for causing the production lines to go down, and demanded he fixed it.  So IT made a new account, with a new password and we used that.  I'm not sure why he thought it was okay to delete that user when I mentioned how the production lines were using it, but he did and I was glad we didn't spend too much time trying to fix it.

Oh and we also have a No Sleep program on our test systems which is an AutoIt EXE.  As for updates we have the systems download updates but not install and then we just need to remember to reboot the machines when testing is finished which might not be for months.  Every once in a while IT comes down and says there is some major new virus and we need to update all machines now, which is a pain but we haven't had any issues yet.

[MarkCG]

This is why i like the cRIOs and cDAQs with the display port.  Self contained system that the IT guys don't really even recognize as a computer.

[A Scottish moose]

The Test & Validation labs I support work with various hazardous materials (mostly combustible/volatile but the occasional chemical or health hazard shows up now and then).  Because of this my team tries to make our test boxes as inert as possible.  Dust and weather proof NEMA boxes with circular connectors is our standard.  Since we are small on space we've been transitioning to micro PCs; Intel's NUC is a great option that we recommended to IT.  Many of them are fan-less (a plus if combustible material does get in the box) and they are ~5x5".  Perfect!  The IT department balked at the idea.  'they aren't safe!, Virus protection?!, it doesn't look like a normal computer, we can't support it!"  When we explained that it's just a tiny windows 10 PC they basically said they didn't want to support a new ghost image (Dell supports the Ghost images for all our standard issue machines) for these boxes because it was too much work... ya I know...

So they offered a different micro PC that was about 2x the surface area.  My NEMA box was starting to get pretty tight. IT agreed to support it (Read: it's under corp's Dell contract so dell will support it) and so we went with it and I redrew all my CADs to make it fit.  It was cozy, I was frustrated, but all in all everyone was appeased with the deal so I considered it a successful bargain.

 

After getting the box built IT guy came down to take a look at it.... First thing he said.... It's a little tight in there don't you think?

 

[MikaelH]

My employer have installed Digital Guardian Windows Agent on all our offices computers in most countries. It monitors all your file access and transfers to and from you PC. The idea is to prevent people to transfer files out from the company to any competitor (INTELLECTUAL PROPERTY PROTECTION).
I guess it’s a good to try to prevent this, but it makes the PC much slower and also breaks LabVIEW from time to time.
I heard that they are not allowed to install it in some countries in Europe, since the monitoring it performed of you breaks some countries laws of “Surveillance in the workplace”.

The DgAgent.exe could have many different modules activated, e.g. monitoring all file movement, hard disk encryption, prevent saving files to any USB device
Some of the problem it has caused us are:
We couldn’t build executables anymore.
The last step when LV zips all files up and adds it to the exe file was treated as a threat, so the exe file got deleted by the agent.
File writing failed, because the folder that was created a millisecond before the file was written to that location, was not really created, it was delayed due to the hard disk encryption. So we had to use the magical delay fairy between create the folder and writing a file inside the folder.
So if your IT department wants to install DGAgent, I’m sure you’re going to have some interesting problems in front of you.

[hooovahh]

I worked at a company that had all USB devices disabled.  Not just USB memory devices, but also USB CAN, USB DAQ, USB Serial, USB GPIB, everything.  For the DAQ stuff we would just simulate the hardware, then deploy it to test it.  It was a pain but not too bad.  The USB memory devices were easy to get around.  It made the USB stick read-only, so to write to it you would map some empty folder like C:\USB to the E:\ drive or where ever your USB was.  Then you could read and write to the C:\USB folder all you wanted which actually wrote to the USB device.  I guess they just setup software to disable writing to drives that were marked as removable. 

That DgAgent sounds nasty.

[Tim_S]

We supplied final test stations to a customer for their production line. The data from our stations gets buffered and sent up to the customer's database system in the sky every five minutes.The stands became quite old (about 10 years) when IT of many companies decided Windows XP would no longer be permitted on the network at all. IT picks a weekend to stop allowing any XP machine from accessing the network, implements the change some o-dark-thirty late one Saturday night and goes home knowing their network is more secure. Come Monday morning, production comes to a stop as final pack-out can't ship product because no record of pass or fail is in the database. IT strolled in three hours after production started and immediately blamed the test stand supplier. This did not fare well for IT after our serviceman opened up the log file and pointed to the exact time when we were no longer able to access the network.

Many years ago we shipped some systems (Windows 2000 based) to a customer with antivirus on them set to update regularly and all that. We installed, checked that the updates happened and went on our merry way. Some months later we get called in to service the machines which started slowing down and behaving erratic. Our investigation determined that IT hadn't locked down any of the network so the test stand operators were surfing porn sites while running the machines. One test stand was so infested with viruses that the antivirus program was just shriveled up in the corner whimpering "help me!" (that one was a wipe and reload as we couldn't even reinstall the antivirus to repair the system). This wound up being a rather expensive paid service call in terms of our bill and lost production.

[Stagg54]

I wrote software that was able to run in nuclear power plants, but was apparently not trustworthy enough to have administrator access on my own laptop.  They had no problem giving me administrator access on the machines that the end software got installed on. So I could run the installer on the actual machine, but couldn't test the installer on my own laptop.  Seemed kind of schizophrenic to me.  One hand they trusted me, and yet on the other hand they didn't.

I also worked in an R&D lab where we were constantly evaluating new hardware, which of course involved installing drivers. I would have to put in a ticket and have the local IT guy install it for me. After several hours he would walk over.  He had no clue what I was installing and didn't really seem to care much either. I would just start the installer and when it asked for admin credentials, he would come over eventually and put his in.  Then he would walk away.  

 

[Stagg54]

On ‎6‎/‎16‎/‎2017 at 7:10 AM, hooovahh said:

I worked at a company that had all USB devices disabled.  Not just USB memory devices, but also USB CAN, USB DAQ, USB Serial, USB GPIB, everything.  For the DAQ stuff we would just simulate the hardware, then deploy it to test it.  It was a pain but not too bad.  The USB memory devices were easy to get around.  It made the USB stick read-only, so to write to it you would map some empty folder like C:\USB to the E:\ drive or where ever your USB was.  Then you could read and write to the C:\USB folder all you wanted which actually wrote to the USB device.  I guess they just setup software to disable writing to drives that were marked as removable. 

That DgAgent sounds nasty.

We got a new head IT security guy.  Shortly after starting, he somehow got a virus on his laptop from a thumb drive he picked up somewhere. The solution was to ban thumbdrives.  So he sent out an email to the entire company that said basically: "Due to security risks, we are banning all USB thumbdrives except for these special ones that are encrypted and have fingerprint readers built into them.  They cost $200 (for a 4 GB drive).  Oh and they are backordered and won't be available for 6 months.  If you want one, let us know and we will put you on the waitlist."  It was almost universally ignored...

[MikaelH]

On 16/06/2017 at 11:10 PM, hooovahh said:

That DgAgent sounds nasty.

Here is just a snapshot how much the DGagent is doing in the background.
These are the task it does during a 5ms time period, it makes your faster computer start crawling.