While I certainly agree there are some areas where SVs can improve (and a few years ago I would have agreed with your assessment when they had much more serious problems) I think SVs now deserve more credit. (I will state upfront that we still see occasional issues with the SVE--mostly having to do with logging, but we have workarounds for these, and I think if there are more users NI will fix these issues more quickly.)
We started using SVs really heavily 3-4 years ago for streaming. We ran into all sorts of trouble with them, and were thrown around by NI a fair bit while they tried to find the spot where we used those SVs wrong. Finally last year we got the verdict: SVs were not designed for streaming, they can break down when heavily loaded - please use Network Streams for this instead. That NI didn't let this info surface before Network Streams were ready as replacement is quite distasteful, it has cost us and our customers millions chasing SVs with the wrong end of the stick. Had NI told us with a straight face that our implementation weren't going to work, but they were working on a replacement, we'd have been in a much better position. Now, as it is, I never want to touch anything that has to do with SVs. I was burned too badly, and now we have our own much better solution.
1) Weight. We've never had an issue with an SVE deployed on Windows. We never deploy the SVE on RT. (We don't have any reason to do so.) Our RT applications (on cRIOs) read and write to SVs hosted on Windows without any problem.
We need to deploy the SVE on RT since the RT system is usually the always-on part in our applications - our Windows hosts are usually optional connect/disconnect type controllers while the RTs run some stuff forever (or for a long while). Simulation, DAQ, control/regulations etc. It could seem SVs fare better on Windows, both due to the (often) higher availability of system resources there, but also due to the more relaxed approach to determinism (obviously).
2) Black box: It is true that we don't see the internal code of the SVE, but we don't have to maintain it either. At some level everything is a black box (a truism, I know)--including TCP/IP, obviously--it just depends on how much you trust the technology. I understand your concerns. By the way, in my tests I haven't noted a departure from the 10 ms or buffer full rule (plus variations since we are deploying on a nonRT system), but there may be issues I haven't seen. The performance has met our requirements.
That's a good thing, and I sense that SVs work for most people. It may just be because SVs were never designed to the load we put on them. It's only when we have problems that it's bad that the toolset is a black box. Otherwise it's just good encapsulation

. But with TCPIP-Link we just flip open the bonnet if anything's acting up.
3) Buffer overflow: I'm not sure I understand this one. SV reads and writes do have standard error in functionality, so they won't execute if there is an error on the error input. We don't wire the error input terminal for this reason but merge the errors after the node. (I don't think this is what you are saying, though. I think you are saying you get the same warning out of multiple parallel nodes? We haven't encountered this ourselves so I can't help here.)
If you have a network enabled, buffered, SV, all subscribers (readers) will have their own copy of a receive buffer. This means a fast reader A (a fast loop executing the read SV node) may keep up with the written data and never experience a buffer overrun. A slow reader B of the same SV may experience backlogging, or buffer overwrite, and start issuing the buffer overflow error code (−1950678981, "The shared variable client-side read buffer overflowed"). A bug in the SVE backend means that read node A will start issuing this error code as well. In fact all read nodes of this SV will output this error code, even though only one of the read buffers overflowed. That is really bad, since it 1) makes it really hard to pinpoint where the failure occurred in the code, and 2) makes it next to impossible to implement proper error handling since some read nodes may be fine with filtering out this error code while other read nodes may need to cause an application shutdown if
its buffer runs full. If the neighbors buffer runs full but you get the blame... how will you handle this? NI just says "That's acknowledged, but it's too hard to fix".
4) While we don't programmatically deploy SVs on RT, as I mentioned, I agree it would be good (and most appropriate!) if LabVIEW RT supported this functionality. For the record, we do programmatically connect to SVs hosted on Windows in our RT applications, and that works fine.
You connect programmatically through DataSocket, right? With a PSP URL? Unfortunately DataSocket needs root loop access, so it blocks all other processes while reading and writing. I don't know if SVs do the same, but I know raw TCP/IP (and thus TCPIP-Link) doesn't. Anyways, for many applications this might not be an issue. For us it often is. But the main reason I'd like to be able to programmatically deploy SVs on RT would be for when variables undeploy themselves for some reason. If we could deploy them again programmatically we could implement an automatic recovery system for this failure mode. As it is now we sometimes need to get the RT system connected to a Windows host with LabVIEW dev on it to deploy the variables again manually. I know this process can be automated some more, but I'd really like the Windows host out of this equation - the RT systems should be able to recover all by themselves.
6) Queues, events, and local variables are suitable for many purposes but not, as you note, for networked applications. (We do, of course, use events with SVs.) TCP/IP is a very good approach for networking, but in the absence of a wrapper does not provide a publish-subscribe system. Networked shared variables are one approach (and the only serious contender packaged by NI) to wrapping TCP/IP with a publish-subscribe framework. If someone wants to write such a wrapper, that is admirable (and that may be necessary and a good idea!), but I think for most users it is a much shorter path to use the SV than to work the kinks out their own implementation. I haven't tried making my own, though, and the basics of the Observer Pattern do seem straightforward enough that it could be worth the attempt--it's just not for everybody. (I'd also prefer that we as a community converge on a single robust implementation of a publish-subscribe system, whether this be an NI option or the best offered by and supported by the community. At the present time I think SVs are the best option readily available and supported.)
It was very hard to write TCPIP-Link. But for us NI simply doesn't deliver an alternative. I cannot sell an application that uses Shared Variables - no matter the level of mission criticality. I can't have my name associated with potential bombs like that.
Single process (intra-process) SVs seems to work fine, but I haven't used them that much. For these purposes I usually use something along the line of VIRegisters (queues basically), but the possibility to enable RT FIFOs on the single process SV could be handy. But again I'm loath to put too many hours into experimenting with them, knowing how much time we've wasted on their network enabled evil cousins

.
Cheers,
Steen