The .rtexe is actually not an executable file. Rather, it is a "bundle" that contains your compiled VIs.
The real executable is /usr/local/natinst/labview/lvrt -- This executable loads your .rtexe bundle and runs the top-level VI(s) from the bundle. The lvrt program checks a config file -- /etc/natinst/share/lvrt.conf -- to find out which .rtexe it should load.
So, in theory, you could edit this file and then shut down the VIs that are currently running. This causes lvrt to re-launch, and it will read your updated config file and load your new .rtexe.
Notes:
Only 1 rtexe can be active at a time.
If you simply kill lvrt (as opposed to triggering a proper shutdown from within your .rtexe), LabVIEW thinks that it crashed. By default, LabVIEW will enter Safe Mode after 2 crashes: https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z000000kFQ6SAM