There are two broad categories of objects/classes:
Values (e.g. numbers, strings, URLs, timestamps, images*, tuples/collections of these)
Identities/entities (e.g. file, physical I/O channel, physical device, database, GUI control/indicator/window, state machine)
Conceptually, values can be copied but identities cannot. (To illustrate what I mean: Branching a string wire can create a new copy of the string, but branching a DAQmx Channel wire cannot create a new channel)
I would use references as a last resort for "value objects", but as a near-first resort for "identity objects".
*IMAQ images are by-reference, but they didn't strictly have to be.
Genuinely curious: Why do you expect reference-heavy G code to be less performant?