It's been a long, long time but I did spend a week trying to get one to work so I'm emotionally scarred.
Part 1 — P2P viability: With no stubs and both devices transformer-coupled internally, is the coupler genuinely redundant? Or are there signal integrity concerns I should be thinking about?
The internal transformers are for isolation rather than impedance matching. They are required so you can have floating differential signals. If you are only transmitting over a couple of metres then you can get away with no external isolation. But you should really be using a dual-stub transformer bus coupler inline (or at one end) with terminators. Everyone starts point to point then adds devices!
Part 2 — Cable spec: We're specifying a custom cable for this (DB9 female one side, HD-DSUB15 male the other — matching the AIM card's connector). What are the things that matter most in the spec — impedance tolerance, shielding, untwisted length inside the backshell, connector grounding? What's easy to get wrong?
Suggestions (in order of importance)
#1 You must terminate both ends. Each end should have a 78 ohm resistor across it. Impedance mismatch is the No.1 Killer. You don't seem to have mentioned termination resistors.
#2 Check differential pair polarity (A and B lines) and that the correct terminals on the connector are soldered. It's been a while but I vaguely recall that the cable is a crossover cable if you are not using couplers so D1A connects to D2B etc. (I may be wrong on that)
#3 Check your cable impedance. It should be 70–85 ohm at 1 MHz.
#4 Make sure grounding is good (360 deg). You will get away with crappy grounding in the lab if the distance is very short, though. Aim for a max of 1-1.5 cm unshielded inside the connector shell. Use metal shells grounded to the chassis, if you can.
#5 Put a scope on the lines looking at the waveform if you don't have an analyser. You are looking for rise/fall times of 100–300 ns, overshoot/ringing limits, etc (I don't remember them all-look at the spec).