Thanks for all your replies - I think I've now tried every possible combination of lens and bellows extension. The Illumitran bellows is simply not usable with a full-frame DSLR such as the D800, even with the top bellows mount removed. It can be made to work with a mirrorless body and a 50mm enlarger lens. I'll go back to using it without the bellows, with a separate copy stand and a micro-Nikkor, which will also let me copy medium format trannys more easily.
A secondary problem is the very high flash intensity (designed for very slow copy film, somewhere around 6 ISO!) - fortunately this can be solved with the aid of a 75mm square gel filter, 3 stop ND, inserted under the opal diffuser.