Good job!
To set the gains, you have to start at the headunit and work along the signal path.
The idea is to turn up the headunit until it starts to distort, then turn it down slightly to produce a distortion free input to the amps. About 3/4 on the volume knob.
Then you slowly increase the gain on each amp in turn until the required output is found for each, making sure not to overdo it and produce distortion. The two amps should be done separately.
But first, turn the gains on the amps down to about 1/3 or lower. Set the sub amp filters to low pass (any frequency around 80Hz). Set the speaker amp filters to around the same.
With headunit tone controls flat, and the sub out level at 1/2 (can you control that?), turn up the headunit volume until you can just hear the speakers distorting.
It's all a matter of trial and error and finding the right balance.
Having two amps complicates things and if there are signal processors in the path, it gets even worse!
It sounds like you're overdriving the amp to make it cut out. Start again with the gains down and work up. It'll take quite a lot of experimentation to get it right.
The filter settings are trial and error too. As lond as the sub is low passed and the speakers high passed, you'll be fine.
Happy fiddling!
Edit. The pro's do it with an oscilliscope! More neighbour freindly.