you can use a script to "synchronize" the heights of the inner and outer table.
Aussumed you have the following hierarchy.
Put this script into the layout:ready event of the row "NestedRow" of the nested table.
var cUnit = "mm",
nRefH = xfa.layout.h(Table.Row, cUnit),
nThisH = xfa.layout.h(this, cUnit);
NestedCell1.minH = nRefH > nThisH === true ? nRefH.toString().concat(cUnit) : "5mm"; // replace 5mm with a minimum height of your choice.
The script will compare the height of the row that contains nested table with the row of the nested table itself and will change the minimum height of the first cell of the nested table. All other cells of it will then automatically use the same height.