This one is hard to explain in print. Let x be the side of the rectangle on the hypotenuse and y be the other side. There is a triangle formed at the right angle of the original which is similar to the original.
So x is to 10 as z is to 8: x/10 = z/8
and z = 4/5 x
The upper part of that leg would be 8 - 4/5 x
The triangle at the top is also similar:
So y is to 6 as (8 - 4/5 x) is to 10: y/6 = (8-4/5 x)/10
and y = 3/5 (8- 4/5 x) = 24/5 - 12/25 x
Now area = xy
Deriv = 0 and solve