If we solve F as a function of a and N

solve 2*(N*9*F)+2*a^2+((b-a)/2)^2=((3*a+b)/2)^2 , a*b=(N*9*F) , 2*(N*9*F)+2*1^2+((a+b)/2+1)^2-((3*a+b)/2)^2=0 ,F,b

->

9*N*F=2*a^2-3*a

multiplying by 2 and imposing 2 * a = A

we will have 18*N*F=A^2-3*A

A0 < sqrt(18*N)

is it possible to apply the Coppersmith method?

https://en.wikipedia.org/wiki/Coppersmith_method