Certainly posting the VI might help. Short of that I've seen this issue when using this function a couple times, all of which were expected if I had stopped to think about it. The first is maybe your front panel has a minimum size set, and the decoration is less than that minimum. In this case you won't be allowed to make the window smaller, so it throws an error.
I've also seen it do odd things when there are splitters and multiple panes on a UI, where it will try to scale the window to the largest decoration, but that decoration is only on one pane. I've also seen where there is a minimum pane size, and again the decoration is too small.
I'm sure there are other reasons this VI can throw an error. Because of these types of issues I don't use this function too often any more. I also heard someone mention this function does nothing, and returns no error on the new cRIOs running embedded linux, with a monitor plugged in, but I didn't get a chance to debug it.