We know that to open a pop-up we use window.open( [sURL] [, sName] [, sFeatures] [, bReplace]).

But, what if we want to open it as a modal window? We can use window.showModalDialog(sURL [, vArguments] [, sFeatures]).

But this approach has certain disadvantages, your client side script does not always behave the way it should otherwise.

So the best possible approach to open a modal pop-up window is,

From your page, open a normal popup as
window.open(‘/_layouts/OpenOpUp.aspx?Mode=New’, ‘pop1’,‘left=80, top=50, center=yes, height=480, width=450,status= no, resizable= no, scrollbars=yes, toolbar=no, location=no,menubar=no’); 

Now, in the modal popup window(here, OpenOpUp.aspx) in the body tag write,
<body onblur=”self.focus()”>
Your job is done, now it will behave as a modal popup window as it will have focus on itself and you have to close the page to get back to the parent page (either through javascript or by clicking the cross button at top) and all your client side scripts will work just fine.