Long response time for clients on custom 404 page

I've got a custom 404 page set up like this on my IIS8:

<httpErrors errorMode="Custom" existingResponse="Replace"><remove statusCode="404" subStatusCode="-1" /><error statusCode="404" path="/404/" responseMode="ExecuteURL"/></httpErrors>

Everything works and if i use the browser on the server and browse to localhost and try to access an url that doesn't exist I get the custom 404 page. If I browse to an url that dosn't exist from another computer or from the server using the real host the correct 404 page is still displayed but the response time is through the roof, it takes about two minutes to load.

I have this behaviour on two different servers.

Any help is appreciated