Posts tagged "warp":

Bending Warp

In the past I've noticed that Warp both writes to stdout at times and produces some default HTTP responses, but I've never bothered taking the time to look up what possibilities it offers to changes this behaviour. I've also always thought that I ought to find out how Warp handles signals.

If you wonder why this would be interesting to know there are three main points:

1. The environments where the services run are set up to handle structured logging. In our case it should be JSONL written to stdout, i.e. one JSON object per line.
2. We've decided that the error responses we produce in our code should be JSON, so it's irritating to have to document some special cases where this isn't true just because Warp has a few default error responses.
3. Signal handling is, IMHO, a very important part of writing a service that runs well in k8s as it uses signals to handle the lifetime of pods.

main :: IO ()
main = Log.withLogger $ \logger -> do
    Log.infoIO logger "starting the server"
    runSettings (mySettings logger defaultSettings) (app logger)
    Log.infoIO logger "stopped the server"
  where
    mySettings logger = myShutdownHandler logger . myOnException logger . myOnExceptionResponse

myOnException :: Log.Logger -> Settings -> Settings
myOnException logger = setOnException handler
  where
    handler mr e = when (defaultShouldDisplayException e) $ case mr of
        Nothing -> Log.warnIO logger $ lm $ "exception: " <> T.pack (show e)
        Just _ -> do
            Log.warnIO logger $ lm $ "exception with request: " <> T.pack (show e)

myOnExceptionResponse :: Settings -> Settings
myOnExceptionResponse = setOnExceptionResponse handler
  where
    handler _ =
        responseLBS
            H.internalServerError500
            [(H.hContentType, "application/json; charset=utf-8")]
            (encode $ object ["error" .= ("Something went wrong" :: String)])

myShutdownHandler :: Log.Logger -> Settings -> Settings
myShutdownHandler logger = setInstallShutdownHandler shutdownHandler
  where
    shutdownAction = Log.infoIO logger "closing down"
    shutdownHandler closeSocket = void $ installHandler sigTERM (Catch $ shutdownAction >> closeSocket) Nothing

The timeout manager exception

The other day I bumped the dependencies of a Haskell project at work and noticed a new exception being thrown:

Thread killed by timeout manager

After a couple of false starts (it wasn't the connection pool, nor was it servant) I realised that a better approach would be to look at the list of packages that were updated as part of the dependency bumping.¹ Most of them I thought would be very unlikely sources of it, but two in the list stood out:

warp since the exception seemed to be thrown shortly after handling an HTTP request, and unliftio since the exception was caught by the handler for uncaught exceptions and its description contains "thread". Also, when looking at the code changes in warp on GitHub² I found that some of the changes introduced was increased use of unliftio for async stuff. The changes contain mentions of TimeoutThread and System.TimeManager. That sounded promising, and it lead me to the TimeoutThread exception in time-manager.

With that knowledge I could quickly adjust the handler for uncaught exceptions to not log TimeoutThread as fatal:

lastExceptionHandler :: LoggerSet -> SomeException -> IO ()
lastExceptionHandler logger e
  | Just TimeoutThread <- fromException e = return ()
  | otherwise = do
      logFatalIoS logger $ pack $ "uncaught exception: " <> displayException e
      flushLogStr logger

I have to say it was a bit more work to arrive at this than I'd have liked. I reckon there are easier ways to track down the information I needed. So I'd love to hear what tricks and tips others have.