IMPORTANT: Upcoming change that could cause some webhooks to fail because of timeouts.
Some users have recently complained about unexplained delays in the trigger of their webhook events or even losing some events.
While digging into these issues we found that the max amount of time allowed for each trigger (currently 6 seconds) was not being enforced. This allowed webhook triggers to take much longer than expected to complete. Under load, this can lead to the slowdown of webhooks deliveries on other sites.
Next Monday we will be releasing a fix to put back the time limit check. This could cause some existing webhooks to fail on some events (if the webhook endpoint takes more than 6 seconds to return an answer to Shotgun).
We are also rolling out a fix for lost deliveries. So, starting today, no delivery should ever be lost.
Please do let us know if you see missing deliveries after this date.
(Note: We built the system so no delivery would ever be lost. The tradeoff for this is that it is possible, in extreme cases, that the same delivery is sent twice to a webhook.)
PS: We are still investigating an issue which can also cause events to be delayed. We are working on getting this fixed asap and will provide an update here when the fix is ready.