I’ve tried re-delivering a bunch of times, but it won’t trigger on this request. It works fine on a normal “New Version Created” webhook though.
Seems to be working now.
HI jeff, I’m glad things are working now but concerned about the failure you experienced.
Can you provide the time frame during which you had issues. I’ll check the logs to see if I can find a smoking gun.
Thanks in advance!
Sure thing. This is the timestamp on the specific message that went undelivered. What’s most odd is that trying to re-deliver that specific message wouldn’t ever go through.
1/18/2020, 11:36:20 AM
I just had a webhook fire and deliver twice with the same event_id.
Can you tell me what your Shotgun site name is?
I’m still looking into these specific cases but, I can say that it is possible that an event gets delivered more than once. This is a tradeoff we had to make in order to guarantee no events are lost.
Yeah, this just happened again :(. I guess I’ll have to add logic to my code to prevent my action on the receiving end from firing twice.
It is surprising that it is happening so often though…
I think it happened a total of around 5x yesterday across almost 27 newly created versions, putting the double delivery rate around 15%.
I opened a ticket for this. Double deliveries can happen byt that should be in error-recovery mode on our side.
So your code should definitely account for these but they should not be in the way that often…
Concerning the re-delivery issue, if you try to re-deliver that same event (I think it’s event id:
55251.4281.0 ?), does it now work?
So that particular event isn’t actually one that I care about. I care much more about the one that came right after it:
55253.4284.0. That’s the one that wouldn’t deliver for me.
I’ll have to SSH into that machine to see if the event comes in because it won’t be able to complete its action since the signed S3 URL which I’m grabbing from the payload will have expired at this point (and thus the next step won’t work).
I will reply here once I’ve had a chance to test that.
Ok, making progress!
At this point, I’m pretty sure that if you do try to re-deliver, it won’t work.
But it would be really great to have a confirmation!
We’re working on a fix for the doubled deliveries and, if my theory is correct, this will also fix the the re-delivery issue…
Awesome! Glad to hear this bug report has proven to be beneficial.
I still plan to do the SSH test at some point today but I’m swamped w/ roto and tracking stuff right now…
Out of curiosity, are you planning to hook this new Webhook system up to Zapier?
As far as I know, there are no short term plans to hook into Zapier right now. But it would be great to hear more about the workflows you’d like to enable through it.
Continuing to dig…
What we’re seeing in the logs is that event ‘55251.4281.0’ (the one you are not interested in) is the one that got re-delivered (6 times) but ‘55253.4284.0’ (the one you are interested in) never did.
I’m trying to see if this is somehow related to the fact that both events were initially delivered pretty much at the same time.
These events are 5 days old so they will get deleted today
If you do have a chance to retry the re-delivery, that would really be awesome!
Thanks again for your help with this!
Just tried re-delivering that one again while watching the log output on the remote server. Didn’t see anything come through.
As for what I would like to use Zapier for, I think it would be awesome to easily hook up other applications to my Shotgun event stream. To be able to do things like send a link to Slack when a new Version is posted, or message users on Slack when there’s a new note that mentions them.
Not to mention their Python support, means that Zapier can act as a simple serverless solution to support ‘minimal code’ rather than no-code workflows.