ShotgunEventDaemon not working anymore

Good morning,

We have been using the ShotgunEvents trigger system you years at our Studio and in the last couple of week it has just stopped working altogether. Not sure if we just finally hit critical mass on the number of events or something else with the server is amiss.

For the last week all and of the triggers do is…

What does this mean?

Daniel

3 Likes

Hi Daniel, thanks for posting. I think those messages may be a red herring. Here’s an explanation from a previous support case, written by @bouchep (the esteemed creator of the ShotgunEvents daemon):

The message Timeout elapsed on backlog event id X is shown when an event has failed to show up in the list of event log ids in the allotted time. Normally you’d only see this once for a given id. Because of the multi threaded nature of the app event log entries don’t appear sequentially. For example:

Process A's db transaction finishes and creates event log entry 1
Process B's db transaction reserves event log entry 2 but doesn't create it right away because the transaction isn't finished.
Process C's db transaction finishes and creates event log entry 3

We’ve got event log entries 1 and 3. If the daemon queries for event log entries at this point, 2 will be “missing” and will be added to a backlog.

At this point process B’s transaction can finish and create event log entry 2 or it can fail leaving a hole in the sequence. In the first case, the event will be processed. In the second case, we won’t wait indefinitely but after some time (default of 5m) you’ll get the message Event X is too old. Last event processed is (Y) and the expected event log entry will be removed from the backlog.

In terms of troubleshooting your event daemon, do you have any more detailed logs of what it may or may not be processing? How many different plugins are active in your daemon?

3 Likes

No useful information in the logs. This has been going on for more than a week and all of the logs are filled will the error above.

We have around a dozen plugins currently running. I will try adding back one plugin at a time to see if I can get more details or anything new in the log files.

Daniel

2 Likes

Hi Daniel,

Have you kept your daemon install up to date with the latest state of the code in the repo? The project doesn’t evolve extremely rapidly but I thought I’d ask anyway.

2 Likes

Hi Daniel,

I also wonder if perhaps the location of your plugins has moved on your network / filesystem?

1 Like

Good morning,

I think I have gotten things straightened out. This was a combination of user and hardware failures.

Out daemon runs on a VM and needed to be re-booted a few times to finally get the ‘kinks’ out. Additionally I had inadvertently set the logging level too high to show any activity whatsoever. So, even when it started working correctly, I thought it was still broken. This was super frustrating and totally my fault.

We had been experiencing severely degraded trigger performance and I was trying to see if some tweaks to the .conf could help. Turned out it was a persistent network/VM issue that I was mis-interpreting as an issue with the daemon.

Several reboots later, and setting logging back to a useful level seems to have straightened things out. I set the event numbers back in time to before the issues started and now that it is caught up I think all is well.

Thanks to everyone for playing along we me.

4 Likes

Hi Daniel,

I’m stoked you got it figured out. Happy plugin writing.

2 Likes