Jira Bridge Bulk Updates missing changes

djhauck · October 8, 2019, 6:14pm

Hi All, I’m curious if anyone out there that is using the SG>Jira bridge has run into an issue with bulk updates on the Jira side. Everything works fine going either direction with individual updates, but when we do a bulk update in Jira (say transitioning multiple issues to done), the Jira>SG bridge only catches a handful of the updates. We think this generally has to do with the fire-and-forget nature of the Jira updates and the bridge can’t keep up. We’re considering creating a log on the Jira side so we can store the updates that come faster than the bridge can process them, but a log to be a database of changes between two other databases seems messy.

philip.scadding · October 9, 2019, 1:15pm

Sounds like it could be a bug, but our QA just ran a test batch updating about 20 things, and he didn’t encounter an issue. How many items were you updating in Jira?

djhauck · October 9, 2019, 3:41pm

The smallest batch I ran was 49, it caught just under 20 of the updates. It also caused a critical error in the plugin. Here is a dump of the error before the section with specific content I can’t post here.

Time: 2019-10-08 11:53:24,220
Logger: plugin.sg_jira_event_trigger.process_event
Path: C:\shotgun-events-jira-bridge\src\shotgunEventDaemon.py
Function: process
Line: 966

An error occured processing an event.

Traceback (most recent call last):
File “C:\shotgun-events-jira-bridge\src\shotgunEventDaemon.py”, line 956, in process
self._callback(self._shotgun, self._logger, event, self._args)
File “C:\shotgun-events\src\shotgun_events_plugins\jira_bridge_plugins\sg_jira_event_trigger.py”, line 173, in process_event
json=payload,
File “C:\shotgun-events-jira-bridge.venv\lib\site-packages\requests\api.py”, line 116, in post
return request(‘post’, url, data=data, json=json, **kwargs)
File “C:\shotgun-events-jira-bridge.venv\lib\site-packages\requests\api.py”, line 60, in request
return session.request(method=method, url=url, **kwargs)
File “C:\shotgun-events-jira-bridge.venv\lib\site-packages\requests\sessions.py”, line 533, in request
resp = self.send(prep, **send_kwargs)
File “C:\shotgun-events-jira-bridge.venv\lib\site-packages\requests\sessions.py”, line 646, in send
r = adapter.send(request, **kwargs)
File “C:\shotgun-events-jira-bridge.venv\lib\site-packages\requests\adapters.py”, line 516, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host=‘slc-shotgun01’, port=9090): Max retries exceeded with url: /sg2jira/default/Task/616545 (Caused by NewConnectionError(’<urllib3.connection.HTTPConnection object at 0x0000000002745710>: Failed to establish a new connection: [Errno 10061] No connection could be made because the target machine actively refused it’,))

Local variables at outer most frame in plugin:

b.croucher · October 29, 2019, 11:23pm

Just wanted to bump this thread to see if anything has been learned, and say that when we do a bulk update in jira we also have shotgun events from the daemon that don’t get processed.

In our case, since we have the jira bridge trigger plugin set to deactivate when an error is logged, this also means that someone has to go in and restart the daemon to reactivate the plugin.

rob · October 30, 2019, 3:25pm

I think this could be coming from the use of BaseHTTPServer.HTTPServer as the default webserver for the bridge:

github.com

shotgunsoftware/sg-jira-bridge/blob/master/webapp.py#L48


            }
            .content { margin: 0 0 15px 15px; }
            .error { margin: 0 0 15px 15px; }
            .details { margin: 40px 0 15px 15px; }
            h2 { margin-bottom: 10px; }
            p { margin-top: 10px; }
        </style>
"""

HMTL_TEMPLATE = """
    <head>
        <title>SG Jira Bridge: %s</title>
        {style}
    </head>
    <body>
        <h1>SG Jira Bridge</h1>
        <div class="content">
            <h2>%s</h2>
            <p>%s</p>
        </div>
    </body>

That server is single threaded and what you are reporting seems consistent with it not keeping up.

It should be possible to update the code to switch to a multi-threaded server with the same general interface. There is a recipe to do this talked about here:
http://brahmlower.io/threaded-http-server.html

And one wrapped up as it’s own package here:

We’ll add getting this fixed to our backlog, but maybe you can try the recipe to see if it helps our during batch changes?

b.croucher · December 4, 2019, 8:41pm

So I implemented a multi threaded server on the jira bridge. It turned out to be fairly easy to set up since the socket server framework has a threading mix in class included. Here are the changes it required:

This mostly worked as shown. It would start a thread for each request that came in and process them in parallel, so we didn’t miss any requests that arrived in quick succession.

However, some of these threads would crash with varying errors. All occurring during shotgun api calls. After doing some digging, it became clear that the shotgun api was not thread safe because it used the same connection across all threads, and it was possible for a call in one thread to close the connection before another was done with it. I then found someone who had run into the same problem in the past, and who modified the api to create a new connection for each thread. Here is his repo with the required changes: https://github.com/westernx/shotgun_api3/commit/dac6a58cb2389f0ec3b002e5eae4b7b80b025a84
I took his diffs and applied them to a new version of the api, and the problem was solved.

Now I’m curious if there is a way to submit a pull request with our changes or otherwise get them integrated. So that other people can make large bulk updates in jira, and so we don’t have to do this again next time we want to update our jira bridge.

Thanks for your help with this!

Patrick · July 26, 2021, 10:41am

This sounds like a pretty useful update. Is there any news on whether this has been implemented yet ?