HTTP Error 503: Slow Down

Flow Production Tracking
1

Hey all,

Just got this fun AWS error while trying to upload a thumbnail from the python Shotgun api.

It appears that we are exceeding requests to the S3 bucket.
Is this something that I should just handle for and retry?
Do I have to delete the created entity and redo it? Does the python shotgun api treat a “create” call as atomic?

Perhaps there are docs/warnings that Shotgun published that I have missed

Thanks
-Alex

self._shotgun.create(entity_type, create_fields) 
File "/opt/shotgun_cache/jangley/distant/p128c10.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/shotgun.py", line 1367, in create 
  self.upload_thumbnail(entity_type, result["id"], upload_image) 

  File "/opt/shotgun_cache/jangley/distant/p128c10.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/shotgun.py", line 2326, in upload_thumbnail 
    return self.upload(entity_type, entity_id, path, field_name="thumb_image", **kwargs) 

    File "/opt/shotgun_cache/jangley/distant/p128c10.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/shotgun.py", line 2432, in upload 
      tag_list, is_thumbnail) 

      File "/opt/shotgun_cache/jangley/distant/p128c10.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/shotgun.py", line 2468, in _upload_to_storage 
        self._upload_file_to_storage(path, upload_info["upload_url"]) 

        File "/opt/shotgun_cache/jangley/distant/p128c10.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/shotgun.py", line 3887, in _upload_file_to_storage 
          self._upload_data_to_storage(fd, content_type, file_size, storage_url) 

          File "/opt/shotgun_cache/jangley/distant/p128c10.basic.desktop/cfg/install/core/python/tank_vendor/shotgun_api3/shotgun.py", line 3985, in _upload_data_to_storage 
            raise ShotgunError("Unanticipated error occurred uploading to %s: %s" % (storage_url, e)) 

ShotgunError: Unanticipated error occurred uploading to <long S3 url>: HTTP Error 503: Slow Down
2

Ah part of the transition to direct s3 uploads mention retries.

I’ll just do that then and hopefully everything works!

From @zoe.glynn

You have not built in retries to your file upload operations .
As with the Shotgun-conduit architecture, failures will sporadically happen when uploading directly to S3. A retry on the spot typically works.

3

Wait actually I still do have a question about the entity. Do I have to run create again or just the upload thumbnail?
How do I get the entity id of the error’d entity?

4

I guess the solution is to not use create for the image/filmstrip fields.
I’ve broken that out similarly to how it’s being done in create but I have added retries to it.

Is this something that might make sense to put in the core python shotgun api?