Shotgun is standardizing the way we handle file uploads for all hosted sites, as we work on our architecture to enable improvements in reliability and quality of image transcoding.
One change coming soon is direct upload to S3. This feature will be progressively enabled on hosted sites over the next months. A dismissible banner will be presented to each site’s admin users with notice of the change a couple of weeks beforehand.
- Python API versions
3.0.37 and aboveare compatible. Support for earlier versions will be dropped in 2020.
- Thumbnail “image” fields will have an additional possible value to indicate transient state. (Exposed to both REST and Python APIs).
Read on for a complete overview of changes.
What is changing?
Previous Upload Operation
Behind the scenes, Shotgun’s original system architecture meant that any (web UI or API) file uploads resulted in files first being uploaded to the Shotgun server, and then sent to S3 for storage.
New Upload Operation
In the new upload operation, supported by Python API versions
3.0.37 and newer, the client application asks Shotgun for an ‘upload link’ and then uses this link to upload content directly to S3.
Shotgun is not involved in the upload process, but is informed once it is complete.
Why this Change?
Since media does not need to hop between servers, the upload chain is greatly simplified and reduces risks of media being lost in interim steps.
Users are guaranteed (by S3 policies) that when an upload succeeds, their media will not be lost.
Since media upload requests do not transit through Shotgun servers, more bandwidth is available to process other user requests.
With content going directly from its source to its target destination the attack surface is greatly reduced.
Do I need to update my code?
From an API client perspective, you should be aware that this change introduces an additional ‘state’ of values returned when querying thumbnail URLs (“image” fields).
Three possible states for values returned are:
null: No thumbnail image uploaded, or thumbnail generation has failed.
transient URL: Serves a placeholder icon. Returned when image is requested after upload, but before thumbnail has been created and made available on S3. Will be a constant string for your site, with the form:
NOTE: Other upcoming features are likely to require the use of other transient thumbnails.
For this reason, it is highly recommended to use the prefix part of the placeholder path (e.g.
https://yoursubdomain.shotgunstudio.com/images/status/transient/) to detect any transient URLs rather than use the full path of the thumbnail.
signed URL for S3 object: Provides access to final processed thumbnail. Returned after transcoding is complete and media is available on S3.
This implementation provides a seamless experience for dynamic user interfaces, and in many cases, no changes will be required from API calling code.
You will need to take action in the following cases:
You are running a Python API version below 3.0.37 (Toolkit Core API below v0.18.149).
Python API versions below 3.0.37 (released July 2018) will be deprecated, and support dropped in 2020. Deprecation notices will be posted to site admins. Please update to the latest version as soon as possible.
You have not built in retries to your file upload operations.
As with the Shotgun-conduit architecture, failures will sporadically happen when uploading directly to S3. A retry on the spot typically works.
You persist image URLs, assuming they are immutable.
This could include:
- generating reports which include thumbnail images
- caching results of image field queries based on entity information only
Developers can explicitly deal with this new behavior in different ways:
- Testing the returned image string
- Using Event Log Entries for thumbnails
- Using the new processing_status field on Attachment (aka File) entities.
Let us know if you have any questions or concerns.