Python API (< 3.0.37) deprecation & Direct Upload to S3

Shotgun is standardizing the way we handle file uploads for all hosted sites, as we work on our architecture to enable improvements in reliability and quality of image transcoding.

One change coming soon is direct upload to S3. This feature will be progressively enabled on hosted sites over the next months. A dismissible banner will be presented to each site’s admin users with notice of the change a couple of weeks beforehand.

  • Python API versions 3.0.37 and above are compatible. Support for earlier versions will be dropped in 2020.
  • Thumbnail “image” fields will have an additional possible value to indicate transient state. (Exposed to both REST and Python APIs).

Read on for a complete overview of changes.


What is changing?

Previous Upload Operation

Behind the scenes, Shotgun’s original system architecture meant that any (web UI or API) file uploads resulted in files first being uploaded to the Shotgun server, and then sent to S3 for storage.

FsPNFITyrFFl4PQ8asHen3dupKFQBOAFVhlQkhMfsaqR2xSgoQp2Swj62Ty3U7PttnW5jw4F0fHe_Ch9afFJu9EqLTDxpPasVrqWyZye4_nlpHjVYtm2j_KGy3TFCnPmvhL4p-DO

New Upload Operation

In the new upload operation, supported by Python API versions 3.0.37 and newer, the client application asks Shotgun for an ‘upload link’ and then uses this link to upload content directly to S3.

Shotgun is not involved in the upload process, but is informed once it is complete.

cr92Hot_lDio6gH1Z99bt_WAgAwfCgEgzDZF7tz67IcbrguMD9wfNY-DxS7nAJvHVXVLXU0qjsk-KqBdGBRYvb2EpZ4WkjThahuGQfc6uHIDicEjzj5bXtEP6ilEpZOprehKa54f

Why this Change?

Durability

Since media does not need to hop between servers, the upload chain is greatly simplified and reduces risks of media being lost in interim steps.

Users are guaranteed (by S3 policies) that when an upload succeeds, their media will not be lost.

Performance

Since media upload requests do not transit through Shotgun servers, more bandwidth is available to process other user requests.

Security

With content going directly from its source to its target destination the attack surface is greatly reduced.

Do I need to update my code?

From an API client perspective, you should be aware that this change introduces an additional ‘state’ of values returned when querying thumbnail URLs (“image” fields).

Three possible states for values returned are:

  • null: No thumbnail image uploaded, or thumbnail generation has failed.

  • transient URL: Serves a placeholder icon. Returned when image is requested after upload, but before thumbnail has been created and made available on S3. Will be a constant string for your site, with the form: <protocol>://<domain>/images/status/transient/thumbnail_pending.png
    NOTE: Other upcoming features are likely to require the use of other transient thumbnails.
    For this reason, it is highly recommended to use the prefix part of the placeholder path (e.g. https://yoursubdomain.shotgunstudio.com/images/status/transient/) to detect any transient URLs rather than use the full path of the thumbnail.

  • signed URL for S3 object: Provides access to final processed thumbnail. Returned after transcoding is complete and media is available on S3.

This implementation provides a seamless experience for dynamic user interfaces, and in many cases, no changes will be required from API calling code.

You will need to take action in the following cases:

  • You are running a Python API version below 3.0.37 (Toolkit Core API below v0.18.149).
    Python API versions below 3.0.37 (released July 2018) will be deprecated, and support dropped in 2020. Deprecation notices will be posted to site admins. Please update to the latest version as soon as possible.

  • You have not built in retries to your file upload operations.
    As with the Shotgun-conduit architecture, failures will sporadically happen when uploading directly to S3. A retry on the spot typically works.

  • You persist image URLs, assuming they are immutable.
    This could include:

    • generating reports which include thumbnail images
    • caching results of image field queries based on entity information only

Developers can explicitly deal with this new behavior in different ways:

Let us know if you have any questions or concerns.

6 Likes

Hey Zoe!

One question I’m not clear on. Is this going to be a new option to upload with, or is this entirely replacing the old upload functionality? I’m not clear on if it’s required that we update our scripts or just that we could change them to use this new functionality.

1 Like

Hi Gary, we’re entirely replacing our architecture behind uploads, and if you utilize the Python API, it will be necessary to run a newer version in order to be compatible.
After that, your scripts themselves will only require revisiting to ensure they can deal gracefully with upload failures (something you may already accommodate) or if you are reporting or caching the values returned for image fields without first inspecting for validity (not a situation we anticipate, because of expiration time for signed URLs).

3 Likes

Hi Zoe, how does this change affect local-installs i.e. the Shotgun app and database and files/storage are all on-premises, with no S3? I’d imagine this is a silly question and the answer is “no change whatsoever”… but it’s nice to be sure. :slight_smile:

1 Like

Hi Tony, correct, “no change whatsoever” is required for local installs with no S3, but we’d still encourage people to upgrade to the latest Python API for its support of Python 3, amongst other niceties :slight_smile:

3 Likes

Hi,
Can you give little correlation between tkcore version and pyhton api version.

Core v0.18.149 bundles python API v3.0.37:


Thanks for the question. Main post now edited to include this info.