Python API/Rest API and authentication with SSO enabled

Pipeline Integrations
, ,
1

Both our APIs—Python and REST—allow the use of a username/password pair for authentication when connecting to Shotgun, but if a site is using SSO, the username/password method is not allowed.

While this may be surprising, there is a good explanation:

Shotgun is no longer the gate-keeper, and cannot tell by itself if your credentials are valid. This task is now handled by your IdP (Identity Provider, like Azure, Okta, etc.).

(Note: scripts that authenticate with a script_name/api_key pair are not impacted by the use of SSO, it is business-as-usual for them)

When using a browser, Shotgun redirects you to your IdP (using a 302) and then the IdP redirects your browser back to Shotgun once authenticated. Unfortunately, this mechanism is not possible when using a API-based communication. The handshake process between the servers involves Javascript being interpreted and the actual login flow varies across IdPs.

But, can't Shotgun just contact the IdP to validate the credentials?

Unfortunately, no. Depending on your specific network topology, Shotgun may not able to directly connect to your IdP (in fact, our SSO implementation assumes that Shotgun cannot connect directly to an IdP). As well, no IdP offers an API to validate a password as it is a potential security risk.

The key take-away is:

Use of username/password in API calls are inherently insecure and highly discouraged.

So, what are my options to authenticate with the API when SSO is enabled?

Use a web browser—or a browser-like widget—to authenticate and then use the session_id from the browser to initiate an API connection with Shotgun. This is what is done by Shotgun Create, Shotgun Desktop, and RV.

These applications use a special path on the Shotgun server (<SERVER_URL>/auth/renew) to start an authentication process for a user. Once the web widget gets back to the <SERVER_URL>/auth/landing, we know that the login process was successful. The widget is then closed and we fetch the _session_id value from the cookie jar. This is used to create an API connection to Shotgun.

It is assumed that any web page shown to the user is part of the authentication process or is reporting back errors to the users with additional instructions. If the widget is closed without reaching the landing page, then we know the user abandoned the process.

The mechanism used by our applications is available to all external clients, but the simplest way to make use of it is to use Shotgun Toolkit in combination with Shotgun Desktop or Shotgun Create installed on your system to use the existing session.

Additional Background and Context

Shotgun's SSO implementation relies on SAML2.0 as this older standard maximizes the number of IdP we can support. This means we do not have to rely on IdP-specific SDKs or APIs—thereby reducing maintenance and support complexity.

We expect Shotgun and the IdP to know about each other, but we do not rely on direct communications between the two. In order to support the case where Shotgun is hosted on the cloud, but the IdP is hosted internally in the company’s infrastructure. For Shotgun to connect to the IdP, a special firewall configuration would be needed to allow.

Instead, we rely on the client application (e.g.—the browser or the web widget) to be the relay for communications between the two servers. The expectation being that the user’s machine is able to reach both Shotgun and the IdP. This also sheds some light on the absence of server-to-server calls to validate the user’s credentials.

Implementation Examples

Here are two potential examples of ways to proceed that connects to a site and gets the list of projects (before.py) using a username and password when assuming a simple initial Python API script:

Sample 1—Using Toolkit for User Authentication Reference

(after_simple.py) We use the toolkit to get a reference to the currently authenticated user and create a new connection to the server (which is the equivalent of a Shotgun() object creating in the Python API). If there are no currenlty authenticated users, we instruct the script invoker to start either the Shotgun Desktop or Shotgun Create and authenticate.

Sample 2—Using Current Toolkit User

(after_self_auth.py) We attempt to use the current toolkit user. If there are none, we create a Qt Application (so that the GUI can be shown, not just command-line prompts), and then ask the user to select a site and authenticate.

These are just simple examples that can easily be modified and tailored to suit the different needs.

They each have pros and cons, and your individual use case will dictate which one makes more sense for you.

-Patrick

8 Likes
3

Finally, here’s an overview of pros & cons of using Toolkit as opposed the the Python API:

Under the hood, the Toolkit uses the Python API, but a lot of scaffolding and abstraction is built on top of it.

1. Using the Python SDK with username/password

Pro:
-You can have multiple scripts connecting to different sites with different users at the same time

Con:
-SSO is not supported

2. the simple integration with the Toolkit

Pros:
-Support all current and future authentication mode, as well as handle MFA and other security mechanisms
-The changes to an existing script is minimal (import sgtk, create connection). The rest of the Python API script stays the same

Cons:
-Can only be connected to the same site as the current Toolkit session
-You need to query which user and site are in play, to get the current context of the connection
-Dependent on another application to initiate the connection (SG Desktop or Create), You do not control the site/user

3. the self-auth integration with the toolkit (as is show in the sample code):

Pros:
-Support all current and future authentication mode, as well as handle MFA and other security mechanisms
-The changes to an existing script is minimal (import sgtk, create connection). The rest of the Python API script stays the same
-No need to defer to the SG Desktop or Create to initiate the authentication

Cons:
-Can only be connected to the same site as the current Toolkit session
-You need to query which user and site are in play, to get the current context of the connection
-You need to have PySide/PySide2 available on your machine (which is not that big of a deal if you rely on an existing SG Desktop install)
-Since a GUI is shown to the user to select the site/user, no control over those values
-Loading PySide and creating a QApplication does induce increased startup times

3 Likes
5

The 3 previous samples I included were using (directly or indirectly via the Toolkit) the Shotgun API.

I have created versions of these, which instead are using the REST API.

The important take-away is that you need a way to get a valid session_token from Shotgun. This can be achieved in a number of different ways, but the common point is that a Web Browser has to be used in that process. Our examples usea Qt, which is used in RV, Create and Desktop. You can use native toolkits for your target platforms, which is what we used for the iOS Review app.

Base sample : before_rest.py (1.4 KB)

Sample 1 : after_simple_rest.py (2.7 KB)

Sample 2 : after_self_auth_rest.py (2.6 KB)

Assuming that you have in hand a valid session token, and you want to use the Python API, you need to initiate your API session like this:

from shotgun_api3 import Shotgun
sg = Shotgun(<YOUR_SITE_URL>, session_token='<YOUR_SESSION_TOKEN>')
projects = sg.find('Project', [], [])

To see how it is done with the REST API, have a peek at samples 1 and 2.

-Patrick

4 Likes
6

I would like to add an important note about our dependency on PySide and PySide2…

Unfortunately, different products ships with different versions, and this will impact your ability to connect to a Shotgun site that is using SSO.

PySide:

  • only version 1.2.2 and up is known to be compatible. And even then, it may be best to use the version bundled with the Shotgun Desktop as apposed to a standalone version or one shipped with another product.
  • based on Qt4, this is very old and going away.
  • the web functionalities are provided by QtWebKit, which is a hard requirement. This is an optional package and it may be absent from your version, which will prevent SSO support.
  • used by the Shotgun Desktop up to version 1.5.9 (version 1.6.1 and later uses PySide2)
  • used by RV up to version 7.3.4 (version 7.5 and later uses PySide2)
  • also used but older applications, but not recent ones
  • only is available for Python2, not Python3
  • there is an NTLM patch applied to versions provided by Autodesk products. It is best to stick to those versions.

PySide2:

  • only version 5.12.1 and up are known to be supported and work. Some version-specific issues may cause problems with the authentication process.
  • version 2.0.0~alpha0 without QtWebKit is NOT supported, no matter if it came from an Autodesk product or another application or standalone.
  • this uses Qt5
  • provides the web environment using QWebEngine, based on Chromium. This is a hard-requirement. This is an optional package, which may not be present in your build. And even when present, must be properly configured.
  • some older version of PySide2 would provide the older QtWebKit. In that case, you will be able to authenticate. But this is a very rare setup, seen in software from 2018 or before.
  • PySide2 is available for Python2 and Python3. The official Qt repository does not offer a Windows version for Python2, but certain applications do.
  • The recent Autodesk applications which offers PySide2 are usually able to support the full authentication flow. Contact our support if you have questions about specific versions.
  • When using the Shotgun toolkit, a recent release of tk-core, version v0.19.16 or more, will be required. You may need to update your configuration.
  • When using the recent versions of RV, Create and Shotgun Desktop, support of SSO will be present.

In short, whenever possible always use the latest version of the application and tk-core.

-Patrick

7

Hi @patrick-hubert-adsk,

I’ve some doubts about it.
I’m thinking to build a layer between my application and the Shotgun. To do that, I’m thinking to authenticate with the Shotgun and save the user auth token on an in-memory database (maybe Redis) on my studio authentication service, once done that, all my other services, don’t need to know the Shotgun user password, only the token. Do you think this a security risk? If yes, have you any idea or suggestion for doing something like this? Because my internal services don’t necessarily will be logged in to the Shotgun and, at the same time, these ones need to communicate with the Shotgun as a user.

Thank you!

8

I fail to see why script users are not good for this use case, but I may be missing a point.
We have a system that keeps the script passwords in an encrypted database, and fetches them on demand. This seems way easier than having to scrape an SSO token, etc., and possibly more secure.

From what I have seen in libraries, they tend to call the auth service, and then scrape the response page for the token. But that seems gnarly.

1 Like
9

Hi @gvalderramos ,

My concern here is the agregation of all of those session tokens into one place (as opposed to using Shotgun Desktop, where only your personal tokens are saved). Anyone that can get their hands on these can do a tons of things while making it look like someone else.

Using a script-name/api-key and sudo, while also very sensitive, at least keeps track of how things were done (e.g. a script was doing thing on the user’s behalf). There is at least some traceability.

Taking a situation where the content of the server is exposed:

  • to address a leak of session tokens you need to revoke all of the existing sessions, Which require help from Shotgun Support (if you are hosted)
  • for a leaked script-name/api-key, you just need to re-generate that api-key. The update should be straight-forward if you kept to the rule of one-script-per-service.

It is difficult for me to speculate on your intended setup, as the devil is in the details. But your mention of actions being taken not necessarily while the user is online/active, to me this scream script-name/api-key + sudo_as

Maybe if you were to post a diagram of what you intent to setup, it’d be easier to evaluation options.

-Patrick

10

Hi,

I have created the following reference implementation of a ShotGrid authenticator, which will work for the legacy (Shotgun, LDAP, SSO) and the Autodesk Identity (with or without SSO) authentication.

This reference implementation requires:

  • Python3
  • PySide2 (because a Web-like environment is needed)
  • ShotGrid Python API (it could have used the REST API, it should be trivial to modify it)

To make my life simpler, and because I know that the PySide package distributed with the ShotGrid Desktop works well, my sample uses the Python3 binaries that are installed with the Desktop.

This is a rough sample, implementing what is described here:

Hoping it helps,

-Patrick

sg_authenticator.py (3.6 KB)

3 Likes
11

Thanks for the SSO/Autodesk ID example @patrick-hubert-adsk, I found it very useful! This might just be what I needed to start migrating away from SG accounts + PAT for stuff we need to authenticate users for.

Small note to other rebels wanting to run this in standalone and PySide6 → You need version >=6.2, and your imports for the web engine stuff will look something like this:

from PySide6.QtWebEngineWidgets import QWebEngineView
from PySide6.QtWebEngineCore import QWebEngineProfile
12

Hi Patrick, this is really helpful thanks. But we still have an issue here and I’m hoping you can help.

When we launch our DCCs, we don’t do so from SG Desktop but from our own launching app (we don’t use any of the engines either e.g. tk-maya etc). Then from inside Maya (for example), we then use Toolkit to talk to SG when we need to. We do something like this to get the user:

ShotgunAuthenticator().get_user()

Now, if the user is logged into SG Desktop before they launch Maya, then this code works fine. But if not, then it won’t (since the Autodesk login change). I can’t replace it with code that uses the method you posted above, because in Maya 2018 (for example) it’s the wrong PySide version. And in any case, I don’t really want to launch the login dialog each time a DCC is started.

So what we really need is for our custom launching app to be able to ensure the user is logged in and for that login to persist when DCCs are launched etc.

Like I say, if a user logs in through SG Desktop (or SG Create) then that login persists and they don’t have to log in again. If they launch Maya, then our code can get gold of the session user with ShotgunAuthenticator().get_user().

But using your code above, although it works to log in and access SG, the login doesn’t persist the way it does when logging in through SGD etc.

That’s what we really need to be able to do, so we’re not replying on all users to log in through SGD before they can use our tools.

13

Hi @TomMelson

That sample is just that… a sample.

You can build on it to:

  • Add cookies persistency with setPersistentCookiesPolicy(QWebEngineProfile.ForcePersistentCookies)
  • Potentially use the toolkit to create a persistent local session with create_session_user() Authentication — tk-core v0.20.6 documentation

Given that you have taken your own path to initiate your DCCs, it is difficult for us to provide a solution that works for everyone else and you.

Hoping that this helps,

-Patrick

14

That’s great thanks! Yes I wasn’t expecting the sample to solve all issues but I wanted to explain what we needed. This looks promising, thanks again.

Yes it’s fine to have to get to grips with extra stuff, given we’re slightly off-piste. But it was disappointing that what was working before, suddenly broke when this new authentication was rolled out. I think it’s caught a lot of people by surprise.

15

Thanks @TomMelson for the very detailed explanation.

It always helps understanding the context in which things are done.

Should you desire to go with using a PAT and the traditional login GUI, the changes required in your version of tk-core is trivial (should you be unable to update).

In file python/tank/authentication/console_authentication.py, commenting out the line with if is_autodesk_identity_enabled_on_site(hostname, http_proxy): and the next, would do the trick.

-Patrick

16

Thanks for the extra info Patrick. However, we are fine creating a QApplication before we authenticate and so don’t need console login. But going this route:

import sgtk
from tank_vendor.shotgun_authentication import ShotgunAuthenticator
cdm = sgtk.util.CoreDefaultsManager()
authenticator = ShotgunAuthenticator(cdm)
user = authenticator.get_user()

What we get is the old style login window, which expects the old login name and password (not the Autodesk registered email). These logins actually still work but presumably this is the wrong approach?

What I would expect to be correct is for the new style (web-linked) dialog to appear, which is why I’m looking at the method in this thread. Like I say though, we will need the login to persist beyond the session - just as when you log in to SGD, it means you’re logged in permanently. i.e. if I then run the code above from a completely new process, the user will return correctly and can be used.

To be honest it feels like tank_vendor.shotgun_authentication.ShotgunAuthenticator has not been brought up to date with the current login system. Why does it launch the old style login dialog if you’re not logged in?

Anyway, I’ll educate myself on persistent cookies etc. and see if I can get where we need with that. Thanks.

17

Hi, I wanted to see why the old style login window was coming up for us when using Toolkit’s ShotgunAuthenticator class. So after doing some digging and debugging in the toolkit code I ended up in “tank.authentication.sso_saml2.core.sso_saml2_core.py” - here it’s throwing an exception on line 154:

raise SsoSaml2MissingQtWebKit(“The QtWebKit module is unavailable”)

This exception is caught and it seems to fall-back to using the old style login. It seems like, if the QtWebKit module was available, it would do exactly what we need and run the correct login process.

Our code is running the python installed with Shotgrid Desktop, here:

“C:\Program Files\Shotgun\Python3\python.exe”

And I can see that the PySide2 module installed under the site_packages is indeed missing the QtWebKit module. I uninstalled Shotgrid Desktop and reinstalled, in case I just needed the very latest version but it’s still missing.

I tried running through a custom python install (with PySide2 and QtWebKit installed) and indeed I got the correct authentication dialog and the exact behaviour we need ! :slight_smile: (The authentication works and whats more the login persists just like it does when logging in through SG Desktop).

So I think this exception is the cause of all our woes and can be fixed by ensuring this QtWebKit module is availabl. I’m not sure why it’s not included in the SGD python install.

18

Hi @TomMelson

If you are running the Python from the SG Desktop as you describe, and getting the error message you cite: The QtWebKit module is unavailable, then it is because you are using a very old version of tk-core.

Getting the Web authentication working is not easy : you need to have a correctly built version of PySide/Qt and a compatible version of tk-core. Unfortunately there are so few working combinations.

If I understand correctly, you want to use the python interpreter provided with Maya 2018, is that the case ? If so, I strongly suggest to only use the corresponding mayapy for your tests.

That version is a bit unusual in that it uses PySide2/Qt5 and QtWebKit. Usually, with PySide2, QtWebEngine is used.

What exact version of tk-core are you hoping to use ?

-Patrick

19

Hi Patrick, thanks for another quick response. This has helped and I now have it working :slight_smile:

I thought I had updated ALL but seems I needed to update tk-core separately? I’m actually not using Maya, I’m just running Toolkit, using the python interpreter installed under Shotgrid Desktop. The issue was that this interpreter’s PySide2 is missing QtWebKit. But the latest core allows for that and uses QtWebEngineWidgets instead.

I was still getting the old style login dialog until I added the following lines to my code:

from tank.authentication import set_shotgun_authenticator_support_web_login
set_shotgun_authenticator_support_web_login(True)

I’m not sure why this is necessary, shouldn’t the web login now be the default? Otherwise it’s going to allow people to log in with their old style user names.

Anyway, thanks again!

20

Hi @TomMelson

My apologies, I thought you were aiming to use an old tk-core and and an old interpreter/PySide/Qt

There was a recent change of behaviour for the tk-core when dealing with Autodesk Identity site: before, it was always forcing the use of the web. But that prevented the use of a setup leveraging a Personal Access Token, which was necessary for older DCCs.

As things stand now, it is the responsibility of the client application to let the tk-core know if it has a setup that is appropriate for Web login (we cannot easily auto-detect that).

By default, when connecting to an Autodesk Identity ShotGrid site we will use the old dialog which requires having setup a PAT.

If set_shotgun_authenticator_support_web_login(True) has been set, then we will try to use the web. All the responsibilities of validating the build of Qt/PySide used is left to the developer.

When using the Qt from the SG Desktop, you can safely set the web support toTrue, which is what we do ourselves in the bootstrap code of the Desktop.

Not all builds of Qt/PySide are equals. There are so may options that can impact the Web Login (SSL support, QtWebEngine/QtWebKit support, etc.) that we cannot take for granted that it will work. It needs to be tested first.

Sorry for the complex nature of this.

-Patrick

21

Thanks Patrick, that all makes sense. It was a bit confusing but I’m all good now. I’ve updated all current jobs to using the latest core and my code that runs in our standalone tools (through the SGD python interpreter) now has that line to set_shotgun_authenticator_support_web_login, so we’re getting consistent behaviour, all working.

Cheers!