Distributed config management workflow

I’ve seen the question come up a few times around how to test configuration changes and push them to production when using a distributed config setup.

My personal recommendation would be to do the following:

  1. Assuming your project already has a “primary” PipelineConfiguration entity with an uploaded config, I would create a second one called dev.
    Set a dev descriptor to define a location to your config on disk. (If you haven’t got a config on disk you can download the primary config and unzip it somewhere convenient where you can work on it and make changes.)
    Make sure you restrict the PipelineConfiguration entity to your self:


    Here is what the config looks like on disk.
    uploaded_config_dev
    Note that I have mine tracked in git (you can see the hidden .git folder in the image) so I can pull down changes and switch branches within this location on disk when testing and developing. I prefer using a dev descriptor rather than using the git or git branch descriptors as it doesn’t require me to change the descriptor each time I want to switch branch or if I make a new commit.

  2. Once you’ve done that you can start making changes to the dev config stored on disk and test them on your project. You can test the config by switching to it in Shotgun Desktop:
    Change_configuration
    Using a dev descriptor also gives the added benefit of being able to access the Check for config updates… option.

  3. Once you’re happy with your changes to the config, it’s a case of zipping it up and reuploading the config to the “primary” configuration’s Uploaded Config field. Then when your user’s open the project in Shotgun Desktop, it will download the latest version of your config automatically for them.

  4. [optional] Instead of zipping it up and uploading it straight to your primary config, I think it’s a good idea to track the config in a version control software such as git, so that you can see what changed and revert back more easily if something goes wrong.

    • I would personally recommend using something like Github and push your dev configuration changes to the remote configuration on Git.
    • Create a release of your config in Github, as we do with our tk-config-default2.
    • Download the zipped release and upload it to your Uploaded Config field on your primary configuration.

    It’s more steps doing this, but now you have the added knowledge of knowing which version of the config you’re using in production and you can more easily roll back to a previous version or check out what’s changed.

Other people may have other suggestions please feel free to share your thoughts.

4 Likes

3 posts were split to a new topic: Multi platform Git descriptor paths

What folder do we zip up though?
the config folder?
– core
– env
– hooks

Or the more root folder?

– config
– install
– cache

2 Likes

Hi Ricardo

Welcome to the forums, and thanks for the question!

In my example, I was assuming you already had an uploaded config, in which case you would be downloading that and then zipping it back up again once you had made your changes.

If you were starting with an already existing centralized config then you would have something that looks like this:
paint_drying
In which you would zip up the highlighted “config” folder, but this is a bit different there would be a few extra steps to take to convert a centralized config to a distributed one.

I’ll update the post above with a few more screenshots to make it clearer, about what you should expect at each stage. It might be good to have a different topic about going from a centralized config to a distributed one.

Cheers
Phil

1 Like

In this type of workflow, is it still possible to use the “tank push_configuration” command? I used the “New config sandbox…” option inside the SG Desktop app. This creates a local copy of the Primary config on disk, but does not localize the core. When I run the tank command, I can only access it by going into the Terminal and doing the following:

cd ~/Library/Caches/Shotgun/_SITE_/pXXXcYYY.basic.shotgun/cfg
./tank

If I run tank with the push_configuration option, I get a Python TypeError: expected str, bytes, or os.PathLike object, not NoneType

Do I need to run ./tank localize first? Change my PYTHONPATH environment variable? Both?

I should mention - zipping up the dev config sandbox and replacing the uploaded primary config works as expected. Just wondering if we should still be using the tank command.

I think only a few tank commands still work in a distributed workflow.
I would recommend setting up your configs as git repos so you can just push/merge/pull that way.

I’ve done this in the past, but now with most users working from home, this can get real tricky. You either have to require the user to be connected with VPN in order to access a private GIT repository, or you have to give them credentials to a public repository. Far easier to manually zip up a config directory and upload it to the PipelineConfiguration in the web UI.

Thanks for the clarification about tank commands, I will proceed accordingly.

You can still do that with my suggestion.

You would do the push/merge/pul etc in a dev pipeline config pointing to a local git folder.

Once you have a new stable release you localize all the apps/frameworks with the tk-core/developer/bake_config.py script and have it zip up a new release for you.

Even better, build a webhook so you can have this all be done automagically from a github release or a push to a specific branch.