Recommended way to share/update pipeline configurations between projects?

Hi again!

Our studio does a lot of small projects that currently all get a pipeline set up individually (copied from the latest developped project at that moment).

Is there a better way to maybe share most of the config but still allow for changes (like nuke writenode customisation)?

Maybe in combination with github?

I’m doing a lot of work on hooks at the moment and would really appreciate it if I didn’t have to manually keep track of what hooks on what project are the latest.

We don’t need to create different workflows for any project, I would like a master project and all projects to derrive their functionality from there (but keep things like colorspaces etc configurable)

3 Likes

Hey Ricardo,
I can explain what I do to solve this exact issue. I work at a small studio and we also get a lot of small projects, sometimes we have projects where only 1 or 2 people work on it, but I like to still have them access our full tool pipeline so what I have done is the following.

All our projects are not localized, they are instead sharing a core, the core stays on a consistent version for 2-3months on average then all of those projects also share a configuration that is attached to a central repository. I have 2 of these configurations running in parallel currently, one for the old style shotgun config, and one for shotgun config-default2, this is so I can have a Primary and secondary config setup on each project encase certain futures are not working as intended under config-default2 and more work needs to be done, it’s sort of a WIP config that is stable enough to test, but isn’t guaranteed to be fully functional.

We do, in fact, use Github, but there are certainly other options

Then within any given project I still have the option to create a branch on either config repository for any small deviations a particular show requires. Those deviations then can be merged back into the master branch or can live as a deviation branch for the life of the project.

If I need to take over any of shotgun’s apps/engines/frameworks I simply fork shotguns repo, then create a custom repository to hold my deviation of that shotgun fork. From there I can hook my repo up to my config and easily distribute it to all shows.

I have a pipeline project where I do all my sandbox testing, it is linked to both the stable active config, and a parallel copy “sandbox” config that has branch deviations of it’s own waiting to either be merged, or worked on further.

This is the basic outline of how I maintain multiple projects while keeping a streamlined and simple to understand config setup.

Let me know if you want more info on any of the outline steps.
-Ross

4 Likes

Thanks for the very detailed explanation @Ross_Macaluso! :slight_smile:

What software do you use to track branches on github since I have found it to be very confusing to manage a branch in Github Desktop.
It doesn’t seem to be able to save a branch somewhere else? (or at least, not in a very elegant way).

and I assume you link these manually to the folder of the branch on disk?

2 Likes

I use powershell for initial remote setup, and initial new repo initalization…

But I use ATOM code editor, there are numerous packages for github specifically, ATOM is a totally modular code editor that allows the install of user created packages. However, Github has also authored a few core Github specific packages that handle commits, branch management, merge conflict resolution and numerous other features that really help visualize your branch/master relationship.
I would take some time and look at ATOM if you aren’t familiar with it already. I have friends that swear by Sublime, but with the integrated Github package support… I really have fallen in love with ATOM haha…

A branch will never have a physical local folder location, it will instead change all the altered files to the current file state held on that branch. Think of a branch just storing the differences, and applying those differences when that branch is checked out (made the active state) That way you can have one projects app repo on a branch, and another project still on the master, both will have the same unedited files, but any files that have deviated will ONLY be on the project where you have checked out that new branch.

I would take some time and look through Github training materials, they have quite an extensive database of how repositories work locally, and virtually. If you do go with ATOM, there is also extensive documents about the ATOM Github packages.

Hope that makes sense,
-Ross

3 Likes

Thank you!

You have given me some ideas! :wink:

2 Likes

I thought I’d chime in with a suggestion as well.

Option 1

Another approach would be to have a site config (a PipelineConfiguration entity with no project set), using a distributed setup. This will mean that all projects that don’t have their own config will use the site config (you will still need to set a tank_name value on each Project entity).Then to take over for a specific project, you can just create a PipelineConfiguration for that project. That will be a full config take over though.

Option 2

If you just want to handle taking over specific hooks or environment yaml files, then you could maybe make use of environment variables. Here is an example:

Here is a config where I just mocked up a test: garden_show.zip (286.4 KB)

In this config, if you were in a project called garden show then it would not show the software launchers in Shotgun Desktop, but for all other projects it would.

In the config, I used environment variables to define the includes path.

I also added code to the pick_environment.py core hook, that contained the following code to set the environment variable based upon the project the configuration was linked to. If a project-specific settings folder was found (named after the project) , then it would define the PROJECT_SETTINGS env var to that. If one isn’t found it would define the env var to settings (which is the default folder.)

        if context.project is None:
            os.environ["PROJECT_SETTINGS"] = "settings"
            return

        # set an environment variable that will later be used to resolve the include path to the project
        # specific settings inside the generic config.
        settings_folder_name = "{0}_settings".format(context.project["name"])
        # remove any spaces from the name
        settings_folder_name = settings_folder_name.replace(" ", "_").lower()

        # Check project settings exist
        config_path = self.sgtk.configuration_descriptor.get_path()
        settings_folder_path = os.path.join(config_path, "env", "includes", settings_folder_name)
        self.logger.debug("checking for project specific settings here: %s" % settings_folder_path)
        if not os.path.exists(settings_folder_path):
            # no project specific settings found use the default location
            settings_folder_name = "settings"

        self.logger.debug("Using settings folder name: %s" % settings_folder_name)

        os.environ["PROJECT_SETTINGS"] = settings_folder_name

I also took over the bootstrap.py core hook and added os.environ["PROJECT_SETTINGS"] = 'settings' so that if the config is used as a site config then it will take the settings block.

This is just an example though, there are probably a number of different ways to set up something similar.

3 Likes

omg, just realized this is what I asked for in the other topic!

:see_no_evil: :see_no_evil: :see_no_evil: :see_no_evil:

2 Likes