Am I alone? Single pipeline config with per-project overrides?

I’ve been moving towards using a single config for all projects in a studio, with the obvious benefit that there’s far less code to maintain.

One of the hurdles to this is that there are ARE certain configurations that currently cannot be abstracted from the shared config on a per-project basis without forking the pipeline-config. The tk-writenode profiles being an obvious one.

I’ve almost implimented a solution that allows us to have optional ymls residing in a config folder in each projects folder where setting overrides can be defined. We’ve managed to do this by adding an additional include in the nuke.yml definition that points to a path with an environment variable for the project folder eg :

includes:
- ./tk-nuke-writenode.yml # Pipeline-config based writenode settings.
- "$PROJECT_PATH/config/shotgun/settings/tk-nuke-writenode.yml" # Project folder based writenode settings. 

This would work if it weren’t for the fact that tk-core implements a validation of include paths, and throws a tankError if the path doesn’t exist. This behaviour prevents SG Desktop from loading as there is no project context when launching SG Desktop, so no project_path.

Am I alone in seeing this approach as being beneficial for how to configure a studios pipeline? Am I storing up problems for myself by complicating where configurations are located?
As I see it, any disadvantage of this approach is easily outweighed by the benefits of having a single config to maintain across an entire studio.

I’m curious to hear your thoughts!
Thanks in advance for your contributions.
p.

3 Likes

Your not alone. A global config with project overwrites is ideal. In the past I have lobbied for this. It’s a challenging thing to implement at a studio that has legacy configs and tools. Please keep sharing your progress on this topic. Would love to see SG/TK migrate to this. All roadblocks you mentioned should be prioritized in the roadmap ASAP in my opinion. As a result this would empower studios with more flexibility to manage the technical debt that builds up with per project configs.

2 Likes

Thanks for your input. Eloquently put!

2 Likes

I know the scenario is a bit different here, but you could use the template core hook, a bit like I suggested here:

You could then provide different setting depending on the project, though it’s maybe not quite a elegant, and is a bit more error prone. Basically you could pass a list of settings through, and maybe add a project list to each, and then in the hook filter out those settings that don’t apply to current context project.

2 Likes

Hi Phil,
Oh my word… interesting suggestion!, but that really calls on me to up my yml-fu! :laughing:

“maybe add a project list to each” - It sounds like this wouldn’t really address the issue of increasing technical debt… quite the opposite in this case?

Thanks for sharing this though, I never knew you could gather settings from a hook… although I’m struggling to understand quite how your example works. Is this documented anywhere else?

I think what I, and I assume others, are looking for, is an elegant solution. Being able to have optional yml files living in the project folder structure would appear to be a good candidate for this.

“Optional” being a key requirement here… we don’t want to have to have redundant files in the schema to get this to work. Plus, if it’s not optional, then backwards compatibility would fail.

3 Likes

I’ve updated a fork of tk-core that appears to work; supporting non-existant includes without any adverse effects. It logs missing includes as log.warning() so it’s clear to any TD that there may be a problem with the config.
I’ll add a pull-request to the official repo so others can test this out and get some feedback.
I’ll also post here with details on how to include your project path in include paths.

3 Likes

Ok, I’ve created a pull-request with a simple, and largely untested implementation of this.
There are instructions for how to get this working in our config in the pull-request description.

Let me know if you have any trouble getting this to work, or if you discover any bugs.

Thanks!

2 Likes