Dynamic Pipeline Configurations

Hi!

I am currently developing our shotgun site configuration based on an episodic project, where we have episodes, sequences and shots.
But for other smaller projects, this configuration will not work since it would be an overload and we might not need episodes or even sequences.

One solution would be to have different pipeline configurations for those solutions,
but that would require probably a lot of redundant work, right?

I wonder if it is possible to combine, for example,
a structure with and without an episode dynamically within one single pipeline configuration.

I found a very interesting topic here:

But how could I, for example, create a folder schema like this (when episodic):
[episode]/[sequence]/[shot]
and when not episodic like this:
[sequence]/[shot]

It looks like the “pick_environment.py” hook could be a entry point into this,
but from then on I struggle to find a way to dynamically set the config data I want to,
especially in the core of the config with the schema and templates.

It would be nice if someone could point me in the right direction here!

Thanks a lot,

cheers
Sebastian

4 Likes

Hi @sebastian.brandhuber!

Not necessarily, you could as you mentioned have a separate PipelineConfiguration entity for each project. But each PipelineConfiguration entity that wants to share the config, could point at the same config bundle. If for example, you were using a path descriptor they could all point to the same config on disk, but the same applies to any of the other descriptors, such as git or shotgun upload, they could point to the same source.

This I think would be more work and becomes messy and confusing, but is doable.
You would have your schema set up so that the Sequence folder in the {Episode}/{Sequence}/{Shot} hierarchy only got created when an episode was provided using the filters.

- { "path": "episode", "relation": "is_not", "values": [ null ] }

And the reverse for the Sequence folder in the {Sequence}/{Shot} structure.

- { "path": "episode", "relation": "is", "values": [ null ] }

That would then mean you would need a set of templates for episodic and a set for non episodic. Note it is possible it define optional fields in a template using the square brackets syntax ([Epsiode] ) but this is only intended to make keys optional in the file name and not in the path, you would likely hit errors if you tried to make the episode field optional, so its best to have two sets of templates, episodic and non episodic.

That would then mean you would need to change your environment depending on whether it was episodic or not. And at that point, you would as you mentioned need to change the pick_environment.py hook to if the sequence had an episode, and then choose the correct environment accordingly.
This means that your environment files need branching depending on whether or not you are working with episodes. Option 2 in the post you linked could help make that easier, but in my opinion, I think what your trying to do with the schema means you are better off having separate configs.
I would be almost like cramming two configs inside one.

I hope that helps?

4 Likes

Hi Philip,
thanks for the explanation.

I have two following questions:

  1. As you described, it would require me to branch the environment files based on working with episodes or not. In the linked post, you suggest using env variables and this part is clear, but how could I, for example, have a templatesA.yml and templatesB.yml and chose between those?
    I could not find a possibility to “change” a path or link for that with a custom variable.
    Or thinking bigger - maybe change folders like “schmema” to “schemaA” and “schemaB”.

  2. You suggest that having separate configs would be the better approach.
    Even if they are two completely different config bundles, would I be able to “share”
    files between them? like hooks or .yml files?

thanks a lot!
cheers

Sebastian

2 Likes

You can only have a templates.yml. Off the top of my head, I think you can maybe include other yamls from within it, I’m not certain, but what I was saying was you would need episodic and non episodic templates within the templates.yml.

This isn’t possible, at this point the answer is really to have a separate config.

Depends on how your configs are setup. Are they centralized or using path descriptors, or are they uploaded or pulled from git?
If they are always in a hardcoded location, then you could, in theory, include files from locations outside of the config, but if they are distributed (downloaded on demand) then you can’t be sure that the other config will exist locally.

2 Likes

Okay, that helps me a lot.
We are using a distributed setup where we develop the config with git, but then
upload the config for distribution.

But I have the feeling that we really should aim for separate configs here.
Thanks,

Sebastian

3 Likes

Sebastian,

We’ve got the same setup, and we’re also looking to use that episodic / non-episodic folder structure. I can see that having two configs greatly reduces the amount of templates needed, but it seems that it’ll require extra work to push updates two two separate configs every time any config updates are made. Do you (or anyone reading this) have a painless way to keep two slightly different configs in sync with future updates? We’re also using distributed config with git.

2 Likes