Customizing Publisher with templates

[nonlosao]

Hi all,

I’m kinda new to SG, and I’m still trying to figure out lots of things. So apologies in advance for any blasphemies I might be writing in this post!

I’m trying to customize my pipeline so that published files are handled unrelated from tasks.

I’ve created a PublishedItem custom entity to represents the items that I want to publish and that I want to keep track of.

I’ve added a “Published Items” field to the Task entity, and a “Tasks” field to the PublishedItem entity.

Basically the logic that I want to follow is:

work files - represent the evolution of Tasks [and are saved through tk-multi-workfiles2 ]

published files - represent the evolution of PublishedItems [and are saved through tk-multi-publish2 ]

And eventually I want to be able to keep track of dependencies between PublishedItems.

Now, I’ve edited the folder schema, the templates.yml and the various environment yml files to reflect these changes, but I still get errors whenever I try to publish something.

For the sake of this example, let’s say I’m using Maya and I’m working on an asset task (so I’m in the asset_step ).

I managed to correctly get the tk-multi-workfiles2 to save the task files in the right location.

Then, when I load the publisher, it automatically understands which context I’m in and passes it to the publish functions to resolve the template.

However, I’m having two issues at this point:

1. I want my published files to be organized by PublishedItem (and then versioned). This is defined in the templates.yml , but the publisher doesn’t know where to take this information from, as the ui does not let me choose any additional information. Should I change the code for the UI or is there some other way to pass on this information (i.e. through fields connected to the task)?
2. When I click on the Publish button, it still saves a file in my maya_asset_work path (plus, it overwrites the latest version and creates a new one…)

Maybe I’m misunderstanding what the publisher is supposed to do, but I really wasn’t expecting it to save any files using the maya_asset_work path.

How should I go about this? Do I have to change UI, hooks and core modules of tk-multi-publish2 ?

Thank you for your help,

Enrico

[Patrick]

I’m not sure I can totally unpack your issues here, but the main thing to understand is that when you publish, the publishedFile inherits the version number of the workfile, and then the workfile is versioned-up (eg save-as). This is to ensure that you set the version in stone and start working on the next version whenever you publish.

(plus, it overwrites the latest version and creates a new one…)

Can you show examples of this? Also can you share the work and publish templates for this task?
It sounds like you might not be using version numbers for workfile templates?

Are you attempting to make a system whereby your publishes use incremental version numbers independent of the workfile versions? (or a single workfile without version?) If so, then that’s just not how SG has been defined. It probably is possible to do this, but I have to ask whether it’s worth the effort. What’s the benefit?

Please share you templates.yml if you can.

[nonlosao]

Hi Patrick,

thanks for your reply.

Yes I do want to have publishes which are versioned separately from working files. For me they’re two different things. Working files represent the evolution of a task. Published files are indeed connected with a task (in which they are generated by it), but in fact they represent the evolution of an asset (asset in a generic sense, not a shotgun asset). To represent this, I created the PublishedItem entity.
The reason behind this choice, is that often the same asset is worked on in several tasks. Therefore if the version number were inherited from the working file, it would be inconsistent. And it would be very hard to find the latest version of a PublishedItem if I had to search through all the tasks folders in which that item could be. That’s why I want to keep track of these PublishedItems separately. So basically when I publish, I want the publisher to save my scene as a working file, and also to export a number of items from the scene as PublishedItems - each one following its own versioning.

That’s the way I’d like to set up our pipeline. If you think this is conceptually wrong, I’d be happy to hear your thoughts though!

What do you think the best approach would be to achieve this?

Here is the relevant sections of my templates.yml
{CustomEntity05} is my PublishedItem

asset_root: prod/assets/{sg_asset_type}/{Asset}

# define the location of a work area
asset_work_area:
    definition: '@asset_root/work'
# define the location of a publish area
asset_publish_area:
    definition: '@asset_root/publish'

# The location of WIP files
maya_asset_work:
    definition: '@asset_work_area/{Step}/{task_name}/{Asset}_{Step}_{task_name}.v{version}.{maya_extension}'
# The location of a project
maya_asset_project:
    definition: '@asset_work_area/{Step}/{task_name}'
# The location of backups of WIP files
maya_asset_snapshot:
    definition: '@asset_work_area/{Step}/{task_name}/snapshots/{Asset}_{Step}_{task_name}.v{version}.{timestamp}.{maya_extension}'
# The location of published maya files
maya_asset_publish:
    definition: '@asset_publish_area/{Step}/{CustomEntity05}/{Asset}_{Step}_{CustomEntity05}.v{version}.{maya_extension}'

And the relevant section in tk-multi-publish2.yml

# asset step
settings.tk-multi-publish2.maya.asset_step:
  - name: Publish to Shotgun
    hook: "{self}/publish_file.py:{engine}/tk-multi-publish2/basic/publish_session.py"
    settings:
        Publish Template: maya_asset_publish
  - name: Publish to Shotgun
    hook: "{self}/publish_file.py:{engine}/tk-multi-publish2/basic/publish_session_geometry.py"
    settings:
        Publish Template: maya_asset_publish
help_url: *help_url
location: "@apps.tk-multi-publish2.location"

(although I’m not really publishing session geometry at the moment).

And regarding this…
(plus, it overwrites the latest version and creates a new one…)

Well, say I have a v001 already saved. When I published a new file, it replaced v001 with the current scene, and saved it as v002 as well… When I published again, it replaced v002 with the new scene, and saved it also as v003…
Coult it be because of my settings in tk-multi-publish2.yml, cause I haven’t touched any of the hooks yet.

Thanks for your help!

[Patrick]

I’m confused by the publish item entity. How does it differ from your asset entity? What fields does it have? Also, you talk about about publish items without once mentioning “publishedFiles”. Are you not duplicating functionality here?

As for work file versioning. You’re thinking about it the wrong way. If you publish from a v2 workfile, sg will save the file, ensuring it is up to date and it then copies this file and renames it based on your publish template, before finally save-as version up. This logic works for me and I’m not understanding the utility of your desired workflow.

Ok, and I see your intention of grouping all task publishes per step. I imagine you can do this with a custom collector or publish_file publisher2 hook. The default behavior is to take the workfile’s version, but it should be fairly easy to replace that with a function that finds the next available publishedFile’s version number. I’d highly recommend ensuring the parent workfile path is stored in the publishedFile entity for debugging purposes.
Im not a big fan of independent versioning like this because of the potential for confusion when talking about versions of things.

If the only reason is to avoid searching through task folders I would ask why you don’t use the web gui which you could easily configure to show you all publishes in chronological order of an asset irrespective of task. If you’re happier living in bash, then I can sympathize with you that it’s still not an ideal solution, but it’s better than having to browse aimlessly as it sounds like you are having to do currently.

I’m interested to here others views on this as I’m pretty sure others will have taken your approach, or something similar, and got it working well.

Finally. Is this a film/tv or games studio?

[nonlosao]

I chose to create the PublishedItem entity because I didn’t know for sure how the PublishedFile was being used by SG. So at least for testing I preferred using my own entity, not to mess things up too much.

We’re mainly doing rigging, mocap, and integration into RT engines. Every client has different requirements, so needless to say that every project turns out to be radically different than the previous one, and our pipeline needs to be able to adapt quickly. Especially as we are often working on several projects at the same time.

My approach is inspired by a combination of the ideas shared on Pipeline Patterns and the book Production Pipeline Fundamentals. I basically want Shotgun to be my asset manager, and to have each of these assets saved in a unique, unambiguous location with consistent versioning. Note that here by asset I mean any file or combination of files that represent data needed by other artists or scripts - not a Shotgun Asset entity.

So what I want to do in the Publisher is: after the maya scene has been analyzed and the items to publish added to the tree, I want the user to specify what PublishedItem entity each item refers to - thus giving Shotgun enough information to correctly publish each item in its folder - as specified in the templates.yml

# The location of published maya files
maya_asset_publish:
    definition: '@asset_publish_area/{Step}/{CustomEntity05}/{Asset}_{Step}_{CustomEntity05}.v{version}.{maya_extension}'

Where CustomEntity05 is my PublishedItem entity.

I hope this helps clarify my intentions.

How should I approach this? Would working on Hooks be enough?

[Patrick]

Right, that’s really helpful to know the context of your work.
It sounds like we need to find a shared language to use that removes any ambiguity of terms.

eg,

where you say “asset”, in SG parlance, this would be a “publishedFile”. A Published file can be any output from a DCC(eg a single file), or it could be a textfile/yaml that defines a list of other publishedFiles(eg a combination of files).

Can you further illustrate the relationship between your maya scene items and the PublishedItem you want to group them by? and how would this be any different to a SG Asset entity? Can Assets entities be assigned to multiple CustomEntity05s? or is it a many-1 relationship.

Can you share a screengrab of your CustomEntity05 page on the SG website showing how you’ve connected the Customentity05 entity to your Asset entities?

Can you also share the CustomEntity05 “Key” from your templates.yml.

Can you share the CustomEntity05.yml file in your schema.

" I want the user to specify what PublishedItem". Is this really a user’s responsibility? wouldn’t you be defining this relationship via the SG website? you would be best to utilise the power of the database to define this relationship rather than having it defined arbitrarily by artists.

Can anyone else chip in here? I’m not familiar with this workflow so don’t want to be leading Enrico away from his desired approach uneccessarily!

[Patrick]

I should add, if you’re going to be using SG, then I would stick to SG vocabulary and definitions… mixing it up with the vocabulary from your external references will lead to a lot of confusion when working with other people or getting support here. I would nip that one in the bud!
If you’re frustrated by this, you’re not alone! One bugbear of mine is the use of the term “Version” in SG to indicate a “reviewable media”. It often causes confusion when people mix it up with actual file version numbers.

[nonlosao]

I totally agree! I’m sorry for the confusion. I’m still getting used to Shotgun’s terminology.
However I have looked more into the PublishedFile entity, and I can confirm now that is not what I was talking about in my previous posts. I’ll explain the difference.

You can see here how SG creates a new PublishedFile for each version # of a publish. My PublishItem however needs to be unique. So each version # of my PublishedItem would be a PublishedFile itself.

@Patrick

Can you further illustrate the relationship between your maya scene items and the PublishedItem you want to group them by?

Let’s say I have defined Tasks A and B, and PublishedItems 1, 2 and 3. Now let’s say that on task A artists will work on PublishedItem 1 and 2, whereas on task B they will work on 2 and 3.
I will specify this relationship in the tasks field of my PublishedItem entity - which is also be reflected in the PublishedItems <-> Tasks field on the Task entity.
So whenever I run the publisher, for each item in the tree I want to be able to chose a PublishedItem, in order for that item to be published in the right location. The publishing logic would have different implementations based on the pipeline step and/or the environment in which the artist is working in.

Here a screengrab of my tasks

and of my PublishedItems.

This is the CustomEntity05.yml file in my schema (but it’s still a WIP)

# the type of dynamic content
type: "shotgun_entity"

# the shotgun field to use for the folder name
name: "code"

# the shotgun entity type to connect to
entity_type: "CustomEntity05"

filters:
    - { "path": "project", "relation": "is", "values": [ "$project" ] }
    - { "path": "sg_linkedentity", "relation": "is", "values": [ "$asset" ]}

and finally the CustomEntity05 key in my templates.yml file

keys:
    CustomEntity05:
        type: str

paths:
    maya_asset_publish:
        definition: '@asset_publish_area/{Step}/{CustomEntity05}/{Asset}_{Step}_{CustomEntity05}.v{version}.{maya_extension}'

where I defined it as a simple string.
However, creating folders from the Tank doesn’t create folders for the CustomEntity05 /PublishedItem s. It stops at @asset_publish_area/{Step}

@Patrick

Is this really a user’s responsibility? wouldn’t you be defining this relationship via the SG website? you would be best to utilise the power of the database to define this relationship rather than having it defined arbitrarily by artists.

I want the artists to choose the PublishedItems from the predefined list of items connected to the task. So this, plus custom validation steps, should ensure some sort of stability, and hopefully don’t drive the user crazy!

This is the general idea. I hope this is enough to understand how I’m trying to set things up.
Do you know if there’s any way to expose more fields in the publisher UI without having to tinker too much with the UI code? Or would you suggest a different approach?

[Patrick]

Thanks for the detail.
I’m really not seeing what published items are doing that asset entities wouldn’t also achieve?

I’m regards to your key, it should be a sg_entity, not str. That way sg can populate the field from the relationship to the asset… Well that might have been the case in certain situations, but perhaps less so I’m your case.

Why not have head, hair, body assets? All linked to a parent asset?

[Patrick]

@nonlosao

I want the artists to choose the PublishedItems from the predefined list of items connected to the task. So this, plus custom validation steps, should ensure some sort of stability, and hopefully don’t drive the user crazy!

Just to add a thought about this. Another way to achieve something similar here, in terms of identifying what “item” from a “task” an artist is publishing… the more SG way of thinking about this would be to have different publish “plugins” for each of your expected “items”. These would appear as seperate Publisher2 “items” in the gui that the user could select from. Each of these items would/could have their own template for where the publish goes, and, crucially, a specific “PublishedFileType”. The plugin might

Someone lend me a hand here… on the one hand I feel like I’m trying to restrict Enrico’s creativity … but on the other I can’t help thinking this overcomplicating something that can be achineved within the more standard models I’ve seen of shotgun setups. I’m always wary of deviating too far from the accepted norms, as one can encounter dead ends in terms of the way that SG templating or the schema can work.

[tannaz]

Hi guys!

@Patrick, thanks for going deep on this stuff with @nonlosao – your tips are really good.

@nonlosao, a couple more thoughts on the paradigms that Toolkit (and Shotgun) set up that might help you sort this out:

• First off, I’m with Patrick: it does seem like the components of the characters that you’re wanting to link publishes to (ie, the “PublishItems” in your setup) are in fact just sub-assets of your character assets.
• As we talked about on our call last week, the primary function that publishing holds in a Toolkit pipeline is to make the work that an artist has been doing available to their colleagues to build upon. So it does stand to reason that if you have a pipeline like model -> rig -> uv creation (or whatever) on each component (body, head, etc.), that each of these would be a task on the sub-asset. It looks like this is what you have with your PublishedItem entity: each one has its own set of tasks.
• Regarding work files and published files: PublishedFile entities represent an individual file (or a per-frame file sequence) that’s linked to a task or other entity that you want to track in Shotgun. Originally, an artist’s work file was considered the “primary” published file on a publish, and you could have secondary publishedFiles as well (eg, alembic models of a character). When we revamped the Publisher a couple years back, this evolved into the Publish items that the user can check/uncheck individually to decide what actually gets published. This could be different representations of the same asset, or, perhaps in your case, these subassets. If you went that route, your workfiles and published files would still be versioned in the same sequence, but for some PublishedFiles, there would be skips in the sequence of version numbers.
• It sounds like the real sticking point here is that your artist work on several tasks from the same workfile. This is admittedly a workflow that Toolkit doesn’t handle well by default. I’m going to run that question by the Toolkit team and see if they have any advice to offer.
• I’ll also run this one by the Street team and see what their advice is on how to best set up your Shotgun entities for subassets. Will report back!

[tannaz]

Hi guys – a few more thoughts to follow up here:

• First off, it sounds like you have better options than I first though on the Toolkit side for publishing to multiple tasks from a single workfile. The PublishItem does have a context attribute (documented here), so at collection time, you could find all the individual publish items, and associate each with the correct sub-asset task.
• Secondly, on the Shotgun side, the Asset entity already has “Parent Assets” and “Sub Assets” fields, so you could just use these to organize your Characters and their components.
  • One note: if you’re doing close tracking of a lot of assets/subassets, you may choose to keep them separate by having a custom entity for the parent characters, so that they don’t clutter each other up with extraneous data.

Hope that helps! Let us know if you have further questions.

[nonlosao]

Hi @tannaz and @Patrick,

Thank you so much for your input. It is very appreciated!
Sorry for disappearing for so long. A new project landed that kept me super busy for the last few weeks.

I am giving this pipeline a lot of thought, and want to make sure I have everything well planned before putting my hands on the code.
However I had to slow down a bit on this idea because unfortunately I have only so much time I can spend on it.

For the time being, I am testing the approach that you have suggested and doing some more in-depth research on the topic.

I will be back as soon as possible with updates