Hierarchical includes in environment config files

Pipeline Integrations
1

Thanks for that Tannaz, this has worked!

The next issue I have, which might warrant another thread, is how to have hierarchical settings ymls; eg

in tk-nuke:

includes:
- ./studio-tk-nuke-writenode.yml
- "$VFX_PROJECT_PATH/config/shotgun/settings/tk-nuke-writenode.yml"

settings.tk-nuke.shot_step:
  apps:
    tk-nuke-writenode: "@settings.tk-nuke-writenode.shot"

In studio-tk-nuke-writenode.yml :-

studio.settings.tk-nuke-writenode.shot: &studio-shot-writenode
  template_script_work: nuke_shot_work
  write_nodes:
  - file_type: exr
    name: Mono Exr, 16 bit
    promote_write_knobs: []
    proxy_publish_template:
    proxy_render_template:
    publish_template: nuke_shot_render_pub_mono_exr
    render_template: nuke_shot_render_mono_exr
    settings:
      datatype: 16 bit half
      metadata: all metadata
      colorspace: "Input - ADX - ADX10"
    tank_type: Rendered Image
    tile_color: []

and in $VFX_PROJECT_PATH/config/shotgun/settings/tk-nuke-writenode.yml :

includes:
- $VFX_PIPELINE_CONFIG_PATH/env/includes/app_locations.yml
- $VFX_PIPELINE_CONFIG_PATH/env/includes/settings/studio-tk-nuke-writenode.yml

    settings.tk-nuke-writenode.shot: #"@studio.settings.tk-nuke-writenode.shot"
      write_nodes:
      - file_type: exr
        name: Mono Exr, 16 bit
        promote_write_knobs: []
        proxy_publish_template:
        proxy_render_template:
        publish_template: nuke_shot_render_pub_mono_exr
        render_template: nuke_shot_render_mono_exr
        settings:
          datatype: 16 bit half
          metadata: all metadata
          colorspace: "sRGB"
        tank_type: Rendered Image
        tile_color: []
      <<: *studio-shot-writenode

but it’s causing the error:-
Could not open file ‘/redacted/jobs/gol/config/shotgun/settings/tk-nuke-writenode.yml’. Error reported: ‘found undefined alias ‘studio-shot-writenode’
in “/redacted/jobs/gol/config/shotgun/settings/tk-nuke-writenode.yml”, line 62, column 7’

What I’m trying to achieve here is a single config with a default writenode setup, and another settings.yml that’s located in the project folder-tree that can be used to over-ride certain settings on a per-project basis.

I’ve tried the usual SG way of using @ to reference another settings definition (you can see it commented out above, as well as the above technique of using aliases. But neither have worked.

I’m importing the studo writenode settings first (albeit without being referenced from tk-nuke.yml), then from the job writenode yml I’m importing the studio writenode yml and referencing the studio wide config with the alias studio-shot-writenode.

Do you know if it is possible to do this at all? I’m pretty sure I’ve exhausted all the options of syntax, but it’s not working.
It would be really awesome to be able to override main pipeline config settings on a per-project basis in the way I’ve outlined above.

Thanks again for you help
p.

3 Likes
2

Further to this, here’s the error if I use the following approach :

the job tk-nuke-writenode.yml :-

includes:
- $VFX_PIPELINE_CONFIG_PATH/env/includes/app_locations.yml
- $VFX_PIPELINE_CONFIG_PATH/env/includes/settings/studio-tk-nuke-writenode.yml

# asset
settings.tk-nuke-writenode.asset: "@studio.settings.tk-nuke-writenode.asset"

# shot
settings.tk-nuke-writenode.shot: "@studio.settings.tk-nuke-writenode.shot"
  write_nodes:
  - file_type: exr
    name: Mono Exr, 16 bit
    promote_write_knobs: []
    proxy_publish_template:
    proxy_render_template:
    publish_template: nuke_shot_render_pub_mono_exr
    render_template: nuke_shot_render_mono_exr
    settings:
      datatype: 16 bit half
      metadata: all metadata
      colorspace: "sRGB"
    tank_type: Rendered Image
    tile_color: []

error:

Could not open file '/redacted/jobs/redacted/config/shotgun/settings/tk-nuke-writenode.yml'. Error reported: 'while parsing a block mapping
  in "/redacted/jobs/redacted/config/shotgun/settings/tk-nuke-writenode.yml", line 13, column 1
expected , but found ''
  in "/redacted/jobs/redacted/config/shotgun/settings/tk-nuke-writenode.yml", line 24, column 3'

It looks like you can’t inherit an entire app settings dict; only individual attributes. Is that right?

2 Likes
3

Hey Patrick –

I’m not sure I totally understand what you’re trying to achieve, but here’s some ifno that is hopefully helpful?

As I understand it, an include stands in as the value in a dictionary, at any level.

So you can have a include like this, where it’s getting a single setting on an app:

  location: "@apps.tk-multi-workfiles2.location"

(this is from config/env/includes/settings/tk-multi-workfiles2.yml)

Or you can have one that covers all of the settings, say for an engine, like this:

engines:
  tk-houdini: "@settings.tk-houdini.asset_step"

(from config/env/asset_step.yml)

But you can’t have an include and a “baked in” value for the same dictionary key – so this block from your second reply doesn’t make sense, as you’re defining both for settings.tk-nuke-writenode.shot:

# shot
settings.tk-nuke-writenode.shot: "@studio.settings.tk-nuke-writenode.shot"
  write_nodes:
  - file_type: exr
    name: Mono Exr, 16 bit
    promote_write_knobs: []
    proxy_publish_template:
    proxy_render_template:
    publish_template: nuke_shot_render_pub_mono_exr
    render_template: nuke_shot_render_mono_exr
    settings:
      datatype: 16 bit half
      metadata: all metadata
      colorspace: "sRGB"
    tank_type: Rendered Image
    tile_color: []

I think that answers your second question. As for the first, am I correct in my understanding of the problem? You have settings coming into a project from your shared studio config via include, but you want to be able to override those settings individually in a given project?

Our include system doesn’t really support overrides – it’s more designed with the purpose of not having to change things in multiple places, as opposed to “cascading” settings.

The one thing that might work (and I’m not suggesting this; it’s a hack) is take advantage of include file order. I’d have to test this to see if it’s actually reliable, but say you have a file set up like this:

includes:
 - file1.yml
 - file2.yml

setting: @value

There’s no explicit definition of which include file @value is coming from. So if it exists in both, which one wins? Again, we’d have to test this, but if the order is reliable – if it will get it in the first file it finds it, in the order they’re listed – then you could do something like this:

includes:
 - project-config.yml
 - studio-config.yml

And have the same include name (settings.tk-houdini.asset_step or whatever) in both. If it doesn’t exist in the first file, it will fall back to the second. Again, though, lots of unanswered questions with this approach.

Let me know if you still have questions; I went a little off-piste with that one :slight_smile:

2 Likes
4

Hey there,
I’ve been working along the lines you suggested, and It’s become apparent this won’t work without an update to core.
If I point to a project config file, using an environment variable, SG Desktop will fail to load because the context is at site, not project context, so the path SG is looking for the yml doesn’t exist. This causes SG Desktop to fail to load.
It should be possible to allow includes to point to non-existent files, so we can have optional overrides on a per-project basis, and so SG Desktop won’t error on launch. It would make sense to log a warning if a file doesn’t exist, rather than a TankError.

Here’s the log
2020-03-17 18:37:50,310 [7172 DEBUG sgtk.ext.shotgun_desktop.startup] Getting the default site configuration.
2020-03-17 18:37:50,493 [7172 DEBUG sgtk.ext.shotgun_desktop.paths] These non-plugin_id based pipeline configurations were found by Desktop:
2020-03-17 18:37:50,495 [7172 DEBUG sgtk.ext.shotgun_desktop.paths]
2020-03-17 18:37:50,496 [7172 DEBUG sgtk.ext.shotgun_desktop.paths] This pipeline configuration will be used:
2020-03-17 18:37:50,497 [7172 DEBUG sgtk.ext.shotgun_desktop.paths] None
2020-03-17 18:37:50,498 [7172 DEBUG sgtk.core.log] Removing TK_DEBUG from the environment for this session. This ensures that subprocesses spawned from this process will inherit the global debug logging setting from this process.
2020-03-17 18:37:50,499 [7172 DEBUG sgtk.core.log] Disabling debug logging.
2020-03-17 18:38:08,419 [7172 INFO sgtk.ext.no_current_bundle.3d09481fd15e4e5e967ac190a76b8244] STORAGE ROOT PATH: <Path win:‘Z:\jobs’, linux:’/redacted/jobs’, macosx:’/Volumes/redacted/jobs’> : [‘SHOTGUN_PATH_FIELDS’, ‘class’, ‘delattr’, ‘dict’, ‘doc’, ‘eq’, ‘format’, ‘getattribute’, ‘hash’, ‘init’, ‘module’, ‘ne’, ‘new’, ‘nonzero’, ‘reduce’, ‘reduce_ex’, ‘repr’, ‘setattr’, ‘sizeof’, ‘str’, ‘subclasshook’, ‘weakref’, ‘_get_current_os’, ‘_get_linux’, ‘_get_macosx’, ‘_get_windows’, ‘_linux_path’, ‘_macosx_path’, ‘_sanitize_path’, ‘_set_current_os’, ‘_set_linux’, ‘_set_macosx’, ‘_set_windows’, ‘_windows_path’, ‘as_descriptor_uri’, ‘as_shotgun_dict’, ‘as_system_dict’, ‘current_os’, ‘from_current_os_path’, ‘from_shotgun_dict’, ‘from_system_dict’, ‘get_file_name_from_template’, ‘get_shotgun_storage_key’, ‘join’, ‘linux’, ‘macosx’, ‘normalize’, ‘windows’]
2020-03-17 18:38:08,419 [7172 INFO sgtk.ext.no_current_bundle.3d09481fd15e4e5e967ac190a76b8244] self.parent: [‘class’, ‘delattr’, ‘dict’, ‘doc’, ‘format’, ‘getattribute’, ‘hash’, ‘init’, ‘module’, ‘new’, ‘reduce’, ‘reduce_ex’, ‘repr’, ‘setattr’, ‘sizeof’, ‘str’, ‘subclasshook’, ‘weakref’, ‘_bundle_cache_fallback_paths’, ‘_bundle_cache_root_override’, ‘_descriptor’, ‘_get_descriptor’, ‘_get_metadata’, ‘_get_pipeline_config_file_location’, ‘_get_templates_config_location’, ‘_get_yaml_cache_location’, ‘_pc_id’, ‘_pc_name’, ‘_pc_root’, ‘_plugin_id’, ‘_populate_yaml_cache’, ‘_preprocess_descriptor’, ‘_project_id’, ‘_project_name’, ‘_published_file_entity_type’, ‘_storage_roots’, ‘_update_metadata’, ‘_use_shotgun_path_cache’, ‘convert_to_site_config’, ‘execute_core_hook_internal’, ‘execute_core_hook_method_internal’, ‘execute_post_install_bundle_hook’, ‘get_all_os_paths’, ‘get_all_platform_data_roots’, ‘get_app_descriptor’, ‘get_associated_core_version’, ‘get_bundle_cache_fallback_paths’, ‘get_config_location’, ‘get_configuration_descriptor’, ‘get_core_hooks_location’, ‘get_core_python_location’, ‘get_data_roots’, ‘get_engine_descriptor’, ‘get_environment’, ‘get_environment_path’, ‘get_environments’, ‘get_framework_descriptor’, ‘get_hooks_location’, ‘get_install_location’, ‘get_latest_app_descriptor’, ‘get_latest_engine_descriptor’, ‘get_latest_framework_descriptor’, ‘get_local_storage_for_root’, ‘get_local_storage_mapping’, ‘get_local_storage_roots’, ‘get_name’, ‘get_path’, ‘get_plugin_id’, ‘get_primary_data_root’, ‘get_primary_data_root_name’, ‘get_project_disk_name’, ‘get_project_id’, ‘get_published_file_entity_type’, ‘get_schema_config_location’, ‘get_shotgun_id’, ‘get_shotgun_menu_cache_location’, ‘get_shotgun_path_cache_enabled’, ‘get_templates_config’, ‘has_associated_data_roots’, ‘is_auto_path’, ‘is_localized’, ‘is_site_configuration’, ‘is_unmanaged’, ‘turn_on_shotgun_path_cache’]
2020-03-17 18:38:08,421 [7172 INFO sgtk.ext.no_current_bundle.3d09481fd15e4e5e967ac190a76b8244] self.parent.get_config_location : [‘add’, ‘class’, ‘contains’, ‘delattr’, ‘doc’, ‘eq’, ‘format’, ‘ge’, ‘getattribute’, ‘getitem’, ‘getnewargs’, ‘getslice’, ‘gt’, ‘hash’, ‘init’, ‘le’, ‘len’, ‘lt’, ‘mod’, ‘mul’, ‘ne’, ‘new’, ‘reduce’, ‘reduce_ex’, ‘repr’, ‘rmod’, ‘rmul’, ‘setattr’, ‘sizeof’, ‘str’, ‘subclasshook’, ‘_formatter_field_name_split’, ‘_formatter_parser’, ‘capitalize’, ‘center’, ‘count’, ‘decode’, ‘encode’, ‘endswith’, ‘expandtabs’, ‘find’, ‘format’, ‘index’, ‘isalnum’, ‘isalpha’, ‘isdigit’, ‘islower’, ‘isspace’, ‘istitle’, ‘isupper’, ‘join’, ‘ljust’, ‘lower’, ‘lstrip’, ‘partition’, ‘replace’, ‘rfind’, ‘rindex’, ‘rjust’, ‘rpartition’, ‘rsplit’, ‘rstrip’, ‘split’, ‘splitlines’, ‘startswith’, ‘strip’, ‘swapcase’, ‘title’, ‘translate’, ‘upper’, ‘zfill’]
2020-03-17 18:38:08,422 [7172 INFO sgtk.ext.no_current_bundle.3d09481fd15e4e5e967ac190a76b8244] dp.SHOTGUN_PATH_FIELDS : [‘windows_path’, ‘linux_path’, ‘mac_path’]
2020-03-17 18:38:08,423 [7172 INFO sgtk.ext.no_current_bundle.3d09481fd15e4e5e967ac190a76b8244] dp.current_os : /redacted/jobs
2020-03-17 18:38:08,424 [7172 INFO sgtk.ext.no_current_bundle.3d09481fd15e4e5e967ac190a76b8244] unnamed:None
2020-03-17 18:38:08,426 [7172 INFO sgtk.ext.no_current_bundle.3d09481fd15e4e5e967ac190a76b8244] project_path : /redacted/jobs/unnamed
2020-03-17 18:38:10,184 [7172 ERROR sgtk.ext.shotgun_desktop.startup] Fatal error, user will be logged out.
Traceback (most recent call last):
File “/home/patrick-macdonald/.shotgun/desktop/install/app_store/tk-framework-desktopstartup/v1.10.5/python/shotgun_desktop/startup.py”, line 726, in main
settings
File “/home/patrick-macdonald/.shotgun/desktop/install/app_store/tk-framework-desktopstartup/v1.10.5/python/shotgun_desktop/startup.py”, line 338, in __launch_app
engine = __start_engine_in_zero_config(app, app_bootstrap, splash, user)
File “/home/patrick-macdonald/.shotgun/desktop/install/app_store/tk-framework-desktopstartup/v1.10.5/python/shotgun_desktop/startup.py”, line 456, in __start_engine_in_zero_config
return mgr.bootstrap_engine(“tk-desktop”)
File “/home/patrick-macdonald/.shotgun/desktop/install/app_store/tk-framework-desktopstartup/v1.10.5/python/tk-core/python/tank/bootstrap/manager.py”, line 479, in bootstrap_engine
tk = self._bootstrap_sgtk(engine_name, entity)
File “/home/patrick-macdonald/.shotgun/desktop/install/app_store/tk-framework-desktopstartup/v1.10.5/python/tk-core/python/tank/bootstrap/manager.py”, line 1076, in _bootstrap_sgtk
self.progress_callback
File “/home/patrick-macdonald/.shotgun/desktop/install/app_store/tk-framework-desktopstartup/v1.10.5/python/tk-core/python/tank/bootstrap/manager.py”, line 674, in _cache_bundles
report_bundle_progress
File “/home/patrick-macdonald/.shotgun/desktop/install/app_store/tk-framework-desktopstartup/v1.10.5/python/tk-core/python/tank/bootstrap/cached_configuration.py”, line 487, in cache_bundles
env_obj = pipeline_configuration.get_environment(env_name)
File “/home/patrick-macdonald/.shotgun/redactedfx/sitec186.basic.desktop/cfg/install/core/python/tank/pipelineconfig.py”, line 1109, in get_environment
env_obj = EnvClass(env_file, self, context)
File “/home/patrick-macdonald/.shotgun/redactedfx/sitec186.basic.desktop/cfg/install/core/python/tank/platform/environment.py”, line 692, in init
super(InstalledEnvironment, self).init(env_path, context)
File “/home/patrick-macdonald/.shotgun/redactedfx/sitec186.basic.desktop/cfg/install/core/python/tank/platform/environment.py”, line 61, in init
self._refresh()
File “/home/patrick-macdonald/.shotgun/redactedfx/sitec186.basic.desktop/cfg/install/core/python/tank/platform/environment.py”, line 75, in _refresh
self._env_data = environment_includes.process_includes(self._env_path, data, self.__context)
File “/home/patrick-macdonald/.shotgun/redactedfx/sitec186.basic.desktop/cfg/install/core/python/tank/platform/environment_includes.py”, line 180, in process_includes
data, _ = _process_includes_r(file_name, data, context)
File “/home/patrick-macdonald/.shotgun/redactedfx/sitec186.basic.desktop/cfg/install/core/python/tank/platform/environment_includes.py”, line 213, in _process_includes_r
included_data, included_fw_lookup = _process_includes_r(include_file, included_data, context)
File “/home/patrick-macdonald/.shotgun/redactedfx/sitec186.basic.desktop/cfg/install/core/python/tank/platform/environment_includes.py”, line 203, in _process_includes_r
include_files = _resolve_includes(file_name, data, context)
File “/home/patrick-macdonald/.shotgun/redactedfx/sitec186.basic.desktop/cfg/install/core/python/tank/platform/environment_includes.py”, line 112, in _resolve_includes
path = resolve_include(file_name, include)
File “/home/patrick-macdonald/.shotgun/redactedfx/sitec186.basic.desktop/cfg/install/core/python/tank/util/includes.py”, line 90, in resolve_include
file_name, include, path
TankError: Include resolve error in ‘/redacted/studio/_repos/tk-config-default2/env/./includes/settings/tk-nuke.yml’: ‘$VFX_PROJECT_PATH/config/shotgun/settings/tk-nuke-writenode.yml’ resolved to ‘/redacted/jobs/unnamed/config/shotgun/settings/tk-nuke-writenode.yml’ which does not exist!