Returning fields other than just {'type','id','name'} from links?

Flow Production Tracking
1

Hi, I bet this question has been asked for years, but I’m gonna ask it again…

When the API returns the results for a single or multi-entity field, like say the “Link” i.e. entity or “Assigned To” i.e. task_assignees fields from a Task, the result always takes the form of a dictionary with three keys {'type', 'id', 'name'}. The 'name' is the normalized display_name_cache of the type/id’d entity. So the question is, is there any way to alter that set of default keys? For example, we virtually always want the 'login' for each of the folks in a Task’s task_assignees, but you can’t flow-thru / dot-syntax on multi-entity fields, so we have to run another query.

Secret black magic ruby hacks are welcome… :wink:

5 Likes
3

Mmm… I’ve wondered about this too… seems like there is no option but to run another query…

Here’s what the docs have to say…

http://developer.shotgunsoftware.com/python-api/usage_tips.html

This post seems to address this problem… still requires a second query but you can get the users login deets in one go

https://groups.google.com/a/shotgunsoftware.com/forum/#!topic/shotgun-dev/Pr5hdzsH6Ss

2 Likes
4

Hi, Tony and Adrian :wave:

Thanks for the question! Unfortunately, you’re right there’s no other way to do this :slightly_frowning_face: I double checked with our engineers to be sure as well. You can drop in a feature request for us on our Shotgun Roadmap link and share more details to our product team. Sorry we don’t have a better answer on this.

Best,
Tram

3 Likes
5

When the API is used directly, with no additional “fields” requested from the call, it will typically return only the 'type' and 'id' fields.
Eg: {'type': 'Asset', 'id': 12345}

I believe this is due to the concrete consistency of fields available across all Entity types in Shotgun.
They will always have a 'type' and an 'id' - these are used together to target a unique record in the database.

One way you can ‘adjust’ the return is to just create an access layer which wraps the default SG.find() / SG.find_one() methods of the shotgun python api to extend the filters argument for the field names you want to return.

# example pseudo-code
# file: shotgun.py (sourced) - python 2.7
import shotgun_api3

class Shotgun(shotgun_api3.Shotgun):
    def find(entity_type, filters, fields=None, **kwargs):
        fields = fields or list()
        if entity_type == 'Task':
            fields.extend(['name', 'custom_field'])
        super(Shotgun, self).find(entity_type, filters, fields, **kwargs)

    def find_one(entity_type, filters, fields=None, **kwargs):
        fields = fields or list()
        if entity_type == 'Task':
            fields.extend(['name', 'custom_field'])
        super(Shotgun, self).find_one(entity_type, filters, fields, **kwargs)

To use, you would just source this module rather than the original shotgun_api3 module.

import shotgun
SG = shotgun.Shotgun(SERVER_PATH, SCRIPT_NAME, SCRIPT_KEY)
SG.find_one('Task', [], [])
# >>> {'type': 'Task', 'id': 1234, 'name': 'Animation', 'custom_field': 'your data'}

It’s much easier to extend for future needs as well - especially things which may be unique fields to your studio instance of Shotgun.

2 Likes
6

Hi @davidma,

Thanks. This is a super useful post showing a neat wrapping trick for Python API users however the OP’s concern was slightly different.

If you have a linked field, the format of data representing records in that field will be {'name':x, 'type':y, 'id':z}. For example:

>>> sg.find_one('Version', [['id', 'is', 6962]], ['entity'])
{'type': 'Version', 'id': 6962, 'entity': {'type': 'Asset', 'id': 1438, 'name': 'test_asset'}}

Currently there is no way to specify extra return fields for the Asset record which is in the entity field of the Version.

You could do some wrapping tricks such that if one of the requested return fields is entity.Asset.shots the wrapper would also add entity automatically in the requested return fields and upon receiving the server’s response it would move or duplicate the entity.Asset.shots information into an injected shots key in the Asset's hash but that still leaves some corner cases for multi-entity fields, among other things.

For now, unfortunately, @taiello’s concern remains a very interesting feature request.

1 Like
7

Ah, I see now

It’s an interesting problem - My main concern with adding too much logic or additional calls via an in-studio wrapper on the Shotgun class is the loss of efficiency per-query from SG

Though that’s counterbalanced if you’re constantly running the same multiple queries to gather all the field data needed.

For my studio, I have written an OOP python API library which defines most entity types we have in order to reduce the amount of code required to reach an operational state.

The objects gather all “standard” data (customized per entity type) during initialization and present to the developer field data and entity links as instance attributes and properties. Entity link field properties are then represented as their class instance.

2 Likes
8

What I was proposing wouldn’t incur any query overhead. This is all absolutely academic at this point but what I was proposing a wrapper could do is something like the following:

Let’s say your schema is that a Shot is linked to a Sequence and a sequence has a string attribute called the prefix. You’d like to query the Shot and have it return the Sequence but you’d also like to have the prefix attribute for the Sequence.

Normally you’d do:

sg.find_one('Shot', [['id', 'is', <id>]], ['sg_sequence', 'sg_sequence.Sequence.sg_prefix'])

And you’d get

{'type':'Shot', 'id':<id>, 'sg_sequence':{'type':'Sequence', 'id':<id>, 'name':'SeqName'}, 'sg_sequence.Sequence.sg_prefix':'some_prefix'}

Now, you could have a wrapper that would accept the following call:

wrapper.find_one('Shot', [['id', 'is', <id>]], ['sg_sequence.Sequence.sg_prefix'])

And that, by virtue of having sg_sequence.Sequence.sg_prefix in the return fields, it would add sg_sequence resulting in the same thing as our first call.

Then, when the return payload comes in, the wrapper would reformat the payload like so before returning to the caller:

{'type':'Shot', 'id':<id>, 'sg_sequence':{'type':'Sequence', 'id':<id>, 'name':'SeqName', 'sg_prefix':'some_prefix'}}

The wrapper basically takes the bubbled information and moves it into the actual entity hash.

Now this is just a thought experiment and the value proposition might not be enough to be worth the trouble implementing it. Not to mention that because this isn’t like asking the API to return “by default” some specified fields when an entity hash is returned, you still don’t have an eloquent way to deal with multi-entity links.

Anyway… I’m just digging the conversation. Have a great day!

1 Like
9

Nice to be chatting with you, Patrick! I agree that the single-entity use-case can be wrapped fairly straightforward-ly. I think the more-interesting use-case is with multi-entity results, where you cannot use dot-syntax – and thus you have to do a second query.

1 Like
10

Yeah, the multi-entity case would probably be better served by some sort of default return fields system or by leveraging the linked field syntax but just in a “single-hop” form as a way to inform which extra fields you want to have returned on the linked entities.

Maybe you could have double-hops in the return fields at which point you’re just nesting the data structures in the returned payload.

sg.find_one('Sequence', [['id', 'is', <id>]], ['sg_shots.Shot.sg_duration'])

Returns:

{
    'type':'Sequence',
    'id':<id>,
    'name':'SomeSeq',
    'sg_shots': [
        {'type':'Shot', 'id':<id>, 'name':'ShotA', 'sg_duration':<duration>},
        {'type':'Shot', 'id':<id>, 'name':'ShotB', 'sg_duration':<duration>}
    ]
}

Anyway, this is squarely in feature request territory but is fun nonetheless.

There’s also a performance implication to returning entity hashes for multi-entity fields. The way the db is structured, when we return records linked via single entity links, we actually don’t need to join the table for the target entity type, we just get all the hash’s info for from from the source table. Not so much for multi-entity links. I know… it wouldn’t be slower than 2 API calls. :stuck_out_tongue:

2 Likes
11

Hi there!

I was wondering if I’m doing something wrong here. If I start to query custom fields from linked entities, I can’t get it’s ‘name’/‘code’ and ‘type’ anymore. Here a small example:

app.sgtk.shotgun.find(
  "Shot",
  [["id", "is", 39190]],
  ["sg_sequence"],
)

Returns me following:

# Results:
#  [{'type': 'Shot', 'id': 39190, 'sg_sequence': {'id': 17062, 'name': 'ANS_105_042', 'type': 'Sequence'}}]

I have my sequence ‘id’, ‘name’ and ‘type’.

But if I do following:

app.sgtk.shotgun.find(
  "Shot",
  [["id", "is", 39190]],
  [
    "sg_sequence.Sequence.id",
    "sg_sequence.Sequence.name",
    "sg_sequence.Sequence.type",
    "sg_sequence.Sequence.task_template",
  ],
)

I get back following:

# Results:
#  [{'type': 'Shot', 'id': 39190, 'sg_sequence.Sequence.id': 17062, 'sg_sequence.Sequence.task_template': None}]

I have my ‘id’ and ‘task_template’, but ‘name’ and ‘type’ is gone, although I requested it. Am I doing something wrong here?

Greets,
Carlo

12

Hi there!

So, I should have read the entire thread more in detail. It already has been mentioned that for single entities, we can use the field that links the entity and for any custom fields of that linked entity, we can add the more complex syntax, e.g.

app.sgtk.shotgun.find(
  "Shot",
  [["id", "is", 39190]],
  [
    "sg_sequence",
    "sg_sequence.Sequence.task_template",
  ],
)

Sorry for the noise, and next time, I’ll read through all comments first.

Greets,
Carlo