Transferring data between Shotgun projects

Shotgun
#1

Hi,

One of our Shotgun project is messy for several reasons (there was never a person dedicated to its maintenance, so several people have been modifying the project in various ways). As a consequence, it is difficult to work with. We then decided to:

  • Create a new, clean and well structured Shotgun project.
  • Transfer all necessary data from the current one to the new one.
  • Switch everyone working to the new project when data transfer will be completed.

The new project has been created by a very experienced Shotgun manager, so it is well crafted and its structure suits our pipeline. We now need to transfer data between the two Shotgun projects.

Here are more information:

  • This is a running project, so data are added on Shotgun everyday.
  • We do not want to modify in any way the current project during this process, and we think the best idea would be to keep both projects completely separated.
  • This project is big enough to make the transfer too tedious to be done manually. We decided to rely on the Python API instead.
  • Our entities are available in the new project: sequences and shots, pipeline steps, tasks and assets.
  • The next step is populating the new entities with status, versions and notes that exist in the current project.

Here are some questions. Please keep in mind that I am new to Shotgun development, so I might miss things that are obvious.

  • As we want to keep both project completely separated, I assume the right way would be to copy existing data as new in the new project, and not link to any existing data. Is this the right way to proceed?
  • When copying entities from the current project to the new one, how far should I go? Should I copy every field, including those related to history and such?
  • What about metadata? Should they be copied too?

Are their any relevant resources I could look into in order to understand better how to approach this transition?
Did some of you have gone trough a similar experience? Would you have feedback to share if you did?

Many thanks.

#2

Hey @Francois

That does not sound like a trivial exercise to me. Depending on the complexity of your project it is constructed from a bunch of different interconnected tables (entities).
Would love to hear if anyone has any easy answer for this as I expect to have to do something like this at some point too.

@Shotgun copying assets/shots is something that would be really useful! For instance when we finish a show we often archive some assets into a library project so they can be reused. Would be awesome to be able to copy an entire shotgun asset entity and all linked data directly to a library project. As it is we have to copy over the data on disk and then republish everything on the library project (when doing this we loose all the history and any extra data).

There are a couple of complications here:
• You will need to re-link the new entities once they have been created
• You may need to handle any differences in fields between your new and old projects

If I was attempting it on our servers I think I would do the following:

  1. Identify all entities that need copying over (Shots, Assets, Tasks, PublishFiles, PublishFilesDependencies, Version etc etc)
  2. Write a tool to pull all that info from shotgun using the api and represent in some kind of data structure.
    a. This will need to include all extra fields
    b. You’ll need to represent any connections to other entities (maybe use some kind of graph representation?)
  3. Write a tool to push your representation onto your new project
    a. Create all the entities with all extra fields except connections (you need the new ids of the entities to be able to connect)
    b. Use the new ids and connection representation to connect up all the new entities.

This is quite a large project and has a lot of potential for things to go a bit awry. I would focus a lot of effort on running dry-runs, tests on sandboxes and a way to validate that everything has moved over correctly.

Hope that helps,
All the best!

1 Like
#3

Hi Jared,

Thank you for your response and your highlights, it will be useful.

I can’t tell you more right now as I needed to jump to something else. I will be back on this subject in a matter of weeks. Then I will be able to share more on about how I deal with it.