RabbitMQ, The Guide

Pipeline Integrations
,
1

Tony Aiello (@taiello) once had a nice reply to Hasiel Alvarez (@hasielhassan) outlining his experience connecting RabbitMQ to Shotgun.

Since this valuable information was accidentally scrubbed from the old ticket system I thought it might be helpful to post it back in here.

The rest of this post are mostly Tony’s words re-formatted and edited from 2018-05-24…

For local-installs

Shotgun can publish EventLogEntries as MQ messages. I don’t think there is any significant documentation. The rest of my overly-long post here is still not going to be an attempt at thorough documentation either :sunglasses: But maybe others will help add to this over time and it’ll become a support Article.

For hosted sites

Shotgun does not support this for cloud-hosted sites! Shotgun folks: yes / no?

Get familiar with the terminology.

There was a great presentation at PyCon 2018 in Cleveland just a couple weeks ago, check it out. Go watch it, I’ll wait…

(by the way, ALL PyCon videos are on YouTube. PyCon is awesome)

Shotgun would be considered the producer

  • Shotgun already has built-in support to publish EventLogEntries as they happen out to an exchange

  • Using a message broker, you create one or more bindings between an exchange and one or more queues.

  • The routing_keys map from an exchange to a queue.

  • Each EventLogEntry gets packed as a json blob into a message. Then you can write consumers to receive the messages from the queues and do stuff with them.

Step #1: Installing a message broker

Here at Laika, our broker of choice is RabbitMQ; we currently run version 3.6.12. You can install a broker directly on your shotgun server, or you can setup dedicated hardware for it. Dedicated hardware is what we do, because we use RabbitMQ for MUCH more than just Shotgun.

Our RabbitMQ setup consists of an 8-node cluster: two servers that are physically redundant for each of our four buildings.

A proxy server provides the physically-closest pair through a common URL. Rabbit only ships data over the building WANs if necessary, otherwise all the traffic is kept within the specific network segment of the cluster. The hosts are eight-core i7 machines with 12G RAM, running FreeBSD 10.3p7 without any tweaks.

We’ve measured the cluster wrangling over 18 thousand simple “hello world” messages per second with a cluster policy of +2 replication. In addition to Rabbit, the hosts also serve DHCP/DNS/LDAP, and even so load is consistently below 1.

Step #2: Enable message production

This Shotgun producer-bit might be the easiest part of this process! If you’re locally-hosted, ssh into your Shotgun server then

cd /var/rails/<your_site>/shared/config

You should find a “bunny.example” file there; take a look inside:

# This file contains contains information Shotgun uses to 
# connect to RabbitMQ.
#
# The host and port variables should match the information needed 
# to connect to the rabbit host. 
# If the host is different then localhost you may need to open
# firewall ports between the Shotgun server and the RabbitMQ server.
#
# DEFINITIONS
# :host: The fqdn or ip address of the host running rabbitmq
# :port: The port at which rabbitmq is listening on the :host:
# :id: This is the exchange prefix. 
#       Typically, this is set to your shotgun site's fqdn. 
#       The exchange is the rabbit process which delivers the message
#       to the appropriate queue(s).
# :no_crud: This is an internal variable which should be set to true
#
# EXAMPLE
---
:bunny:
  :host: localhost
  :port: 5672
  :id: shotgun.yourstudio.com
  :no_crud: true

You should now:

  • make a copy of the bunny.example file, named bunny.yml – that specific name is required for this to work

  • edit that copy to set the values for host and port to point at your broker install

  • edit that copy to set the id value

  • save the file and exit your editor, then

  • restart your site.

To clarify the :id: key in this basic config – whatever value you specify for :id: will get concatenated with _events to form the exchange name.

So if you specify :id: fubar.studio.com then the exchange into which the Shotgun EventLogEntries will go will be named fubar.studio.com_events.

Exchange Type

Shotgun uses the fanout exchange type by default.

Assuming you were paying attention to the video, this means that all messages from Shotgun going into the exchange defined by your bunny.yml will be routed to every queue you bind to that exchange.

So if you’re paying close attention, you might notice there is no way to specify a routing_key in that bunny.yml.

Fanout is like an ethernet hub – any one thing coming-in gets copied back out every exit, so the responsibility is on the consumer(s) of the queue(s) to decide whether the message (copy) is interesting or not.

As opposed to topic and direct exchange types, which are analogous to an ethernet switch, where there’s efficiency gained by smart routing thus lightening the workload for your consumer(s).

Routing Keys

Regardless, Shotgun creates the routing_keys for event messages for you. The first part of the string is either the event['project']['code'] field value or global if the event doesn’t have a ['project'] value.

Then it appends a converted form of the event['event_type']
e.g. Shotgun_Asset_Change by stripping off the leading Shotgun_ then converting underscores to periods then lowercasing it all: Shotgun_Asset_Change becomes asset.change.

So for an Asset change event in a project whose ['code'] is abc, the routing_key for the MQ message created would be abc.asset.change and for a non-project event like creating a new HumanUser entity, the routing_key would be global.humanuser.new.

One other bit

If you’re running anything older than Shotgun v6.3, which uses Ruby 1.9 and thus bunny v0.8.0, you will need to upgrade the bunny gem in your installation. cd /opt/ruby/bin then sudo ./gem install bunny should bring you up to the right version. Again, local-hosted only…

If you’re cloud-hosted, you would have to ask Shotgun Support to do all this for you… and again, I don’t know if Shotgun even officially supports this at all for cloud-hosted sites.

If you do get to this point, new EventLogEntry messages should start hitting your broker.

SPOILER WARNING…

Speaking just for Laika, we don’t recommend nor use that basic bunny.yml file. Since you watched the video, you know that one might want to completely control the exchange name and probably set an exchange_type.

The exchange name of {id}_events and fanout exchange type might not be ideal for your studio. And if your IT dept cares about security like ours, they probably will want to SSL-secure the broker site and create virtual hosts within it with username:password creds.

I do know of at least two studios that are successfully using Shotgun’s basic bunny.yml file and hardcoded defaults, but it was not enough for our environment.

So a few years ago, we worked with Shotgun on custom code to add support for those additional (in our case, essential) properties. Now our bunny.yml file looks like this:

:bunny:
  :uri: amqp://user:pword@rabbitmq.studio.com/%2Fa_virtual%2Fhost%2Fname
  :exchange_type: topic
  :exchange: an_exchange_name
  :id: routing_key_prefix
  :no_crud: true

Notice that the custom code adds three new keys:

  • :uri:
  • :exchange:
  • :exchange_type:

Also notice that by using a specific :uri: , it’s not necessary to specify a :host: or :port: .

You or your IT dept would configure the virtual host(s) in RabbitMQ, in the form of something like maybe /your_studio/your_department/maybe_another_noun , and notice that the slashes have to be converted to hex %2F when entered in the :uri: key in the bunny.yml file. You’d define the :exchange: name and :exchange_type: in RabbitMQ, then enter those values in the bunny.yml file. (If you leave the exchange_type as fanout in RabbitMQ, you can skip entering an :exchange_type: key/value in the .yml.) You’d define routing keys in RabbitMQ as described in the video – if you use a “topic” type, you’ll probably use at least one # wildcard in your key definition – and bind the exchange(s) to queue(s) with those keys.

More on routing keys

If you use a topic :exchange_type, the :id: key in the YML becomes a prefix value for Shotgun’s routing_keys instead of the exchange name.

So for example with an :id: value of foo, the routing_keys for messages will start with foo and end with the rest of the key that Shotgun constructed, for example foo.abc.asset.change or foo.global.humanuser.new using the example above.

Again, using the #-wildcard functionality in RabbitMQ, you might bind an exchange to an asset-changes-only-queue for all projects with a key like #.asset.change… and maybe another queue for all events in the abc project with a key like #.abc.#… and maybe if you’re running multiple Shotgun sites (for example production and staging) and thus have multiple bunny.yml files, you might use that :id: field to specify the site after all and thus have a queue per site. All sorts of combinations are possible!

If you want these additional RabbitMQ properties, regardless of whether you’re cloud- or local-hosted, you need to talk to Shotgun about the custom code. I don’t know if it has become part of the standard shotgun app code in the years since, or if it needs to be specially-activated, or if it has to be a totally different app software cut for you.

Disclaimer

[Tony] I hope I’m not making trouble for the Shotgun folks by talking about this!

Certainly not my intent my friends, and feel free to delete my post, or just this part of it, with my apologies in advance.

[Dougm] …ditto! Likely Shotgun Software would appreciate me noting that this is posting might never be discussed by anyone other than clients that have local installs and enough engineers to dabble in this.

After installing

You’ve got EventLogEntries from Shotgun hitting your message broker! Great! Route them from the exchange(s) to some queues via some key bindings, and start consuming the messages from those queues.

Step 3: Create one or more queues

…in your broker app

Step 4: Create bindings

…from the exchange(s) to those queues in your broker app using some form of key specification probably involving the # wildcard

Step 5: Write some consumer apps

…to pull the messages off those queues… see the video for examples in python

Step 6: Profit!

Consumer Apps

Notice that your consumer apps will probably have to open a shotgun python API connection (or hit Shotgun v7.9 REST API endpoint woohoo!) to retrieve more data about the EventLogEntries, similar to what you usually have to do with the shotgun event daemon or your own apps.

Finally, EventLogEntries that fail to get into an exchange get logged into

/var/rails/<your_site>shared/log/rabbitmq_failed_events.log

Sample

Here’s a sample RabbitMQ message from Kubo and the Two Strings production. Note the Exchange and Routing Key are custom-code spoiler-style:

  Exchange:  shotgun
  Routing Key:  production.kbo.version.change
  content_type:   application/json
  Payload: 756 bytes
  Encoding: string
  {"user":{"name":"GDPR Redacted","type":"HumanUser","id":<redacted>,"login":"gdpr_redacted"},"id":39871767,"event_type":"Shotgun_Version_Change","description":"GDPR Redacted changed \"Status\" from \"aprdir\" to \"aprtk\" on Version 1500.0490.test.001","meta":{"type":"attribute_change","attribute_name":"sg_status_list","entity_type":"Version","entity_id":407523,"field_data_type":"status_list","old_value":"aprdir","new_value":"aprtk"},"entity_id":407523,"entity_type":"Version","created_at":"2015-11-10 00:46:14.314755","image_id":0,"attribute_name":"sg_status_list","project":{"name":"Kubo","type":"Project","id":125},"notification_trial":1,"session_uuid":"6bd44946-8744-11e5-a82e-1adeadbeef01","sg_processed":"f","filmstrip_image_id":0,"cached_display_name":null}
4 Likes
2

Hi Doug,

This is absolutely fantastic, thank you so much for re-sharing this.

Cheers,
Andrew