Compiling cross-platform python modules

Hi,

We work with a lot of remote teams and are finding a way to distribute our plugins in a cross-platform manner keeping in mind the TK’s agnostic approach to code delivery while ensuring code obfuscation.

Current Environments:

  • Windows 10
  • CentOS 7
  • Core v0.19.3

What we are hoping to achieve:

  • Compile all our hooks/python files into *.so and *.pyd files
  • Bundle the compiled code in our config, which is using a centralized setup

Approach:

  • We are locally versioning the centralized config on git
  • On updates to master a CI job will trigger to traverse the tree and compile relevant python files into platform specific binaries
  • The compiled version of the config is then made available to remote teams as a zip which can be downloaded and extracted to the appropriate location

Questions:

  • Will it be alright to keep both the *.pyd and *.so files in the same location, setting aside the duplication of data which would add to the bundle size would there be any other concerns by doing this?
    OR
  • Would it be better to break apart the binaries by os and have a wrapper script which would then be able to import the appropriate path based on the running?

We are just trying to understand the recommended approach to this.

Thank you.

4 Likes

Hi @Sreenathan_Nair,

I’ll ask one of our expert’s to have a look at this for you.

  • David
1 Like

Hi @DavidMason,

Thank you very much.

1 Like

Hi!

[EDIT: Welcome to our forums!]

This is a problem we actually have ourselves with the tk-framework-desktopserver framework, as we need to package a couple of binary modules in addition to pure Python modules in order for the browser integration to work. We initially tried using virtual envs, one per platform, and pay the price of redundantly including several pure Python modules.

Due to the size of some of those pure Python modules like Twisted, the framework’s size would have more than doubled. So we took the hard road of enumerating all dependencies and their dependencies and then deciding where each of them should be installed. So in the end we have a source folder and 3 bin/<os> folders. This will get even worse when Python 3 comes around. :frowning:

When the framework is loaded by Toolkit, we append the right folders to the PYTHONPATH.

Ideally, we wouldn’t commits those files to git, but we’ve decided that our repositories should work out of the box, so we’re paying that price.

JF

6 Likes

Hi @jfboismenu,

Thank you very much for the response and the links, we had a similar implementation in mind and this really helps move things along! I agree about the repos working out of the box and being self-contained, I guess the bundle size is something we will have to live with :slight_smile:

Best wishes,
Sreenathan