• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
Data Dependence

Data Dependence

Learn about software development. Covering topics such as Coding in Python and JavaScript, developer productivity and more.

  • Home
  • Top Resources
  • Who Are You?
  • Show Search
Hide Search
You are here: Home / Programming / [Video Series] Taking Your Python Skills to the Next Level With Pythonic Code – Hacking Python’s Memory With __slots__

[Video Series] Taking Your Python Skills to the Next Level With Pythonic Code – Hacking Python’s Memory With __slots__

· · Leave a Comment

This is the second post in a series covering Pythonic code written by Michael Kennedy of Talk Python To Me. Be sure to catch the whole series with 5 powerful Pythonic recommendations and over 45 minutes of video examples.

What if I told you there was a simple technique you can apply to your custom classes that would dramatically decrease the memory usage of your application? Well, there is; it involves the keyword __slots__ and that’s what this post is all about.

The story starts with a blog post from Oyster.com with a powerful title and image:

SAVING 9 GB OF RAM WITH PYTHON’S __SLOTS__

Physical memoray usage after python slots

Oyster.com is a travel website with lots of traffic. They were able to use a very simple technique to “hack” python’s type memory management using the __slots__ language feature. Read oyters.com’s story here.

To understand slots, it’s best to start by discussing how fields are associated with standard Python classes. Consider this class:

class Measurement:
    def __init__(self, x, y, value):
        self.x = x
        self.y = y
        self.val = value

If we allocate a few of them, setting some values via the initializer and even dynamically after allocation on one of the instances, you’ll have something like this in memory:

m1 = Measurement(1, 2, "Happy")
m2 = Measurement(7, 10, "Crazy")
m2.other = True

typical python class

Notice that each instance has a pointer to the __dict__ containing the underlying field names and values. We can look at them like this:

print(m1.__dict__) # {'x': 1, 'y': 2, 'val': 'Happy'}
print(m2.__dict__) # {'x': 7, 'y': 10, 'other': True, 'val': 'Crazy'}

This is how custom types in Python work and how they are meant to work.

However, if we aren’t customizing the fields after creation of our types (e.g. the line we set m2.other = True), and we have millions of instance of that type, it can be inefficient. We have millions of copies of those dictionaries including the duplicated key names.

USING SLOTS

We can adjust our Measurement class slightly to change how fields are stored to remove this duplication and 1-to-1 dictionary allocation.

class Measurement:
    __slots__ = ['x', 'y', 'val']
    def __init__(self, x, y, value):
        self.x = x
        self.y = y
        self.val = value

Then we have a new picture in memory:

Python class with slots

Now the field names are associated with the type Measurement, not the instances of Measurement. This is great because we only have one instances of the type (i.e. Measurement) and potentially millions of Measurement instances (i.e. m1, m2, …).

To see this in action, check out my video on this demonstrating this technique as well as comparing it to plain tuples and the trade-offs associated.

VIDEO: HACKING PYTHON’S MEMORY WITH __SLOTS__

You can find the code from this video on GitHub.

Michael Kennedy
@mkennedy

Don't forget to share and follow!

Remember, don't forget to share this post so that other people can see it too! Also, make sure you subscribe to this blog's mailing list and follow me on Twitter so that you don't miss out on any useful posts!

I read all comments, so if you have something to say, something to share or questions and the like, leave a comment below!

‘Ello, I’m Jamal – a Tokyo-based, indie-hacking, FinTech software developer with a dependence on data.

I write Shakespeare-grade code, nowadays mostly in Python and JavaScript and productivity runs in my veins.

I’m friendly, so feel free to say hello!

Twitter is the best place for a chat.

Share this:

  • Click to share on Twitter (Opens in new window)
  • Click to share on Facebook (Opens in new window)
  • Click to share on Reddit (Opens in new window)
  • Click to share on LinkedIn (Opens in new window)
  • Click to share on Pinterest (Opens in new window)
  • Click to share on Pocket (Opens in new window)

Related

Programming programming, python, pythonic, tips and tricks

Primary Sidebar

LEVEL UP with the Data Dependence Newsletter! Subscribe!


Top Posts & Pages

  • How to Find Unclosed Tags and Brackets Using a Stack
  • How to Build a GUI in Python 3.5
  • Mocking in Python - How to Bypass Expensive and External Code Within Your Tests
  • How to Logically Group Your Python Code into Modules
  • A Quick Guide to Slicing in Python - Become a Python Ninja

Data Dependence

Copyright © 2023 · Monochrome Pro on Genesis Framework · WordPress · Log in

  • Home
  • Top Resources
  • Who Are You?
 

Loading Comments...