Is that Python code of yours running a little slow? Are you thinking about rewriting the
algorithm or maybe even in another language? Well, before you do, you'll want to listen to
what Davis Silverman has to say about speeding up Python code using profiling.
This is show number 28, recorded Wednesday, September 16th, 2015.
Welcome to Talk Python To Me, a weekly podcast on Python, the language, the library,
the ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter
where I'm @mkennedy. Keep up with the show and listen to past episodes at talkpython.fm
and follow the show on Twitter via at Talk Python. This episode is brought to you by Hired and
CodeShip. Thank them for supporting the show on Twitter via at Hired underscore HQ and at CodeShip.
There's nothing special to report this week, so let's get right to the show with Davis.
Let me introduce Davis. Davis Silverman is currently a student at the University of Maryland working
part-time at the Human Geo Group. He writes mostly Python with an emphasis on performant
Pythonic code. Davis, welcome to the show.
Hello.
Thanks for being here. I'm really excited to talk to you about how you made some super slow Python
code much, much faster using profiling. You work at a place called the Human Geo Group. You guys do a
lot of Python there, and we're going to spend a lot of time talking about how you took some of your
social media sort of data, real-time analytics type stuff, and built that in Python and improved
it using profiling. But let's start at the beginning. What's your story? How did you get into programming
in Python?
I originally, when I was a kid, obviously, I grew up and I had internet. I was lucky, and I was very into
computers, and my parents were very happy with me building and fixing computers for them, obviously.
So by the time high school came around, I took a programming course, and it was Python, and I fell in
love with it immediately. And I've since always been programming in Python. I've been programming in Python
since sophomore year of high school. So it's been quite a few years now.
I think all of us programmers unwittingly become tech support for our families and whatnot, right?
Oh, yeah. My entire family, I'm that guy.
Yeah. I try to not be that guy, but I end up being that guy a lot.
So you took Python in high school. That's pretty cool that they offered Python there. Do they have
other programming classes as well?
Yeah. So my high school, I live in the DC metro area. I live in Montgomery County. It's a very nice
county, and the schools are very good. And luckily, the intro programming course was taught by a very
intelligent teacher. So she taught us all Python. And then the courses after that were Java courses,
the college-level advanced placement Java, and then a data structures class after that.
So we got to learn a lot about the fundamentals of computer science with those classes.
Yeah, that's really cool. I think I got to take basic when I was in high school, and that was about it.
It was a while ago.
I wrote a basic interpreter, but it wasn't very good.
Cool. So before we get into the programming stuff, maybe you could just tell me,
what is the Human Geo Group? What do you guys do?
Yeah. So the Human Geo Group, we're a small government contractor, and we deal mostly in government contracts,
but we have a few commercial projects and ventures that I was working on over the summer that we'll be
talking about. We're a great small little company in Arlington, Virginia, and we actually just won an award for one of the best places to work in the DC metro area for millennials and for younger people.
If you go to thehumangeo.com, you guys have a really awesome webpage. I really like how that is. It's like, bam, here we are. We're about data, and it just has kind of this live page. So many companies want to get their CEO statement and all the marketing stuff and the news. You guys are just like, look, it's about data. That's cool.
Yeah. There's a recent, I don't remember how recent, but there is the website rewrite and the website, this one guy, he decided he took the time and he's like, I really want to do something, you know, show some geographic stuff.
So he used leaflet.js, which we do a lot of open source with and on our GitHub page with leaflet.js. And he made it very beautiful. And he, and that even there's some icons of all the people at human geo. And I think it's much better than any of those. Like you said, a generic, you know, contractor site. It's, it's much better and much more energetic.
It looks to me like you do a lot with social media and like sentiment analysis and tying that to, to location, the geo part. Right. And all that kind of stuff. What's the story there with what kind of stuff do you guys do?
Yeah. So one of, what I was working on was we took one of our, one of our customers is a billion dollar entertainment company. And, I mean, you've probably heard of them. I think we talk about them on our site.
And what we do is we analyze various social media sites like, you know, Reddit and Twitter and YouTube, and we gather geographic data if available. And we gather sentiment data using specific algorithms from things like the natural language toolkit, which is an amazing Python package.
Then we show it to the user in a, in a very nice, website that we have created.
So you say you're, you work for this entertainment company as well as like a government contractor.
What is the government interested in with all this data? The U S government that is right for the international listeners.
Yeah, it's, it's definitely a United States government. we, we do, not, we do less social media analysis for the government. We do, we do some, but it's nothing.
It's not what people think the NSA does. definitely. I think, you know, just like anything, like a company would want, like your, like you'd search on something and then it would have like, Oh, there are Twitter users talking about this.
This in the, you know, in these areas. Yeah. I guess the government wouldn't know, would want to know that, especially in like emergencies or things like that possibly. Right.
Yeah. We also do some, platform stuff. Like we create, certain platforms for, the government. That's not necessarily social media stuff.
Right. Sure. So how much of that is Python and how much of that is other languages?
So at the human geo, we do, we do tons of Python at the backend. for some of the government stuff, we do Java, which is big in the government, obviously. so I think we, I mean, we definitely do have a lot of Python. We use it a lot in the pipeline, for various tools and things that we use internally and externally at human geo. I know that the project that I was working on is exclusively Python.
Python and all parts of the pipeline for, for gathering data and for representing data for, you know, the backend server and the front end server. So that was all Python.
Right. So it's, it's pretty much Python end to end other than it looks, I don't know specifically the project you were working on, but it looks like it's very heavy, like D3, fancy JavaScript stuff on the front end for the visualization.
But other than that, it was more or less Python, right?
We do use a lot. I mean, yeah, we do have some amazing JavaScript people. They, they do a lot of really fun looking stuff.
Yeah. You can tell it's, it's a fun place for like data display, front end development. That's cool.
So Python two or Python three.
So we use Python two, but I was making sure as, I mean, when I was working on the code base, I was definitely writing Python three compatible code using the proper future imports.
And, and I, and I was testing it on the Python three and we're, we're probably closer to Python three than a lot of companies are. we just haven't expanded the time to do it.
We're probably will in 2018 when Python two is nearing end of line.
That's seems like a really smart, smart way to go. Did you happen to profile it under Python, CPython two and CPython three?
I didn't. It doesn't fully run in CPython three right now. I wish I could.
It would just be really interesting since you spent so much time looking at the performance if you could compare those, but yeah, if it doesn't run.
That would be interesting. You're right. I wish I could know that.
I suspect that most people know what profiling is, but there's a whole diverse set of listeners. So maybe we could just say really quickly, what is profiling?
Yeah. So profiling in any language, anything is, knowing heuristics about what's running in your program. And, you know, for example, how many times is a function called or how long does it take for this section of code to run? And it's simply like a statistical thing. Like you, like you get a profile of your code. You see all the parts of your code that are fast or slow. For example.
You wrote a really interesting blog post and we'll talk about that in just a second. And I think like all good profiling articles or topics, you know, you, you sort of point out what I consider to be the first rule of profiling or more like the first rule of optimization, which profiling is kind of the tool to get you there, which is to not prematurely optimize your stuff. Right.
Yeah, definitely.
Yeah. I, you know, I've spent a lot of time thinking about how programs run and why it's slow or why it's fast or worrying about this little part of that little part. And, you know, most of the time it just doesn't matter. Or if it is slow, it's slow for reasons that were unexpected. Right.
Yeah, definitely. Yeah. I always make sure that there is a legitimate problem to be solved before spending time doing something like profiling or optimizing a code base.
Definitely. So let's talk about your blog post because that kind of went around on Twitter in a pretty big way and on the Python newsletters and so on. And I read that. Oh, this is really cool. I should have Davis on and we should talk about this.
So, so, so on the human geo blog, you wrote an article or post called profiling in Python, right? What motivated you to write that?
Yeah. So when I was right, when I was working on this code, basically my, my coworkers, my boss, they said, you know, this piece of code, we have this pipeline for, for gathering data from the customers specifically that they give us.
And we run it, we ran it about like 2am every night. The problem was it took 10 hours to run this, this piece of code. It was doing a very heavy text processing, which I'll talk about more later, I guess.
It was doing a lot of text processing, which, which ended up taking 10 hours and they look at it and they said, you know, it's updating at noon every day and the workday starts at like nine.
So we should probably try to fix this and get it working faster. Davis, you should totally look at this as a great first project.
Here's your new project, right?
Yeah. It was like, here's day one. Okay. Make this thing 10 times faster. Go.
Yeah. Oh, is that all you want me to do?
Yeah. So I start, so I wrote this and I did what any guy would do. I, first thing I did was, you know, profile it, which is the first thing you should do to make sure that there's actually a hotspot.
And I ran through the process that I, that I talked about in the post post. I talk about what I ran through the tools I use. And I realized that it was, it wasn't a simple thing for me to do all this. I don't do this often. And I figure that a lot of people, like you said, maybe don't know about profiling or haven't done this.
So I said, you know, if I write a blog post about this and hopefully somebody else won't have to Google like 20 different posts and read all of them to come up with one coherent idea of how to do this in one fell swoop.
Right. Maybe you can just lay it out. Like, these are the five steps you should probably start with. And, you know, profiling is absolutely an art more than it is, you know, engineering, but it, at least having the steps to follow is super helpful.
You started out by talking about the CPython distribution. And I think it'd be interesting to talk a little bit about potentially alternative implementations as well. Cause you, you did talk about that a bit.
Yep. You said there's basically two profilers that come with CPython profile and CPython.
Yeah. So there, the two profilers that come with Python, as you said, profile and CPython, all have the same exact interface and include the same heuristics.
But the idea behind profile is that it's written in Python and is portable across Python implementations, hopefully, and CPython, which is written in C.
And it's pretty specific to using a C interpreter, such as a CPython or even PyPy because PyPy is, is very, interoperable.
Right. It does have that, that interoperability layer. So maybe if we were doing like Jython or we were doing IronPython or PyStone or something, we couldn't use CPython.
Yeah. I wouldn't, I wouldn't say that you can.
I'm just guessing. I don't really, I haven't tested it.
I would say that you would use the, for Jython and IronPython, you could use the standard Java or .NET profilers and use those instead.
I'm pretty sure those will work just fine because I mean, they've, they're known to work great with their respective ecosystems.
Do you know whether there's a significant performance difference? Let me take a step back. You know, it seems to me like when I've done profiling in the past, that it's a little bit like the Heisenberg uncertainty principle.
And that if you observe a thing by the fact you've observed it, you've altered it. Right. You know, when you run your code under the profiler, it might not behave in the same exact timings and so on as if it were running natively, but you can still get a pretty good idea usually. Right. So is there a difference across the C profile versus profile in that regard?
Oh yeah, definitely. Profile is much slower and it has a much higher latency and as overhead, I mean, than C profile because it has to do a lot of different worker.
I mean, because Python exposes internals to C to CPython in, in some Python modules, but they're a lot slower than just using straight C and getting straight to it. So if you're using regular profile, I'd recommend, well, if you're using it in CPython or pipe, I'd recommend using C profile instead because it has much lower overhead and it gives you much better numbers that you can work with that make more sense.
Okay. Awesome. So how do I get started? Like, suppose I have some code that is slow, but how do I run it in this in C profile?
Yeah. So the first thing that I would do is, I mean, I, in the blog posts, which I'm pulling up right now just to see.
And I'll be sure to link to that in the show notes as well so that if people can just jump to the show notes and pull it up.
Yeah. Well, so one of my favorite things about C profile is that you can call it using, you know, the Python dash M syntax, the column module syntax, and it will print out to standard out your profile for you when it's done.
It's super easy. I mean, all you need to do is just give it your, you know, main Python file and it'll run it. And then at the end of the running, it'll give you the profile. It's super simple. And one of the reasons why the blog post is so easy to write.
Yeah, that's cool. So by default, it looks like it gives you basically the total time spent in methods, all of the methods, you know, looking at the call stack and stuff, a number of times it was called, stuff like that, right? The cumulative time, the individual time per call and so on.
It gives you the default, like you said, you're correct. And it's also really easy. You can give it a sorting argument. So that way you can, if you want to call it on C, you know, how many times this is called, like if it's called 60,000 times, it's probably a problem in a, you know, a 10 minute run.
And if it's, you know, it could be only called twice, but it may take an hour to run. That would be very scary. In which case you definitely want to try, you want to, you want to sort it both ways. You want to sort it every way. So you can see what, you know, just in case you're missing something important.
Right. You want to slice it in a lot of different ways. How many times was it called? What was the, you know, maximum individual time, cumulative time, all those types of things.
All right. Maybe you're calling a database function. Right. And maybe that's just got tons of data.
Yeah. It's slow. Yeah. Yeah. Maybe, maybe the database is slow. And so that says, well, don't even worry about your Python code. Just go make your database fast or use Redis for caching or something. Right.
Or yeah, work on your query. Maybe you're, maybe you're, maybe you can make it distinct, a distinct query and get a much smaller data set that ends up coming back to you.
Yeah, absolutely. Yeah. It could just be a smarter query you're doing. So this all is pretty good. I mean, this, this output that you get is pretty nice, but in a real program with many, many functions and so on, that can be pretty hard to understand, right?
Yes. I definitely, I found, so when I was working on this, I had the same issue. There was, there was so many lines of code. It wasn't, it was filling up my terminal, you know, and what I had to do is I had to save it to an output file and that was too much work.
So I was searching around more and I found PyCallGraph, which is amazing at showing you the flow of a program and it gives you great graphic representation of what C profile is also showing you.
That's awesome. So it's kind of like a visualizer for the C profile output, right?
Yeah. It even colors in the more red it is, the more hot of a call it is, or the more times it runs and the longer it runs.
Yeah. That's awesome. So just pip install PyCallGraph to get started, right?
Super simple. It's one of the best things about, pip as a package manager.
Yeah. I mean, that's part of why Python is awesome, right? pip install.
Definitely.
Whatever. Anti-gravity. So then you say, you, invoke it, you say basically PyCallGraph and you say GraphFizz.
And then how does that work?
So PyCallGraph is, it supports outputting to multiple different file formats. GraphFizz is simply a file format for, I mean, it's a program that can show called .file.
I mean dot dot files
I don't really understand how to say it out loud
so I mean the first argument for
graph is the
style of how it's going to be read the program that's going to
read it and then you give it
the double dash which means it's not
part of the PyCallGraphCall options
it's now the Python call
that you're calling it with
and those arguments
so it's almost basically the same
as cProfile but it's kind of inverted
right and you can get really simple
call graphs that are just this function called
this function which called this function and it took this
amount of time or you can get really
quite complex
call graphs
like you can say you know this module
calls these functions which then reach out
to this other module and then they're all
interacting in these ways
that's pretty amazing
yeah it shows
exactly what module you're using
I mean like if you're using regex it'll just
it'll show you each part of the regex
module like the regex compile
or you know the different
modules that are using the regex module
and then it'll show you how many times each is called
and they're boxed all nicely
and
and it gives you I mean the image is so
easy to look at
and you could just zoom in at the exact part you want
and then look at what calls it
and where and what it calls to see
you know how the program flows much simpler
yeah that's really cool
and you know it's something that just
came to me as an idea
I'm looking at this
this episode is brought to you by
hired
hired is a two-sided
curated marketplace that connects the world's
knowledge workers to the best opportunities
each offer you receive
has salary and equity presented right up front
and you can view the offers to accept or reject them
before you even talk to the company
typically candidates receive five or more offers in just the first week
and there are no obligations
ever
sounds pretty awesome doesn't it
well did I mention there's a signing bonus
everyone who accepts a job from hired gets a two thousand dollar signing bonus
and as talk python listeners
it gets way sweeter
use the link hired.com slash talk python to me
and hired will double the signing bonus to four thousand dollars
opportunities knocking
visit hired.com slash talk python to me
and answer the call
because it colors the hot spots and all it's really good for profiling but
even if you weren't doing profiling it seems like that would be pretty interesting for just
understanding new code that you're trying to get your head around
oh yeah that's definitely true that's uh
i've employed that since as a method to look at the control flow of a program
right how does these methods how did these
modules and all this stuff just how do they relate like just run the main method and see what happens right
exactly it's
it's uh it's become a useful tool of mine
that i'll definitely be using in the future
i always have it my virtual end of nowadays
so
we've
taken c profile we've
applied it to our program
we've sort of gotten some textual version of the output that tells us where we're spending our time in various ways and then we can use py call graph to visualize that understand it more quickly
so
then what like how do you fix these problems what are some of the common things you can do
yeah so
as i outlined in the blog post there's there's a plethora of methods that you can do depending on what
your profile is showing for example if
if you're spending a lot of time in
in python code
then you can definitely look at
things like using a different interpreter for example an optimizing compiler like
pypy
uh will definitely make your code run a lot faster as it'll translate to machine code at runtime
or you can also look at the algorithm that you're using and see if it's for example like an on cubed
uh time complexity algorithm that would be terrible and you might want to fix that
yeah those are the kinds of problems that you run into when you have small test data
and you write your code and it seems fine and then you give it real data and it just dies right
exactly that's
that's why the thing
they gave me the code that took i mean they gave me like five gigabytes of data and they said this is the data that we get like on a
nightly basis and i said
oh my god this will take all day to run
so
i i use smaller test test pieces
and then luckily i i used big enough that it showed
uh some statistically significant numbers for me
right it was something you could actually execute in a reasonable amount of time as you
through your exploration but not so big that
i'd rather not do like c plus plus
times of run you know compile times but at runtime because it just kind of sits there while it's processing
right so
rather not only do it once a day
you mentioned some interesting things like
pi pi is super interesting to me
we're going to talk more about how
how you well you chose not to use that but
you know on show 21 we had
macha from the
the pi pi
oh yeah i have it up right now i'm gonna watch it later
yeah i'm so excited yeah that was super interesting and we talked a little bit about
optimization there and like why you might choose an alternate
interpreter that was cool
then there's some other interesting things you can do as well like
you could use like name tuples instead of actual classes or you could use built-in data
structures instead of creating your own because a lot of times the built-in structures like list and array
and dictionary and so on are implemented deep down in c and they're much faster
yeah definitely
uh one of the best examples of this is that i saw some guy who wrote his own dictionary class
and uh it was a lot slower than
and this isn't in the human geo code base just so you know we have we have good code at human geo
you guys don't show up on the daily wtf
oh please no no we're we're much better than that no this is a another another place that i'm that i saw some code
and uh yeah i mean they have a lot of optimizations like in the latest python release
they actually made them the dict class i mean it's actually now an ordered dict in the latest python releases
because they basically copied what pypy did the same they did the same thing
yeah so you should always trust uh the internal stuff yeah and that's really true and if you're going to stick
with c python as you know the majority of people do understanding how the interpreter actually works
is also super important and i i talked about it several times in the show and um had philip guau on the show
he did a 10 hour he recorded basically a graduate's uh graduate course at the university of rochester he did
and put it online so uh definitely i can try to link to that and check it out that's show 22
i'd love to see that yeah i mean it's you really understand what's happening inside the c runtime
and so then you're like oh i see why this is expensive and all those things can help you know another thing
that you talked about that was interesting was io like i gave you my example of databases
or maybe you're talking about a file system or you know a web service call something like this
right and you had some good advice on that i thought yeah i definitely uh basically the c python
c python's jill the global interpreter lock is very uh precluding you can't you can't do multiple
multiple computation and computations at the same time because c python only allows to be used at one core
at a time right basically computational parallelism is not such a thing in python if you're you got
to drop down to c or fork the processes or something like that right exactly and those are all fairly
expensive for a task that we're running on an aws server on an aws server that we're trying to spend as
little money as possible because it runs at like the break of dawn so we don't want to be running multiple
python instances but when you when you're doing io which doesn't involve any python you can use
threading to do things like you know file system you know expensive io tasks like getting a getting
the data off a url or things like that that would that's great for python's threading but otherwise
you don't really want to be using it right basically the built-in functions that wait on io they
release the global interpreter lock and so that frees up your other code to keep on running more or less
right you definitely want to make sure that if you're doing io that it's not the bottleneck
i mean as long as everything else is not the bottleneck right and you know we've had um
a lot of cool advances in python 3 around this type of parallelism right and they just added async and
await the new keywords to uh is that three five i think it was right yeah just three fives just
came out like two days yeah two days ago so yeah i mean that's super new but these are the types
of places where async and await would be ideal for helping you increase your performance yeah it's uh
it's the new syntax is like a special case it's syntactic sugar for yielding but it makes things
much simpler and easier to read because if you're using yields to do complex async stuff that isn't just
like a generator then it's very ugly so they added this new syntax to make it much simpler yeah it's great i'm
looking forward to doing stuff with that you also had some guidance on regular expressions and the
first one the first bit i also really like kind of like your premature optimization bit is once you
decide to use a regular expression to solve a problem you now have two problems yeah i always have that
issue um when i whenever whenever i do anything and i talk to students and they're like and i'm like oh
look at this text and you could do this and like oh i'll just use regex to solve it i'm saying please know
you know you'll end up with something that you won't be able to read in the next two days and then
you know just find a better way to do it for goodness sakes yeah i'm i'm totally with you a friend of mine
he has this really great way of looking at complex code both around regular expressions and like
parallelism and it says when you're writing that kind of code you're often writing code right at the limit
of your understanding of code or your ability to write complex code and debugging is harder than
writing code and so you're writing code you literally can't debug so you may maybe maybe not
go quite that far right yeah i think uh you should always try to make code as simple as possible because
debugging and profiling and looking through your code will be uh much less fun if you try to be as
clever as possible yeah absolutely clever code it's not so clever when it's your bug to solve later
yeah and i i also had try to give special mention to python's regex engine as much as i dislike regex
i think python's read that verbose flag is amazing and if you haven't uh looked into it python has great
support for i'm greatly annotating the regex so if you have to use it you can be very verbose about it
and that way it's much better documented in the code yeah that's great advice so let's maybe talk
about how you solve your problem after you like what was the problem what did you find out the issue
to be and then how do you solve it yeah so what we were doing in this code is we were taking gigabytes
upon gigabytes of plain text data like words um from users of you know various forums and we
got we processed all this data for sentiment analysis and to do that you need to stem each word to its base word
so that way you can have you can make sure you you're analyzing the correct word because there's like five different
forms of every single word right like running ran all those are kind of basically the same meaning right exactly
yeah okay yeah so so we get the base word of all those words and the thing is gigabytes of like let's say
five gigabytes of words if it's if a word is like four bytes you know that's like billions of that's so many
words i don't even want to think about it and then for every single word we stemmed it and we analyzed it and it was
slow arduous process and as i was profiling and i realized we run the same uh stemming stemming function
which is an nltk it's a porter stemmer which is amazing and it's what everyone uses for stemming
we ran it about in my you know 50 megabyte test case which is still so many words thousands upon thousands
of words it ran about like 600 000 times and i was like my my goodness this is this is running way too much
and there's only like 400 000 words in the english language there's no way you know each of these
words is needs to be stemmed because you know gamers on the internet aren't very you know linguistically
amazing yeah exactly so so what we so i figured you know i can create a cache or as it's called you know
in more functional you know terms i can create a memoization algorithm that saves these answers i mean this
saves the computation so i don't need to recompute the function because stemming is a pure function
if you i mean if you're into functional programming you can you don't need to recompute it every single
time you get the same input right if you're guaranteed with the same inputs you get the same output
then you can cash the heck out of this thing right exactly so i i went from about like 60 600 000 calls to
like you know like 30 000 and i was like it was an immediate you know these these words like the whole
pro program ran orders of magnitude faster that's awesome and you know what i think i really love about
this two things i love about it one is you know you talked a little bit about the solution on the
blog and it's like i don't know nine lines of code no yes that's it's python yeah it's so awesome and
the other thing is you didn't even have to change your code necessarily you're able to like create
things that you can just apply to your code you have to re you have to rewrite the algorithms or things
like this right yeah i i really find that you know the algorithm worked i mean it got it got things
done it did it correctly there had to be you know i i mean i wasn't opposed to changing the algorithm
obviously if that was the hot part of the code but once i found out that the hot part of the code wasn't
even code that we wrote you know it was just code that we were calling right from another library and
it's probably really optimized but if you're calling it 600 000 times well nothing is optimized when you're
calling it hundreds of thousands of times you know at that point you got to not call it that many times
within that time span you basically created a decorator that will just cache the output and
only call the function if it's never seen the inputs before right exactly so i mean what it does is it
you know internally a decorator all it does is it wraps the function in another function so it
adds an internal cache which is just a python dictionary which which keeps the function arguments as the key
and the output of the function as the value and if it's been computed then it's in the dictionary so
it all it needs to do is a simple dictionary call you know it's like just like one or two python bytecode
instructions which are i mean as opposed to calling an entire function um itself which would which would
be hundreds of python bytecode instructions right yeah that's fantastic yeah and i'll and when it when it
doesn't find the answer when i mean when it doesn't find the arguments in the function it i mean it's just
one computation and for the amount of words in the english language um that are primarily used it's it'll
be called much less right yeah so your typical data set maybe i don't know 30 000 20 000 times something like that
yeah yeah yeah
this episode is brought to you by code ship code ship has launched organizations create teams set
permissions for specific team members and improve collaboration in your continuous delivery workflow
maintain centralized control over your organization's projects and teams with code ship's new organizations
plan and as talk python listeners you can save 20 off any premium plan for the next three months
just use the code talk python all caps no spaces check them out at code ship.com and tell them thanks
for supporting the show on twitter where they're at code ship
the other yeah the other thing i i like is that the solution was so simple but but you probably needed
the profiling to come up with it right you know so i have an example that i've given a few times about
these types of things and just you know choosing the right data structure can be really important i worked
on this project that did real time processing of data coming in at 250 times a second so that leaves four
milliseconds to process each segment right which is really not much and but it had to be real time and if
it couldn't keep up well then you can't write a real time system or you need some insane hardware to run it or
something right and it was doing crazy math like wavelet decomposition and all sorts of stuff okay this is
like i was saying earlier it's the verge of understanding what we're doing right yeah and it
was too slow the first time we ran like oh no please don't make me try to optimize a wavelet
decomposition you know it's kind of like four year analysis but worse yeah and i'm like there's got to
be a better way right so break out the profiler and it turned out that we had to sort of do lookups
back in the past on our data structures yeah and we happened to use a list to go back and look up
and we were spending 80 of our time just looking for stuff in the list we just switched it to an on
yeah exactly we just switched it to a dictionary and it went five times faster and i mean it was like
almost one line of code change it was like ridiculously simple but if you don't use the
profiler to find the real problem you're going to go muck with your algorithm and not even really
make a difference because it had nothing to do with the algorithm right it was somewhere else
yeah i i definitely find that there's also a lot of a lot more push like in the java world
to make things like final by default to try to make them immutable unless they don't need to be
and a lot of languages are also embracing immutable by default and trying to keep as strict as possible
so that way you can you know be linear more lenient when you need to and i find the same thing in
languages like python whereas i try to use a set before like unless i absolutely need a list if i'm just
containing elements a set is much better for finding things right if you just want to know have i seen
this thing before a set exactly it's maybe the right data structure or if you're going to store
integers in a list you'd be much better off using an array of integers or an array of floats because
it's much much more efficient you said one of the things you considered was py py just for those who
don't know maybe what's the super quick elevator pitch on py py and what would you find out about using it
yeah so py py is a very compliant python interpreter that at runtime turns the python code into machine
code it finds the hot parts of your code or what's being run a lot and it finds a better way to run
that for you so that way it'll um it'll run a lot faster than c python because c python doesn't do many
optimizations by default or just in general right it it runs it through the interpreter which is a fairly
complex thing rather than turn it straight to machine instructions yeah yeah there's a lot more
overhead you tried out py py did it make it faster or did it matter at all oh yeah oh yeah py py is i i
actually use py py before even profiling just to see you know what i could like just because i was like
oh let's just see how faster py py is in this case and it ran about like five times faster um because it
figured out what to do uh but the thing is that under our constraints uh we didn't we wanted to stick
with a little aws instance that we were just running every night and the thing is py py uses more memory
than c python to support its garbage collector and its just-in-time compiler that both need to run at the
runtime so it uses a little bit more memory and we didn't really want to spend that money whereas you know
because if i can get it down to an hour in c python and it runs at 2 a.m no one's going to be looking at
that stuff at 3 a.m right absolutely and if you can optimize it in c python and then feed it into py py
maybe it could go exactly faster still right that that would have run it at about like as opposed to
from 10 hours to one hour it'd be from like if i was running in py py with the cache optimization
it would probably run in like 30 minutes 20 minutes like i said it was unnecessary like it would it would
have been nice and but we didn't need it so we didn't really feel like spending them spending the
time to add that to the pipeline right sure it seems like i mean if you need it to go faster it seems like
you could just keep iterating on this right so for every execution your your decorator will wrap it
for every run of your python app but you know you could actually keep those stemmed words yeah it could
save it like to a redis cache or to a file or to a database or right you could just keep going on this
idea right but yeah that was definitely something i wanted to i thought about doing that yeah and
definitely it was fast enough already and that's that is where we'll go next after that i always find
that you use whenever when you profile and you have a flat profile with nothing sticking out and with
nothing um you know that looks like it needs to be optimized that's when you need to change the runtime
that's when you need to look into ffi or py py or jython with a proper jit interesting what's ffi
so ffi is the foreign function interface it's just the general term used by all languages
in which case you can drop down into c that's right or any c compatible language yeah that's the c so you
would basically go right in c and then hook into it that's right so or use like cython for example
which compiles python down to c with a weirdly python plus c syntax if you have to right yeah i've never
tried it but i've seen it and i'm like you're saying like you annotate um python variables with
c like you say double i equals zero in python syntax that's right it's really strange yeah that is that is
strange uh so you know i talk about PyCharm on the show a lot i'm a big fan of PyCharm
and uh they just added built-in profiling in their latest release 4.5 sounds nice yeah i don't know
if it's out or not it has a visualizer too kind of like py call graph but they said they're using
something called the yappy y-a-p-p-i profiler and it'll fall back to c profile if it has to do you know
yappy i haven't i've not seen this yeah i i was looking at all these profiles uh also i mean py py comes
with some profiler that works on Linux called vm prof um and those are all different profiles and i
looked at them and and they're nice sure but i really loved how simple it was i mean and i got
the results that i needed from c profile and it comes with python it was that you didn't need to
install anything you just ran the module and that's why i was so happy with using it and didn't need to
try a different profiler right that's cool just python space dash m space you know c profile and
then your your app boom exactly and if you're using something like PyCharm i'm sure i mean if it comes
with uh yappy or any other profile it like definitely use that you know i'm sure it's i'm sure it's great
because that PyCharm people are awesome uh but for the purposes i mean of the simplicity that i wanted
keeping the blog post i only used the word pip once i mean like i only used it once to install PyCharm
and that's just how simple it needs to be right you maybe could have gotten away without it if you
really wanted to look at just the text output the other thing is you know this is something you
can do really well on your server right you can go over and profile something or maybe it has real
data it can get to like maybe it works fine on my machine right but not in production or not in qa or
something like and so you don't have to go install PyCharm on the server which exactly probably you
don't want to do on you know sshed into aws yeah and that's why python is amazing for being
batteries included is that it includes all of these nice things that you that you could need that any
pro that any developer was going to need to use eventually do you want to talk a little bit about
open source and what you guys are doing with open source at human geo yeah so human geo we do it we
have a GitHub profile github.com/human geo and we mostly the most of our open source stuff is
the javascript leaflet uh leaflet stuff that we've incorporated we add themes and stuff so if you do any
if you want any map visualizations i like it much better than the google maps api um using leaflet so
i'd recommend looking at the art the stuff we've done there and we also have uh one or a couple of python
libraries uh you know we have a an elastic search binding that we that we've used which has since
been superseded by the mozilla elastic binding so we definitely love open source of human geo
and we make and use libraries in open source and that's one of my favorite parts about human geo
that's really awesome and you guys are hiring right oh yeah we're we're always hiring uh we're looking for
you know great developers at any age python java like i said we won one of the best places to work
for millennials in the dc metro area right and for the international listeners that's washington dc
in the united states yes washington dc yeah no worries yeah whether uh whether you have
a government clearance or not we'd love you know just send send an email or call or get in touch if
you want to work at an amazing place that does awesome python i love their code bases yeah that's
really cool it should definitely reach a bunch of uh enthusiastic python people so if if you're
looking for a job give those guys a ring that's awesome definitely so you guys are big fans of python
right oh yeah we we've i think they i mean they've been using python since the company started and i was
looking at the original commits like when the company started and it was for these projects and they're
all python it's and it's really exciting we've been using since the beginning it's amazing to rapidly
iterate uh it's fast enough obviously and you can uh you can look at it and it's super easy to profile
when you need to that's another reason why it's amazing it's not just like you can say it's slow
but then it's easy to optimize in that case yeah that's really cool so are there some
practices or techniques or whatever that you guys have sort of landed on or focused in on if you've
been doing it for so long that you could recommend at human geo we uh you know we make sure to
i mean we don't go like full agile or anything i mean we definitely consider ourselves fast moving
and we work uh at a very great pace uh we so i guess you could call it agile and then we have
great git practices we use git flow and we make sure to have good code review and any code base including
in python you know you got to have good code review and when you write python code uh i do
write a lot of unit tests for my code things like that nice so is that a unit test module or pipe or um
py test or oh yeah the standard library just i love standard library stuff yeah it's the batteries
included right yes definitely so anything else you want to touch on before we uh kind of wrap things up
davis uh no i was just uh i'm i'm just really excited to be given this opportunity and i wanted
to give a shout out to all my amazing colleagues at the imageo yeah that's awesome you guys are doing
cool stuff so i'm glad to shine a light on that so final two questions before i let you out of here
favorite editor what do you what do you use to write python these days i'll say that for all my open
source like i work on you know a website for the dc python community and i use sublime text for all
that open source stuff all my tools that i use and when i'm given when i work for a company i i ask them
for a PyCharm license yeah nice because it's uh PyCharm is great for big projects that you can really
focus on yeah i you know like i said that's my favorite editor as well and it definitely there's a
definitely a group of people that love the lightweight editors like vim and emacs and sublime
and then people that like ides it's a bit of a divide but i you know i feel like when you're on a huge
project you can just understand it sort of more definitely more in its entirety using something
like PyCharm so yeah i like it my favorite thing about PyCharm is that you can like control click
or command click and you can go like on a module and it'll take you to the source for that module
so you can really fast like look at where the code is flowing um in the in the source yeah absolutely
or hey you're importing a module but it's not specified in the requirements do you want me to add
it for you for this package you're writing right stuff like that it's just it's sweet they have great
support for the tooling and the tool chain of python yeah awesome davis this has been a really
really interesting conversation and hopefully some people can go make their python code faster
yeah i definitely hope that they will and i hope they learned a lot from this yeah thanks for
being on the show man no problem thank you so much yeah talk to you later this has been another
episode of talk python to me today's guest was davis silverman and this episode has been brought to you
by hired and coachip thank you guys for supporting the show hired wants to help you find your next big
thing visit hired.com slash talk python to me to get five or more offers with salary and equity
presented right up front and a special listener signing bonus of four thousand dollars
coachip wants you to always keep shipping check them out at coachip.com and thank them on twitter
via at coachip don't forget the discount code for listeners it's easy talk python all caps no spaces
you can find the links from today's show at talkpython.fm/episodes slash show slash 28
be sure to subscribe to the show open your favorite podcatcher and search for python we should be right at the top
you can also find itunes and direct rss feeds in the footer of the website our theme music is developers
developers developers developers by cory smith who goes by smix you can hear the entire song at talkpython.fm
this is your host michael kennedy thank you very much for listening smix take us out of here
stating with my voice there's no norm that i can feel within haven't been sleeping i've been using
lots of rest i'll pass the mic back to who rocked it best
developers developers developers developers developers developers developers developers developers developers
developers developers developers developers developers developers developers developers