Parallelising your Python Code

So your friend suggests that you and they take turns digging…Let's say it takes you 100 minutes to finish this task when done turn-wise.

Illustration designed by Macrovector / FreepikBut, What if you both could hustle together and that would get the job done simultaneously-Illustration designed by Macrovector / FreepikIt would result in -Less timeMore efficiencyNo wasting of available resourcesLess sweat!Same happens for your computers, sequentially giving CPU time to 2 processes will basically make the process-in-waiting starve till given a CPU cycle, which is a waste of possible to-use resources.

This post is about how to overcome that.

Let’s Dig in to Parallel programming!Parallel computing/programming is essentially the use of ≥ 2 processors/cores/computers in combination to solve a single problem.

It is a type of computing architecture in which several processors compute simultaneously by dividing the workload between processors.

How to have your code to work parallelly for you!In python, we have a *GIL-led* monster — Literally called the ‘GIL’ {which stands for Global Interpreter Lock }.

It is — a mutex { MUTual EXclusion } that protects access to Python objects simultaneously by the same thread.

It also prevents multiple threads from executing Python bytecodes at once.

‘Multiprocessing’ → Our rescuer from GIL!The multiprocessing package in python is similar to the threading module; The ‘Pool’ class is an epic example that offers really convenient ways to parallelise the execution of a function across various input values.

The Pool class will allow us to create a pool of processes/workers which can handle tasks given to them simultaneously.

For example, a function is taken- let’s say we distribute the calls of that function across all/some of the cores/processors we have available and each of the processors will be responsible for some part of the subset of problems we throw at it, and we can have another function which will run all of these things serially and finally compare how multiprocessing is helpful in processing big chunks of IO intensive computation.

> To check the number of CPU cores you can use the following:Now Let’s get cod-inSimply using a for-loop to loop through all the values given by the user.

In this following gist, we see that it is possible to simply pass the same function into the ‘.

map’ method that makes it all an easy-peasy-cake-walk!Note – in ‘Pool()’ an optional argument lets us define a number of processors we want our program to use, default is the maximum.

Yet another Note— To check out the number of CPUs available using this little codeThere are other awesome methods to map, check out the official documentation of multiprocessing package in python.

Yet another shameless Note – You must’ve noticed when trying out the code above, non-parallel execution gives better results for small number of input whereas, for a large number of input, parallel is the way to go!Do test this out and comment — at what number of values does parallelism perform better than concurrency!Hope this was an interesting read enough for you to dig-into to gold-mine of multiprocessing!!That’s it!.You are good to go with parallel programming.

If you have some questions please feel free to tweet them at me.

Follow me to stay updated with my posts.

Have a great day!.????.

. More details

Leave a Reply