since my first time to dabble at python, i began to keep an eye on its performance in multi-core enviornment. As everyone knows, python adapt GIL to prevent thread from interrupting the data.
the pros is learning and using thread become easy and error-less, the downside is you are unable to take any benefit from using multi-core/cpu. A few years ago, i thought probably the next generation of python (next generation of python refers to python 3000 in context) may remove this bar. However, it is not true.
There are few ways of bypassing the GIL and taking benefit from multi-core, after years study and search, i conclude some ways which are listed as follow:
1. use OS-related API to write your code where you need multi-thread, and wrap it as module. the downside is the development speed is slowly and error-prone. using os api let cross-platform become more difficult.
2. use multi-process instead of multi-thread. If you live in Linux, you will be fine with this choice. The python provide os.fork for multi-process usage, furthmore, you may also use a package called pyprocessing. The usage is similar with threading package. The author claims that the package can be used in both of *nix and win32. However, after a few days of use in win32, i found out that it probably was not a feasible choice. The restrictions of pyprocess in win32 are more tense than Linux, the argument, shared object, or data put in a Queue should be pickable (io serialization), that imply some un-plain objects like com-object, proxy-object can not be passed or communicate interprocess. It quickly become a harassment. I personally don't recommend use this package in production development.
3. it is said that use twisted and prospective broker can take benefit from multi-core, however, i havent' gave it a try. I found someone reported that the use of multi-core in twisted is boloney.
If your app is related to server, it is not a bad choice
4. the final choice is to use jython or ironPython instead, these versions of python are based on VM/CLR which supports multi-thread naturally.
分享到:
相关推荐
Multi-core processors are considered now the only feasible alternative to the large single-core processors which have become limited by technological aspects such as power consumption and heat ...
learning, multi-task active learning, multi-task reinforcement learning, multi-task online learning and multi-task multi-view learning. For each setting, representative MTL models are presented. In ...
Assuming you have some iterable of transactions (which are themselves iterables of items) called `transactions` and an integer minimum support value `minsup`, you can find the frequent itemsets in ...
Some thoughts on differential equation notation - functional vs classical.pdf
Recognizing arbitrary multi-character text in unconstrained natural photographs is a hard problem. In this paper, we address an equally hard sub-...and in some cases exceeds, that of human operators.
This book introduces some machine learning approaches about multi-agent learning. There are a number of algorithms that are typically used for system identification, adaptive control, adaptive signal ...
together and so on. But the computing time of all these methods are too long to solve large scale problem. In this paper SVMs architectures for multi-class problems are discussed, in particular we ...
We exploit some of the advanced features of MPI and CUDA programming to overlap both GPU data transfer and MPI communications with computations on the GPU. We sustain approximately 2.4 TeraFLOPS on ...
He has presented at the last four PyCon AU conferences, at multiple international research conferences, and has been training in some capacity for five years. He has a PhD in cybercrime analytics ...
It is shown that, the noise can make the particles escape over the fluctuating potential barrier in some circumstances; but in other circumstances, it can not. If the noise can make the particle ...
This book is for Data Science practitioners as well as aspirants who have a basic foundational understanding of Machine Learning concepts and some programming experience with Python. A mathematical ...
before and just want to add Python to your list of languages, Introducing Python will get you started. It’s an unhurried introduction, taking small steps from the basics to more involved and varied ...
Therefore, MRSim was created primarily to allow users to develop multi-robot applications, which would benefit working with some specific requirements such as multi-hop communication. Moreover, most...
Some of the python tutorial - 《Python学习笔记》
Central Gateway Architecture A new Network Protocol: FlexRay A new Architecture: Backbone ...Dynamic Multi-PDU-to-Frame Mapping Ethernet and SOME/IP Testing and Analyzing Dynamic PDUs Summary
Get a comprehensive, in-depth introduction to the core Python language with this hands-on book. Based on author Mark Lutz’s popular training course, this updated fifth edition will help you quickly ...
I had been doing a series of articles on my blog called PyDev of the Week that inspired us into crafting a book based on interviewing core members of the Python community. I spent some time hashing ...
python-redis-lock 多个redis客户端访问同一个redis服务端,控制并发。 github:https://pypi.org/project/python-redis-lock/ 在使用这个库之前,需要安装如下: pip install python-redis-lock 使用锁的示例: ...
Python 深度学习实战教程 The field of Artificial Intelligence (AI), which can definitely be considered to be the parent field of deep learning, has a rich history going back to 1950. While we will not ...
Some Topics on Monte Carlo and Quasi-Monte Carlo Methods (PhD Thesis)