As an Engineer — Optimize Python performance
閱讀時間約 12 分鐘
When I was doing the development of the company’s data dashboard, I often pulled tables from various databases for calculation. API response time decreases when the pull time range is wide or when there are multiple tables. This post documents what I’m trying to optimize the process and other tricks that can improve program performance.
1. Change data storage structure
In addition to data types such as str and int, common Python data storage structures include list, tuple, and dict.
List should be a data structure that anyone who has learned any programming languages will be familiar with. A list is an array. Assume that there are a lot of data stored in the list today, and I need to retrieve specific data, and I have to search all over, as follows.
for i in someList:
print(i)
Time complexity is O(n).
If your data storage today is two-dimensional, such as user data in different countries. The first thing you need to do is to find the country, and then find the data you want, as follows:
for i in someList:
if i == someCountry:
for j in someList[i]:
print(j)
This time yime complexity up to O(n²).
Can tuples vs lists improve performance? As far as I know, there is no clear indication that tuples reduce time complexity. Because the principle of tuple is list, but tuple is a list whose length cannot be changed. When he first creates it, he requests a fixed length from memory. In addition to needing space for initialization and memory, List also needs a temporary storage area for future expansion of data. Therefore, in a tuple of the same data size, the required storage space is smaller than that of a list, and the space complexity is also lower.
Next, I will introduce a powerful function of python, which is dict. dict is short for dictionary. Since it is called a dictionary, there is a relationship between key and value. That is, the value is stored in a unique key. So if you want to query a specific information today, it is as follows:
print(someDict["yourKey"])
Time complexity is O(1).
Even with the example above, the data storage by country is as follows:
print(someDict["someCountry"]["yourKey"])
Time complexity still is O(1).
So in terms of data storage, I prefer Dict > Tuple > List
2. Use Redis
Redis — Powerful caching database. If your data doesn’t need to be updated immediately. (Not the Monitor). Redis is used in many places. When an API pulls a Table from DB to use, other APIs may also need it. At this time, if you have to re-enter SQL or NoSQL commands, the performance will be wasted. If we pull out the data of this table and temporarily store it in Redis, it will take about ten minutes. During these ten minutes, if your other API needs to use the same data, there is no need to re-download the SQL and DB request data. Requesting data from Redis is much faster.
Alternatively, return from your API can also be temporarily stored in Redis. Sometimes the user may refresh the page and resend the request. If these requests need to be recalculated in the background and re-request data from DB. This is very wasteful of performance. If you have saved the returned data in Redis at this time, you can return the result directly without any calculation.
3. Asyn functions for FastAPI
In the FastAPI framework, there is a very novel function, which is Async. Basically equivalent to Node.js’ async. Async allows you to wait for response times, temporarily store state, release thread, and handle other needs.
async def someFunc():
await something
Remember that await can only be used in async functions. I’ve tested it myself so far. If your function does not need the await object, it is recommended to remove async. No need to wait for processing to remove async functions for better performance.
4. Use multiprocessing
The literal meaning of Multiprocessing is easy to understand, that is, multiple process handle different tasks separately. It has been discussed on the Internet that multiprocessing is more efficient than multithreading in python. So far, I haven’t tested it myself, so I can’t draw conclusions. This article first introduces the two multiprocessing methods I currently use.
from multiprocessing import Pool
with Pool(processes=2) as p:
print(p.map(someFunc, [1, 2])
Pool can call some functions in parallel through the map function. Remember that the function must be written in the outermost layer.
from multiprocessing import Process
p = Process(target=someFunc, args=(1,2))
p.start()
#
# The original process continues to run
#
# Wait for p process done
p.join()
The difference between the Process method and the Pool is that after the Process is created, the original Process will continue to execute the program. Pool means that after the original process enters the pool, it will be divided into several processes according to the number of tasks.
Have fun~
為什麼會看到廣告
留言0
查看全部
你可能也想看
Google News 追蹤
這個秋，Chill 嗨嗨！穿搭美美去賞楓，裝備款款去露營⋯⋯你的秋天怎麼過？秋日 To Do List 等你分享！
秋季全站徵文，我們準備了五個創作主題，參賽還有機會獲得「火烤兩用鍋」，一起來看看如何參加吧～
11/20日NVDA即將公布最新一期的財報, 今天Sell Side的分析師, 開始調高目標價, 市場的股價也開始反應, 未來一週NVDA將重新回到美股市場的焦點, 今天我們要分析NVDA Sell Side怎麼看待這次NVDA的財報預測, 以及實際上Buy Side的倉位及操作, 從
Hi 大家好，我是Ethan😊
相近大家都知道保濕是皮膚保養中最基本，也是最重要的一步。無論是在畫室裡長時間對著畫布，還是在旅途中面對各種氣候變化，保持皮膚的水分平衡對我來說至關重要。保濕化妝水不僅能迅速為皮膚補水，還能提升後續保養品的吸收效率。
曾經，我的保養程序簡單到只包括清潔和隨意上乳液
可能包含敏感內容
比照各自的性癖好，猜想一副手銬也許會是一份皆大歡喜的交往紀念禮物，於是在春末的深夜，下了網購訂單；然而怎麼也沒料到，本該春宵盡歡的那晚，卻要從小北百貨開始......
是情慾，更是自我定義的探索。
After hosting English speaking groups with different people for the past 3 months, I found some simple tips to share with you guys!
本文主要在討論以 Docs as Code 方法來撰寫技術文件，此做法能否滿足企業內部對知識管理的需求。
神啊，我的心切慕你，
shén a, wǒ de xīnqiè mù nǐ
如鹿切慕溪水。
rú lù qiè mù xīshuǐ
惟有你是我心所愛，
wéiyǒu nǐ shì wǒ xīn suǒ ài
我渴慕來敬拜你。
wǒ kěmù lái jìng bài nǐ
你是我的力量盾牌，
小時候耳熟能詳的的童話長大後紛紛變身，加入更多新世紀元素，來吸引日益刁鑽的大小讀者眼球。
於是「格林童話」”糖果屋”裡的Hansel and Gretel一下子化身電影裡的女巫獵人；一下子又變成2021年動畫“神秘魔法部（Secret Magic Control Agency）”裡的搞笑探員。
去年在科技成長股板塊，有兩隻股票非常受到觸目。一隻是Upstart(UPST)，另一隻是Asana(ASAN)，全年股價都漲非常多，但現在都來回天堂又折返人間。去年年初時候UPST股價約50塊左右，到年底最高漲到400塊，漲幅700%；ASAN也不遑多讓，去年年初時候股價30塊，到年底......
「溝通的認知前提」很重要，用「不能理解」形容會比「不知道」好很多，比認為「對方是故意的」好更多。溝通是雙向的，代表不僅是神經人要懂AS的世界，AS也要學習神經星球的運作方式，「平等互相」才是關係經營之道。因此預設讀者為兩個世界的人都可以看。
這個秋，Chill 嗨嗨！穿搭美美去賞楓，裝備款款去露營⋯⋯你的秋天怎麼過？秋日 To Do List 等你分享！
秋季全站徵文，我們準備了五個創作主題，參賽還有機會獲得「火烤兩用鍋」，一起來看看如何參加吧～
11/20日NVDA即將公布最新一期的財報, 今天Sell Side的分析師, 開始調高目標價, 市場的股價也開始反應, 未來一週NVDA將重新回到美股市場的焦點, 今天我們要分析NVDA Sell Side怎麼看待這次NVDA的財報預測, 以及實際上Buy Side的倉位及操作, 從
Hi 大家好，我是Ethan😊
相近大家都知道保濕是皮膚保養中最基本，也是最重要的一步。無論是在畫室裡長時間對著畫布，還是在旅途中面對各種氣候變化，保持皮膚的水分平衡對我來說至關重要。保濕化妝水不僅能迅速為皮膚補水，還能提升後續保養品的吸收效率。
曾經，我的保養程序簡單到只包括清潔和隨意上乳液
可能包含敏感內容
比照各自的性癖好，猜想一副手銬也許會是一份皆大歡喜的交往紀念禮物，於是在春末的深夜，下了網購訂單；然而怎麼也沒料到，本該春宵盡歡的那晚，卻要從小北百貨開始......
是情慾，更是自我定義的探索。
After hosting English speaking groups with different people for the past 3 months, I found some simple tips to share with you guys!
本文主要在討論以 Docs as Code 方法來撰寫技術文件，此做法能否滿足企業內部對知識管理的需求。
神啊，我的心切慕你，
shén a, wǒ de xīnqiè mù nǐ
如鹿切慕溪水。
rú lù qiè mù xīshuǐ
惟有你是我心所愛，
wéiyǒu nǐ shì wǒ xīn suǒ ài
我渴慕來敬拜你。
wǒ kěmù lái jìng bài nǐ
你是我的力量盾牌，
小時候耳熟能詳的的童話長大後紛紛變身，加入更多新世紀元素，來吸引日益刁鑽的大小讀者眼球。
於是「格林童話」”糖果屋”裡的Hansel and Gretel一下子化身電影裡的女巫獵人；一下子又變成2021年動畫“神秘魔法部（Secret Magic Control Agency）”裡的搞笑探員。
去年在科技成長股板塊，有兩隻股票非常受到觸目。一隻是Upstart(UPST)，另一隻是Asana(ASAN)，全年股價都漲非常多，但現在都來回天堂又折返人間。去年年初時候UPST股價約50塊左右，到年底最高漲到400塊，漲幅700%；ASAN也不遑多讓，去年年初時候股價30塊，到年底......
「溝通的認知前提」很重要，用「不能理解」形容會比「不知道」好很多，比認為「對方是故意的」好更多。溝通是雙向的，代表不僅是神經人要懂AS的世界，AS也要學習神經星球的運作方式，「平等互相」才是關係經營之道。因此預設讀者為兩個世界的人都可以看。