Benchmarking async vs. thread in #python

This compares the performance of different approaches to retrieve documents from a web server. Web server is an nginx (more data below) serving a simple 40bytes document.

The approaches we are interested in are:

thread version
async version
multiple async workers version

as a point of reference we include apaches ab tool which does the same.

Results

	client	total	#/s	ms/req<sup>1</sup>	ms/req<sup>2</sup>
apache	`ab`	0.352s	3014.13	6.6	0.33
python	threads	8.00s	125.07	160	8.0
python	async	2.62s	381.76	-	2.6
python	async-workers	2.15s	465.32	43	2.1

Server info

Server Software:        nginx/1.10.3
Server Hostname:        192.168.23.4
Server Port:            88
Document Path:          /
Document Length:        40 bytes

Apache bench

   Requests per second (#/s)
          This is the number of requests per second. 
          This value is the result of dividing the 
          number of requests by the total time taken

   Time per request (ms/req)
          The  average  time  spent  per request. 
          The first value is calculated with the 
          formula: 
              concurrency * timetaken * 1000 / done 
          while the second value is calculated with 
          the formula:
              timetaken * 1000 / done

without keep-alive

Client: ab -c 20 -n 1000 192.168.23.4:88:

Concurrency Level:      20
Time taken for tests:   0.332 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      271000 bytes
HTML transferred:       40000 bytes
Requests per second:    3014.13 [#/sec] (mean)
Time per request:       6.635 [ms] (mean)
Time per request:       0.332 [ms] (mean, across all concurrent requests)
Transfer rate:          797.68 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    3   0.6      3       4
Processing:     1    4   0.8      4       5
Waiting:        1    3   0.8      3       4
Total:          3    6   1.1      7       8

Python

thread version

# multiple_sync_request_threaded.py
import threading  
import requests
import time

def make_requests(session, n, url):  
    for i in range(n):
        resp = session.get(url)
        if resp.status_code == 200:
            pass

def main():  
    n_threads = 20
    n_requests = 1000
    n_requests_per_thread = n_requests // n_threads

    url = "192.168.23.4:88"
    session = requests.Session()

    threads = [
        threading.Thread(
            target=make_requests,
            args=(session, n_requests_per_thread, url)
        ) for i in range(n_threads)
    ]

    start = time.time()
    for t in threads:
        t.start()
    for t in threads:
        t.join()
    end = time.time()
    total = end-start
    print("took %02.02f seconds" % (total))
    print("%02.02f requests x second" % (n_requests*1.0/total))
    print("average requests time %02.04f s" % (total*1.0/n_requests))
main()

aweil@pizarnik:~/client-bench$ python3 threads.py
took 6.74 seconds
148.38 requests x second
average requests time 0.0067 s

async version

# multiple_async_requests.py
import asyncio  
import aiohttp
import time

async def make_request(session, req_n):  
    url = "192.168.23.4:88"
    async with session.get(url) as resp:
        if resp.status == 200:
            await resp.text()

async def main():  
    n_requests = 1000
    start = time.time()
    async with aiohttp.ClientSession() as session:
        await asyncio.gather(
            *[make_request(session, i) for i in range(n_requests)]
        )
    end = time.time()
    total = end-start
    print("took %02.02f seconds" % (total))
    print("%02.02f requests x second" % (n_requests*1.0/total))
    print("average requests time %02.04f s" % (total*1.0/n_requests))

loop = asyncio.get_event_loop()  
loop.run_until_complete(main())

aweil@pizarnik:~/client-bench$ python3 async.py
took 2.62 seconds
381.76 requests x second
average requests time 0.0026 s

async-workers

No es la mejor version.. no usa una cola asincronica sino mas bien una cantidad de concurrencia fija.. o sea, imitando la version de threads, y para poder comparar contra la version de apache que tiene limite de concurrencia fijo..

import asyncio  
import aiohttp
import time

async def make_request(session, qty_req, req_n):  
    url = "192.168.23.4:88"
    for idx in range(qty_req):
        async with session.get(url) as resp:
            if resp.status == 200:
                await resp.text()


async def main():  
    n_threads = 20
    n_requests = 1000
    start = time.time()
    async with aiohttp.ClientSession() as session:
        await asyncio.gather(
            *[make_request(session, int(n_requests/n_threads), i) for i in range(n_threads)]
        )
    end = time.time()
    total = end-start
    print("took %02.02f seconds" % (total))
    print("%02.02f requests x second" % (n_requests*1.0/total))
    print("average requests time %02.04f s" % (total*1.0/n_requests))
    print("|python|async-workers| %02.02f s | %02.02f |  %02.04f s| %02.04f s| (units in seconds here)" % (
        total,
        n_requests*1.0/total,
        n_threads*total*1.0/n_requests,
        total*1.0/n_requests
    ))

loop = asyncio.get_event_loop()  
loop.run_until_complete(main())

$ python3 async-queue.py
took 2.15 seconds
465.32 requests x second
average requests time 0.0021 s
|python|async-workers| 2.15 s | 465.32 |  0.0430 s| 0.0021 s| (units in seconds here)

Benchmarking async vs. thread in #python

Results

Server info

Apache bench

without keep-alive

Python

thread version

async version

async-workers

To do and notes

To do

Notes

Product

Explore

Company

Blogs

Support