Sign in
Log inSign up

Benchmarking async vs. thread in #python

david weil's photo
david weil
·May 24, 2019

This compares the performance of different approaches to retrieve documents from a web server. Web server is an nginx (more data below) serving a simple 40bytes document.

The approaches we are interested in are:

  • thread version
  • async version
  • multiple async workers version

as a point of reference we include apaches ab tool which does the same.

Results

clienttotal#/sms/req<sup>1</sup>ms/req<sup>2</sup>
apacheab0.352s3014.136.60.33
pythonthreads8.00s125.071608.0
pythonasync2.62s381.76-2.6
pythonasync-workers2.15s465.32432.1

<!-- |apache| `ab -k` | 0.153s|7660.96| 2.6|0.13| -->

Server info

Server Software:        nginx/1.10.3
Server Hostname:        192.168.23.4
Server Port:            88
Document Path:          /
Document Length:        40 bytes

Apache bench

   Requests per second (#/s)
          This is the number of requests per second. 
          This value is the result of dividing the 
          number of requests by the total time taken

   Time per request (ms/req)
          The  average  time  spent  per request. 
          The first value is calculated with the 
          formula: 
              concurrency * timetaken * 1000 / done 
          while the second value is calculated with 
          the formula:
              timetaken * 1000 / done

<!-- ### using keep-alive Client: `ab -k -c 20 -n 1000 192.168.23.4:88` ````console Concurrency Level: 20 Time taken for tests: 0.131 seconds Complete requests: 1000 Failed requests: 0 Keep-Alive requests: 994 Total transferred: 275970 bytes HTML transferred: 40000 bytes Requests per second: 7660.96 [#/sec] (mean) Time per request: 2.611 [ms] (mean) Time per request: 0.131 [ms] (mean, across all concurrent requests) Transfer rate: 2064.64 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.2 0 2 Processing: 0 3 9.7 1 78 Waiting: 0 2 9.7 1 78 Total: 0 3 9.9 1 79 ```` -->

without keep-alive

Client: ab -c 20 -n 1000 192.168.23.4:88:

Concurrency Level:      20
Time taken for tests:   0.332 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      271000 bytes
HTML transferred:       40000 bytes
Requests per second:    3014.13 [#/sec] (mean)
Time per request:       6.635 [ms] (mean)
Time per request:       0.332 [ms] (mean, across all concurrent requests)
Transfer rate:          797.68 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    3   0.6      3       4
Processing:     1    4   0.8      4       5
Waiting:        1    3   0.8      3       4
Total:          3    6   1.1      7       8

Python

thread version

# multiple_sync_request_threaded.py
import threading  
import requests
import time

def make_requests(session, n, url):  
    for i in range(n):
        resp = session.get(url)
        if resp.status_code == 200:
            pass

def main():  
    n_threads = 20
    n_requests = 1000
    n_requests_per_thread = n_requests // n_threads

    url = "192.168.23.4:88"
    session = requests.Session()

    threads = [
        threading.Thread(
            target=make_requests,
            args=(session, n_requests_per_thread, url)
        ) for i in range(n_threads)
    ]

    start = time.time()
    for t in threads:
        t.start()
    for t in threads:
        t.join()
    end = time.time()
    total = end-start
    print("took %02.02f seconds" % (total))
    print("%02.02f requests x second" % (n_requests*1.0/total))
    print("average requests time %02.04f s" % (total*1.0/n_requests))
main()
aweil@pizarnik:~/client-bench$ python3 threads.py
took 6.74 seconds
148.38 requests x second
average requests time 0.0067 s

async version

# multiple_async_requests.py
import asyncio  
import aiohttp
import time

async def make_request(session, req_n):  
    url = "192.168.23.4:88"
    async with session.get(url) as resp:
        if resp.status == 200:
            await resp.text()

async def main():  
    n_requests = 1000
    start = time.time()
    async with aiohttp.ClientSession() as session:
        await asyncio.gather(
            *[make_request(session, i) for i in range(n_requests)]
        )
    end = time.time()
    total = end-start
    print("took %02.02f seconds" % (total))
    print("%02.02f requests x second" % (n_requests*1.0/total))
    print("average requests time %02.04f s" % (total*1.0/n_requests))

loop = asyncio.get_event_loop()  
loop.run_until_complete(main())
aweil@pizarnik:~/client-bench$ python3 async.py
took 2.62 seconds
381.76 requests x second
average requests time 0.0026 s

async-workers

No es la mejor version.. no usa una cola asincronica sino mas bien una cantidad de concurrencia fija.. o sea, imitando la version de threads, y para poder comparar contra la version de apache que tiene limite de concurrencia fijo..

import asyncio  
import aiohttp
import time

async def make_request(session, qty_req, req_n):  
    url = "192.168.23.4:88"
    for idx in range(qty_req):
        async with session.get(url) as resp:
            if resp.status == 200:
                await resp.text()


async def main():  
    n_threads = 20
    n_requests = 1000
    start = time.time()
    async with aiohttp.ClientSession() as session:
        await asyncio.gather(
            *[make_request(session, int(n_requests/n_threads), i) for i in range(n_threads)]
        )
    end = time.time()
    total = end-start
    print("took %02.02f seconds" % (total))
    print("%02.02f requests x second" % (n_requests*1.0/total))
    print("average requests time %02.04f s" % (total*1.0/n_requests))
    print("|python|async-workers| %02.02f s | %02.02f |  %02.04f s| %02.04f s| (units in seconds here)" % (
        total,
        n_requests*1.0/total,
        n_threads*total*1.0/n_requests,
        total*1.0/n_requests
    ))

loop = asyncio.get_event_loop()  
loop.run_until_complete(main())
$ python3 async-queue.py
took 2.15 seconds
465.32 requests x second
average requests time 0.0021 s
|python|async-workers| 2.15 s | 465.32 |  0.0430 s| 0.0021 s| (units in seconds here)

To do and notes

To do

  • make python tests use keep-alive connections

Notes