1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
|
From 6e49c314cd5050763f983759d156f26ffbc59368 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?= <mgorny@gentoo.org>
Date: Sat, 7 May 2022 23:38:31 +0200
Subject: [PATCH] Allow referencing thread identifier in `compiledir_format`
Add a new `thread_id` key that permits referencing the current thread ID
(`threading.get_ident()`) in `compiledir_format`. This makes it
possible to use per-thread compiledirs in multithreaded programs,
and therefore make it possible for aesara to run multiple compiler
instances in parallel.
Combined with pytest-xdist, this change makes it possible to run
aesara's tests in parallel and better utilize multiprocessor (multicore)
systems. On my Ryzen 5, this provides around 4x speedup (down from 82
minutes to 20 minutes per test run). Given that we are running the test
suite multiple times to test different Python versions, this literally
saves hours at a time.
Prior to this change, xdist would not be able to fully utilize
the available resources since the compiledir lock would prevent aesara
from starting multiple compiler instances in parallel, and therefore
practically serialize tests on compiler invocations.
---
aesara/configdefaults.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/aesara/configdefaults.py b/aesara/configdefaults.py
index 8121919958..60eb4f997e 100644
--- a/aesara/configdefaults.py
+++ b/aesara/configdefaults.py
@@ -7,6 +7,7 @@
import socket
import sys
import textwrap
+import threading
import numpy as np
@@ -1345,6 +1346,7 @@ def _get_home_dir():
"numpy_version": np.__version__,
"gxx_version": "xxx",
"hostname": socket.gethostname(),
+ "thread_id": threading.get_ident(),
}
|
