Speed Optimization Expert for Forward Passing Through 200 PyTorch Neural Networks in <20ms, on Linux

Quick Description: I need a professional who would know how to forward pass through 200 already trained PyTorch neural networks, and obtain their outputs, in <20ms, on a provided Linux server. This task is only for the ML inference step, and not for the training stage. Key Requirements: - Expertise in Python and Machine Learning: You should be well-versed in Python and have a strong understanding of machine learning principles. - Previous Experience in Speed Optimization: Experience in optimizing machine learning inference for high-speed applications is crucial. - Proficiency in PyTorch: Experience working with PyTorch specifically is highly preferred, as this is the library we are currently using. - Expertise in Linux - Knowledgeable about how to parallelize computations. Your responsibilities will include: - Implementing necessary software/code changes to reduce execution time to under 20 ms. - Ensuring the accuracy and reliability of the algorithm are not compromised during optimization (for example, don't employ data quantization) If you can deliver high-quality results within a reasonable timeframe, please reach out. And if you have additional clarifying questions and would like more details from me, I'm happy to help. ____________________________________________________________________________________________________ Detailed Description: The ML algorithm that was developed consists of 200 individual pytorch neural networks (i.e. uses python). The goal is to be able to forward pass through these already trained 200 neural networks, and obtain their outputs, in <20 ms. In the zipped file attached, I include the weights of the 200 neural networks, I include the architecture of the neural network, I include the 2D data matrix that is used as input, and I include various example scripts of different things I've tried. I describe the contents in more detail below (See "Zipped file contents"). On my end, I have set up a Linux server on OVH, where all the experiments are to be conducted; this standardizes the hardware, so the emphasis is on software solutions. Upon accepting this project, we will provide the login information required to ssh into this server. In the home directory of "/home/almalinux", you'll find two folders: miniconda3 and PySpeedTest. The miniconda3 is the folder containing the conda version I installed, and is how I am using Python and managing my packages. With miniconda, I created an environment called "ML", which has PyTorch installed, and allows my scripts to run. I recommend that you create your own new environment if you wish to try additional things out. The PySpeedTest folder is the unzipped folder which is attached to this post, and has all the data and example scripts, if you wish to play around with them. Zipped file contents (in the file PySpeedTest): - Long: Folder that contains 100 of the total 200 neural networks - Short: Folder that contains the other 100 of the total 200 neural networks - [login to view URL]: Pytorch architecture of each of the 200 neural networks. - test_data: numpy array of size 2000 by 22, that represents the input data, for each NN - [login to view URL] - Easiest example to start with; shows how to load the test data and the neural networks. Then, it times how long it takes to forward pass through the 200 NNs in sequence, i.e. back to back. To run in command line: python [login to view URL] - The rest of the python scripts are the various experiments I personally tried, via various multithreading and multiprocessing techniques, to parallelize the computations in an effort to reduce the runtime. The time benchmarks I achieved are: a) Sequential: 94ms b) HighLevelMultiThread (5 threads): 67ms c) LowLevelMultiThread (5 threads):~60-65 ms d) HighLevelMultiProcess (max_workers ? chunksize ?): Didn't stop running for a while. Didn't let it time out. Tried a few different combos of max_workers and chunksize. e) LowLevelMultiProcess (num_cores=4): Didn't stop running for a while. Didn't let it time out. f) MultiProcessThroughBashSpeedTestScript (4 processes): 160 - 500ms. Processes start at the same time, but finish at different times. Core issues I've ran into: 1) Windows versus Linux issue. On my own windows server with identical hardware to the linux server, I achieved the same speeds as on the linux server; BUT all 3 of the multiprocessing scripts (d, e, f) actually worked, where (d) had a runtime of 2.2 seconds, and (e, f) had a runtime of ~47ms. It is only on the Linux server, that 2/3 multiprocessing scripts (d and e) take very long to run, while 1/3 (f) runs significantly slower than the windows server. Seems to be some OS related issue that hampers the multi-processing on the linux server.