Is it just performance that puts you off Python? If so, did you try writing a native extension to accelerate it?
Where i work, we also have analysis work which involves sequentially reading gigabytes of binary data. We came up with a workflow where a tool written in Java runs a daily job to parse the raw data and write a simplified, fixed-width binary version of it, then we have a small Python C extension which can read that format. We can push a bit of filtering down into the C as well.
This has worked out pretty well! We get to write all the tricky parsing in a fast, typesafe language, we get to write the ever-changing analytics scripts in Python, and we only had to write a small amount of C, which is mostly I/O and VM API glue.
The Java and C parts were both written by an intern in a month or two. He's a sharp guy, admittedly, but if an intern can do it, i bet an HFT developer could too.
>Is it just performance that puts you off Python? If so, did you try writing a native extension to accelerate it?
This is what I did originally, but it was way slower to have to write and maintain C++ and a Python interface for it than to just write Julia. Particularly because the some of the business/trading logic basically has to be in the native layer (can't backtest a HFT algo without running it on HFT data, and that volume of high frequency tick data is too slow to process in pure Python).
Where i work, we also have analysis work which involves sequentially reading gigabytes of binary data. We came up with a workflow where a tool written in Java runs a daily job to parse the raw data and write a simplified, fixed-width binary version of it, then we have a small Python C extension which can read that format. We can push a bit of filtering down into the C as well.
This has worked out pretty well! We get to write all the tricky parsing in a fast, typesafe language, we get to write the ever-changing analytics scripts in Python, and we only had to write a small amount of C, which is mostly I/O and VM API glue.
The Java and C parts were both written by an intern in a month or two. He's a sharp guy, admittedly, but if an intern can do it, i bet an HFT developer could too.