Asynchronous Financial Data processing using F#

Hi everyone,

I was looking at a book to find some algorithm to detect clusters within financial data. I managed to find a decent algorithm for that matter, but I then wanted to test it on some real data.

Hence, I had to get the data from somewhere and I chose to use Yahoo! Finance. The thing is, I wanted to do it in an automatized way in order to be able to re-use the module for further projects.

I remembered Luca Bolognese’s presentation of F# a few years back (certainly one of the best presentations I’ve ever seen) and I thought it was a good time to give it a try.

The idea is quite simple: you can import data from Yahoo! Finance by parsing the CSV files the website produces. These files are actually generated automatically, which means that you can access them by querying the right URL with the right parameters.

The following module shows how to generate the right query to get the CSV file from Yahoo! Finance:

<br />
module QueryModule<br />
open System</p>
<p>let dateToList (d:DateTime) = [d.Month;d.Day;d.Year]</p>
<p>let parameters =['a';'b';'c';'d';'e';'f']</p>
<p>let parametersToString (d1:DateTime) (d2:DateTime)=<br />
    let datesList = (dateToList d1) @ (dateToList d2)<br />
    let rec innerFunc (dList: int list) pars =<br />
        match pars with<br />
        | [] -&gt; &quot;&quot;<br />
        | x::[] -&gt; x.ToString() + &quot;=&quot; + dList.Head.ToString()<br />
        | x::y -&gt; x.ToString() + &quot;=&quot; + dList.Head.ToString() + &quot;&amp;&quot; + (innerFunc dList.Tail y)</p>
<p>    innerFunc datesList parameters</p>
<p>let getFileUrl (ticker:string) start stop =<br />
    &quot;http://ichart.finance.yahoo.com/table.csv?s=&quot;+ticker+&quot;&amp;&quot;+(parametersToString start stop)+&quot;&amp;ignore=.csv&quot;<br />

The idea is now to write a module to download this data. This would be pretty straightforward and I wouldn’t make you waste your time by reading further code right here.

What I’m trying to do here is to download these files asynchronously. Doing this using C# is possible but requires some heavy coding and the result will most likely contain some bugs. F# (and functional programming languages in general) allows you to do it much more easily.

Let’s see how to implement an asynchronous function in F# then:

<br />
module DownloadModule<br />
open QueryModule<br />
open Microsoft.FSharp.Control.WebExtensions<br />
open System<br />
open System.Net<br />
open System.IO</p>
<p>let parseData (rawData:string) =<br />
    rawData.Split('\n')<br />
    |&gt; Array.toList<br />
    |&gt; List.tail<br />
    |&gt; List.map (fun x -&gt; x.Split(','))<br />
    |&gt; List.filter (fun x -&gt; x.Length = 7)<br />
    |&gt; List.map (fun x -&gt; (Convert.ToDateTime(x.[0]),float x.[6]))</p>
<p>let getCSV ticker dStart dEnd =<br />
    async   {<br />
            let query = getFileUrl ticker dStart dEnd<br />
            let req = WebRequest.Create(query)<br />
            use! resp = req.AsyncGetResponse()<br />
            use stream= resp.GetResponseStream()<br />
            use reader = new StreamReader(stream)<br />
            let content = reader.ReadToEnd()<br />
            let ts = parseData content<br />
            return ts<br />
            }<br />

The whole trick in this code resides in the “async” block and the “use!” keyword. Basically, “use!” tells to F# not to wait for the result, that is, to proceed asynchronously if possible.

You know can run this function in parallel to download multiple ticker as follows:

<br />
let testPrices=<br />
    [&quot;MSFT&quot;;&quot;YHOO&quot;]<br />
    |&gt;List.map (fun x -&gt; getCSV x (DateTime.Parse(&quot;01.01.2000&quot;)) (DateTime.Parse(&quot;01.01.2010&quot;)))<br />
    |&gt; Async.Parallel<br />
    |&gt; Async.RunSynchronously;;<br />

This works fine and it’s much faster than it would have been if the list was processed sequentially.

I pushed the example a little further by thinking: “What if I want to do some operation on the time series now?”.

Well, I could wait until the parallel execution is done and the run the following function to get the returns synchronously:

<br />
module AnalysisModule<br />
open System</p>
<p>let getReturns (prices:(DateTime *float)list) =<br />
    [for i in 1..(prices.Length-1) -&gt; i]<br />
    |&gt; List.map (fun i -&gt;(fst (List.nth prices i), (snd (List.nth prices i))/(snd (List.nth prices (i-1) )) - 1.0))</p>
<p>

That wouldn’t be optimal; once the download of MSFT is done, I don’t need to wait until the download of YHOO is done as well before starting to process the returns…

So, how can I do this easily, without modifying any of the previous function I wrote.

<br />
let testReturns =<br />
    [&quot;MSFT&quot;;&quot;YHOO&quot;]<br />
    |&gt; List.map (fun ticker -&gt; async {<br />
                        let! prices = getCSV ticker (DateTime.Parse(&quot;01.01.2000&quot;)) (DateTime.Parse(&quot;01.01.2010&quot;))<br />
                        return getReturns prices<br />
                   })<br />
    |&gt;Async.Parallel<br />
    |&gt;Async.RunSynchronously;;<br />

This way, everything is computed in parallel and I have a really great performance!

Hope you enjoyed this little demo!

See you next time!

3 thoughts on “Asynchronous Financial Data processing using F#”

  1. Great script, very useful and fast! One small oversight, though. Yahoo! Finance’s URL parameters take the month as an integer between 0 and 11 as opposed to 1 through 12, so you need to change

    let dateToList (d:DateTime) = [d.Month;d.Day;d.Year]

    to

    let dateToList (d:DateTime) = [d.Month-1;d.Day;d.Year]

    in line 4 of your first code block.

Leave a Reply