The technology that’s in place right now is 20 years old, and SCORE is state-of-the-art. It’s going to be H2O on a Java process (instead of an old-school Windows DLL) and it’s going to use machine learning instead of a rules based engine, along with a GUI in a browser that’s friendlier to work with. It’s also hugely faster and more accurate than what’s gone on before.

The other side of it is: we’re solving a very human problem. We’re finding people who are attempting to cheat (or successfully cheating) the stock markets today. There’s a steady stream of people who are attempting to defraud people out of the markets and who are by and large successful because the tools for detecting them are really old. Plus, they know how to work around the tools and they do so on a regular basis.

So as soon as we turn on SCORE in a new trading house, we immediately find people who have been cheating for a long time. It’s very obvious that they’re cheating as soon as you look at the visualisations that are coming out of the tool. We are catching people who are doing big dollar cheating.

I took the existing prior toolchain on SCORE and threw out the database (MariaDB and Hadoop)-- I’m running on a single structured file system with a single JVM process running H2O. This combination goes a long way to solve problems that stack up to tens of terabytes (although not to petabytes and beyond).

There’s a GUI with Elm - which will be covered during the talk - but the key takeaway is that using a single JVM on a single machine with a structured file system can scale to handling terabytes worth of data.

I’ll say that constant shrinking of memory cells and the constant increase in size of memory on a single node means that a lot more problems don’t need clustering in order to be solved. Originally I used H2O for its potential clustering ability, but unless you’re Google you can solve a lot of problems on a single fat node. In addition, nodes are getting fatter faster than the problems are getting larger. So today I can buy a 512Gb machine and tomorrow I can buy a 1Tb machine, which is sufficient to solve most problems.

There’s a huge market opportunity for Tb big data problems using non-clustered technology. Hadoop is solution for a giant filesystem or giant mapreduce. Since the disks are getting so big I don’t need a giant filesystem to be able to store the data in a distributed fashion for a lot of problems.