Technology Professionals help OMS customers by improving workflows through configuration and customization of financial software. We use tools like C#, Java, SQL and ETL for customization of various components and configuration system settings via software integration frameworks.
But before I approach you with any more scrabble board tech buzzwords, please indulge me in a side story about a recent trip I took to the Rockies.
It’s a wonderful July 4th summer day here in Wyoming. We have family that has traveled from around the country to celebrate Independence Day. From the ranch we can see Heart Mountain off in the distance summiting the Bighorn Basin. Its majestic peak towers in the distance and the name “Heart” resonates comfort and warmth for everyone who talks about its geological and historical mysteries. The region also has a darker historical significance that is not so warm and comforting. Tens of thousands of Japanese Americans were moved from their homes, mostly from California and held here in internment camps during World War II.
My brother in-law and I are designated grill masters and charged with staging the implements of fire. In no short order we were corralling large wooden handled tongs, giant two-pronged forks, steel bristled brushes, propane tanks and the great fire breathing rack that shall adorn our selections of meats.
“Medium rare please”, “well done for me”, don’t forget the vegetable kababs. We had an audience of 12 to cook for and we were only one part of the production. In the kitchen was corn, potatoes, salad, drinks and eventual deserts. Knowing when to start which cuts of meat and what can be worked simultaneously is key. Timing is everything.
This family scene reminds me of the common challenge that my customers face in getting their systems ready for trading every morning. The morning batch is easy for most, a slight challenge for many and a dreaded nightmare for others. The orchestration of positions and tax lots from accounting along with securities from the security master group and enrichment from different vendors can make your head spin. Timing is everything.
We had a wealth customer recently who was struggling to meet their morning batch SLA. The system is used for managing portfolios with impressive numbers including 400,000 customer accounts, 4 million positions and 8 million tax lots. Numerous indices are loaded for portfolio analysis as well which means updates to tens of thousands of fixed income and equity securities. With various other require data points, the customer is loading 15 million rows of data with interdependencies every morning.
The accounting system would start its update process at midnight ensuring that even the latest of trades would make it in the night before. This was especially important to cover the first few hours of trades executed on the Tokyo market. After several iterations of processing, the accounting system would deliver updated positions around 4am. Import of those positions along with tax lots, securities and indices typically took up to three and a half hours, after which a security enrichment would be initiated.
Assuming everything goes right, the entire batch process would barely end in time for trading to start at 9:30am. Analysts would like to start their research much earlier but cannot use the system until it is done with the morning data import rituals. Occasionally something does go wrong; a head trader wants to get a last minute trade in, the accounting system takes longer than expected, maybe a database backup doesn’t complete in time, or a drive runs out of disk space, etc. There is very little wiggle room in the morning plan, every minute counts, unfortunately the opening of trading is constantly at risk of being delayed. When something does go wrong, there are well established contingency plans put in place so that trading can commence, however the delays look bad on IT and the operations staff.
Each and every step in the complex dance of the morning batch is often critical. From a high level view, the batch may seem over-complicated. But the truth is that each major process has good reason for running. A task like running table statistics is essential but often misunderstood and misused. It is easy to say that we need more speed and throw hardware at the problem. Improved hardware and software is often a valid solution and does indeed help. But the most overlooked aspects of the batch process are the little tricks of the trade that avoid many collective minutes of processing. This includes knowing when to run or avoid timely database table statistics, knowing what tables can be loaded in parallel, and performing certain row validations ahead of time.
Row validations include tying foreign keys from one table to another, such as mapping the security identifier from position and tax-lot tables to valid known securities in the security table. This mapping process is typically done during the import process, however there are opportunities to pre-populate these mappings saving lots of time during the import process.
My customer was struggling to load their data set in a little over three hours. This didn’t include the necessary enrichment of securities from Bloomberg and Reuters. With a few tweaks and a lot of testing, we were able to cut their load times in half. The new 90 minute import cycle meant delivering the trading system to analysts and portfolio managers 90 minutes earlier. This also meant that if something went wrong with the batch, we had an additional 90 minutes of recovery time. The customer was very happy with these results.
In another unique project, we had a customer that was buying mutual funds on a daily basis on behalf of their large client base. Daily infusions of cash came from clients of various amounts and mutual fund purchases would ensue. Mutual funds usually settle in dollar values, however we had an intermediary system that insisted on rounding purchases for specific mutual funds. Conversion of dollars to shares and back always left remainders. These fractions, usually just a few pennies per purchase added up to thousands very quickly and would then need to be managed. The solution was to hold fractions as preliminary orders and merged them into the next day’s trades. This process necessitated custom software to split off orders by the hundreds in a well-organized and trackable manner. Custom applications were written to help Traders mange the process. Processes were put in place to help to manage the work by automating the splitting and merging of allocations into appropriate orders.
As technology professionals in the financial world, we are faced with interesting challenges. As technologist, we thrive on the complexity of the challenges and endeavor to figure out how to meet our customer’s business needs. A few examples include:
- Performance tuning – Identifying the causes for a slow system and develop a strategy for fixing the issues. In our experience, it is unfortunately never a single issue that slows a system, but rather a complex aggregate of challenges.
- FIX Protocol tweaking – Adding tags, repeating tags, broker rules, etc.
- XML message delivery to downstream systems and accounting systems
- ETL for creation of custom indices and aggregating existing indices for roll ups by industry codes
- Streaming prices from an in-house system pricing source – As most prices come from Bloomberg or Reuters, there are times when you may like prices to stream from your own internal ESM.
- SQL optimization in Oracle and MSSQL to improve blotter query and workflow rule performance
Our Wyoming holiday turned out just fine with plenty of food to go around. Despite our best efforts at the grill, the mid-west grown corn was the hit at the party getting all the raves. My brother in-law and I got only a few small praises for our grillwork. At least we didn’t get any complaints. I suppose that is par for the course for a job well done. Much like a morning batch cycle that goes wrong, any burnt steaks would have gotten me in hot water with my customer, my family.
We’re motivated and hungry, bring us your technology challenges
Ron Benchetrit – RBench Software Consulting, LLC