Apache: Big Data 2016

When building a data management platform, flexible and effective workflows are key to the scalability and effectiveness of the platform.

OODT (originally developed by NASA JPL) has a very flexible and powerful workflow engine and is at the core of pretty much any data processing you will do within the platform but understanding it can sometimes be a challenge.

In this talk we’ll take a deep dive into guts of workflows inside OODT using CAS PGE to help lower the barrier for entry. We’ll run through a number of real world examples. How you build them, how you deploy and trigger them.

We’ll also look at monitoring and feedback. Lastly we’ll tackle resource management and how you make sure your workflows run in the correct server pool, without swamping your resources.