preview

Symphony Driver Essay

Satisfactory Essays
Symphony Driver

The Driver is the main project of the Symphony framework that contains the entry point - main method, for a symphony job to get executed. This is also the core engine of the framework where the orchestration of the entire job takes place. The driver needs a workflow to execute the job which is defined in the form of a JSON file. The JSON contains 3 process sequences namely pre-processing, job-processing and post-processing that are executed in sequence by Symphony as part of the workflow execution.

Overview: How driver works?
 The driver’s main class WorkflowApp is invoked by a Spark-submit command with 2 arguments: job instance JSON and environment properties.
 The driver then parses the first argument i.e. job
…show more content…
In addition to the above conventional logging, Symphony also logs the critical job updates to a centralized logging platform called Common Logging Environment (CLE). This platform provides a user interface that facilitates the user to monitor the workflow execution in real-time and debug the jobs. For more details on Common Logging Environment (CLE), please visit: https://pulse.kdc.capitalone.com/docs/DOC-189214.

Password Management – Chamber of Secrets (CoS):
The driver module is required to utilize services like Nebula and Common Logging Environment (CLE) which needs a batch id and password. Prior to the integration of CoS with Symphony, passwords were stored on the EMR using “ftppwd” utility, which are not secure. Symphony has incorporated Capital One's enterprise-wide credential management solution - Chamber of Secrets, to make the password storage and retrieval more secure. Please visit this pulse page for more details on Chamber of Secrets: https://pulse.kdc.capitalone.com/docs/DOC-184992.

Class Diagrams:
The opening section of this page provides a high-level overview of how the driver works. Now let us dive into how a workflow is executed in the driver module.

As mentioned previously, the workflow contains 3 process sequences: pre-processing, job processing and post-processing and each process sequence can be
Get Access