Falls Church, VA
Follow us

Streamlining FDIC Data Operations via an enterprise Data Orchestration Platform

KEY SCOPE AREAS

  • Develop an enterprise Data Orchestration solution that includes data pipelines for various FDIC customers.
  • Modernize data planning for short-, medium-, and long-term efforts.
  • Acquire, extract, transform, load, and publish datasets in a hybrid cloud environment, with a particular focus on Azure and Oracle ecosystems.
  • Document designs, technical requirements, and standard operating procedures.
 

KEY STATS

  • First team to design, build, and maintain a hosting and data orchestration services platform in the FDIC Azure Cloud – the Data Orchestration Platform (DOP).
  • Transform a proof-of-concept protype into a formal FDIC “Board Case” system. DOP is now a fully funded Mission Essential capability within FDIC.
  • Mature the DOP through its infancy to its first ever Authority to Operate (ATO) on Azure Government.

KC Customer in Focus

The Business Intelligence Services Section (BISS) of the FDIC contracted with Karthik Consulting (KC) to plan, design, develop, and deploy the DOP for short-, medium-, and long-term modernization efforts. By moving FDIC data into the Azure Data Lake within the DOP, the FDIC prepares its data for advanced analytics and new technologies using Bronze, Silver, and Gold Data Lake structures, each improving data quality and structure. KC addressed the need to implement hybrid cloud solutions while migrating applications and data to Azure. KC developed processes to acquire, extract, transform, load, and publish datasets on Azure from Oracle ecosystems. KC’s DOP solution plays a key role in migrating FDIC applications and data, enhancing resiliency, flexibility, and scalability

Significance of Requirements

The FDIC’s mission requires a variety of applications to serve a broad consumer base. To streamline operations and reduce costs, FDIC is migrating Mission Essential / Mission Critical (ME/MC) applications and on-premises data into Azure Cloud services, improving data quality and availability for advanced analytics. KC implemented a tailored DOP to handle FDIC’s diverse datasets, such as Web, Oracle DB, SQL Server, and File/Blob storage, on Azure Cloud. This innovative solution helps FDIC reduce costs and automate business processes and workflows.

Delivered Results

KC delivered solutions to the FDIC that aligned with the client’s Performance Goals, streamlining business operations.

Image link

1. Data Orchestration/Azure Solution

  • KC developed and maintains the DOP, which hosts multiple ME/MC applications with full disaster recovery functionality.
  • The solution achieved ATO in 4 months, meeting key FDIC Performance Goals (FPGs) and Division of Information Technology (DIT) milestones.
  • As customer requirements evolved, KC added the Azure Data Lake Hydration Service, leveraging Azure Data Factory (ADF) to support a hybrid cloud environment.
    KC provides expertise to enhance DOP capabilities through development, testing, and deployment, enabling the FDIC to efficiently onboard diverse datasets.
    KC delivers Extract, Transform, and Load (ETL) services to migrate data from on-premise systems to Azure Data Lake within DOP.

2. Data Modernization

  • KC’s ADF-based Data Orchestration strategy exceeded FDIC’s needs by identifying automation opportunities throughout the data management lifecycle.
  • Use data enrichment and transformation tools like Databricks and Azure Functions to prepare data for advanced processing and predictive analytics.
  • Treat each FDIC request as a separate project, defining a tailored individual approach, from planning through implementation.

3. Management and Support

  • Utilize Azure DevOps (AzDO) for management and Continuous Integration/Continuous Delivery (CI/CD) to streamline onboarding new datasets into DOP.
  • Use standardized agile Product Backlog Items and tasks to guide the team through project phases, refining processes and documentation after each project.
  • Create a Wiki Governance framework and over 400 Standard Operating Procedures (SOPs), serving as a gold standard for deploying ME/MC applications onto DOP. These resources act as a blueprint for structuring use cases.
  • Continue to add engineering documentation, including requirements, designs, architectures, and workflows, streamlining the onboarding of new ME/MC applications and datasets.

Image link
Image link
Image link
Image link
Image link
Image link
Image link

Key Highlights of KC Impact

  • Extracted ~1TB of data via 5 dynamic ADF pipelines to pull 50 different tables from on-prem Oracle DW.
  • Extracted incremental data via ADF pipelines from 750 on-prem SQL Server tables.
  • Built Dynamic ADF pipelines to connect to Economy.com Moody’s data buffet via API, ordered and extracted 19 different data sets.
  • Built Dynamic ADF pipelines to connect to GFIEF public dataset via API and extracted 3 different golden copy data sets.
  • Spawned dynamic ADF pipelines to process each FRB dataset (containing 200-300 files each).
  • Built ADF pipelines that load and process data into Azure Data Lake Storage (ADLS) Gen2 daily.
  • Reduced costs of FDIC’s Qualified Financial Contracts (QFC) Database by 50%.
  • Led initiatives that onboarded 3 ME/MC applications and over 7 datasets onto DOP.
  • Currently averaging eight (8) production software releases a month compared to 4 releases in 2023, when DOP was first established; a 2400% increase in release velocity.
  • Within one year KC has become the team with the 2nd highest number of published software package releases across FDIC.

Other Case Studies

Take a look at what KC is doing to contribute to global cyber security, agile software development and cloud services.

ABOUT US

Karthik Consulting was founded in 2008 to be a reliable and trusted advisor for our customers, providing independent, unbiased, and proven solutions that mitigate risk and help solve enterprise-wide IT challenges.

Our Cyber Security, Software Development and Program Management focus areas (and work methodology) ensure that we can deliver not just solutions, but architecture that scales and grows with the customer's needs over time. We are able to assist in projects ranging from short advisory engagements to assembling a full team to deliver a solution from concept through implementation and on-going management. KC has access to industry experts in various technologies and teaming partners to meet any of your IT challenges.

The vision of KC is to bring the innovation, passion and agility of the commercial IT industry to meet the unique challenges of the federal government. We are a DOD Cleared Facility with a DCAA-approved accounting system.
CONTACT

Felix Martin, 571 435 7632 fmartin@karthikconsulting.com

CAGE: 56GH3
DUNS: 828199880 UEI: FGNNM7KNUPF6

PRIME CONTRACT VEHICLES:

GSA MAS
GSA OASIS Pool 1 and 3
NIH CIO-SP3 8(a) & SB
GSA STARS III 8(a)
Air Force SBEAS
Army RS3
Navy Seaport-NexGen 
FAA eFAST