Senior Data Engineer

• 7+ years of experience in data engineering and cloud data architecture, with hands-on leadership in project delivery.

• Proven expertise in AWS (Glue, Lambda, S3, EventBridge, Batch, SQS) and Azure (Data Factory, Databricks, Synapse, Event Grid, ADLS).

• Advanced proficiency in PySpark, Delta Lake, and large-scale data processing frameworks.

• Strong skills in Python, SQL, and shell scripting for ETL, automation, and data validation.

• Experience designing end-to-end data solutions for analytics, ML, and AI applications.

• Deep understanding of data governance, metadata management, and compliance (HIPAA, GDPR).

• Strong grasp of serverless, event-driven, and microservices-based data architectures.

• Exposure to graph databases (Neo4j, Cosmos DB Graph).

• Expertise in data security, IAM, and encryption standards across AWS and Azure.

• Experience with CI/CD pipelines and Infrastructure as Code (CloudFormation, CDK, Terraform, Azure DevOps).

• Excellent communication and leadership skills with the ability to collaborate across technical and nontechnical teams.

• Proven ability to work with sales and business teams to define and deliver client-focused technical solutions.

• Demonstrated success in building and leading data engineering teams.

• Proven experience in establishing organizational data capability, defining processes, and overseeing execution across multiple projects.

Preferred Qualifications

• Master’s degree in Computer Science, Data Engineering, or a related field.

• Certifications such as AWS Data Analytics Specialty, Azure Data Engineer Associate, or Databricks Certified Data Engineer.

• Experience in regulated industries (e.g., healthcare, finance, or government).

• Experience in building organizational data frameworks, capability development, and process standardization.

Leadership & Solution Ownership

• Design and deliver comprehensive data solutions across AWS and Azure environments — _from ingestion and transformation to storage, governance, and analytics.

• Lead and guide data engineering teams to implement scalable, maintainable, and secure data architectures.

• Collaborate with business leaders to translate complex business problems into actionable data solutions.

• Drive architectural decisions that ensure performance, scalability, and reliability in enterprise data systems.

• Provide strategic direction in adopting modern tools, frameworks, and practices across the data ecosystem.

• Partner with sales and pre-sales teams to design and present technical solutions to clients, supporting proposals, RFPs, and client demos.

• Participate in proof-of-concept (POC) initiatives to validate technical feasibility and demonstrate business value.

• Build and implement organizational data engineering capability, including frameworks, best practices, and reusable components for scalability and consistency.

• Oversee execution of multiple data engineering related initiatives, ensuring technical alignment, on-time delivery, and adherence to quality and compliance standards.

• Mentor and upskill team members by planning and executing structured training programs for junior engineers, fostering a culture of learning, innovation, and technical excellence.

Core Engineering Responsibilities

• Build & Maintain Pipelines: Develop and optimize ETL/ELT pipelines using AWS Glue, Lambda, Batch, PySpark, Azure Data Factory, Synapse, and Databricks.

• Delta Lake Architecture: Implement multi-layer Delta Lake (Bronze, Silver, Gold) on AWS S3 and Azure ADLS with strong governance and lifecycle management.

• Data Transformation: Design and run high-performance data ingestion and transformation workflows using PySpark on Glue, EMR, and Databricks.

• Metadata & Governance: Manage metadata and data governance with AWS Glue Catalog and Azure Purview to ensure traceability, compliance, and security.

• Multi-Source Ingestion: Integrate data from APIs, flat files, relational and NoSQL databases, and onprem systems using AWS DataSync, Transfer Family, and Azure Data Factory connectors.

• Workflow Orchestration: Build event-driven and scheduled workflows with AWS EventBridge, Step Functions, SQS, Azure Event Grid, and Logic Apps.

• Graph Data Support: Collaborate with teams to support graph data integrations (Neo4j, Cosmos DB Graph) for AI and knowledge graph use cases

. • Data Quality & Security: Implement strong DQ checks, encryption, IAM, and access controls (AWS KMS, CloudTrail, Azure Key Vault, Defender for Cloud).

• CI/CD & IaC: Automate data infrastructure deployment using CloudFormation, CDK, Terraform, and Azure DevOps pipelines.

• Monitoring & Optimization: Continuously monitor and fine-tune data pipelines for performance, costefficiency, and reliability. 

We are looking for a senior data engineering leader with strong expertise in AWS and Azure, capable of building scalable, secure data pipelines and leading teams to deliver enterprise-grade data solutions.