Harish Kesava Rao

Principal Data Engineer & Data Architect @ Atlassian
Previously: Data Engineering @ Databricks, Amazon, Indeed, Informatica

Hello, I am a Principal Data Engineer & Data Architect for the Data Engineering organization at Altassian.

My most recent role, prior to Atlassian’s was Staff Software Engineer and Data Engineering Team Lead for Security at Databricks, where I was involved in overseeing the execution of multi-quarter Data Engineering projects, including design, automation and scaling of multi-TB/day data in AWS and Azure.

Earlier, I was in Senior Data Engineering(Individual Contributor) roles at Salesforce’s Tableau, Amazon’s Prime Video and Indeed.com. In Amazon, I was hired as the first Data Engineer to help build a Data Lake for Prime Video Search. At Indeed, I worked for Marketing, Sales and Finance to help build their datawarehouses.

Prior to Indeed, I was in various roles are Informatica, focusing mainly on building ETL pipelines between relational databases and MPP databases.

My interests and key areas of work include:

• Understanding short-term & long-term data engineering requirements, converting them into high-level data architecture roadmaps for execution.
• Designing, building and scaling big data pipelines to ingest large volume data via Spark and AWS, Azure. I also have experience deploying big data pipelines on-premise.
• Performance tuning for high volume ingestion pipelines with SparkStreaming / AWS Kinesis, Azure Eventhubs on Databricks.
• Automation of workflows (such as Apache Airflow and other equivalent tooling on the cloud), automated infrastructure management on the cloud using Terraform (HCL).
• Experience in configuring and deploying Docker containers for specialized use cases.
• Extensive experience in writing Data Engineering code (primarily Python and Scala), adhering to good software engineering practices.
• I also contribute to Open Source projects on a regular basis. I developed the first DAG Sensors for the Databricks Provider in Apache Airflow.