Concentrix is seeking a highly skilled Data Engineer to be part of a team of Senior Data Engineers to implement a cutting-edge Azure Databricks solution focused on automating tax reporting. This role will involve managing the ingestion of daily data feeds and producing outputs on the scale of billions of forms per client, ensuring high performance and scalability. The position is full-time permanent, and you will report to our regional lead.
You will:
- Successfully execute data engineering projects.
- Design and implement advanced data solutions utilizing Azure Databricks to automate tax reporting processes.
- Manage and optimize the ingestion and transformation of daily data feeds, ensuring seamless data flow and integration.
- Develop and maintain scalable data pipelines and architectures for high-volume data processing.
- Collaborate with cross-functional teams to ensure alignment and integration with broader IT and business strategies.
- Drive the use of best practices in data engineering, leveraging modern tools and technologies.
What you need to have is:
- Senior with 8+ years of relevant experience
- Advanced Python knowledge and software engineering skills.
- Proven experience with Spark and SQL for data engineering and analysis.
- Experience with Databricks for data ingestion and transformation.
- Hands-on experience with one or more cloud services Azure.
- Experience with Azure Data Factory, Azure Cosmos DB, Azure functions, Event store, ADLS, Key Vault.
- Experience with OpenAI and Fabric
- Proficiency in RDBMS/NoSQL data stores and appropriate use cases.
- Experience with Data as Code; version control, small and regular commits, unit tests, CI/CD, packaging, familiarity with containerization tools such as Docker (must have) and Kubernetes (plus).
- Good communication skills, fluent English
CallCenterNinja