DataBricks Engineer Job at LatentView Analytics, San Jose, CA

MmlYZGFuV2QwdlBBV1BReHNidk1xc0oyaUE9PQ==
  • LatentView Analytics
  • San Jose, CA

Job Description

LatentView Analytics is a leading global analytics and decision sciences provider, delivering solutions that help companies drive digital transformation and use data to gain a competitive advantage. With analytics solutions that provide a 360-degree view of the digital consumer, fuel machine learning capabilities and support artificial intelligence initiatives., LatentView Analytics enables leading global brands to predict new revenue streams, anticipate product trends and popularity, improve customer retention rates, optimize investment decisions, and turn unstructured data into valuable business assets.

Description:

We are seeking a talented and passionate Databricks Engineer with 6+ years of experience to join our growing data engineering team. The resource must act as a bridge between the offshore and the stakeholders. In this role, he/she must help architect, implement, and manage cloud-based data solutions using Databricks and associated technologies (Azure, AWS). The ideal candidate will have experience in developing, optimizing, and maintaining data pipelines, analytics workflows, and data lakes, leveraging Databricks to drive business insights.

Responsibilities:

Data Pipeline Development: Build and maintain scalable and optimized data pipelines using Apache Spark, Databricks, and other cloud-based tools to ensure efficient data flow across systems.

Cloud Infrastructure: Work with cloud providers (Azure, AWS) to design and implement cloud-native solutions for data storage, processing, and analytics. Experience with Databricks in a cloud-based environment is essential.

Collaboration with Data Scientists/Analysts: Collaborate with data scientists, analysts, and business stakeholders to transform business requirements into data solutions and deliver meaningful insights.

Optimization: Continuously optimize Spark and Databricks workflows for performance and cost efficiency.

ETL & Data Integration: Implement ETL processes to integrate data from various sources (SQL, NoSQL, REST APIs) into Databricks environments and data lakes.

Data Governance: Ensure data security, privacy, and compliance policies are followed, including managing data access, monitoring usage, and implementing data lineage.

Monitoring & Troubleshooting: Proactively monitor and troubleshoot data pipelines, performance bottlenecks, and integration issues.

Documentation & Best Practices: Document solutions, architectures, and workflows while enforcing coding best practices within the team.

Skills:

  • Hands-on experience with Databricks (Apache Spark) for large-scale data processing and analytics. Strong experience working with cloud platforms such as Azure or AWS (Azure Databricks is highly preferred). Proficiency in SQL, Python, Scala, or Java for data processing and automation. Familiarity with data storage solutions like Delta Lake, Data Lakes, Azure Data Lake Storage, or AWS S3.
  • Proficient in building data pipelines and using tools such as Apache Spark, Databricks, and Airflow.
  • Experience with DevOps and CI/CD processes for data workflows. Familiarity with containerization technologies (e.g., Docker, Kubernetes) is a plus.
  • Familiarity with machine learning pipelines and frameworks (e.g., MLflow, TensorFlow, scikit-learn) is a plus.
  • Strong problem-solving skills and the ability to work independently as well as collaboratively.
  • Excellent communication skills and the ability to articulate complex technical solutions to non-technical stakeholders.

Job Tags

Offshore,

Similar Jobs

State of Illinois

HUMAN RIGHTS INVESTIGATOR TRAINEE Job at State of Illinois

 ...Job Requisition ID:46281 Closing Date: 05/21/2025Agency: Department of Human Rights Class Title: HUMAN RIGHTS INVESTIGATOR TRNEE - 19768 Skill Option: None Bilingual Option: None Salary: Anticipated Starting Salary $4,412 monthly; Full Range: $... 

Spencer Dental & Braces - a Benevis company

Endodontist Job at Spencer Dental & Braces - a Benevis company

 .... Our Offer: Competitive Compensation Never any Lab Fees 401(k) Retirement Plan with Company match Flexible Work Schedule: Offering part-time and moonlighting roles with variable schedule choices, perfect for professionals available for... 

GradImages

Graduation Event Photographer - Mobile Job at GradImages

 ...Are you looking for a flexible part-time job that allows you to earn extra income? GradImages is seeking photographers to capture moments during graduation events in and around Mobile, AL. If you have reliable transportation, a flexible schedule, and a dark-colored business... 

Wood Violet Recovery

Clinical Supervisor - LPC, LCSW, LMT, ICS, SAC, CSAC Job at Wood Violet Recovery

 ...Provide supervision to clinical team and sign off on Clinical Intern/Trainee hours as needed for State/Federal/Private licensing....  ...active mental health professional licensure in state of Wisconsin (LPC, LCSW, LMT, Psychologist) with knowledge of addiction treatment.... 

Medix™

Medical Scribe - 238593 Job at Medix™

 ...Compliance with OSHA and HIPAA are a must as well as the adherence to emergency and safety protocol. Qualifications: Medical scribe experience in Retina or Ophthalmology OR Ophthalmic Technician experience Employees that feel comfortable traveling with doctor...