Trusting Social is looking for a passionate and committed Big Data Engineer who will work on the Data processing platform (Trust-Data) for collecting, storing, processing, and analysing huge data sets. The primary focus will be on designing and developing optimal solutions to launch and use them for the deep analysis by the Data Scientists of Trusting Social. Post the launch the ownership and onus of maintaining, enhancing, and monitoring lies with you. You will also be responsible for integrating them with the architecture used across the company. Why is this job key to Trusting Social - The data sets that are ingested and transformed by the Data engineering team form the crux of the source for the Data scientists at Trusting Social, who would then run their complex algorithms and routines on the quality output enabled by the Trust-Data Engineering team. 

Key Job Responsibilities : 

A Data engineer in TrustinSgocial works on the data pipeline infrastructure that is veritably the backbone of our business. In a day's job you would be writing elegant functional SCALA code to crunch TBs of data on Hadoop clusters, mostly using Spark. In a week's job, you would be owning a data pipeline deployment to clusters : on-prem or AWS or Azure or more. In a month's job, you would be managing Hadoop clusters right from security to reliability to HA. Did we mention we are building a pluggable,unified data-lake from scratch? Time to time you have new challenges for automating and scaling tasks for Data science team. We constantly look to improve our framework and pipelines, hence learning on the job is sort of a given. This is the big picture of our big data system. 

Our expertise and requirements include but are not limited to Spark, Scala, HDFS, Yarn, Hive, Kafka, Distributed Systems, Python, Datastores (Relational and NoSql) and Airflow.