Hadoop Administrator

  • 17 July
  • England - Greater London
  • CDD
 HSBC
GB&M Big Data is a Global Markets and Banking Initiative that is part of the Group Data Strategy to transform the way we govern, manage and use all our data to its full potential across HSBC.

Assets that are being developed as part of GB&M Big Data are being designed to support HSBC at a Group level. These assets include the creation of a "Data Lake" for GBM and CMB, or a single virtual pool of client, transaction, product, instrument, pricing and portfolio data. Supporting the "lake" is a set of operational processes to deliver Big Data as an end-to-end Service: a Data Factory to take in data into the Lake and clean it, and an Analytics Factory staffed with Data Scientists and Analysts to support businesses in turning data into actionable insights

The Hadoop Administrator will be part of a joint venture POD consisting of Group Big Data Services and the GBM Big Data Services team members. The role is responsible for designing, building, managing, monitoring, troubleshooting, tuning, job management, backups, expanding and maintaining GB&M's Hadoop infrastructure. Required characteristics are a team player, with good inter-personal and technical skills, able to communicate and resolve issues quickly.

As a key member of the joint venture POD, you will be expected to participate and contribute to many aspects in maintaining, managing and supporting the GB&M platform:

* Understanding all aspects of the Hortonworks Hadoop ecosystem so that informed choices can be made regarding network and hardware, operating system and Hadoop configuration through to end user tooling and provisioning.
* Implementing and maintaining infrastructure, as designed by the Big Data Architects, in accordance with the group operational and security standards and policies.
* The management of core eco system components HDFS, Hive, HBase and related eco system tools YARN, HQL, Map Reduce, TDE
* Understanding the interaction between the Hadoop eco system and other data sources. For example ensure other systems are not impacted by Hadoop work profiles.
* Ensuring standard operational requirements are met through implementation of infrastructure to perform monitoring, contingency, user provisioning



ESSENTIAL EXPERIENCE

* Experience with most of the following technologies (Hortonworks, Hadoop, Apache Spark (including pyspark), Spark streaming, YARN, Kafka, HDFS, Hive, HBase, Avro, Python, MapReduce, SQL, Impala, Oracle Relational & XML Database, Java, Spring, RESTful services, Falcon, Storm, Nagios, Linux, Redhat, operating system, DBMS
* Hands on experience working with data migration projects and writing and executing complex SQL queries.
* Highly desirable - NoSQL, Cloud, Elastic, Continuum Anaconda experience (Installing and supporting Anaconda Eterprice Notebooks and Repositories).
* Experience with industry standard version control tools (Git, GitHub, Subversion) and automated deployment tools (Ansible & Jenkins)
* Understanding of data modelling techniques using relational and non-relational techniques.
* Coordination between Onsite and Offshore.
* Ability to triage issues, react well to changes, work with teams and ability to multi-task on multiple projects
* Excellent communication, collaboration, reporting, analytical and problem solving skills
* Experience with Defect Management tools (JIRA).
* Experience on Debugging the Code issues and then publishing the highlighted differences to the development team/Architects.
* Knowledge of 3rd party data science tooling products
* Experience working in relevant environments, for example IT or technical environment
* Availability to work the shift pattern required for this role, for example not shift, but out of hours on call work.
* A flexible and adaptable approach to change and will support others to respond in a similar way
* Achieved targets and met expectations whilst supporting and encouraging others to do the same
* Both spoken and written communication skills with experience of adapting your style and approach to the audience and message to be delivered
* Contributed productively to team objectives, sometimes supporting and sometimes leading activities to achieve shared goals
* Planned projects and activities systematically in line with business priorities, using a variety of analysis and problem solving techniques
* Supported project teams and managers, showing an understanding of the basic principles of project management