Software Tech Enterprises Inc. is seeking a highly skilled technology consultant in the area of Hadoop administration and processing. The individual will work closely with our customer's development teams to create, manage, upgrade, and secure Hadoop clusters. Key success metrics for this individual include: 1) proven experience with automation, 2) advanced Linux / Windows system administration capabilities, 3) ability to thrive within a dynamic technology environment, and 4) successful demonstration of leadership qualities.
Main responsibilities may include:
Create Hadoop ecosystem (Hadoop, Hive, Pig, Oozie, Hue, Hbase/Cassandra, Flume) using both automated toolsets as well as manual processes.
Identify, assess, and recommend appropriate solutions to advise customer on cluster requirements and any limitations by applying industry best-practices and expertise regarding emerging technologies, risk mitigation, and continuity planning to address back-up and recovery.
Provide enterprise-level information technology recommendations and solutions in support of customer requirements.
Use customer defined data sources and prototype processes to satisfy proof of concept.
Develop design patterns for specific data processing jobs.
Maintain, support, and upgrade Hadoop clusters.
Monitor jobs, queues, and HDFS capacity.
Balance, commission & decommission cluster nodes.
Apply security (Kerberos / Open LDAP) linking with Active Directory and/or LDAP.
Enable users to view job progress via web interface.
On boarding users to use Hadoop – configuration, access control, disk quota, permissions etc.
Address all issues, apply upgrades and security patches.
Commission/de-commission nodes backup and restore.
Apply "rolling" cluster node upgrades in a Production-level environment.
Assemble newly bought hardware into racks with switches, assign IP addresses properly, firewalling, enable/disable ports, VPN etc.
Work with virtualization team to provision / manage HDP cluster components.
- Minimum 5 years Linux/Unix System Administrator experience
- Minimum 3 years' experience with Hadoop Administration in the area of setup, configuration, or management (Cloudera/Hortonworks)
- Minimum 11 years in Networking
- Extensive experience in Hadoop ecosystem including Spark, MapReduce, HDFS, Hive, HBase, and Zeppelin.
- 1 year experience with Hadoop-specific automation (e.g. blueprints).
- 1 year technical experience managing Hadoop cluster infrastructure environments (e.g. data center infrastructure).
- Demonstrable scripting experience in one or more of Python, bash, PowerShell, Perl.
- 1 year experience with Puppet and / or Chef.
- 1 year virtualization experience in any of VMware / Hyper-V / KVM.
- Kerberos and Active Directory
- Identity Manager
- Apache Sentry
- Data Encryption
- Certified Hadoop Admin (Cloudera/Hortonworks).
- Certified Linux System Administrator (e.g. Red Hat Linux)
- Networking (TCP/IP, Routers, IP addressing, use of network tools).
- Analyzing data with Hive, Pig and/or HBase.
- Data Ingestion, streaming, or Importing/exporting RDBMS data using Sqoop.
- DBA experience.
- RDBMS SQL Development.
- Hadoop experience with the setup, configuration, or management of a multi-node (50 to 100) Hadoop cluster, specifically working with Cloudera's Hadoop distribution
- Experience with Kerberos and Securing Hadoop Clusters
- Experience with monitoring concepts and tools in a multi-platform data center environment
- Experience with systems monitoring tools (Nagios), in helping tune, configure, and administer cluster components parameters like CPU, I/O load, memory usage and network topology/bandwidth for identified workloads.
- Experience with performance tuning of applications in a distributed cluster environment
- Ability to investigate/research issues, determine impact, and provide solutions.
- Strong verbal and written communication skills are critical. Candidate must be able to effectively interact with all levels of customer staff, and be able to flourish in a government contract setting.
- Ability to clearly present information, verbally and in writing, to audiences of diverse technical backgrounds.
Certified Linux System Administrator (e.g. Red Hat Linux)
Bachelor's degree from an accredited college in a related discipline, or equivalent experience/combined education, with 11 years of professional experience.
Software Tech Enterprises offers a very competitive salary and benefits package that includes medical, dental, vision, life, STD, LTD, paid vacation and personal days, 401(k) and an EAP program.
To learn more about Software Tech Enterprises, visit us at www.software-tec.com.