Role Summary We are seeking a mid to senior Splunk Data Administrator to own and continuously improve Splunk data onboarding, normalization, and quality across a complex hybrid Splunk environment (on‑prem and cloud). The ideal candidate is hands-on with CIM alignment, data source onboarding, field extractions (regex/props/transforms/ingest actions), TA deployment, and end-to-end operational management of Splunk data pipelines. You will act as the key point of contact for ensuring log sources are onboarded correctly, parsed and normalized consistently, and made usable for security/IT operations, dashboards, correlation searches, and reporting. Splunk: - Good understanding of Splunk architecture and its components (Search Heads, Indexers, Deployers). - Experience in managing and troubleshooting Splunk distributed environments (clusters), Splunk upgrade and migration . Operating Systems & Cloud Platforms: - Expertise in Linux systems, specifically RHEL and Amazon Linux. - Experience with AWS services, including EC2, S3, IAM, VPC, Subnets, Security Groups and CloudWatch. DevOps & Automation Tools: - Experience with Jenkins pipelines and CI/CD processes, Ansible for configuration management and automation, Terraform for infrastructure provisioning. - Ability to write custom Ansible playbooks and Terraform modules for system management and scripting languages like Bash, Python, or Shell for automation tasks. Certifications (Optional): - Splunk Certified Admin. - AWS Certified Solutions Architect – Associate or Professional Required Skills & Experience • 5–10 years experience with Splunk administration and data onboarding (or equivalent depth). • Strong practical knowledge of: - CIM normalization, tags/eventtypes, datamodel alignment - Field extraction (regex, JSON/KV extraction), and troubleshooting parsing issues - props.conf / transforms.conf, sourcetypes, timestamps, line-breaking - TA installation/configuration and deployment patterns across Splunk tiers • Experience with complex Splunk architectures: - Indexer clusters, SH/SHC, forwarder management, deployment server - Hybrid patterns (on-prem cloud), connectivity, and ingestion strategies • Comfortable writing and validating SPL for data quality and CIM compliance. • Strong log source knowledge across common domains: - Security: EDR, firewall, proxy, IAM/auth, VPN, email security - Infrastructure: Windows, Linux, network devices, virtualization - Cloud: AWS/Azure/GCP logging patterns (nice-to-have) Key Responsibilities Data Onboarding & Lifecycle Management • Lead onboarding of new log sources end-to-end: requirements gathering, source validation, parsing strategy, TA selection/deployment, CIM alignment, testing, and release. • Partner with Security/IT teams to translate use-cases into data requirements, ensuring sources deliver the right fidelity, timeliness, and coverage. • Manage onboarding at scale using best practices for source types, metadata strategy, index & sourcetype governance, and naming conventions. • Define and enforce data quality standards (field completeness, timestamps, event consistency, parsing accuracy, duplication control). CIM Normalization & Data Modelling • Normalize data to Splunk Common Information Model (CIM) with strong understanding of data models (e.g., Authentication, Network Traffic, Endpoint, Change, etc.). • Ensure fields are aligned to CIM requirements to support Splunk Enterprise Security (ES) and other CIM-based content. • Validate normalization using SPL and develop reusable onboarding checklists. Field Extraction, Parsing & Enrichment • Design and implement robust field extractions using: - props.conf / transforms.conf, REPORT/TRANSFORMS stanzas - regex and structured parsing (KV_MODE, JSON, XML) - ingest-time vs search-time extraction strategy - sourcetype / timestamp / line breaking configuration • Implement enrichment and routing using event breaking, host/source normalization, lookups, and tagging. • Troubleshoot parsing issues (timestamp drift, multi-line events, encoding, truncation, duplicate ingestion, broken extractions). TA Installation & Configuration (Complex / Hybrid) • Install, configure, and maintain Splunk Add-ons (TAs) and apps across: - Heavy Forwarders / Universal Forwarders - Indexers / Search Heads / SHC - Deployment Server / Cluster Manager (where applicable) • Maintain version compatibility and upgrade strategies for: - Splunk Enterprise / Splunk Cloud - Add-ons, apps, and content packs • Package and deploy TAs using deployment pipelines and change management controls. • Ensure fields are aligned to CIM requirements Hybrid Splunk Architecture Operations • Operate and support Splunk in complex environments: - On-prem Indexer Cluster, Search Head Cluster, Forwarder tiers - Splunk Cloud integrations where applicable (e.g., Heavy Forwarder, VPN, PrivateLink, data forwarding patterns) • Configure and troubleshoot data ingestion pipelines: - Syslog (UDP/TCP), API-based collection, HEC, file monitors, Windows Event Logs, cloud sources • Ensure performance and reliability across the pipeline, including indexing throughput, parsing overhead, and search impact. Monitoring, Troubleshooting & Governance • Monitor ingestion health and pipeline performance: - Forwarder health, queue saturation, parsing/indexing delays, dropped events • Maintain governance for indexes, sourcetypes, retention, RBAC and data access boundaries (as required). • Contribute to operational runbooks, SOPs, and documentation; drive continuous improvement in onboarding and normalization standards. Preferred / Nice-to-Have • Experience with Splunk Enterprise Security (ES) and ES add-ons / CIM compliance expectations. • Knowledge of Splunk Ingest Actions / Edge Processor (or modern ingestion tools, where applicable). • Familiarity with: - HEC, API ingestion, message queues - ITSI / Observability (bonus) • Splunk certifications (preferred): - Splunk Core Certified Power User / Admin - Splunk Enterprise Certified Admin - Splunk ES Admin (bonus)
