Quantcast
Channel: SystemSpeek.com » splunk
Viewing all articles
Browse latest Browse all 2

Virtualizing Splunk and BigData – Common Concerns & Pitfalls

0
0

Virtualization has become the mainstay in enterprise data centers, enabling agility and standardization of infrastructure. A lot of organizations are adopting a ‘virtualize first’ policy to maximize utilization and reduce their operating expenses (OpEx) by enabling ease of deployment and maintenance. Infrastructure customers are happier since admins are now sizing a Virtual Machine instead of infrastructure and have the flexibility to accommodate changes in resource requirements. Higher availability for applications and standardization of processes and tools helps alleviate pressure on admin staff and helps drive a better return on investment (ROI).

Unfortunately when it comes to certain workloads, there’s apprehension and concern of a performance impact due to virtualization, and BigData workloads like Splunk & Hadoop are among those. Let’s look at those concerns and then look at how Nutanix enables you to combine the best of both worlds.

Most concerns for Splunk/BigData performance and virtualization are rooted around:

CPU Performance

Virtualization overhead on the CPU is often called out as a key reason to avoid virtualization. With the recent developments from Intel, with hardware assist virtualization technology (Intel VT), their tick-tock development cycle providing us higher frequency and core counts every six months, the effective overhead to CPU is approx. 5%.

If you consider average utilization for CPU in the datacenter is about 15-20%, a 5% cost for dramatically increased manageability is an easy trade-off to make.

The other concern often voiced is around BigData workloads not having the tolerance for the performance loss. Typical Splunk and Hadoop use-cases involve large queries or batch processing across a large dataset. Unlike transactional systems, the requirement for sub millisecond queries is almost non-existent. Even for such latency-sensitive workloads, virtualization is very capable of providing a compelling solution. We all know how critical latency is for stock-trading – NYSE has created a trading-in-a-box solution using virtualization that dramatically improves uptime and availability while reducing latency: NYSE Trading-in-a-Box

VMware has published a detailed report that documents how virtualization might help drive CPU utilization and improve performance for BigData workloads by ensuring better NUMA access.

In summary, CPU performance issues are definitely not the reason to stay away from virtualization, the management benefits far outweigh the cost.

IO Performance

Splunk, Hadoop and other BigData workloads depend heavily on IO performance for driving timely reports and analysis. As such concerns around IO performance under virtualization are key, and typically stem from:

 Use of a SAN/NAS

Splunk and Hadoop workloads rely on ‘data-locality’ (compute processes read and write data from direct attached storage) to drive maximum performance. Splunk indexers perform best when they read and write from local storage providing for fast access, but more importantly enable organizations to start small and scale out their indexing tier as their usage grows.

Virtualization architectures today use SANs or NAS to enable some key benefits of virtualization like vMotion and HA/DRS etc – this breaks the scale-out model for BigData workloads and affects scalability in the indexing tier.

IO Blending

With multiple BigData VMs reading and writing to either shared-storage or to local disks, the IO for the various VMs ends up blending, converting sequential IO to random and causing seek-hell – thus dramatically reducing disk throughput and performance. Working around that problem requires additional configuration on the SAN/NAS which adds additional complexity.

One could avoid the IO blending problem by using 1 VM per server and local storage, but that exposes the hypervisor to disk failures – which can be fairly common in a large deployment with hundreds of disks. Moreover, restricting to 1 VM per server takes away the ability to drive higher utilization, a key benefit of virtualization, which is undesirable.

The Nutanix Way

Nutanix solves for both the above concerns via it’s distributed file system (NDFS), thus enabling virtualization of Splunk and Hadoop without affecting performance. Infact, due to the way NDFS leverages SSDs, one might realize much better performance across typical working sets in a BigData deployment.

NDFS is a distributed system made of services running in a Controller VM that is run on every Nutanix server. All IO for all VMs on a Nutanix server is passed to the Nutanix Controller VM on that same server. Every Nutanix controller uses local server attached SSDs, coalesces IOs and ensures large sequential writes to local server attached HDDs to ensure optimal performance while maintaining architectural integrity with Hadoop/Splunk. This same loopback IO technique between workload/user VMs and the Nutanix Storage Controller occurs on all servers in the Nutanix cluster, enabling high-performance local direct attached IO.

Moreover, the Nutanix controllers also expose a highly-available NFS datastore to the hypervisor, ensuring all virtualization benefits of vMotion and HA/DRS etc are available.

IO Path on a Nutanix Server

 

An admin can choose to scale a Nutanix deployment by adding additional servers, which each have their own compute, storage, and software controller and thus provide a linear increase in resources and performance. The admin can then deploy additional Splunk indexers on the new Nutanix servers thus scaling out existing indexing tiers and increasing capacity, while improving search and indexing performance.

 

Deploy additional Indexers on new Nutanix Servers

 

Summary

This scale out deployment model enables easy to manage, multi-petabyte Splunk deployments while maintaining the flexibility and granularity of procurement of bare-metal servers. Nutanix provides additional enterprise grade features that isolate hardware failures from Splunk, ensuring higher uptime on such a business critical workload. A single pane of management for all Splunk infrastructure, support for out-of-band compression, data placement across SSDs and HDDs enables faster indexing and searching delivering better ROI for a Splunk deployment.

Ability to snapshot entire data sets, and support for infrastructure level disaster recovery makes it possible for organizations to ingest valuable business data into Splunk for advanced analytics. Self-contained infrastructure enables isolation for data security and privacy enabling financial and healthcare industries to deploy Splunk securely.

Click here for more information on Splunk on Nutanix: http://www.nutanix.com/splunk/

For more information on Nutanix, check out the Nutanix Bible at: http://stevenpoitras.com/the-nutanix-bible/

Image Credits: Steven Poitras @ The Nutanix Bible.

The post Virtualizing Splunk and BigData – Common Concerns & Pitfalls appeared first on SystemSpeek.com.


Viewing all articles
Browse latest Browse all 2

Latest Images

Trending Articles





Latest Images