Introduction to the Watson & Walker Support Page for Spark on z/OS and SMF
Welcome to our Spark on z/OS support page. Cheryl is a world renowned expert on SMF and we have been working closely with IBM and Rocket Software to investigate ways to maximize the value that Spark brings to the world of SMF. While there are already many powerful tools to process SMF data, most of them are designed on the basis that SMF data provides a window into what happened in the past. Most of them also concentrate on a subset of SMF record types, based on their particular area of expertise.
In Watson & Walker we don’t see Spark as a competitor to those products; rather, we see it as augmenting them, providing the ability to use a common tool set against all SMF record types. It also enables the use of real-time SMF data for installation-developed applications such as detecting hacking attempts in real time.
There is no one time charge or monthly license charge for Spark (IBM Product number 5655-AAB), and Subscription & Support is optional. All of the Spark code is zIIP-eligible, as is 95+% of the MDS processing of sequential data sets, log streams, and SMF in-memory buffers. In addition, IBM is offering very attractive processing on zIIPs and memory purchased for use by Spark. The financial incentives from IBM make the business case for Spark on z/OS nearly irresistible.
We plan to use this website to develop an ecosystem around using Spark to process SMF data. The site will contain our own documentation and presentations, links to other material, a description of Spark-related services that we offer, and a repository of Spark queries developed by us and other Spark users that would like to contribute their queries for use by the Spark SMF community.
Spark is a new offering that requires a number of skills to fully exploit it. Cheryl Watson’s Tuning Letter is carrying a series of articles about setting up and using Spark on z/OS, the first of which is available here in its entirety. Subsequent installments will be provided here in summary, and in their entirety to Tuning Letter subscribers.
- Spark for z/OS and SMF Part 1 Tuning Letter 2016 No. 3
- Spark for z/OS and SMF Part 2, Tuning Letter 2016 No. 4
There are also a number of presentations that might prove helpful:
- Spark and SMF – Fasten Your Seatbelt, This is Going To Be a Wild Ride! by Frank Kyne
- SHARE 2016 in Atlanta Session 19406 z/OS Platform for Apache Spark Install and Usage courtesy of IBM’s Joe Bostian and Mike Gildein © IBM and SHARE
- SHARE 2017 in San Jose Session 20200 LIVE: Big Data and z Systems — IBM z/OS Platform for Apache Spark by Peter Fandel and Theresa Tai © IBM and SHARE
- SHARE2017 in San Jose Session 20710 LIVE: Live Streaming SMF Data for Operational Insights by Theresa Tai, Elpida Tzortzatos, and Gregg Willhoit © IBM and SHARE
- SHARE 2017 in San Jose Session 20788 z/OS Platform for Apache Spark Install and Usage by Joe Bostian and Mike Gildein © IBM and SHARE
You can also find more information about Spark on z/OS in the IBM Spark library at http://www.ibm.com/support/knowledgecenter/SSCTFE_1.1.0/com.ibm.azk.v1r1.azklp/azk.htm
The following IBM Redbooks provide additional supporting information about Spark on z/OS:
- Apache Spark Implementation on IBM z/OS, SG24-8325
- Apache Spark for the Enterprise: Setting the Business Free, REDP-5336
IBM Flashes and White Papers
- Installing IBM zOS Platform for Apache Spark, WP102609
- Resource Tuning Recommendations for IBM z/OS Platform for Apache Spark, WP102684
- Apache Spark 2.0.2, shipped by PTF UI43106, FLASH10872
Watson & Walker Services to Get Up and Running Quickly
Watson & Walker has been working with IBM and Rocket Software on using Spark on z/OS with SMF data since early 2016. And Cheryl has been working with SMF data for over 40 years, including five years supporting MICS in Morino Associates prior to establishing Watson & Walker. She is famous around the world for her knowledge, experience, and her integrity.
We would be happy to provide consulting services about how to get the most value from Spark with the minimum effort. Whether you need help with planning for Spark on z/OS installation, getting it up and running, or configuring it to deliver optimum performance while protecting your other workloads and your software bill, come and talk to us.
If you like the power and flexibility that Spark on z/OS brings to the processing of SMF, but you have SMF records from ISV products that are not included in the set of maps provided with the IBM z/OS Platform for Apache Spark, we can help.
Sample Spark Queries for SMF Data
When is the last time anyone created the JCL for a batch job from scratch? We all take an existing working example and use that as the base for our new job. Why not do the same with your Spark SMF queries? We are working with IBM and Rocket Software to create real world examples of SMF queries that we will share on this website.
NOTE: Make sure to install PTF UI44906 to pick up the latest additions and corrections to the Spark SMF support.
But we certainly don’t have a monopoly on ideas for valuable SMF queries. If you develop a Spark SMF query and you would like to share it with your peers, please use the form below to submit your queries.