Big Data

AWS Well-Architected

Folks at AWS publish a really great resource for anyone, who is designing cloud architecture. Even if you are using already or thinking about Azure or GCP, it is a really good read and it is not your typical sleep-provoking dry white-paper. AWS did an awesome job packing a lot

Big Data

Building Near Real-time Big Data Lake: Part 2

This is the second part of the series. In Part 1 [https://boristyukin.com/building-near-real-time-big-data-lake-part-i/] I wrote about our use-case for the Data Lake architecture and shared our success story. Requirements Before we embarked on our journey, we had identified high-level requirements and guiding principles. It is crucial to think

Big Data

Building Near Real-time Big Data Lake: Part I

Preface A lot has been said and done about a Data Lake architecture. It was 10 years ago when James Dixon defined a Data Lake concept in his viral blog post [https://jamesdixon.wordpress.com/2010/10/14/pentaho-hadoop-and-data-lakes/]. I know more people who can explain what a Data Lake

Big Data

How to ingest a large number of tables into a Big Data Lake, or why I built MetaZoo

MetaZoo what?? When I started learning about Big Data and Hadoop, I got excited about Apache Sqoop [https://sqoop.apache.org/]. I was naïve enough to believe that the ingestion of database tables was a very easy task. All I needed to do was to use Sqoop and it would

Big Data

Connecting to Kafka on VirtualBox from Windows

After many hours of frustration, I was finally able to push messages into Apache Kafka, running on my VirtualBox guest machine, from Windows host. Since I did not find complete steps on the web, I wanted to document them quickly, hoping to save someone's time. Big thanks to my teammate

Healthcare Featured

Sepsis Dashboard

A silent killer > Sepsis is the body’s extreme response to an infection. It is a life-threatening medical emergency. Sepsis happens when an infection you already have —in your skin, lungs, urinary tract, or somewhere else—triggers a chain reaction throughout your body. Without timely treatment, sepsis can rapidly lead

Big Data

How to hot swap Apache Kudu tables with Apache Impala

Sometimes, there is a need to re-process production data (a process known as a historical data reload, or a backfill). Source table schema might change, or a data discrepancy might be discovered, or a source system would be switched to use a different time zone for date/time fields. One

Groovy

Dynamic SQL query with Groovy

Groovy’s groovy-sql module provides a higher-level abstraction over Java’s JDBC technology and it is very easy to use. You can find good examples, using groovy.sql.Sql class in API documentation [http://docs.groovy-lang.org/latest/html/api/groovy/sql/Sql.html] or Groovy documentation [http://groovy-lang.org/

NiFi

How to run Sqoop from NiFi

Sqoop + NiFi = ? Apache Sqoop [https://sqoop.apache.org/] is still the best tool to do a bulk data transfer between relational databases and Apache Hadoop [https://hadoop.apache.org/]. One sunny day in Florida, I was able to reluctantly ingest 5 billion rows from a remote Oracle database in just

NiFi

How to connect Apache NiFi to Apache Impala

I spent 4 interesting hours, trying to connect Apache NiFi to Apache Impala. It turned out to be very easy and not really any different from a JDBC compliant database, but at the same time frustrating enough to make me post about it, hoping it will save someone's time. First,

Alteryx

Quick evaluation of Alteryx In-Database tools

Back in 2015, Alteryx announced [https://www.youtube.com/watch?v=GGkEd3KoMj0] a brand new set of In-Database tools, available to all customers with no additional license required. Alteryx keeps bringing amazing value to its customers without making them pay every time a great new future is announced. This is

Big Data Featured

Benchmarking Impala on Kudu vs Parquet

Why Apache Kudu Apache Kudu is a recent addition to Cloudera's CDH distribution, open sourced and fully supported by Cloudera with an enterprise subscription. Created by Cloudera and HBase veterans and getting so much traction and press recently, Kudu is worth considering for your next Big Data architecture platform. Apache

Hadoop

Watch out for timezones with Sqoop, Hive, Impala and Spark

My head was spinning as I tried to accomplish a simple thing (as it seemed at first). I load data from 3 Oracle databases, located in different time zones, using Sqoop and Parquet. Then I load data to Hive using external tables and finally do some light processing and load

Cerner

Healthcare Analytics with Cerner: Part 2 - Cerner Millennium Data Model

This is the second part of my blog series, dedicated to healthcare analytics with Cerner. In the first part [https://boristyukin.com/healthcare-analytics-with-cerner-part-1-data-acquisition/] we've looked at different options on how to extract data from Cerner and this time we will focus on Cerner Millennium Data Model. > If you do not

Qlik

2017 Qlik Luminary

Qlik just broke the news and congratulated me for being selected among with other 60 Qlik enthusiasts from around the globe! I am very proud and grateful to be part of this group of extremely talented and passionate people who I've learned a lot from! According to Qlik: > The Qlik

Cerner Featured

Healthcare Analytics with Cerner: Part 1 - Data Acquisition

Introduction. I am starting a series of blog posts, dedicated to healthcare analytics with Cerner. My intent is to get someone new to Cerner started with her/his analytics projects whether it is a Business Intelligence (BI) project, ad-hoc research or a quick proof of concept using Cerner's data. I

QlikView

How to raise or throw an error in QlikView or Sense

Sometimes you need to stop your LOAD script if some conditions are not met and raise a custom exception error (for example, required files are not found). You will also want to show a user friendly error message so it is clear why process was stopped and record the error

Hadoop

Is Snappy compressed Parquet file splittable?

I spent a few hours trying to find a definite answer on this question and hopefully my post will save someone time and trouble. The short answer is yes, if you compress Parquet files with Snappy they are indeed splittable Read below how I came up with an answer. First

QlikView

QlikView Qlik Sense and Oracle Database - Tips for Performance

Here is a few tips how to load data faster to QlikView or Qlik Sense from Oracle database. 0. Optimize performance of your SQL Statement. This is really a step 0 and assumes that you have basic knowledge and experience working with Oracle Databases. If you are coming from other

QlikView

Alteryx: To QVX, or Not To QVX

Question Alteryx [http://alteryx.com] is an awesome tool for Data Analysts and can do a lot of things. As of version 10, not only Alteryx can write data to QVX file, but can also read QVX files. It is a bit difficult to find information about QVX format itself

QlikView

QlikView Readmission Tracking App

I was super excited when one of my QlikView apps was nominated for a webinar, organized by Qlik. Even more surprising, that this application was built in just 6 weeks showing Qlik's agility and an amazing time-to-market value. This was also one of the few operational dashboards we've built for

Find and replace text in a very large file fast

I needed to replace a specific text string in a 6.5Gb file for one of my projects. This is a pretty easy task if you are on Linux (using tool like sed) but it is not that easy if you are on Windows. First, I tried my favorite PowerShell.

QlikView

QlikView How to check if table exists already

Here is a simple (and fast) one-line code to check in QlikView load script if a table exists already and then drop it: If (len(TableNumber('Table Name To Check')) > 0) THEN; DROP TABLE [Table Name To Check]; End If Note, there is ; after THEN. TableNumber function returns a number

Qlik

Book review: QlikView Your Business

Just finished reading a recently published book for QlikView developers and super users QlikView Your Business: An expert guide to Business Discovery with QlikView and Qlik Sense [http://www.wiley.com/WileyCDA/WileyTitle/productCd-1118949552.html] by Oleg Troyansky, Tammy Gibson and Charlie Leichtweis. I met Oleg Troyansky [https://www.linkedin.

QlikView

How To drop multiple fields using pattern (wildcard)

I was looking for a good neat solution to remove multiple fields from my QlikView data model with similar names. These were temporary fields, needed during execution of the load script but no longer needed once script is finished. Unfortunately QlikView 11 does not allow something like that: DROP FIELD